1.4.1 SSL VPN issue - Tornado error

pingywon · August 5, 2020, 5:45pm

I have a SSL VPN set up to all 3 octo instances at the office. I have used octo 1.4.0 to remotely get to them with out any issues. I just upgraded to 1.4.1 to get some fixes for the Read-Only user issue and now I can not log in to the printers. One of the three printers is runnin an old v. of octo and has no issues.

I can get to the octo instances and when I input a valid username and password the login screen just refreshes.

Here is the valid portion of the log:

2020-08-05 13:20:09,891 - tornado.application - ERROR - Uncaught exception GET /static/webassets/packed_client.js?c542e09b (::ffff:192.168.0.2)
Traceback (most recent call last):
  File "/home/pi/oprint/local/lib/python2.7/site-packages/tornado/web.py", line 1590, in _execute
    result = method(*self.path_args, **self.path_kwargs)
  File "/home/pi/oprint/local/lib/python2.7/site-packages/octoprint/server/util/tornado.py", line 1013, in get
    if self.should_use_precompressed():
  File "/home/pi/oprint/local/lib/python2.7/site-packages/octoprint/server/util/tornado.py", line 1002, in should_use_precompressed
    and "gzip" in self.request.headers.get("Accept-Encoding"))
TypeError: argument of type 'NoneType' is not iterable
2020-08-05 13:20:09,893 - tornado.access - ERROR - 500 GET /static/webassets/packed_client.js?c542e09b (::ffff:192.168.0.2) 2.03ms
2020-08-05 13:20:11,373 - octoprint.plugins.dashboard - INFO - cmd_results: 9V
2020-08-05 13:20:14,445 - octoprint.plugins.dashboard - INFO - cmd_results: 9V
2020-08-05 13:20:17,529 - octoprint.plugins.dashboard - INFO - cmd_results: 9V
2020-08-05 13:20:20,658 - octoprint.plugins.dashboard - INFO - cmd_results: 9V
2020-08-05 13:20:21,383 - tornado.application - ERROR - Uncaught exception GET /static/webassets/packed_client.js?c542e09b (::ffff:192.168.0.2)
Traceback (most recent call last):
  File "/home/pi/oprint/local/lib/python2.7/site-packages/tornado/web.py", line 1590, in _execute
    result = method(*self.path_args, **self.path_kwargs)
  File "/home/pi/oprint/local/lib/python2.7/site-packages/octoprint/server/util/tornado.py", line 1013, in get
    if self.should_use_precompressed():
  File "/home/pi/oprint/local/lib/python2.7/site-packages/octoprint/server/util/tornado.py", line 1002, in should_use_precompressed
    and "gzip" in self.request.headers.get("Accept-Encoding"))
TypeError: argument of type 'NoneType' is not iterable
2020-08-05 13:20:21,385 - tornado.access - ERROR - 500 GET /static/webassets/packed_client.js?c542e09b (::ffff:192.168.0.2) 2.78ms

Again it is worth reitterating...nothing on my network has changed and this all worked perfectly under Octo 1.4.0

pingywon · August 5, 2020, 9:27pm

Gonna have to try and rollback to 1.4.0 and lose the fixes made to users groups until I can get some help with this.

pingywon · August 5, 2020, 10:19pm

Rolled back one of the 1.4.1 instances and its working perfectly again.

pingywon · August 6, 2020, 1:25pm

Looks like the error boils down to this

  File "/home/pi/oprint/local/lib/python2.7/site-packages/octoprint/server/util/tornado.py", line 1013, in get
    if self.should_use_precompressed():
  File "/home/pi/oprint/local/lib/python2.7/site-packages/octoprint/server/util/tornado.py", line 1002, in should_use_precompressed
    and "gzip" in self.request.headers.get("Accept-Encoding"))
TypeError: argument of type 'NoneType' is not iterable

Trying to compare a running 1.4.0 that works to a 1.4.1 that does not work...but this is not my cup of tea

foosel · August 6, 2020, 2:08pm

For some reason the requests coming in through your SSL VPN do not contain an "Accept-Encoding" header, and that is causing this issue since 1.4.1 now supports precompressing assets and such if gzip is an accepted encoding on the requesting client.

That particular line needs a change so that "no header" doesn't cause an issue. In the meantime you need to figure out how to tell your reverse proxy to set this header. Can be an empty string, but needs to be present.

As a side-note: considering the amount of issues you have now run into, it would maybe be a good idea to help test future release candidates to ensure they are compatible to your apparently quite particular environment and requirements.

pingywon · August 6, 2020, 2:17pm

I will happily accept your invitation and begin testing RCs going forward. Im really good at reporting issues and pulling logs just beyond that I wouldnt be much help which is why I havent in the past. I think I was on one of the RCs a while back.

The SSL VPN is run off an enterprise Fortigate gateway....im not sure how much control I have over changing it or even how to make the change.

Is there anything I can do on the Octo side to get it working? or no?

I have rolled back to 1.4.0 but I can gladly upgrade again if there is a hack that will make it work.

foosel · August 6, 2020, 2:19pm

Based on the logs above I'm assuming you run on OctoPi and you are actually connecting to the haproxy on that through the VPN? In that case it might be possible to change the haproxy configuration to ensure the header in question is set.

pingywon · August 6, 2020, 2:21pm

Correct on both counts. Im digging through the Fortigate now to see if there is a config option for header options.

I have never messed with the haproxy...

foosel · August 6, 2020, 2:25pm

Try the following.

First SSH into your OctoPi instance. Then sudo nano /etc/haproxy/haproxy.cfg.

There's a block in there that should look like this:

backend octoprint
        acl needs_scheme req.hdr_cnt(X-Scheme) eq 0

        reqrep ^([^\ :]*)\ /(.*) \1\ /\2
        reqadd X-Scheme:\ https if needs_scheme { ssl_fc }
        reqadd X-Scheme:\ http if needs_scheme !{ ssl_fc }
        option forwardfor
        server octoprint1 127.0.0.1:5000
        errorfile 503 /etc/haproxy/errors/503-no-octoprint.http

Turn that into

backend octoprint
        acl needs_scheme req.hdr_cnt(X-Scheme) eq 0

        reqrep ^([^\ :]*)\ /(.*) \1\ /\2
        reqadd X-Scheme:\ https if needs_scheme { ssl_fc }
        reqadd X-Scheme:\ http if needs_scheme !{ ssl_fc }
        reqadd Accept-Encoding:\ *
        option forwardfor
        server octoprint1 127.0.0.1:5000
        errorfile 503 /etc/haproxy/errors/503-no-octoprint.http

by adding a line reqadd Accept-Encoding:\ * right after the X-Scheme lines.

Save and exit. Restart haproxy: sudo service haproxy restart. Test if connecting to the OctoPi instance in question now works.

pingywon · August 6, 2020, 3:42pm

I spent this time trying to figure it out on the Fortinet side of things. Then I refreshed the forum and saw your reply.

I'am very happy to say that that worked like a dream!

Thank you foosel

Going forward this is something that would need to be changed on the Octopi code base?

foosel · August 6, 2020, 3:45pm

Nah, in OctoPrint, and in fact I already pushed a fix for the next maintenance release for that. But glad to hear that worked! Can you mark it as solution so others may find it easier?

pingywon · August 6, 2020, 5:04pm

For sure! Thanks Gina