Cannot reach octoprint from PC, but can reach from another Raspberry Pi

[robbery525]

What is the problem?

I have 3 Raspberry Pis on my home network, 2 are connected via LAN and the Octoprint device is connected via wifi.
My windows 10 computer normally cannot reach the octoprint device at all, not even through ping the local IP address. But that same computer can reach both of the other RPIs. So in order to reach octoprint, I had to set up 1 of the RPIs to act as a reverse proxy and forward the octoprint web data. In order to SSH, I have to first SSH into a different RPI, then SSH from that device into the octoprint Pi. This issue has been ongoing pretty much since I set up octoprint on that device.

I have the remote proxy set up so I can view the webcam from my phone when I am away from home. I have lived with using my proxy setup to connect when I'm not home, but also to connect when I am home (despite the fact that the computer should be able to connect without the proxy).

This odd setup worked well enough until today, when I upgraded to the latest version of Octoprint when the web page prompted me to do so. Now, I cannot access it using my proxy at all. Instead, I get this message:
" Socket connection failed. Please check your browser's error console and octoprint.log for possible reasons.Also make sure that the server is actually running by reloading this page."

This is when I then followed the procedure to turn OctoPrint into safe mode. As I state above, I had to connect via SSH after SSH'ing into a different device since the main computer cannot see octoprint. Once I had safe mode up and running, all of a sudden my main computer can see octoprint just fine.

In safe mode, the PC can ping, ssh, and even connect into the web GUI, none of which was possible if not in safe mode. The proxy method still returns that same error though.

What did you already try to solve it?

Reinstalled octoprint, set up DHCP, tried static IP address, tried connecting with a LAN cable, no effects.

Logs (syslog, dmesg, ... no logs, no support)
Here is some data from octoprint.log from attempting to log in using the proxy:

2020-03-18 14:02:00,092 - octoprint.server.util.sockjs - INFO - New connection from client: [IP]
2020-03-18 14:03:57,910 - octoprint.server.util.flask - INFO - Passively logging in user [USERNAME] from  [IP]
2020-03-18 14:03:57,912 - octoprint.access.users - INFO - Logged in user: [USERNAME] 
2020-03-18 14:03:58,544 - tornado.access - WARNING - 400 GET /sockjs/363/uvx2v4lb/websocket ( [IP]) 1.12ms
2020-03-18 14:04:00,143 - tornado.access - WARNING - 400 GET /sockjs/508/iibkiv5v/websocket ( [IP]) 1.98ms
2020-03-18 14:04:00,760 - octoprint.server.util.sockjs - INFO - New connection from client:  [IP]
2020-03-18 14:04:25,281 - octoprint.server.util.sockjs - INFO - Client connection closed: [IP]
2020-03-18 14:04:44,703 - octoprint.server.util.sockjs - INFO - Client connection closed:  [IP]
2020-03-18 14:04:45,519 - octoprint.server.util.sockjs - INFO - Client connection closed:  [IP]
2020-03-18 14:06:08,368 - tornado.access - WARNING - 400 GET /sockjs/260/bvylnqa2/websocket ( [IP]) 3.02ms
2020-03-18 14:06:08,575 - octoprint.server.util.sockjs - INFO - New connection from client:  [IP]
2020-03-18 14:06:59,371 - tornado.access - WARNING - 400 GET /sockjs/903/rdmhoqpu/websocket ( [IP]) 2.16ms
2020-03-18 14:07:00,063 - octoprint.server.util.sockjs - INFO - New connection from client:  [IP]

Additional information about your network (Hardware you are trying to connect to, hardware you are trying to connect from, router, access point, used operating systems, ...)
Noted above in issue..

Made a thread here as well:

r/HomeNetworking - 1 PC - 2 Pis Pi #1 can reach Pi #2, but PC can reach Pi #...

[foosel]

@robbery525

In safe mode, the PC can ping, ssh, and even connect into the web GUI, none of which was possible if not in safe mode. The proxy method still returns that same error though.

So that sounds like whatever issue you are having you are having due to some third party plugin.

We could take guesses at which one if you'd shared a full log instead of just an excerpt

[robbery525]

Alright, I have disabled all plugins that safemode also disabled and restarted.

Now that I am out of safe mode, I can reach it using the local IP address. Going through the proxy still results in a 'Socket Connection Failed'.

Here is my proxy configuration I am using with LetsEncrypt (which uses an nginx for the proxy):
Note that it connects and begins loading, it just results in a socket connection failed.

Nginx Proxy Config

server {
listen 443 ssl;
listen [::]:443 ssl;

server_name octoprint.*;

include /config/nginx/ssl.conf;

client_max_body_size 0;

# enable for ldap auth, fill in ldap details in ldap.conf
#include /config/nginx/ldap.conf;

location / {
    # enable the next two lines for http auth
    #auth_basic "Restricted";
    #auth_basic_user_file /config/nginx/.htpasswd;

    # enable the next two lines for ldap auth
    #auth_request /auth;
    #error_page 401 =200 /login;

    include /config/nginx/proxy.conf;
    resolver 127.0.0.11 valid=30s;
    set $upstream_octoprint 192.168.1.105;
    proxy_pass http://$upstream_octoprint;
}

}

Here is the firefox debug when attempting to connect via the proxy:

Plugins

List of 3rd Party Plugins:

• Bed Visualizer (0.1.12)
• Creality Temperature (1.2.3)
• Display ETA 1.0.2)
• Enclosure Plugin (4.13.1)
• FileManager (0.13)
• Firmware Updater (1.6.1)
• GcodeEditor (0.2.8)
• HeaterTimeout (0.0.1)
• Resource Monitor (0.2.2)
• The Spaghetti Detective (1.1.4)

Currently Disabled Plugins:

• Bed Leveling Wizard (0.2.4)
• Detailed Progress Plugin (0.1.4)
• MultiCam (0.2.6)
• Multi Colors (1.0.16)
• Telegram Notification (1.5.0)

I've enabled each one (3 at a time, including the ones listed as 'currently disabled') and performed restarts of the server to attempt to narrow down which may be the culprit. Even with all of them enabled, the issue did not crop back up, and I am still able to access using local IP (currrently).

I wonder if there was some back-end setting that got screwed up somehow, and restarting in safe mode forced it to fix itself.

[robbery525]

Figured out how to upload logs here.

octoprint.log (556.3 KB)

Also, here is a text file with the console debug from firefox (since it won't let me post due to excessive links)

FirefoxDebug.log (3.5 KB)

I'm a bit confused about one line in particular, though I don't think it has to do with the issue.

Why would doubleclick be attempting to load?!

That said, thank you for the quick reply!

[foosel]

Cross check that with

Reverse proxy configuration examples

If you want to run OctoPrint behind a reverse proxy such as Nginx, HAProxy or Apache's mod_proxy you can find some configuration examples here. Nginx For Nginx the basic forwarding configuration is something like this: worker_processes 1; events { worker_connections 1024; } http { include mime.types; default_type application/octet-stream; sendfile on; keepalive_timeout 65; map $http_upgrade…

I see at least the line proxy_http_version 1.1; missing there.

[robbery525]

LetsEncrypt has a base 'Proxy.conf' file that already included that parameter in it, as well as a bunch of other defaults..

Thank you for the assistance and pointing to that page.
I can't explain why it used to work with the config i posted but broke just recently.

That said, I have since updated my config to this, which loaded right up. (Basically I told it to not import the base defaults from LetsEncrypt and defined everything in the Octoprint.subdomain.conf file nginx was looking at)

LetsEncrypt Config

make sure that your dns has a cname set for deluge and that your deluge container is not using a base url

server {
listen 443 ssl;
listen [::]:443 ssl;

server_name octoprint.*;

include /config/nginx/ssl.conf;

client_max_body_size 0;

# enable for ldap auth, fill in ldap details in ldap.conf
#include /config/nginx/ldap.conf;

location / {
    # enable the next two lines for http auth
    #auth_basic "Restricted";
    #auth_basic_user_file /config/nginx/.htpasswd;

    # enable the next two lines for ldap auth
    #auth_request /auth;
    #error_page 401 =200 /login;

    # DO NOT INCLUDE THE DEFAULT - OVERRIDES BELOW
	#include /config/nginx/proxy.conf;
	
	#Settings per this link: https://community.octoprint.org/t/reverse-proxy-configuration-examples/1107
	proxy_set_header Host $http_host;
	proxy_set_header Upgrade $http_upgrade;
	proxy_set_header Connection "upgrade";
	proxy_set_header X-Real-IP $remote_addr;
	proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
	proxy_set_header X-Scheme $scheme;
	proxy_http_version 1.1;
	client_max_body_size 0;    
	
	#Settings from the proxy.conf that need to be imported since import was disabled
		client_body_buffer_size 128k;
		#Timeout if the real server is dead
		proxy_next_upstream error timeout invalid_header http_500 http_502 http_503;

		# Advanced Proxy Config
		send_timeout 5m;
		proxy_read_timeout 240;
		proxy_send_timeout 240;
		proxy_connect_timeout 240;

		# TLS 1.3 early data
		proxy_set_header Early-Data $ssl_early_data;

		# Basic Proxy Config
		proxy_set_header X-Forwarded-Proto https;
		proxy_set_header X-Forwarded-Host $host;
		proxy_set_header X-Forwarded-Ssl on;
		proxy_redirect  http://  $scheme://;
		proxy_cache_bypass $cookie_session;
		proxy_no_cache $cookie_session;
		proxy_buffers 32 4k;
		proxy_headers_hash_bucket_size 128;
		proxy_headers_hash_max_size 1024;

	
	#Required in this file
    resolver 127.0.0.11 valid=30s;
    set $upstream_octoprint 192.168.1.105;
    proxy_pass http://$upstream_octoprint;
}

}

If the device becomes unreachable on the local network again, I'll re-open this thread. For now everything is working. Thanks. <3

Last edit: for what its worth, the LetsEncrypt container proxy is running on a separate raspberry pi from octoprint.

[robbery525]

@foosel
Hey, thanks for your help yesterday. The issue with the octoprint server being unreachable on the local network has come up again. (It is now accessible via firefox through the proxy I discussed earlier, so atleast that is still working.)
I'll post a
Similar to the last time this issue cropped up though, I performed a print and throughout the print it was accessible. After print completed, it became inaccessible. (Last time I tested this, I had I think a few minute grace period after the print where I was able to still access it. If starting another print, everything was fine, but if I didn't get to it immediately it would go offline per this discussion.)

Sidenote: Prior to software update I was able to access octoprint web gui on my iphone, now it returns the loading failed debug page I mentioned in the first post. But accessing web gui from a PC works properly (both using the proxy). That said, the OctoClient iphone app still can connect via the proxy, so the API isn't down.

TCPDump incoming (I'll edit this post once I'm on the proper computer that can SSH into the device)

[robbery525]

Ok so here is the troubleshooting data so far.

I've performed a TCPdump on the pi prior to restating it, so I can capture the fault state.
Then I restarted it and communication was back online, so I captured more.
I also had wireshark running on a PC, and used that to capture packets data as well to see what was outgoing.

TCP_DUMP.log (113.2 KB)

Note that I made a line in the picture below as to when I restarted the RPI running octoprint. You can see that it immediately starts communicating. The red ICMP packets is me performing a ping test.
Prior to restarting though, there is very minimal outgoing according to wireshark, despite attempting to load the web page using the local IP, and also attempting to perform pings against the local IP (like I did after the reboot).

Its also noteworthy that I tried 2 different computers and neither could reach octoprint locally, but the proxy (going through the other RPIs on my network) worked well.

[robbery525]

This issue just surfaced again. It was working fine for a while there. but before bed I had to abort a print last night, allowing the printer to cool down, then I shut the printer off. Now in the morning, it was not reachable using the local IP from the computer, but (just like in my previous posts) I can reach it via a proxy device.

I had suspected it was TheSpaghettiDetective causing this issue, so I disabled it. After which it ran well for a few days, until it did it again.

[MercurieVV]

Same thing. Worse of this that logs dont contain any error

[nebbywan]

If anyone's still looking for a solution, disabling IPv6 on the OctoPrint host has worked for some as found here:

Can't Connect to UI from Windows -- Android is fine

Thanks. Been trying to do that, but not sure what sudo command to use, and chmod 777 to get to cmdline.txt is failing due to permissions.

[robbery525]

It happened again, so I'm unsure how long it takes to occur, since I honestly don't have to access my device this way that often. But:

I'm using this guide to Disable IPV6 on my device, hopefully this fixes the issue permanently.
https://cwesystems.com/?p=231

[LeonsDev]

I had an issue with websockets not being found after logging in.

enabling CORS under Settings > API made it work for me.

spend hours on this, hope it helps someone