OctoPrint is currently not running, requires daily reboot

Get Help Networking
#1

What is the problem?
Octoprint disconnects every evening, and requires a SSH reboot or a manual power cycle to reconnect. It had previously no issues for over a year. I verified the python process as suggested by the error message. Please note, this is a 3D Printer we are using at work, but my IT said he made no changes to the network when the disconnects started to occur.

What did you already try to solve it?

  1. Installed a clean image of Octopi 0.17.0, and updated to Octoprint 1.5.2
  2. Installed a clean image of Octopi 0.16.0, and did not update Octoprint.
  3. Installed a clean image of Octopi 0.16.0, and updated to Octoprint 1.5.2
  4. Was previously using an ethernet switch, and changed to direct connection with a new cable.

Logs (syslog, dmesg, ... no logs, no support)
(Please note, the log attached below was edited to hide IP Addresses)
I don't know which device kept trying to run "404 GET", but I will check in with IT when I find out.
octoprint.log (3.1 MB)

Additional information about your network (Hardware you are trying to connect to, hardware you are trying to connect from, router, access point, used operating systems, ...)
Raspberry Pi 3B, Ender 3 Printer with latest Marlin, I don't have much knowledge with the network since it is maintained by IT.

#2

My IT stated that IP Address XXX.XXX.XX.98 is an awn-risk-scanner. Arctic Wolf Pen testing server.

#3

If I had to take a guess, it is the multiple-call-per-second from whatever this is that you have setup to call over 12,000 endpoints in 5 mins is knocking the server offline. There's nothing else that looks off in the logs, and given you have an IT maintaining this network you could ask them to fix it for you?

Maybe they have good reason, but I don't think you can just say 'I have these problems, triggered after 12000 requests by X' and not make sure that is not the thing causing the issue. Turn it off, whitelist the OctoPi device, whatever you do my first step would be rule out this first.

#4

Thanks Charlie! I was afraid that might be the problem. I'll see if I can convince IT to whitelist and will let you know that fixes it.

1 Like
#5

You could also adjust haproxy to require a path be specified for octoprint instead of sending all requests to a default.

Here's a working diff

--- haproxy.cfg.1610587101	2021-01-13 20:18:21.502548481 -0500
+++ haproxy.cfg	2021-01-13 20:26:35.377357456 -0500
@@ -23,14 +23,15 @@
         bind :::443 v4v6 ssl crt /etc/ssl/snakeoil.pem
         option forwardfor except 127.0.0.1
         use_backend webcam if { path_beg /webcam/ }
-        default_backend octoprint
+        use_backend octoprint if { path_beg /op/ }
 
 backend octoprint
         acl needs_scheme req.hdr_cnt(X-Scheme) eq 0
 
-        reqrep ^([^\ :]*)\ /(.*) \1\ /\2
+        reqrep ^([^\ :]*)\ /op/(.*) \1\ /\2
         reqadd X-Scheme:\ https if needs_scheme { ssl_fc }
         reqadd X-Scheme:\ http if needs_scheme !{ ssl_fc }
+        reqadd X-Script-Name:\ /op
         option forwardfor
         server octoprint1 127.0.0.1:5000
         errorfile 503 /etc/haproxy/errors/503-no-octoprint.http
#6

My IT unfortunately is locked out of the scanning service since it is implemented by corporate. I was able to confirm however, that the server went offline right after the latest scan, so I'm confident that this is the root cause.
For some reason Repetier Server is able to stay connected during these scans, so I just migrated the 3D Printer to that server instead. I appreciate the help guys. I'll definitely continue using Octoprint for my personal 3D Printer due to the customization options.

#7

The haproxy config above might help, if it is only looking at the base URL. If it is better than that and picks up the changed URL then fair play and you can admit defeat. Sure there would be a way around this (does haproxy have a blacklist you could use?) somehow, if you really wanted it to. That said, it doesn't appear that OctoPrint is erroring or anything, so maybe it is Haproxy dying first.