Something crashed inside the serial connection loop

What is the problem?
Octoprint failed to connect to my printer after successfully connecting many times before. It sat overnight disconnected and this happened on the next connect the next morning.

What did you already try to solve it?
After getting this error, I tried clicking on the Connect button in the GUI. Nothing happened so I clicked it again and, again, nothing happened. So, I tried restarting Octoprint from the pulldown menu and got this:

image

Using the SSH interface, I executed an 'sudo systemctl restart octoprint' command. That completed successfully and, after refreshing the GUI, I was able to connect to the printer successfully.

Logs (octoprint.log, serial.log or output on terminal tab at a minimum, browser error console if UI issue ... no logs, no support!)

2020-05-08 13:49:06,822 - octoprint.server.heartbeat - INFO - Server heartbeat <3
2020-05-08 13:49:18,585 - octoprint.plugins.tracking - INFO - Sent tracking event ping, payload: {'octoprint_uptime': 184514}
2020-05-08 13:50:41,007 - octoprint.plugins.mqtt - ERROR - Connection to mqtt broker refused, not authorised
2020-05-08 13:50:41,010 - octoprint.plugins.mqtt - ERROR - Disconnected from mqtt broker for unknown reasons (network error?), rc = 5
2020-05-08 13:52:41,017 - octoprint.plugins.mqtt - ERROR - Connection to mqtt broker refused, not authorised
2020-05-08 13:52:41,019 - octoprint.plugins.mqtt - ERROR - Disconnected from mqtt broker for unknown reasons (network error?), rc = 5
2020-05-08 13:54:41,027 - octoprint.plugins.mqtt - ERROR - Connection to mqtt broker refused, not authorised
2020-05-08 13:54:41,029 - octoprint.plugins.mqtt - ERROR - Disconnected from mqtt broker for unknown reasons (network error?), rc = 5
2020-05-08 13:56:41,036 - octoprint.plugins.mqtt - ERROR - Connection to mqtt broker refused, not authorised
2020-05-08 13:56:41,037 - octoprint.plugins.mqtt - ERROR - Disconnected from mqtt broker for unknown reasons (network error?), rc = 5
2020-05-08 13:58:41,044 - octoprint.plugins.mqtt - ERROR - Connection to mqtt broker refused, not authorised
2020-05-08 13:58:41,051 - octoprint.plugins.mqtt - ERROR - Disconnected from mqtt broker for unknown reasons (network error?), rc = 5
2020-05-08 14:00:41,061 - octoprint.plugins.mqtt - ERROR - Connection to mqtt broker refused, not authorised
2020-05-08 14:00:41,066 - octoprint.plugins.mqtt - ERROR - Disconnected from mqtt broker for unknown reasons (network error?), rc = 5
2020-05-08 14:02:41,076 - octoprint.plugins.mqtt - ERROR - Connection to mqtt broker refused, not authorised
2020-05-08 14:02:41,079 - octoprint.plugins.mqtt - ERROR - Disconnected from mqtt broker for unknown reasons (network error?), rc = 5
2020-05-08 14:04:06,831 - octoprint.server.heartbeat - INFO - Server heartbeat <3
2020-05-08 14:04:18,548 - octoprint.plugins.tracking - INFO - Sent tracking event ping, payload: {'octoprint_uptime': 185414}
2020-05-08 14:04:41,086 - octoprint.plugins.mqtt - ERROR - Connection to mqtt broker refused, not authorised
2020-05-08 14:04:41,087 - octoprint.plugins.mqtt - ERROR - Disconnected from mqtt broker for unknown reasons (network error?), rc = 5
2020-05-08 14:06:41,096 - octoprint.plugins.mqtt - ERROR - Connection to mqtt broker refused, not authorised
2020-05-08 14:06:41,100 - octoprint.plugins.mqtt - ERROR - Disconnected from mqtt broker for unknown reasons (network error?), rc = 5
2020-05-08 14:07:45,276 - octoprint.util.comm - INFO - Changing monitoring state from "Offline" to "Opening serial port"
2020-05-08 14:07:45,306 - octoprint.util.comm - INFO - Changing monitoring state from "Opening serial port" to "Connecting"
2020-05-08 14:07:45,339 - octoprint.util.comm - INFO - M110 detected, setting current line number to 0
2020-05-08 14:07:45,364 - octoprint.util.comm - INFO - M110 detected, setting current line number to 0
2020-05-08 14:07:45,367 - octoprint.util.comm - ERROR - Something crashed inside the serial connection loop, please report this in OctoPrint's bug tracker:
Traceback (most recent call last):
  File "/home/pi/oprint/local/lib/python2.7/site-packages/octoprint/util/comm.py", line 2175, in _monitor
    self._onConnected()
  File "/home/pi/oprint/local/lib/python2.7/site-packages/octoprint/util/comm.py", line 2482, in _onConnected
    self._temperature_timer.start()
  File "/usr/lib/python2.7/threading.py", line 736, in start
    _start_new_thread(self.__bootstrap, ())
error: can't start new thread
2020-05-08 14:07:45,383 - octoprint.util.comm - INFO - Changing monitoring state from "Connecting" to "Offline (Error: See octoprint.log for details)"
2020-05-08 14:07:45,613 - octoprint.plugins.influxdb - ERROR - Disconnected from InfluxDB. Attempting to reconnect.
Traceback (most recent call last):
  File "/home/pi/oprint/local/lib/python2.7/site-packages/octoprint_influxdb/__init__.py", line 155, in influx_emit
    self.influx_db.write_points([point])
  File "/home/pi/oprint/local/lib/python2.7/site-packages/influxdb/client.py", line 599, in write_points
    consistency=consistency)
  File "/home/pi/oprint/local/lib/python2.7/site-packages/influxdb/client.py", line 676, in _write_points
    protocol=protocol
  File "/home/pi/oprint/local/lib/python2.7/site-packages/influxdb/client.py", line 410, in write
    headers=headers
  File "/home/pi/oprint/local/lib/python2.7/site-packages/influxdb/client.py", line 369, in request
    raise InfluxDBClientError(err_msg, response.status_code)
InfluxDBClientError: 400: {"error":"partial write: max-series-per-database limit exceeded: (1000000) dropped=1"}
2020-05-08 14:07:45,628 - octoprint.plugins.influxdb - INFO - connecting: database='octoprint', host='10.248.0.30', ssl=False, use_udp=False
2020-05-08 14:07:45,738 - octoprint.plugins.influxdb - INFO - Using existing database `octoprint`
2020-05-08 14:07:45,748 - octoprint.plugins.stats - INFO - Printer Stats - on_event
2020-05-08 14:07:45,980 - octoprint.plugins.stats - INFO - Printer Stats - on_event
2020-05-08 14:07:46,297 - octoprint.plugins.tracking - INFO - Sent tracking event commerror_crash, payload: {'commerror_text': u'See octoprint.log for details'}

The MQTT errors are due to the fact that I had forgotten to configure Octoprint with username and password in the MQTT plugin. I configured those values and the errors went away.

After seeing that the problem seemed to be related to thread creation, I checked the memory state with 'top' and here is the result:

top - 14:19:16 up 3 days, 58 min,  1 user,  load average: 0.26, 0.33, 0.43
Tasks: 110 total,   1 running, 109 sleeping,   0 stopped,   0 zombie
%Cpu(s):  3.4 us,  4.4 sy,  0.0 ni, 92.2 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem :    864.6 total,    256.9 free,    161.7 used,    445.9 buff/cache
MiB Swap:    100.0 total,    100.0 free,      0.0 used.    634.2 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
21113 pi        20   0 2073136 111700  19620 S  31.7  12.6 627:57.63 octoprint
 9699 pi        20   0   10216   3016   2592 R   1.4   0.3   0:00.24 top
   10 root      20   0       0      0      0 I   0.5   0.0   2:02.89 rcu_sched
  420 root      20   0   26516   4296   4088 S   0.5   0.5   9:24.08 mjpg_streamer
 6984 root      20   0       0      0      0 I   0.5   0.0   0:01.78 kworker/0:1-events_power_efficient
 9249 pi        20   0   12544   4584   3704 S   0.5   0.5   0:00.14 sshd
    1 root      20   0   33808   8100   6428 S   0.0   0.9   0:12.19 systemd
    2 root      20   0       0      0      0 S   0.0   0.0   0:00.36 kthreadd
    3 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 rcu_gp
    4 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 rcu_par_gp
    8 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 mm_percpu_wq
    9 root      20   0       0      0      0 S   0.0   0.0   0:45.98 ksoftirqd/0
   11 root      20   0       0      0      0 I   0.0   0.0   0:00.00 rcu_bh
   12 root      rt   0       0      0      0 S   0.0   0.0   0:00.14 migration/0
   13 root      20   0       0      0      0 S   0.0   0.0   0:00.00 cpuhp/0
   14 root      20   0       0      0      0 S   0.0   0.0   0:00.00 cpuhp/1
   15 root      rt   0       0      0      0 S   0.0   0.0   0:00.18 migration/1
   16 root      20   0       0      0      0 S   0.0   0.0   0:00.53 ksoftirqd/1
   19 root      20   0       0      0      0 S   0.0   0.0   0:00.00 cpuhp/2
   20 root      rt   0       0      0      0 S   0.0   0.0   0:00.21 migration/2
   21 root      20   0       0      0      0 S   0.0   0.0   0:00.39 ksoftirqd/2
   24 root      20   0       0      0      0 S   0.0   0.0   0:00.00 cpuhp/3
   25 root      rt   0       0      0      0 S   0.0   0.0   0:00.19 migration/3
   26 root      20   0       0      0      0 S   0.0   0.0   0:07.78 ksoftirqd/3
   29 root      20   0       0      0      0 S   0.0   0.0   0:00.01 kdevtmpfs
   30 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 netns
   34 root      20   0       0      0      0 S   0.0   0.0   0:00.42 khungtaskd
   35 root      20   0       0      0      0 S   0.0   0.0   0:00.00 oom_reaper
   36 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 writeback
   37 root      20   0       0      0      0 S   0.0   0.0   0:00.00 kcompactd0
   38 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 crypto
   39 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kblockd
   40 root      rt   0       0      0      0 S   0.0   0.0   0:00.00 watchdogd
   41 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 rpciod
   42 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker/u9:0-hci0
   43 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 xprtiod
   46 root      20   0       0      0      0 S   0.0   0.0   0:00.00 kswapd0
   47 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 nfsiod
   58 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kthrotld
   60 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 iscsi_eh
   61 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 dwc_otg
   62 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 DWC Notificatio
   63 root       1 -19       0      0      0 S   0.0   0.0   0:00.07 vchiq-slot/0
   64 root       1 -19       0      0      0 S   0.0   0.0   0:00.00 vchiq-recy/0
   65 root       0 -20       0      0      0 S   0.0   0.0   0:00.00 vchiq-sync/0
   66 root      20   0       0      0      0 S   0.0   0.0   0:00.00 vchiq-keep/0
   67 root      10 -10       0      0      0 S   0.0   0.0   0:00.00 SMIO
   69 root     -51   0       0      0      0 S   0.0   0.0   0:00.01 irq/86-mmc1

Given that there did not appear to be a memory issue, I did not dig into the proc and sys filesystems for more clues.

The serial log looked like this:

2020-05-08 14:07:45,258 - Connecting to: /dev/ttyACM0
2020-05-08 14:07:45,276 - Changing monitoring state from "Offline" to "Opening serial port"
2020-05-08 14:07:45,297 - Connected to: Serial<id=0x6545f550, open=True>(port='/dev/ttyACM0', baudrate=115200, bytesize=8, parity='N', stopbits=1, timeout=10.0, xonxoff=False, rtscts=False, dsrdtr=False), starting monitor
2020-05-08 14:07:45,304 - Changing monitoring state from "Opening serial port" to "Connecting"
2020-05-08 14:07:45,340 - Send: N0 M110 N0*125
2020-05-08 14:07:45,350 - Recv: ok
2020-05-08 14:07:45,366 - Send: N0 M110 N0*125
2020-05-08 14:07:45,375 - See octoprint.log for details
2020-05-08 14:07:45,383 - Changing monitoring state from "Connecting" to "Offline (Error: See octoprint.log for details)"
2020-05-08 14:07:45,437 - Connection closed, closing down monitor
2020-05-08 14:17:53,356 - Connecting to: /dev/ttyACM0
2020-05-08 14:17:53,367 - Changing monitoring state from "Offline" to "Opening serial port"
2020-05-08 14:17:53,372 - Connected to: Serial<id=0x6733ead0, open=True>(port='/dev/ttyACM0', baudrate=115200, bytesize=8, parity='N', stopbits=1, timeout=10.0, xonxoff=False, rtscts=False, dsrdtr=False), starting monitor
2020-05-08 14:17:53,379 - Changing monitoring state from "Opening serial port" to "Connecting"
2020-05-08 14:18:23,432 - No answer from the printer within the connection timeout, trying another hello
2020-05-08 14:18:33,453 - There was a timeout while trying to connect to the printer
2020-05-08 14:18:33,472 - Changing monitoring state from "Connecting" to "Offline"
2020-05-08 14:18:33,484 - Connection closed, closing down monitor

Additional information about your setup (OctoPrint version, OctoPi version, printer, firmware, browser, operating system, ... as much data as possible)

Octoprint: 1.4.0
Octopi: octopi-buster-lite-0.17.0
Printer: DIY cartesian using an SKR v1.3 (LPC1768-based) control board running Marlin 2.0.5.3.
Browser: Chrome Version 81.0.4044.129 (Official Build) (64-bit)
OS: Microsoft Windows 10 Pro version 10.0.18362 Build 18362

Try in Safe Mode. If it's happy and doesn't fail again then it's possible that one of the third-party plugins is giving you difficulties.