Server eventually can't create threads when my service that uses the Rest API is running


#1

What is the problem?
I wrote a script that monitors the connection status of my printer, then tries to connect to it when it is not connected. The flow is basically: use the api/connection API to get the connection status. If it is closed initiate a connection, then sleep for 30 seconds. Once the printer is connected, the script then queries every 30 seconds for the print status using the api/printer API. If there is a print, it writes to a file. That file is checked in the connection process and no connection is attempted if the printer lost connection while printing.

Should this use case work with the current rest api or is it too taxing? Should I file a bug?

Here is the info from octoprint.log
2018-05-27 05:27:46,923 - tornado.access - ERROR - 500 POST /api/connection (127.0.0.1) 395.82ms
2018-05-27 05:28:00,929 - octoprint.util.comm - INFO - Changing monitoring state from "Connecting" to "Offline"
2018-05-27 05:28:03,188 - octoprint - ERROR - Exception on /api/connection [POST]
Traceback (most recent call last):
File "/home/pi/oprint/local/lib/python2.7/site-packages/flask/app.py", line 1817, in wsgi_app
response = self.full_dispatch_request()
File "/home/pi/oprint/local/lib/python2.7/site-packages/flask/app.py", line 1477, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/home/pi/oprint/local/lib/python2.7/site-packages/flask/app.py", line 1381, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/home/pi/oprint/local/lib/python2.7/site-packages/flask/app.py", line 1475, in full_dispatch_request
rv = self.dispatch_request()
File "/home/pi/oprint/local/lib/python2.7/site-packages/flask/app.py", line 1461, in dispatch_request
return self.view_functionsrule.endpoint
File "/home/pi/oprint/local/lib/python2.7/site-packages/octoprint/server/util/flask.py", line 1132, in decorated_view
return flask_login.login_required(func)(*args, **kwargs)
File "/home/pi/oprint/local/lib/python2.7/site-packages/flask_login.py", line 758, in decorated_view
return func(*args, **kwargs)
File "/home/pi/oprint/local/lib/python2.7/site-packages/octoprint/server/api/connection.py", line 67, in connectionCommand
printer.connect(port=port, baudrate=baudrate, profile=printerProfile)
File "/home/pi/oprint/local/lib/python2.7/site-packages/octoprint/printer/standard.py", line 226, in connect
self._comm = comm.MachineCom(port, baudrate, callbackObject=self, printerProfileManager=self._printerProfileManager)
File "/home/pi/oprint/local/lib/python2.7/site-packages/octoprint/util/comm.py", line 587, in init
self.sending_thread.start()
File "/usr/lib/python2.7/threading.py", line 736, in start
_start_new_thread(self.__bootstrap, ())
error: can't start new thread
2018-05-27 05:28:03,256 - tornado.access - ERROR - 500 POST /api/connection (127.0.0.1) 633.30ms
2018-05-27 05:28:03,263 - octoprint.util.comm - INFO - Changing monitoring state from "Offline" to "Opening serial port"
2018-05-27 05:28:03,297 - octoprint.util.comm - INFO - Changing monitoring state from "Opening serial port" to "Connecting"
2018-05-27 05:28:19,378 - octoprint - ERROR - Exception on /api/connection [POST]
Traceback (most recent call last):
File "/home/pi/oprint/local/lib/python2.7/site-packages/flask/app.py", line 1817, in wsgi_app
response = self.full_dispatch_request()
File "/home/pi/oprint/local/lib/python2.7/site-packages/flask/app.py", line 1477, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/home/pi/oprint/local/lib/python2.7/site-packages/flask/app.py", line 1381, in handle_user_exception
r
eraise(exc_type, exc_value, tb)
File "/home/pi/oprint/local/lib/python2.7/site-packages/flask/app.py", line 1475, in full_dispatch_request
rv = self.dispatch_request()
File "/home/pi/oprint/local/lib/python2.7/site-packages/flask/app.py", line 1461, in dispatch_request
return self.view_functionsrule.endpoint
File "/home/pi/oprint/local/lib/python2.7/site-packages/octoprint/server/util/flask.py", line 1132, in decorated_view
return flask_login.login_required(func)(*args, **kwargs)
File "/home/pi/oprint/local/lib/python2.7/site-packages/flask_login.py", line 758, in decorated_view
return func(*args, **kwargs)
File "/home/pi/oprint/local/lib/python2.7/site-packages/octoprint/server/api/connection.py", line 67, in connectionCommand
printer.connect(port=port, baudrate=baudrate, profile=printerProfile)
File "/home/pi/oprint/local/lib/python2.7/site-packages/octoprint/printer/standard.py", line 226, in connect
self._comm = comm.MachineCom(port, baudrate, callbackObject=self, printerProfileManager=self._printerProfileManager)
File "/home/pi/oprint/local/lib/python2.7/site-packages/octoprint/util/comm.py", line 581, in init
self.monitoring_thread.start()
File "/usr/lib/python2.7/threading.py", line 736, in start
_start_new_thread(self.__bootstrap, ())
error: can't start new thread
2018-05-27 05:28:19,402 - tornado.access - ERROR - 500 POST /api/connection (127.0.0.1) 439.98ms

What did you already try to solve it?
I tried increasing the sleep time to 1 minute but still got the same error.

Additional information about your setup (OctoPrint version, OctoPi version, printer, firmware, octoprint.log, serial.log or output on terminal tab, ...)

I only ever print from SD and this script is to prevent me from logging into a computer to connect to octoprint when I turn on my printer. PortLister did not work for me and I can't use udev to check if the printer is on because my printer appears the same when powered on or off.

My printer is a Prusa MK3 running fimrware 3.2.1

Here is info on octoprint version and plugins:
2018-05-26 20:53:59,671 - octoprint.server - INFO - --- Log roll over detected ---------------------------------------------------
2018-05-26 20:53:59,675 - octoprint.server - INFO - OctoPrint 1.3.8
2018-05-26 20:53:59,693 - octoprint.plugin.core - INFO - 22 plugin(s) registered with the system:
| Announcement Plugin (bundled) = /home/pi/oprint/lib/python2.7/site-packages/octoprint/plugins/announcements
| Autoscroll (0.0.2) = /home/pi/oprint/local/lib/python2.7/site-packages/octoprint_autoscroll
| Core Wizard (bundled) = /home/pi/oprint/lib/python2.7/site-packages/octoprint/plugins/corewizard
| CuraEngine (<= 15.04) (bundled) = /home/pi/oprint/lib/python2.7/site-packages/octoprint/plugins/cura
| Detailed Progress Plugin (0.1.4) = /home/pi/oprint/local/lib/python2.7/site-packages/octoprint_detailedprogress
| Discovery (bundled) = /home/pi/oprint/lib/python2.7/site-packages/octoprint/plugins/discovery
| DisplayZ (0.1.0) = /home/pi/oprint/local/lib/python2.7/site-packages/octoprint_displayz
| Firmware Updater (1.0.0) = /home/pi/oprint/local/lib/python2.7/site-packages/octoprint_firmwareupdater
| Fullscreen Plugin (0.0.4) = /home/pi/oprint/local/lib/python2.7/site-packages/octoprint_fullscreen
| Gcodebar Plugin (0.1.4) = /home/pi/oprint/local/lib/python2.7/site-packages/octoprint_GCodeBar
| HeaterTimeout (0.0.1) = /home/pi/oprint/local/lib/python2.7/site-packages/octoprint_HeaterTimeout
| Logging (bundled) = /home/pi/oprint/lib/python2.7/site-packages/octoprint/plugins/logging
| Navbar Temperature Plugin (0.9) = /home/pi/oprint/local/lib/python2.7/site-packages/octoprint_navbartemp
| OctoPi Support Plugin (bundled) = /home/pi/oprint/lib/python2.7/site-packages/octoprint/plugins/octopi_support
| Plugin Manager (bundled) = /home/pi/oprint/lib/python2.7/site-packages/octoprint/plugins/pluginmanager
| PortLister (0.1.7) = /home/pi/oprint/local/lib/python2.7/site-packages/octoprint_portlister
| Printer Safety Check (bundled) = /home/pi/oprint/lib/python2.7/site-packages/octoprint/plugins/printer_safety_check
| Prusa Mesh Leveling (0.2.1) = /home/pi/oprint/local/lib/python2.7/site-packages/octoprint_PrusaMeshMap
| Software Update (bundled) = /home/pi/oprint/lib/python2.7/site-packages/octoprint/plugins/softwareupdate
| Telegram Notifications (1.4.2) = /home/pi/oprint/local/lib/python2.7/site-packages/octoprint_telegram
| !TouchUI (0.3.11) = /home/pi/oprint/local/lib/python2.7/site-packages/octoprint_touchui
| Virtual Printer (bundled) = /home/pi/oprint/lib/python2.7/site-packages/octoprint/plugins/virtual_printer
2018-05-26 20:53:59,801 - octoprint.environment - INFO - Detected environment is Python 2.7.13 under Linux (linux2). Details:
| hardware:
| cores: 1
| freq: 1000.0
| ram: 388034560
| os:
| id: linux
| platform: linux2
| plugins:
| octopi_support:
| model: Zero W
| revision: 9000c1
| version: 0.15.0
| python:
| pip: 10.0.1
| version: 2.7.13
| virtualenv: /home/pi/oprint

Here is my python script that I run as a service:

#!/usr/bin/env python

import requests
import sys
import time
import json
import os
import subprocess
import datetime

LogLevel = 0
PrintLevel = 1
LOG_LEVEL_ERROR = 0
LOG_LEVEL_TRACE = 1

def Log(log, message, level):
	if(level <= PrintLevel):
		print(message + '\n')

	if(level <= LogLevel):
		logEntry = datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S") + ' - ' + message + '\n'
		log.write(logEntry)
#end Log

def CheckPrinting(printerApiUrl, headers, tempDetectEnabled, tempThreshExtruder, tempThreshBed, log, lockPath):
	isPrinting = False
	
	r = requests.get(
		printerApiUrl,
		headers=headers
		)	
		
	Log(log, ('Check Printing status code (%s)' % r.status_code), LOG_LEVEL_TRACE)
	
	if(r.status_code == 200):
		forcePrinting = False
		try:
			jsonOut = r.json()
			printing = jsonOut['state']['flags']['printing']
			paused = jsonOut['state']['flags']['paused']
			extruderTarget = jsonOut['temperature']['tool0']['target']
			bedTarget = jsonOut['temperature']['bed']['target']
			extruderTemp = jsonOut['temperature']['tool0']['actual']
			bedTemp = jsonOut['temperature']['bed']['actual']
			Log(log, ('printing(%s) paused(%s) extruder(cur %s/tgt %s/thresh %s) bedTarget(cur %s/tgt %s/thresh %s)\n' % (printing,paused,extruderTemp,extruderTarget,tempThreshExtruder,bedTemp,bedTarget,tempThreshBed)), LOG_LEVEL_TRACE)			
		except:
			#assume we are printing if we can't read the print status to be safe
			Log(log,("Error reading print status - %s"), LOG_LEVEL_ERROR)
			forcePrinting = True

		#we write this to a file so if we crash or restart during a print, we will know we used to be printing, will require a manual reconnect in these cases
		f = open(lockPath,"w")
		doLock = False
		if(printing == True or paused == True or extruderTarget > 0 or bedTarget > 0 or forcePrinting == True):
			doLock = True
		else:
			if(tempDetectEnabled):
				if(extruderTemp > tempThreshExtruder or bedTemp > tempThreshBed):
					doLock = True
		if(doLock == True):
			isPrinting = True
			f.write('printing')	
		f.close()
		#os.chmod(lockFile, stat.S_IREAD | stat.S_IWRITE | stat.S_IRGRP | stat.S_IWGRP | stat.S_IROTH | stat.S_IWOTH)
	else:
		Log(log, ('Failed to get print status (%d)\n' % r.status_code), LOG_LEVEL_ERROR)
	
	if(isPrinting):
		Log(log, ('Printing'), LOG_LEVEL_TRACE)
	else:
		Log(log, ('Not Printing'), LOG_LEVEL_TRACE)
	return isPrinting
#end CheckPrinting()

#Main()
LOCK_FILENAME = '.printlock'
lockPath = sys.path[0] + '/' + LOCK_FILENAME

LOG_FILENAME = 'oprintmon.log'
logPath = sys.path[0] + '/' + LOG_FILENAME

CONFIG_FILENAME = 'oprintmon.config'
configPath = sys.path[0] + '/' + CONFIG_FILENAME

log = open(logPath, "w+")

Log(log, ('oprintmon start'), LOG_LEVEL_ERROR)

if(os.path.isfile(configPath) == False):
	Log(log, ('Failed to load config (%s) - exiting' % configPath), LOG_LEVEL_ERROR)
	sys.exit(1)
	
configFile = open(configPath, "r")

try:
	configJson = json.load(configFile)
except:
	Log(log, ('Failed to read % - exiting' % configPath), LOG_LEVEL_ERROR)
	configFile.close()
	sys.exit(1)

octopiConfig = configJson['config']['octopi']
apiKey = octopiConfig['api-key']
baseUrl = octopiConfig['url-base']
port = octopiConfig['serial-port']
try:
	baudRate = int(octopiConfig['baud'])
except:
	Log(log, ('baudRate must be a number - exiting'), LOG_LEVEL_ERROR)
	sys.exit(1)

configConnect = configJson['config']['connect']
try:
	sleepTimeConnecting = float(configConnect['sleep-time'])
except:
	Log(log, ('sleep-time must be a number - exiting'), LOG_LEVEL_ERROR)	
	sys.exit(1)

printmonConfig = configJson['config']['printmon']
tempDetectEnabled = printmonConfig['detection-enabled'] == "true"
if(tempDetectEnabled) :
	try:
		tempThreshBed = int(printmonConfig['thresh-bed'])
		tempThreshExtruder = int(printmonConfig['thresh-hotend'])
	except:
		Log(log, ('temp thresholds must be numbers - exiting'), LOG_LEVEL_ERROR)	
		sys.exit(1)
try:
	sleepTimePrinting = float(printmonConfig['sleep-time'])
except:
	Log(log, ('sleep-time must be a number - exiting'), LOG_LEVEL_ERROR)	
	sys.exit(1)
	
Log(log, ('api-key (%s), base_url(%s), baud (%s), port (%s), sleeptime(connect) (%s), sleeptime(print) (%s) tempDetect(%s), bedThresh(%s), extruderThresh(%s)' % (apiKey, baseUrl, baudRate, port, sleepTimeConnecting, sleepTimePrinting, tempDetectEnabled, tempThreshBed, tempThreshExtruder)), LOG_LEVEL_TRACE)
	
headers = {'X-Api-Key': apiKey}
connectionApiURL = baseUrl + 'api/connection'
printerApiUrl = baseUrl + 'api/printer'

isPrinting = False

log.flush()
isConnected = False		
while(1):
	connectionState = 'Invalid'
	wasConnected = isConnected
	wasPrinting = isPrinting
	status_code = 0
	sleepTime = sleepTimeConnecting
	
	if(os.path.isfile(lockPath)):
		f = open(lockPath, "r")
		lockResult = f.read()
		isPrinting = lockResult == 'printing'
		Log(log, ('Read (%s) isPrinting = %s' % (lockResult, isPrinting)), LOG_LEVEL_TRACE)
		f.close()
		
	try:
		Log(log, ('%s\n%s' % (connectionApiURL, headers)), LOG_LEVEL_TRACE)
		r = requests.get(connectionApiURL, headers=headers)
	except:
		Log(log, ("failed to send request, server may not be up yet\n"), LOG_LEVEL_ERROR)
		status_code = -1
		
	if(status_code != -1):
		status_code = r.status_code
		
	#received valid connection info
	if(status_code == 200):
		jsonOut = r.json()
		#get connection state
		connectionState = jsonOut['current']['state']
		currentPort = jsonOut['current']['port']
		Log(log,('State = %s Port = %s' % (connectionState, currentPort)), LOG_LEVEL_TRACE)
						
		connecting = False			
		#if we are not connected and not printing, try to connect
		if(not isPrinting):	
			if(connectionState == 'Closed'):
				connecting = True
				json = {
					"command": "connect",
					"port": port,
					"baudrate": baudRate,
				}

				Log(log, ("Connecting to url %s on port %s with baud rate %s\n" % (connectionApiURL, port, baudRate)), LOG_LEVEL_TRACE)

				r = requests.post(
					connectionApiURL,
					json=json,
					headers=headers
				)	
	
				if(r.status_code == 204):
					Log(log, ("Connect Request Successful\n"), LOG_LEVEL_TRACE)
				else:	
					Log(log, ("Connect Request Failed (%d)\n" % r.status_code), LOG_LEVEL_ERROR)
		else:
			sleepTime = sleepTimePrinting
			
		if(not connecting):
			sleepTime = sleepTimePrinting
			if(connectionState != 'Closed' and connectionState != 'Connecting' and connectionState != 'Disconnecting'):
				isPrinting = CheckPrinting(printerApiUrl, headers, tempDetectEnabled, tempThreshExtruder, tempThreshBed, log, lockPath)

	if(connectionState != 'Closed'):
		isConnected = True
		if(not wasConnected):
			Log(log, 'Connected to Server', LOG_LEVEL_ERROR)		
	else:
		isConnected = False							
		if(wasConnected):
			Log(log, 'Disconnected from Server', LOG_LEVEL_ERROR)			

	if(wasPrinting != isPrinting):
		if(isPrinting):
			Log(log, 'Started Printing', LOG_LEVEL_ERROR)
		else:
			Log(log, 'FinishedPrinting', LOG_LEVEL_ERROR)

	Log(log, ('Sleeping for %s seconds' % sleepTime), LOG_LEVEL_TRACE)
	log.flush()
	time.sleep(sleepTime)
#end Main

#2

Please reformat your message, it's really hard to read without being in code blocks.

When it can't create threads, how many processes do you have running? Anything weird in the process list? My first guess is you aren't closing your connections properly but I'm unwilling to read the unformatted code.


#3

Also- you might take a look at the MQTT plugin, the output is really handy for seeing the current status of the printer. Here are some messages from mine:

octoprint/e10/event/ZChange {"new": 0.7, "_event": "ZChange", "old": 0.5, "_timestamp": 1527611897}
octoprint/e10/progress/printing {"progress": 3, "_timestamp": 1527612236, "location": "local", "path": "tues_thurs_250_75_spd40_12-infill.gcode"}
octoprint/e10/event/ZChange {"new": 0.9, "_event": "ZChange", "old": 0.7, "_timestamp": 1527612364}

#4

The first thing I see here is that you're using a Raspberry Pi Zero W and it only has one core (as compared to the four on a Raspberry Pi 3). At best, the Zero can serve up a vanilla instance of OctoPi but you have several added plugins also running. It's going to be rather busy just managing the printer without adding more code on top of this. Consider upgrading to the Raspi 3.

I'd like to read the Python code but it's not formatted. I've attempted to copy/paste here but it's clear to me that some of the indenting is lost.

#!/usr/bin/env python

import requests
import sys
import time
import json
import os
import subprocess
import datetime

LogLevel = 0
PrintLevel = 1
LOG_LEVEL_ERROR = 0
LOG_LEVEL_TRACE = 1

def Log(log, message, level):
if(level <= PrintLevel):
print(message + '\n')

if(level <= LogLevel):
	logEntry = datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S") + ' - ' + message + '\n'
	log.write(logEntry)
#end Log

def CheckPrinting(printerApiUrl, headers, tempDetectEnabled, tempThreshExtruder, tempThreshBed, log, lockPath):
isPrinting = False

r = requests.get(
	printerApiUrl,
	headers=headers
	)	
	
Log(log, ('Check Printing status code (%s)' % r.status_code), LOG_LEVEL_TRACE)

if(r.status_code == 200):
	forcePrinting = False
	try:
		jsonOut = r.json()
		printing = jsonOut['state']['flags']['printing']
		paused = jsonOut['state']['flags']['paused']
		extruderTarget = jsonOut['temperature']['tool0']['target']
		bedTarget = jsonOut['temperature']['bed']['target']
		extruderTemp = jsonOut['temperature']['tool0']['actual']
		bedTemp = jsonOut['temperature']['bed']['actual']
		Log(log, ('printing(%s) paused(%s) extruder(cur %s/tgt %s/thresh %s) bedTarget(cur %s/tgt %s/thresh %s)\n' % (printing,paused,extruderTemp,extruderTarget,tempThreshExtruder,bedTemp,bedTarget,tempThreshBed)), LOG_LEVEL_TRACE)			
	except:
		#assume we are printing if we can't read the print status to be safe
		Log(log,("Error reading print status - %s"), LOG_LEVEL_ERROR)
		forcePrinting = True

	#we write this to a file so if we crash or restart during a print, we will know we used to be printing, will require a manual reconnect in these cases
	f = open(lockPath,"w")
	doLock = False
	if(printing == True or paused == True or extruderTarget > 0 or bedTarget > 0 or forcePrinting == True):
		doLock = True
	else:
		if(tempDetectEnabled):
			if(extruderTemp > tempThreshExtruder or bedTemp > tempThreshBed):
				doLock = True
	if(doLock == True):
		isPrinting = True
		f.write('printing')	
	f.close()
	#os.chmod(lockFile, stat.S_IREAD | stat.S_IWRITE | stat.S_IRGRP | stat.S_IWGRP | stat.S_IROTH | stat.S_IWOTH)
else:
	Log(log, ('Failed to get print status (%d)\n' % r.status_code), LOG_LEVEL_ERROR)

if(isPrinting):
	Log(log, ('Printing'), LOG_LEVEL_TRACE)
else:
	Log(log, ('Not Printing'), LOG_LEVEL_TRACE)
return isPrinting
#end CheckPrinting()

#Main()
LOCK_FILENAME = '.printlock'
lockPath = sys.path[0] + '/' + LOCK_FILENAME

LOG_FILENAME = 'oprintmon.log'
logPath = sys.path[0] + '/' + LOG_FILENAME

CONFIG_FILENAME = 'oprintmon.config'
configPath = sys.path[0] + '/' + CONFIG_FILENAME

log = open(logPath, "w+")

Log(log, ('oprintmon start'), LOG_LEVEL_ERROR)

if(os.path.isfile(configPath) == False):
Log(log, ('Failed to load config (%s) - exiting' % configPath), LOG_LEVEL_ERROR)
sys.exit(1)

configFile = open(configPath, "r")

try:
configJson = json.load(configFile)
except:
Log(log, ('Failed to read % - exiting' % configPath), LOG_LEVEL_ERROR)
configFile.close()
sys.exit(1)

octopiConfig = configJson['config']['octopi']
apiKey = octopiConfig['api-key']
baseUrl = octopiConfig['url-base']
port = octopiConfig['serial-port']
try:
baudRate = int(octopiConfig['baud'])
except:
Log(log, ('baudRate must be a number - exiting'), LOG_LEVEL_ERROR)
sys.exit(1)

configConnect = configJson['config']['connect']
try:
sleepTimeConnecting = float(configConnect['sleep-time'])
except:
Log(log, ('sleep-time must be a number - exiting'), LOG_LEVEL_ERROR)	
sys.exit(1)

printmonConfig = configJson['config']['printmon']
tempDetectEnabled = printmonConfig['detection-enabled'] == "true"
if(tempDetectEnabled) :
try:
tempThreshBed = int(printmonConfig['thresh-bed'])
tempThreshExtruder = int(printmonConfig['thresh-hotend'])
except:
Log(log, ('temp thresholds must be numbers - exiting'), LOG_LEVEL_ERROR)	
sys.exit(1)
try:
sleepTimePrinting = float(printmonConfig['sleep-time'])
except:
Log(log, ('sleep-time must be a number - exiting'), LOG_LEVEL_ERROR)	
sys.exit(1)

Log(log, ('api-key (%s), base_url(%s), baud (%s), port (%s), sleeptime(connect) (%s), sleeptime(print) (%s) tempDetect(%s), bedThresh(%s), extruderThresh(%s)' % (apiKey, baseUrl, baudRate, port, sleepTimeConnecting, sleepTimePrinting, tempDetectEnabled, tempThreshBed, tempThreshExtruder)), LOG_LEVEL_TRACE)

headers = {'X-Api-Key': apiKey}
connectionApiURL = baseUrl + 'api/connection'
printerApiUrl = baseUrl + 'api/printer'

isPrinting = False

log.flush()
isConnected = False	
while(1):
connectionState = 'Invalid'
wasConnected = isConnected
wasPrinting = isPrinting
status_code = 0
sleepTime = sleepTimeConnecting

if(os.path.isfile(lockPath)):
	f = open(lockPath, "r")
	lockResult = f.read()
	isPrinting = lockResult == 'printing'
	Log(log, ('Read (%s) isPrinting = %s' % (lockResult, isPrinting)), LOG_LEVEL_TRACE)
	f.close()
	
try:
	Log(log, ('%s\n%s' % (connectionApiURL, headers)), LOG_LEVEL_TRACE)
	r = requests.get(connectionApiURL, headers=headers)
except:
	Log(log, ("failed to send request, server may not be up yet\n"), LOG_LEVEL_ERROR)
	status_code = -1
	
if(status_code != -1):
	status_code = r.status_code
	
#received valid connection info
if(status_code == 200):
	jsonOut = r.json()
	#get connection state
	connectionState = jsonOut['current']['state']
	currentPort = jsonOut['current']['port']
	Log(log,('State = %s Port = %s' % (connectionState, currentPort)), LOG_LEVEL_TRACE)
					
	connecting = False			
	#if we are not connected and not printing, try to connect
	if(not isPrinting):	
		if(connectionState == 'Closed'):
			connecting = True
			json = {
				"command": "connect",
				"port": port,
				"baudrate": baudRate,
			}

			Log(log, ("Connecting to url %s on port %s with baud rate %s\n" % (connectionApiURL, port, baudRate)), LOG_LEVEL_TRACE)

			r = requests.post(
				connectionApiURL,
				json=json,
				headers=headers
			)	

			if(r.status_code == 204):
				Log(log, ("Connect Request Successful\n"), LOG_LEVEL_TRACE)
			else:	
				Log(log, ("Connect Request Failed (%d)\n" % r.status_code), LOG_LEVEL_ERROR)
	else:
		sleepTime = sleepTimePrinting
		
	if(not connecting):
		sleepTime = sleepTimePrinting
		if(connectionState != 'Closed' and connectionState != 'Connecting' and connectionState != 'Disconnecting'):
			isPrinting = CheckPrinting(printerApiUrl, headers, tempDetectEnabled, tempThreshExtruder, tempThreshBed, log, lockPath)

if(connectionState != 'Closed'):
	isConnected = True
	if(not wasConnected):
		Log(log, 'Connected to Server', LOG_LEVEL_ERROR)		
else:
	isConnected = False							
	if(wasConnected):
		Log(log, 'Disconnected from Server', LOG_LEVEL_ERROR)			

if(wasPrinting != isPrinting):
	if(isPrinting):
		Log(log, 'Started Printing', LOG_LEVEL_ERROR)
	else:
		Log(log, 'FinishedPrinting', LOG_LEVEL_ERROR)

Log(log, ('Sleeping for %s seconds' % sleepTime), LOG_LEVEL_TRACE)
log.flush()
time.sleep(sleepTime)
#end Main

#5

Sorry about the formatting error! I edited my original post to clean it up.

Thanks for the advice.

A couple of notes:

I am using a Pi Zero W which I know is not recommended but I'm hoping it is powerful enough to just monitor my sd prints through telegram and stream video. Until now it has been fine. I suspect some sort of leak since it runs for 5 hours or so before it starts to run into errors. I do have a Pi 3 that I've repurposed for testing and will let it run overnight to see if I get the same errors. If I do I'll take a look at the process list as well.

The requests library that I am using doesn't seem to have any way to close the connection, so either I am missing it, or it doesn't require it (or there is a bug in the library). If the Pi 3 test has problems I'll take a look at sending the http requests a different way to rule that out.


#6

First thing I'd check here is if the behaviour is reproducible in safe mode (that's also something that would be part of the steps to try before even opening a ticket). I've so far not encountered that particular behaviour, and in principle it should be perfectly fine to ping the API twice per minute.


#7

Thanks. I ran it on my Pi 3 last night with the same results. I turned off my service and I am running it today to see if it is related to one of the plugins I have running. I'll do the safe mode test tonight. This morning when it was in the state of not being able to create threads I ran htop and saw that there were a lot of instances of this running:

/home/pi/oprint/bin/python2 /home/pi/oprint/bin/octoprint serve --host=127.0.0.1 --port=5000

pi@octopi:~ $ htop

  1  [||                                                                                                      1.3%]   Tasks: 35, 242 thr; 1 running
  2  [||||||                                                                                                  4.5%]   Load average: 0.11 0.07 0.01
  3  [                                                                                                        0.0%]   Uptime: 10:51:40
  4  [                                                                                                        0.0%]
  Mem[|||||||||||||||||||||||||||||||||||||||||||||                                                      154M/876M]
  Swp[                                                                                                   0K/100.0M]

  PID USER      PRI  NI  VIRT   RES   SHR S CPU% MEM%   TIME+  Command
 6260 pi         20   0 28820  1488    92 S  0.0  0.2  0:00.00 (sd-pam)
 6270 pi         20   0  6108  4136  2852 S  0.0  0.5  0:00.26 -bash
  421 pi         20   0  5120  2988  2700 S  0.0  0.3  0:01.26 /bin/bash /root/bin/webcamd
  419 pi         22   2 2021M 95148 13936 S  0.6 10.6 14:04.24 /home/pi/oprint/bin/python2 /home/pi/oprint/bin/octoprint serve --host=127.0.0.1 --port=5000
  618 pi         20   0 2021M 95148 13936 S  0.0 10.6  2:37.46 /home/pi/oprint/bin/python2 /home/pi/oprint/bin/octoprint serve --host=127.0.0.1 --port=5000
  587 pi         20   0 2021M 95148 13936 S  0.0 10.6  1:19.32 /home/pi/oprint/bin/python2 /home/pi/oprint/bin/octoprint serve --host=127.0.0.1 --port=5000
  615 pi         20   0 2021M 95148 13936 S  0.0 10.6  0:52.72 /home/pi/oprint/bin/python2 /home/pi/oprint/bin/octoprint serve --host=127.0.0.1 --port=5000
  603 pi         20   0 2021M 95148 13936 S  0.0 10.6  0:49.50 /home/pi/oprint/bin/python2 /home/pi/oprint/bin/octoprint serve --host=127.0.0.1 --port=5000
  606 pi         20   0 2021M 95148 13936 S  0.0 10.6  0:36.60 /home/pi/oprint/bin/python2 /home/pi/oprint/bin/octoprint serve --host=127.0.0.1 --port=5000
  584 pi         20   0 2021M 95148 13936 S  0.0 10.6  0:32.85 /home/pi/oprint/bin/python2 /home/pi/oprint/bin/octoprint serve --host=127.0.0.1 --port=5000
  488 pi         20   0 2021M 95148 13936 S  0.0 10.6  0:32.54 /home/pi/oprint/bin/python2 /home/pi/oprint/bin/octoprint serve --host=127.0.0.1 --port=5000
  693 pi         20   0 2021M 95148 13936 S  0.0 10.6  0:18.01 /home/pi/oprint/bin/python2 /home/pi/oprint/bin/octoprint serve --host=127.0.0.1 --port=5000
  612 pi         20   0 2021M 95148 13936 S  0.0 10.6  0:05.63 /home/pi/oprint/bin/python2 /home/pi/oprint/bin/octoprint serve --host=127.0.0.1 --port=5000
  611 pi         20   0 2021M 95148 13936 S  0.0 10.6  0:04.94 /home/pi/oprint/bin/python2 /home/pi/oprint/bin/octoprint serve --host=127.0.0.1 --port=5000
  486 pi         20   0 2021M 95148 13936 S  0.0 10.6  0:02.53 /home/pi/oprint/bin/python2 /home/pi/oprint/bin/octoprint serve --host=127.0.0.1 --port=5000
  586 pi         20   0 2021M 95148 13936 S  0.0 10.6  0:00.00 /home/pi/oprint/bin/python2 /home/pi/oprint/bin/octoprint serve --host=127.0.0.1 --port=5000
  601 pi         20   0 2021M 95148 13936 S  0.0 10.6  0:00.00 /home/pi/oprint/bin/python2 /home/pi/oprint/bin/octoprint serve --host=127.0.0.1 --port=5000
  602 pi         20   0 2021M 95148 13936 S  0.0 10.6  0:00.00 /home/pi/oprint/bin/python2 /home/pi/oprint/bin/octoprint serve --host=127.0.0.1 --port=5000
  604 pi         20   0 2021M 95148 13936 S  0.0 10.6  0:00.00 /home/pi/oprint/bin/python2 /home/pi/oprint/bin/octoprint serve --host=127.0.0.1 --port=5000
  605 pi         20   0 2021M 95148 13936 S  0.0 10.6  0:00.00 /home/pi/oprint/bin/python2 /home/pi/oprint/bin/octoprint serve --host=127.0.0.1 --port=5000
  691 pi         20   0 2021M 95148 13936 S  0.0 10.6  0:00.00 /home/pi/oprint/bin/python2 /home/pi/oprint/bin/octoprint serve --host=127.0.0.1 --port=5000
 2179 pi         20   0 2021M 95148 13936 S  0.0 10.6  0:00.00 /home/pi/oprint/bin/python2 /home/pi/oprint/bin/octoprint serve --host=127.0.0.1 --port=5000
 2187 pi         20   0 2021M 95148 13936 S  0.0 10.6  0:00.00 /home/pi/oprint/bin/python2 /home/pi/oprint/bin/octoprint serve --host=127.0.0.1 --port=5000
 2201 pi         20   0 2021M 95148 13936 S  0.0 10.6  0:00.00 /home/pi/oprint/bin/python2 /home/pi/oprint/bin/octoprint serve --host=127.0.0.1 --port=5000
 2209 pi         20   0 2021M 95148 13936 S  0.0 10.6  0:00.00 /home/pi/oprint/bin/python2 /home/pi/oprint/bin/octoprint serve --host=127.0.0.1 --port=5000
 2223 pi         20   0 2021M 95148 13936 S  0.0 10.6  0:00.00 /home/pi/oprint/bin/python2 /home/pi/oprint/bin/octoprint serve --host=127.0.0.1 --port=5000
 2231 pi         20   0 2021M 95148 13936 S  0.0 10.6  0:00.00 /home/pi/oprint/bin/python2 /home/pi/oprint/bin/octoprint serve --host=127.0.0.1 --port=5000
 2247 pi         20   0 2021M 95148 13936 S  0.0 10.6  0:00.00 /home/pi/oprint/bin/python2 /home/pi/oprint/bin/octoprint serve --host=127.0.0.1 --port=5000
 2255 pi         20   0 2021M 95148 13936 S  0.0 10.6  0:00.00 /home/pi/oprint/bin/python2 /home/pi/oprint/bin/octoprint serve --host=127.0.0.1 --port=5000
 2283 pi         20   0 2021M 95148 13936 S  0.0 10.6  0:00.00 /home/pi/oprint/bin/python2 /home/pi/oprint/bin/octoprint serve --host=127.0.0.1 --port=5000
 2291 pi         20   0 2021M 95148 13936 S  0.0 10.6  0:00.00 /home/pi/oprint/bin/python2 /home/pi/oprint/bin/octoprint serve --host=127.0.0.1 --port=5000
 2307 pi         20   0 2021M 95148 13936 S  0.0 10.6  0:00.00 /home/pi/oprint/bin/python2 /home/pi/oprint/bin/octoprint serve --host=127.0.0.1 --port=5000
 2317 pi         20   0 2021M 95148 13936 S  0.0 10.6  0:00.00 /home/pi/oprint/bin/python2 /home/pi/oprint/bin/octoprint serve --host=127.0.0.1 --port=5000
 2331 pi         20   0 2021M 95148 13936 S  0.0 10.6  0:00.00 /home/pi/oprint/bin/python2 /home/pi/oprint/bin/octoprint serve --host=127.0.0.1 --port=5000
 2339 pi         20   0 2021M 95148 13936 S  0.0 10.6  0:00.00 /home/pi/oprint/bin/python2 /home/pi/oprint/bin/octoprint serve --host=127.0.0.1 --port=5000
 2353 pi         20   0 2021M 95148 13936 S  0.0 10.6  0:00.00 /home/pi/oprint/bin/python2 /home/pi/oprint/bin/octoprint serve --host=127.0.0.1 --port=5000
 2361 pi         20   0 2021M 95148 13936 S  0.0 10.6  0:00.00 /home/pi/oprint/bin/python2 /home/pi/oprint/bin/octoprint serve --host=127.0.0.1 --port=5000
 2376 pi         20   0 2021M 95148 13936 S  0.0 10.6  0:00.00 /home/pi/oprint/bin/python2 /home/pi/oprint/bin/octoprint serve --host=127.0.0.1 --port=5000
 2384 pi         20   0 2021M 95148 13936 S  0.0 10.6  0:00.00 /home/pi/oprint/bin/python2 /home/pi/oprint/bin/octoprint serve --host=127.0.0.1 --port=5000
 2398 pi         20   0 2021M 95148 13936 S  0.0 10.6  0:00.00 /home/pi/oprint/bin/python2 /home/pi/oprint/bin/octoprint serve --host=127.0.0.1 --port=5000
 2406 pi         20   0 2021M 95148 13936 S  0.0 10.6  0:00.00 /home/pi/oprint/bin/python2 /home/pi/oprint/bin/octoprint serve --host=127.0.0.1 --port=5000
 2420 pi         20   0 2021M 95148 13936 S  0.0 10.6  0:00.00 /home/pi/oprint/bin/python2 /home/pi/oprint/bin/octoprint serve --host=127.0.0.1 --port=5000
 2429 pi         20   0 2021M 95148 13936 S  0.0 10.6  0:00.00 /home/pi/oprint/bin/python2 /home/pi/oprint/bin/octoprint serve --host=127.0.0.1 --port=5000
 2443 pi         20   0 2021M 95148 13936 S  0.0 10.6  0:00.00 /home/pi/oprint/bin/python2 /home/pi/oprint/bin/octoprint serve --host=127.0.0.1 --port=5000
 2452 pi         20   0 2021M 95148 13936 S  0.0 10.6  0:00.00 /home/pi/oprint/bin/python2 /home/pi/oprint/bin/octoprint serve --host=127.0.0.1 --port=5000
 2466 pi         20   0 2021M 95148 13936 S  0.0 10.6  0:00.00 /home/pi/oprint/bin/python2 /home/pi/oprint/bin/octoprint serve --host=127.0.0.1 --port=5000
 2475 pi         20   0 2021M 95148 13936 S  0.0 10.6  0:00.00 /home/pi/oprint/bin/python2 /home/pi/oprint/bin/octoprint serve --host=127.0.0.1 --port=5000
 2491 pi         20   0 2021M 95148 13936 S  0.0 10.6  0:00.00 /home/pi/oprint/bin/python2 /home/pi/oprint/bin/octoprint serve --host=127.0.0.1 --port=5000
 2499 pi         20   0 2021M 95148 13936 S  0.0 10.6  0:00.00 /home/pi/oprint/bin/python2 /home/pi/oprint/bin/octoprint serve --host=127.0.0.1 --port=5000
 2513 pi         20   0 2021M 95148 13936 S  0.0 10.6  0:00.00 /home/pi/oprint/bin/python2 /home/pi/oprint/bin/octoprint serve --host=127.0.0.1 --port=5000
 2521 pi         20   0 2021M 95148 13936 S  0.0 10.6  0:00.00 /home/pi/oprint/bin/python2 /home/pi/oprint/bin/octoprint serve --host=127.0.0.1 --port=5000
 2535 pi         20   0 2021M 95148 13936 S  0.0 10.6  0:00.00 /home/pi/oprint/bin/python2 /home/pi/oprint/bin/octoprint serve --host=127.0.0.1 --port=5000
 2543 pi         20   0 2021M 95148 13936 S  0.0 10.6  0:00.00 /home/pi/oprint/bin/python2 /home/pi/oprint/bin/octoprint serve --host=127.0.0.1 --port=5000
 2558 pi         20   0 2021M 95148 13936 S  0.0 10.6  0:00.00 /home/pi/oprint/bin/python2 /home/pi/oprint/bin/octoprint serve --host=127.0.0.1 --port=5000
 2566 pi         20   0 2021M 95148 13936 S  0.0 10.6  0:00.00 /home/pi/oprint/bin/python2 /home/pi/oprint/bin/octoprint serve --host=127.0.0.1 --port=5000
 2580 pi         20   0 2021M 95148 13936 S  0.0 10.6  0:00.00 /home/pi/oprint/bin/python2 /home/pi/oprint/bin/octoprint serve --host=127.0.0.1 --port=5000
 2589 pi         20   0 2021M 95148 13936 S  0.0 10.6  0:00.00 /home/pi/oprint/bin/python2 /home/pi/oprint/bin/octoprint serve --host=127.0.0.1 --port=5000
 2603 pi         20   0 2021M 95148 13936 S  0.0 10.6  0:00.00 /home/pi/oprint/bin/python2 /home/pi/oprint/bin/octoprint serve --host=127.0.0.1 --port=5000
 2612 pi         20   0 2021M 95148 13936 S  0.0 10.6  0:00.00 /home/pi/oprint/bin/python2 /home/pi/oprint/bin/octoprint serve --host=127.0.0.1 --port=5000
 2626 pi         20   0 2021M 95148 13936 S  0.0 10.6  0:00.00 /home/pi/oprint/bin/python2 /home/pi/oprint/bin/octoprint serve --host=127.0.0.1 --port=5000
 2634 pi         20   0 2021M 95148 13936 S  0.0 10.6  0:00.00 /home/pi/oprint/bin/python2 /home/pi/oprint/bin/octoprint serve --host=127.0.0.1 --port=5000
F1Help  F2Setup F3SearchF4FilterF5Tree  F6SortByF7Nice -F8Nice +F9Kill  F10Quit

#8

yeah, htop is showing that you are leaving connections open. Pretty safe to bet that your script is to blame. A couple of suggestions:

  • since you are using a longrunning script, use a session object for your request. It'll maintain a connection pool.
  • use with requests.get().. as r to ensure that r goes out of scope
  • likewise, you could wrap requests into a method, which'll ensure everything goes out of scope as you'll only be returning text or json.

Obviously, while a Zero may be underpowered, that's not the reason you are having problems. Your problem is leaving too many connections open to octoprint, which will take down a server of any size.


#9

I suspect that Telegram Notifications did it in the Library with the Candlestick.


#10

Thanks for the suggestions, looks like I have some stuff to dig into now!


#11

ping, @HeySideburns.


#12

I have the same issue..
I wrote a very simple bash script and put it in my crontab to run every minute.
It checks the connection status and tries to connect if it's offline. It works fine at the beginning but after about 3 hours it bugs (can't create threads).

Here's my script.
checkconnection.sh:
if (( $( curl -s http://localhost/api/connection?apikey=MYAPIKEY | grep -c "Closed") == 1 )) ; then /home/pi/autoconnect.sh; fi
autoconnect.sh:
curl -X POST http://localhost/api/connection -H "X-Api-Key:MYAPIKEY" -H 'Content-Type: application/json; charset=utf-8' --data "@/home/pi/connectioninfo"

Anyone has an idea how this can be solved or if there is a better way to automatically connect to a printer without PortLister (lsusb is the same with printer off or on...)?


#13

If it were me, I think I'd run some sort of ps command on the server to make sure that these connections aren't staying open. Not sure, but you could try ps -ax|grep curl and see if they're still there.

In my case, I was trying to do some remote ssh commands and at that time, I didn't know to add a -t which tends to help ssh top not stay open on the server-side. So I was shooting myself in the foot, so-to-speak.