Better way to run asynchronously?

Jamiek · March 16, 2022, 4:14pm

I'm writing a plugin that is a scanner of sorts. It takes a picture, analyzes the picture, moves the machine, and repeats. This is only intended for when the machine is not printing. This can't be a gcode program because the movement and stopping condition are dependent on the image analysis.

Currently I have it so it starts from a trigger that's exposed by a flask blueprint. It queues motion and then G4 and then a 'special' M117 command. The M117 command is picked up with a gcode sending hook, based on the assumption that the M117 won't be sent until the G4 returns 'ok', meaning all the previous commands are complete.

Within the gcode hook, it does its work, taking a picture, doing the analysis, queuing the next movement, and then releasing control, waiting to be triggered again by the 'special' M117 command. It has a state machine to track its progress within the scan.

I have a simple prototype that works, but I am wondering about the advisory to not perform any long-running operations within the gcode hooks, or it would stall the send loop. I can understand how stalling during a print job could easily degrade print performance. If I am not printing, but just using the machine as a Python-driven motion platform, do I need to worry about stalling the send loop? Is there a risk that the server might freak out and cause other bad things to happen if the loop is stalled too long?

And if so, what is a better alternative? I could imagine a separate process that sends and receives messages, so the hook just notifies the other process and does not block the send loop. The other process gets the notification and then does its work asynchronously and then somehow inserts gcode back into the queue. I have avoided multiprocessing in Python so I don't exactly know how to do this, but I could figure it out if it is the proper way.

So the question is, how bad is it to block the send loop? And if it is important not to block the queue, what is the best alternative?

I am unwilling to push any of the responsibility onto the client javascript (if that is even possible). The server must continue operating even if the browser disconnects.

foosel · March 16, 2022, 4:40pm

It depends. If your printer does regular autoreporting, then no. If it doesn't, then OctoPrint will attempt to query the temperature from it regularly, and if it can't do that for a couple of tries (as in, it doesn't get a response) it will consider the printer dead and disconnect.

A process, or just a simple thread really. Spawn a worker thread that sleeps most of the time, have your gcode hook set some event or add something to a queue that the thread waits for, let it work, send it back to sleep, rinse, repeat.

Jamiek · March 16, 2022, 5:25pm

Thank you. I will go with the background thread approach. I'm sure I could turn off autoreporting, but I can't guarantee that others trying to use the plugin (or future me) won't trip over a timeout issue that blows the application. I will need to learn a bit, but it's pretty clear this is the proper approach.

OllisGit · March 17, 2022, 8:46am

Hi @Jamiek,

from my experience you should never ever block the gcode-hooks!

I did this in my DisplayLayerProgress-Plugin and it not just decrease the performance or stall, it could create ugly artifacts during the print (e.g. layer-shifts..maybe not relevant for your usecase).

So, I added a command-queue that use its own thread and the assigned worker could process the gcode in an async way.

github.com

OllisGit/OctoPrint-DisplayLayerProgress/blob/86ae2a443482e3a348b386221e7b13bdd9c4c684/octoprint_DisplayLayerProgress/CommandQueue.py

# coding=utf-8
from __future__ import absolute_import

import re
from queue import Queue
from queue import Empty
from threading import Thread

import time
import logging

class CommandQueue():

    def __init__(self):
        self._logger = logging.getLogger(__name__)
        self._worker = None
        self._workerMethod = None
        self._executionCommandQueue = Queue()
        self._isQueueRunning = False
        self._isPrintJobRunning = False

This file has been truncated. show original

Jamiek · March 19, 2022, 7:36pm

I got the threading working, and it ended up much easier that way because I could ditch all the state machine stuff and just write regular blocking code in the worker thread.

It is at a point where it is just beginning to work, but not very polished:

This is using Raspberry Pi and USB webcam, although a Raspberry Pi camera should also work if you have a ribbon cable long enough.

markeby · March 22, 2022, 2:43pm

Very impressive. Please share with us the exact design, camera mounting, software, and details. This is something I am definitely interested in!

Jamiek · March 24, 2022, 5:12am

This is now on Github here, although it's probably not ready for prime-time. The camera mount was a custom mount I made that fits an older version of MPCNC + DW611 mount and adds an adapter for a Logitech C170 and holds a 38mm diameter lens with a focal length of 100 mm. Extremely specific to what I have on hand and probably not generally useful. More useful would be a Raspberry Pi camera mount, and those have adjustable lenses that can achieve decent focus at closer distances.

The code and the blurb on Github is probably the best place to start, and I can answer specific questions when you get there.