The "safe" gcodes for afterPrintPaused and beforePrintResumed hooks?


#1

We wrote an OctoPrint plugin and here is what we want it to do:

  1. after print is paused, retract filament, home the print head, and turn off extruder heater.
  2. before print is resumed, re-heat extruder heater until the previous temperature, move print head the previous position, and feed filament the same retraction distance.

Based on OctoPrint documents we feel afterPrintPaused and beforePrintResumed hooks are the right place (let me know if we are wrong). But since the user can also set custom gcodes for these 2 hooks, we are not sure how we can craft the gcodes that they won't interfere the ones that user sets.

I'm an experienced programmer but a gcode noob. So any help is much appreciated!

PS. A bit more background about our plugin: it is a machine learning-based algorithm that automatically detects print failures based on webcam feed. When a possible failure is detected, it'll automatically pause the print and alert user. We have implemented everything in python but it caused a lot of problems and hence we believe that gcodes returned by the hooks should be the right way to do it.


#2

I wouldn't use those hooks at all for this purpose since your plugin will be the one to pause the print. What I would do is put the print job on hold and manage the resume process yourself. That way you can just send whatever gcode you like. It probably won't be easy to make 100% compatible pause/resume script, so consider allowing the user to customize that.

As a note, job_on_hold also stops any OctoPrint custom scripts from executing. See this issue for details.

Be careful with that lock, however. If you don't release it, OctoPrint will stop working since it won't be able to send any gcodes.

Edit: Neat plugin, by the way! Did I send you guys timelapse videos of failures a long time ago for this project, or was that another failure detection project? I uploaded a bunch to some FTP server a while back. If it was yours, I'm glad you've made such progress!


#3

And while we're on the subject of safe pausing/resuming do note that it's difficult at the moment to ask the firmware what mode(s) it was in with respect to axis and extruder movement: absolute or relative. And if you don't know and change this in your script (and don't put it back correctly when you're finished and zero out any necessary settings) then up to two of these four bad things could potentially happen:

  1. the print will resume in the wrong place, ruining the part
  2. the hotend will drill into the bed, ruining the bed in some cases
  3. the hotend will come down and plop a bunch of wasted filament before continuing
  4. the hotend will dry-print for, say, 10mm before actually extruding filament

#4

Once could use a position/state tracker, like the c++ version we were discussing in another thread, or the older and much much slower version that is in the current Octolapse release. It would be more efficient in their case to to write one that only cares about the XYZ and E axis modes, which is pretty easy to track. Now knowing the initial state is another matter entirely. I recently added a feature to my plugin that forces G90/91 M82/83 commands to be sent BEFORE the first line of gcode from the file is sent. I still can't believe there is start gcode that contains neither of those commands, but I'll save that for another rant.

Also, knowing the Z height might be important since some kind of z-lift is likely necessary to prevent a hot nozzle from melting part of the print, or to keep the nozzle from snagging on a partially completed layer. Don't want to z-hop into the top of the z axis either! It is trickier than it seems to track this due to things like firmware retract (which might hop) and the need to acquire the initial z position after a home or mesh bed level, since the final X, Y, and Z position can vary each time you run it depending on the printer. However, for this purpose I'd say it would be OK to waits to perform any Z height checks (also need to know the builld volume for this!) until AFTER the plugin receives an absolute Z position. Not many people use firmware retract either, so that could probably be ignored (or better yet, a warning could be generated).

TLDR: I agree that axis mode probably can't be ignored safely if they are generating this gcode. If it's user supplied, it becomes more reasonable, but also harder to configure.


#5

Hi @FormerLurker, I love your plugin too. Easily one of the coolest, if not THE coolest, OctoPrint plugins.

job_on_hold sounds like what we were looking for ! Thank you so much for the pointer! I'll play with it a bit to see if it functions as how I understand it by the document.

I don't think we got timelapse videos from you. Will really appreciate it if you can share them with us! My email is k@thespaghettidetective.com.


#6

Oh @FormerLurker @OutsourcedGuru Since you are both well-respected in OctoPrint circle and spending time to help us figure this out - hit me up when/if you want a free account with The Spaghetti Detective, or need personal tech support in case you want to run your own server.


#7

And don't forget the scenario when you're within 5mm of the top of the available print volume and the user asks for a pause/resume event and you're coded to "lift the hotend 10mm and then lower it 10mm" = recipe for fail.


#8

Thanks for your kind words Kenneth!

One final thought: if you use job_on_hold, make sure you edit your setup.py and require at least OctoPrint 1.3.9, else you may run into some problems related to custom OctoPrint scripts (prior to this job_on_hold did not stop these scripts per the issue I posted earlier).


#9

Will do. Thank you for the tip!

I tested job_on_hold and have a feeling that it was designed for a brief moment of pause. The reason why I said that is that both "pause" and "cancel" buttons are still clickable in OcotPrint. When I clicked these buttons, they actually kicked in AFTER the hold is lifted.

Since my plugin will potentially put job on hold for long time (several hours, if it happens in the middle of the night), I'm wondering if job_on_hold is the right solution for me.

Is the reason why you don't think afterPrintPaused and beforePrintResumed hooks are the way to go because other custom scripts may get in the way? @FormerLurker


#10

That was the primary reason Kenneth. It would be possible to disable the pause button (Octolapse does this during snapshots) by sending a message to the client, though there would be a delay so you still would need to deal with the pause script in some edge cases. Why would you want to prevent cancel though? I'd imagine you would look for any Cancel events and release your job lock and cancel the print in that case.

Here's an alternative that I just thought of. Pause the print via the API yourself, and then monitor the tags parameter in the on_gcode_queuing hook, and just return (None, ) to prevent those gcode lines from being sent to the printer. This will effective kill them so they will never reach the printer. How does that sound?

Edit: Not sure what the tag for the pause script is exactly, but I think it probably will look something like this:

source:SOMETHING_HERE

If you log the tags and pause the print, it should be obvious. When Octolapse sends gcode that tag looks like this: plugin:octolapse, that much I know :slight_smile:


#11

I'm testing my code to set_job_on_hold then retract filament and lift Z. The piece of code that does this is here: https://github.com/TheSpaghettiDetective/OctoPrint-TheSpaghettiDetective/blob/d6dd34433f8c87555d1e3342bd2dc80d68f5c344/octoprint_thespaghettidetective_beta/commander.py#L24

However, in one of the tests I did on my printer, OctoPrint kept on sending more gcodes after set_job_on_hold(True) is called. The gcode log is like this:

226 Recv: ok 40
227 Recv: ok 41
228 Recv: ok 42
229 Send: N43 G91*38  <--- These lines obviously came from my code, which should be AFTER `printer.set_job_on_hold(True)` is called.
230 Send: N44 M83*40  <---
231 Send: N45 G1 E-5.0*122 <---
232 Send: N46 G1 Z5.0*75  <----
233 Recv: ok 43
234 Recv: X:100.00 Y:100.00 Z:0.40 E:-4.48
235 Recv: ok 44
236 Recv: ok 45
237 Recv: ok 46
238 Send: N47 G1 X62.288 Y62.279*20 <--- But here more gcodes were sent, and sent my print head off boundary.
239 Send: N48 M400*27
240 Recv: ok 47
241 Recv: ok 48
242 Send: N49 M114*26
243 Recv: ok 49
244 Recv: X:162.29 Y:162.28 Z:5.40 E:-9.48
245 Recv: wait
246 Send: N50 G1 E4.50000 F2400.000*41
247 Send: N51 M82*45
248 Recv: ok 50
249 Recv: ok 51
250 Send: N52 G1 F1200.000*100
251 Send: N53 G1 X63.060 Y61.602 E0.08838*95

Any reason why more gcodes can be sent even after printer.set_job_on_hold(True) is called @FormerLurker ? Or maybe this is a question for @foosel ?


#12

I can answer this, no need to bother @foosel. The reason that gcode sent was that it was queued before (or while) you paused the print. I hadn't thought about that, but Octolapse needs to deal with this too. Here's an outline of what might work, but note that I typed this in without checking for errors of logic or syntax:

    def __init__(self, printer):
        self.job_is_on_hold = False
        # save a reference to the octoprint printer object
        self.printer = printer
        # member to hold a command that will need to be sent when we unpause
        self.save_command = None
        # set this to true somewhere to pause the print, wherever that is
        self.pausing = False
        # I think you'll need to deal with cancelling as well.  Not sure
        self.cancelling = False

    # you would call this from an event handler that is looking for cancelling events.
    # Probably from your main __init__ file
    def print_cancelling(self):
        # we only need to handle this if the job is on hold.
        if self.job_is_on_hold:
            # just in case, clear the pausing flag
            self.pausing = False
            # release the lock.  This is really important
            self.printer.set_job_on_hold(False)
            # you may have to reset the axis mode here if you've changed it in pause print
            reset_axis_mode_gcode = self.get_reset_axis_mode_gcode()
            # only send commands if there are commands to send.
            if len(reset_axis_mode_gcode) > 0:
                # Since I'd expect cancel/end gcode to work at this point, reset the axis mode.
                # also, FYI, if the extruder is cold, and the end gcode moves the E axis at all (retract maybe)
                # you could get jams here.  Not sure what you want to do here exactly, but I think this is
                # surmountable.  You could probably just suppress the end/cancel gcode since that would be simpler.
                self.printer.commands(reset_axis_mode_gcode, tags=set(['TSD']))

    def get_reset_axis_modes_gcode(self):
        # need to implement this
        return []

    def get_set_axis_modes_gcode(self):
        # need to implement this
        return []

    def pause_print(self, save_command):
        # why the mutex here?  I admit I haven't looked at the rest of your source though, so maybe you need it?
        with self.mutex:
            self.printer.set_job_on_hold(True)
            self.job_is_on_hold = True
            self.pausing = False

        # you need to save any commands that are queuing, else they will be lost
        self.save_command = save_command
        # send your pause commands now, you'll get no interference
        self.printer.commands(self.get_pause_gcode(), tags=set(['TSD']))

        # now this might not work.  Octolapse used to use the pause function, but I ran into issues.
        # It might work in your case.  If not, you can just deal with the pause button manually.
        # do this last so that you can get that retract command to the printer ASAP!
        self.printer.pause_print()

    def get_pause_gcode(self):
        # create your pause gcode array here
        # set the axis mode
        pause_gcode = self.get_set_axis_modes_gcode()
        # some gcode generation code you'll need to write goes here
        pause_gcode.extend(GCODES_NEEDED_TO_PAUSE)
        return pause_gcode

    def resume_print(self):
        # send your resume commands now, you'll get no interference
        self.printer.commands(self.get_resume_gcode(), tags=set(['TSD']))

        # whatever the resume command is, send it here.  I can't recall off the top of my head either :)
        # The same caveats apply to resume as I mentioned for pause above.
        self.printer.resume_print()

        # why the mutex here?  I admit I haven't looked at the rest of your source though, so maybe you need it?
        # release the lock last so that you can be assured all of your gcodes have sent
        with self.mutex:
            self.printer.set_job_on_hold(False)
            self.job_is_on_hold = False

    def get_resume_gcode(self):
        # some gcode generation code you'll need to write goes here
        resume_gcode = [GCODES_YOU_NEED_TO_RESUME]
        # reset the axis mode to what it was before you paused.
        resume_gcode.extend(self.get_reset_axis_modes_gcode())
        # Send the command that was queuing when you paused.
        resume_gcode.append(self.save_command)
        return resume_gcode


    def on_gcode_queuing(self, comm_instance, phase, cmd, cmd_type, gcode, subcode=None, tags=None, *args, **kwargs):

        # you may need to rethink this part if you decide to reset the axis mode on cancel.  In that case
        # your axis mode tracking will need to know that you switched axis modes, but your code below won't
        # see any of the G90/G91 M82/M83 commands.  There are other ways to handle this, of course, like just
        # storing the original axis mode when you pause.
        if 'TSD' in tags:
            return

        # somewhere in your code you will set self.pausing = True to indicate you want to pause.
        # the actual pause will occur when the next gcode is queuing.  You may have to deal with an edge case
        # where you pause AFTER the last line of gcode is sent to the printer.  That would be extremely unlikely
        # though :)

        # need to check for for pause BEFORE you do any axis mode checks since
        # this command will be suppressed if we are pausing
        if self.pausing:
            # you may need a reference to the printer here, not sure where that is coming from
            self.pause_print(cmd)
            # suppress the gcode command so that it doesn't send (this prevents the problem you were having)
            # you will have to send it when you resume
            return (None ,)

        # you might have to check for and suppress cancel/end gcode here if the print was cancelled while your
        # plugin is paused.  Depends on how you handle cancelling.

        # now you can track the axis mode!
        with self.mutex:  # why are you using a mutex here?  I admit I haven't looked at the calling code though.
            if re.match('G9[01]', cmd, flags=re.IGNORECASE):
                self.last_g9x = cmd
                print('commander setting: {}'.format(self.last_g9x))
            if re.match('M8[23]', cmd, flags=re.IGNORECASE):
                self.last_m8x = cmd

It's not complete, but it should get you started. I tried to comment everything as much as possible.


#13

Wow THANK YOU @FormerLurker. Feel like I can just cut and paste your code and call it a day. :smiley:

How I currently handle cancelling is to just blindly reset mode to the "last tracked" one. Since I filtered my own gcode in on_gcode_queuing, the "last tracked" one should be whatever it should be whether or not it's currently on hold by TSD plugin. The only caveat seems to be that G90/91 and M82/83 need to be idempotent because TSD plugin will always resend the last tracked mode upon job cancelling. Hope this is not a problem for any printer.

A quick note on the mutex: the reason why I added was because I assumed on_gcode_queuing will be called from another thread than the rest of the methods (called from a thread TSD plugin created). Although I don't have a hard evidence on whether thread-safety is a real issue for this scenario, but my other plugin (OctoPrint Anywhere) used to randomly cause OctoPrint to hang (with heaters still on). After I added mutex all over the place in my code, there hasn't been a single report of such incident.


#14

The info regarding the mutex is Interesting Kenneth. I don't see how this can be the case in on_gcode_queuing since it expects you to return None in order to mutate or suppress a gcode. This might be a question for @foosel.


#15

It's probably more about how my plugin is implemented. Since my plugin has to have a consistent communication with the server, I had to create a thread (https://github.com/TheSpaghettiDetective/OctoPrint-TheSpaghettiDetective/blob/84d7bcc75b776d118ed44d591a2949d843ff22dc/octoprint_thespaghettidetective/init.py#L187) and the bulk of my code will be executed in that thread. This means any "callback" will be running in a different thread. Will love to know if I can get away without using a separate thread.


#16

Ok, that makes sense. I can't think of a way to remove that thread. You might consider using a queue to send commands to the main thread, however. That way you could just check the queue for commands (pause, resume, etc) from on_gcode_queuing, which could run from the main thread. The queue class is thread safe, and quite useful.