What is the best way to modify g-code file before printing (not using g-code hooks)?

Problem:
Since Version 1.15.0 my „DisplayLayerProgress-Plugin (DLP)“ use the FilemanagerPreprocessor-Hook during upload to replace layer-comments (like ;LAYER:10) with M117 layer indicators (M117 INDICATOR-Layer10).
The layer-comments were generated from the slicer (mostly out of the box).

The replacement is necessary, because during the print-job only g-code commands were send to plugin-hooks and not the comments.

The behaviour with the file-preprocessor has some drawbacks:

  • if uploading large files (>50MB) the user received sometimes a timeout, because pre-processing takes some time. But the files is processed -> user needs to do a reload
  • Using Cura with the „Ultimaker Format Package)“-Plugin is not working, because a UFP-File (a.k.a Zip-File) is send to octoprint and the DLP-preprocessor could not handle the file during upload
  • G-Code files which copied directly (via scp or Git-Plugin) to the upload-folder were not processed

So, I changed the behaviour from upload- to file-selection event in Version 1.16.0.

Now, all mentioned drawbacks were gone, yeah……but now I have new ones:

  • If the user use the small button next to each file in octoprint „Load and Print“ or
  • using the autostart-start features from some slices, then it is not working!

Reason:
The Eventmanager works in a async-way.
In may case: File-Selected event fired...fine, I can start analysing the layer-comments, but also Print-Started immediately, so my plugin has not enough time to modify the M117 layer indicators in the selected code.

The print job run with the orig. g-code without layer indicators.

Current tryouts:

  • Prevent/Disable the „Load and Print“ button -> only one issue solved
  • Use of AnalysisQueue my assumption was that this analyse is done in sync way and always before starting a print-job, so I can place a sleep until the file-selected event is processed. BUT it is also executed in async..grrrr!
  • Implement in MachineCom my custom callback, because then I have the opportunity to overwrite the method on_comm_print_job_started. But I was not able to inject my custom callback. Has someone an idea how to do this?

Overall:
I am stuck, every idea will be welcome!

Thx, in advance
Olli

My general approach is to bring all this out of OctoPrint. I would guess that a slicer-based post-processing script is the best location. But it doesn't solve your needs if you're trying to build a plugin.

And I also write a variety of Go language pre-processors. They run pretty fast. You might even consider writing something in assembly since you're talking about a simple edit. But that would require you knowing which processor they're on.

The user is expected to drop the file(s) into the watch folder if they want them analyzed, for what it's worth.

Yeah, I could change the UFP plugin to put those files in the watched folder rather than directly in the upload folder. It was something that I've been meaning to do anyway, which would possibly resolve one of the issues. I guess the question is if a gcode file is put into the watched folder does your plugin get triggered when it's transferred to the uploads folder?

I'd have to walk the code to find out if it did.

If I were doing the UFP plugin I'd create another folder, upload the file into here, then process it there. When you have the gcode unzipped and pre-processed then drop it into the watch folder and you should be done at this point.

I don't process gcode, just extract it.

Hi @OutsourcedGuru, @jneilliii

thanks for the discussion.

I think my current solution will look like this:

  • I will switch back to the preprocessor-hook (instead of file-selection). This should fix the autostart feature, if someone call upload + print api (like slicers) it is processed in a sync way.
  • To support UFP my preprocessor will check for this format and if uploaded, the plugin will delegate to the UFP-plugin for extraction (some kind of preprocessor-cascading)
  • But then I have the upload-problem with large files, so to fix that, I will try to optimise the layer-analyse (switching from regex to character-comparison (I looked to Octolapse for inspiration)) or switching to external parsing or...maybe the easiest solution: find the right flask-webserver config for increase the timeout..any ideas?
  • For the watched/upload folder issue, I will simply show a button after selecting the file if no layer information could be found. So the user could manually start the layer processing by pressing the button.

@OutsourcedGuru: As you already mentioned I am looking for a userfriendly "out-of-the-box solution", so not all users are willing or knowing how to "attach" additional scripts to there print-workflow....I will try to make a performance test regarding: line by line processing with python vs. delegating to external processing via script-call...and maybe I will go for a "go-programm" or maybe I ask @FormerLurker about his C++ parser mentioned here Idea: CoolRunning-Plugin :wink:

C Ya,
Olli

You might take a look at GcodeEdit to see how easy it is to write a Go lang program. The syntax is a lot like C# but there are times when it feels like Python coding.

In theory it would probably be easiest to compile it for Raspbian, Windows, Linux, macOS and then apply the correct one during installation. You'd have to talk to foosel about the approach though. If it were me, I'd have the user self-pull their program from my repository in the init part of the Settings screen. Here's one that I did like that. The user either wants to do that or not. In your case, the user either wants to pull a compiled program to their Pi or not.

If/when I do write assembly languages programs for the Pi I intend to do it that way. I'm not going to try to educate end-users how to compile/link one.

Have you seen this? You'd think that one of these hooks would be useful.

I'm already using those hooks to process the ufp for extraction if I remember correctly.

The new parser i built can be found in the devel branch of octolapse if you want to give it a try. It is quite fast, and already can calculate layer changes, works in vase mode, can be called via python (both as a preprocessor and line by line in the gcode stream), supports progress callbacks, and could be used to output a new gcode file. However, there are a few printer settings you will need to supply that octoprint does not supply (but not too many).

However, i dont think you need to modify the gcode. Octolapse generates a snapshot plan, which is essentially a list of file line numbers and printer positions where snapshots will occur. For your plugin, you could output a list of layer change line numbers to a metadata file, and watch for them in the queuing phase, then update your state accordingly. Octolapse actually shows layer progress in V4 as well, since it knows not only the line number of the layer change, but also how many gcodes have been read, so there are examples of how it could be done in your use case.

It would require some rework, since it is designed for octolapse, but I have been wanting to package it up. I think it might be useful as an addin to octoprint so that other plugins can query from it, but that is just a hope. I would bet that it is much faster and has better position tracking than octoprint's current method, but i could be wrong.

Let me know if you want to take a look. I could create a new repo with just the c++.

1 Like

That's a good point @FormerLurker. @OllisGit, why couldn't you use that gcode hook while the print is happening and add the M117 line to the output of the callback?

http://docs.octoprint.org/en/master/plugins/hooks.html#octoprint-comm-protocol-gcode-phase

@jneilliii that would be great, but as I mentioned in my initial post:

The replacement is necessary, because during the print-job only g-code commands were send to plugin-hooks and not the comments.

So, in the queuing-phase (also in all other phases as well) there is no layer-comment from the slicer.
Of course I could ask @foosel, "please add a Hook that provides all lines and not just the G-Code lines", I'm not sure why I haven't done that in the past...hmmm, maybe a feature request for octoprint or better a pull-request :wink:

It could be an option to parse the layer informations out of the g-code instead using slicer-comments. Other plugin like OctoLapse or LayerDisplay were doing it.
But I thought it is not "precise enough". You need to detect z movement with extraction (related to printed object and not just nozzle-cleaning or other stuff) and not only simply z-movement. So, from my point of view this is the job for the slicer.

@FormerLurker

  • modifying is not the problem (from performance point of view). Identifying is the issue, so it doesn't matter if I add a M117 LAYERINDICATOR or if I create a separate metafile with such information. It takes the same amount of time.
  • I will take a look how you implement the layer-detection in V4. Maybe I can find some "inspirations" for my plugin...I already found some improvements in your older version for my layer-parser:
    now I can parse the layer-comments 40-50% faster (tested with a 50MB sliced Cura G-Codefile). Improvement will be in my next release.
  • Wow...heavy shit...I looked into your C++ source code....holy moly...how long did it take you to do that ?

Don't forget purge-stack activities at the back of the print bed.

So, in the queuing-phase (also in all other phases as well) there is no layer-comment from the slicer. Of course I could ask @foosel, " please add a Hook that provides all lines and not just the G-Code lines ", I'm not sure why I haven't done that in the past...hmmm, maybe a feature request for octoprint or better a pull-request :wink:

I've already asked for this, and there are apparently some complications. It would be EXTREMELY useful, however. Maybe the complications can be eliminated somehow.

Wow...heavy shit...I looked into your C++ source code....holy moly...how long did it take you to do that ?

Answer: A long ass time. Most of the complications were figuring out how to communicate to and from Python. That stuff is crazy. I've written other extensions like this in other languages, and Python is by far the most difficult I've worked with. In fact, there still might be a memory leak in the portion of the code that sends the snapshot plan information back to python. It is very difficult to tell, actually, though I plan to get to the bottom of that soon. The straight c++ was mostly simple (I just translated my python code, more or less, it somewhat, and added what was needed to pre-calculate where snapshots would be taken), except for the fact that c++ versions were very difficult to keep straight. As a general rule, any features in c++ v11+ needed to be removed, else many compilers would fail. However, it is SO SO much faster than python. Also, it is a bit messy because I couldn't really plan how it worked up front since I was learning to interact with python via trial by fire. I have been refactoring things, and will eventually fix it up so that it doesn't smell so much or make one's eyes bleed (which it does, and so does Octolapse).

I don't mean to hijack the thread, but I figure I'll explain how the processor can be used to help you determine if it's worth investing the time, or if a simpler method would be better. I can edit this comment and remove this wall of text if it distracts from your purpose, or I can move it into another thread in the development category. Let me know if I should do this!

Calling the extension from python is easy (using the gcode_processor.py module, currently tailored for Octolapse).

Initialization:

GcodeProcessor.initialize_position_processor(cpp_position_args)

The args are a bit complicated, but not everything is needed in most cases (i.e. just layer change detection):

{
            "location_detection_commands": self.get_location_detection_command_list(),
            "xyz_axis_default_mode": self.xyz_axes_default_mode,
            "e_axis_default_mode": self.e_axis_default_mode,
            "units_default": self.units_default,
            "autodetect_position": self.auto_detect_position,
            "slicer_settings": {
                 "extruders": [
                      "z_lift_height": 0,
                      "retraction_length": 0,
                 ] 
            },
            "zero_based_extruder": self.zero_based_extruder,
            "priming_height": self.priming_height,
            "minimum_layer_height": self.minimum_layer_height,
            "num_extruders": num_extruders,
            "shared_extruder": self.shared_extruder,
            "default_extruder_index": default_extruder - 1,  # The default extruder is 1 based!
            "extruder_offsets": extruder_offsets,
            "home_position": {
                "home_x": None if self.home_x is None else float(self.home_x),
                "home_y": None if self.home_y is None else float(self.home_y),
                "home_z": None if self.home_z is None else float(self.home_z),
            }

You could set all axis to absolute by default, leave the home positions as None (they will autodetect after homing). A minimum layer height of 0.05MM seems to work in most cases for vase mode if you don't know the layer height. The slicer settings per extruder can fudged if you don't care to track z_lift or retraction, and you can pretty safely just configure a single extruder and it will still work. Priming height is required to detect purges, but in 99% of the cases a setting of 0.5MM will do the trick. This will work for purging on the bed, or in mid-air for pretty much all of the gcode I've encountered.

Updating the current position with gcode is super easy:

GcodeProcessor.update(gcode, self.current_pos)

It spits a dict of the current printer and extruder state into the second parameter (see gcode_processor.py for the Pos and Extruder classes, which explain what is returned).

You can also undo commands (a fixed number, currently up to 10), in case they aren't sent to the printer:

GcodeProcessor.undo()

I plan to add an undo that takes gcode, which would reverse a gcode update, giving unlimited undo, but that will take some work.

You can fetch the current and previous position and state:

self.current_pos = GcodeProcessor.get_current_position()
self.previous_pos = GcodeProcessor.get_previous_position()

You can even manually override the current location, which is useful in some situations:

GcodeProcessor.update_position(self.current_pos, x, y, z, e, f)  # all in absolute I believe.

Also, there is a comment processor in the c++ code for extracting comment based information per slicer (it auto-detects the slicer type), but right now it only tracks things like feature type (infill, perimeters, etc), but not layer changes (it does that via a different method). It could be easily modified to do that too, however.

The main problem with it right now is that it needs to be compiled. See setup.py for info there. I would rather it be installed via pip, but that will take some work, and would require some refactoring to separate out the Octolapse stuff (snapshot plan generation) from the rest (parsing and position/state processing).

I hope that helps, and even if it does not I look forward to seeing what you come up with!