Hi,
I'm working on making a binary encoded g-code format that should reduce file sizes significantly for SD card uploads, but it requires a custom serial format that's not covered in standard Marlin. I've already implemented the protocol on the firmware side, as well as python utility for encoding the format, but for it to be useful I need to write a plugin for octoprint to run the encode script, as well as method for grabbing the octoprint serial stream and sending binary data out. As far as I can tell most plugins can enqueue standard g-code which obviously won't be handled correctly with raw binary data.
It seems the strategy used for the firmware upload plugins is to connect to a serial port directly through a third party utility, but I think this is kind of a bad experience for uploading a file since it requires triggering a disconnect/reconnect and synchronizing printer state externally.
The behavior I'd like to implement is something like this:
-user clicks a new binary upload button and selects a g-code file
-plugin calls an encoding script that makes a new .binGcode file
-plugin takes exclusive use of the open serial port, disabling timers for background communication like temperature queries and keep-alives
-plugin streams the binary encoded file through a custom protocol
-plugin releases serial port and returns things to normal operation
is this kind of behavior feasible with the current plugin system? It's not immediately clear how plugins trigger actions. Some behavior seems to be triggered by javascript and some are handled by callbacks run by the main octoprint instance.
This is what's not possible with the current plugin system. You could have the instance disconnect, reconnect from your plugin, do your upload, then disconnect and let OctoPrint reconnect again, but you can't take over exclusive control over the serial port and basically shut out OctoPrint from a plugin without having to basically reimplement the whole communication layer.
I was afraid of that. What's the best method for triggering disconnect /reconnect? Can i just access the octoprint instance directly or does there need to be some argument passed into the plugin / callback action from the plugin. What i don't want to do is kill the serial connection without octoprint knowing and risk it trying to auto reconnect or throw up a bunch of errors.
What is the interest in supporting a more streamable protocol? Marlin is really inefficient in terms of latency / throughput because you have to pause after every command to send "ok" with every line. You could potentially reduce stuttering on low-power hardware if the protocol were changed.
Personally, I think I would avoid a computer which is low end instead, to be honest.
For what you're doing—and not spending too much time thinking about this—it strikes me that a man-in-the-middle computer could receive the entire GCODE file from the OctoPrint installation, encode it and then stream it to the printer's controller board in the way you've described. To OctoPrint, this might seem like an SD card upload event to this in-between device.
Yeah, I don't know how adding another communication link in between the printer and Octoprint emulating the original high-latency interface will somehow improve communication latency. Part of the issue is also on the firmware side, you have to allocate an entire line worth of memory because you don't know how long the next command will be, limiting the amount of buffering you can do. It's largely a Marlin issue, but it can't be solved at a protocol level without having support from the streamer. Uploading a complete file to the printer SD card is the lowest hanging fruit, you don't need any of the printer monitoring, and all integrity checks can be saved until the end with a CRC.
The fundamental issues I am trying to solve are:
-It should not take 5+ minutes to upload a 2MB file to the SD card on a 32-bit processor
-It should not require a >1Ghz processor to stream g-code live without stuttering.
I believe both of these issues can be solved at a protocol level, but it needs to be implemented on both sides.
It doesn't technically need to be another IoT device. Imagine another process that runs on the Raspberry Pi, pretends to be something that OctoPrint is familiar with (SD card on the controller board), accepts an SD card upload from OctoPrint and then streams it to the controller board.
I realize this is how Klipper does it, but it seems more like a workaround than an actual solution. It would require also forwarding all normal commands in an appropriate manner, which is not a trivial undertaking. Seeing as there's little interest, I'll probably just implement the binary SD upload as a Cura/Pronterface plugin for now.