Hi! I'm curious to find out whether hosts (and serial clients generally) really need senders (in this case, Marlin Firmware) to add a delay after sending strings. One of the Marlin devs insists that without delays Printrun will "blow up" but I have sent piles of characters to Printrun on my laptop without ever seeing a problem — not even lost characters.
As I understand it, when the serial output buffer on an Arduino gets full, the serial.print function will simply block until there's more room in the buffer. So I don't think delay is needed for the benefit of the sender. But if Marlin was to send a large block of output — let's say 2K — all at once, would that have any negative effect on Octoprint or other hosts, assuming they're handling serial in a standard manner?
Thanks for any guidance you can provide. We want to make sure the firmware is behaving optimally and with the best possible compatibility.
I'm definitely seeing problems which seem to be due to an overflowing host side rx buffer (which I can't control) if temperature auto reporting is used on multi extruder systems, see
A first attempt at increasing read speeds by replacing pyserial's read line with a custom implemented bulk read sadly hasn't worked around this. I plan to switch to a constant line push in the future, but I'm not sure that will solve this either.
I'm going to research and see if there's some clear limitation to the number of characters we can send before we have to pause, and try to come up with a well-defined minimum time period for that pause. I feel it would be a lot nicer on the firmware side if we had an intermediate method that counts up bytes as they're sent and automatically inserts the minimum needed delay. It's very haphazard as it is now, where we add delay(10) calls (even up to delay(75)) for every instance of serial printing. We ought to be doing longer delays for slower baud rates, and that isn't being accounted for either.
Maybe an interesting data point: as far as I can tell these overflows really only have started to show themselves with the introduction of temperature auto reporting.
During boot the firmware usually dumps quite a large blob of text on the host, also with some quite long lines, but I don't think I've ever seen anything that looked like an overflowing in those. Of course, it's hard to be sure about this.
Well, we did have an issue (now fixed) where those reports were getting into the middle of other lines. I'm not sure if that was prevalent in 1.1.8 or if it was an intermediate problem during current development. That bug would have been enough to confuse any host's parser and maybe even overflow the line buffer. But also, could too many bytes without a newline (or carriage return) cause problems in the host side rx buffer, or is it "line agnostic"?
I'm not sure what you mean with "line agnostic". Up until and including 1.3.6, OctoPrint used PySerial's built-in readline method which (in the version currently still used by OctoPrint) basically reads one byte after the other from serial until it sees a newline (well, or a timeout), then returns what it found. For 1.3.7 I've overwritten this method and made it so that OctoPrint will now always read everything that is currently available to read and only hand over the first line from that, keeping the rest in an internal (unlimited) buffer. However that doesn't appear to have solved the issue observed in the ticket I linked further up according to its reporter.
I've also patched the "missing ok" issue. That was due to a typo from a contributor that we missed, and was only in the bugfix-1.1.x branch. The re-send function now includes an "ok" as it should.