Th3dstudio.com report of problem with Octoprint

I got this email today: https://mailchi.mp/8a6c1327da03/whats-new-at-th3d-firmware-updates-discounts?e=553166e0f2

Essentially Tim found an issue with Octoprint and issues where the opening the UI causes some printer issues. The above email goes in to a fair amount of detail and shows some photo's. Just wanted to make sure you were aware.

Regards,
Tony

You might tell Tim to stop making 8-bit printer controller boards. Or, he could learn how to adjust Marlin's receive buffer. (Either way ought to work.)

EZboard isn't an 8 bit board. He's not blaming the issue on development but rather on Pi. I just wanted to make Gina aware of it in case there IS something she can do. Tim has a tendency to rub people the wrong way, but he really does mean well.

1 Like

image

I'm hoping he is wrong here.

Lets not take everything as a personal attack please.

1 Like

We're not using 8 bit boards. This was seen when the UI was being loaded on a 32 bit. Adjusting the BUFSIZE and MAX_CMD_SIZE to a higher value "resolves" the issue.

Here is the problem. The whole reason to change to larger buffer sizes is because the Pi stops sending serial data to the printer. The printer board regardless of 8 bit or 32 bit has no control over the data being sent to it. Adjusting the buffer and cmd sizes just gives the machine more commands in the buffer so that when Octoprint stops sending data because the UI is being loaded up it can continue to print without stopping. Once Octoprint loads back up and the UI stops interrupting the serial data flow the print resumes. These blobs on the prints are a result of the printer pausing because the command buffers are empty. Adjusting the firmware buffers still doesn't change the fact that Octoprint is stalling the serial data when the UI is being loaded.

Maybe it is Octoprint or maybe its just how the Pi CPU management is. Either way its a real issue and upping the buffers on the printer still doesn't change the fact that at some point the serial data transmission from the Pi to the printer board is stalled or "lags" when the Web UI is loaded.

I have no Octoprint setups on anything less than the 3B boards. Most are running 3B+ and more recent additions are running 4B 2GB boards. I did not notice this issue because we start a print and then close the browser. It's only when the UI is loaded again from the computer or whatever is hitting the UI that the issue pops up.

So please do tell me how the printer board regardless of CPU, firmware, or brand has anything to do with Octoprint halting the transmission of serial data to the board. Increasing the buffers is to hide the fact to the user that the data stream is being interrupted for a short period of time. I'll gladly admit it if I end up being wrong but all the data and testing points to the issue of the serial data being stalled/"lagging" when the UI is loaded.

1 Like

Someone else also brought up this link as a possible cause of poor multithreaded handling: https://wiki.python.org/moin/GlobalInterpreterLock

Which in my limited experience with the low level of the Pi and how its all running with Python seems to be something worth looking into optimizing. Maybe have the serial data process on it's own core since the recommended boards are all quad core boards. I am not an expert with Python but I do know that the serial data transmission is stalled/interrupted when it shouldn't be from loading the UI.

2 Likes

First of all, welcome to the forums, Tim. It's nice to have a discussion about the topic. Hopefully we collectively can speak about blobbing-as-an-issue and anything else you might want to discuss.

  • Sorry if I thought you were selling 8-bit boards; having recently reviewed your product page I saw all-but-one product which didn't mention the bit-ness of the processors. I've found that this is usually the thing that is omitted when it's in fact 8-bit. You did though mention 8-bit in the first paragraph of your blog post so I must have been confused there.
  • Indeed, adjusting the BUFSIZE—as I've personally suggested—is the #1 fix and seems to be the underlying cause (IMHO) for the blobbing in a correctly-hardware'd rig. I link again the exact spot in your own repository where it's set to 4. As I communicate with end-users, I try to list this in order, regardless of what they may feel about anything. Unfortunately, most of them are very daunted about the task of editing/reflashing their Marlin. And since they're afraid of making this sort of adjustment, they simply... don't. I guess what I'm trying to communicate is that if you adjust the receive buffer so that these users get the correct settings out-of-the-box then they shouldn't have to go through that pain.

I believe I understand what you're saying in the "Here is the problem" paragraph. Paraphrasing it back to you I believe what you're saying is: "OctoPrint's processing of a new browser connection, as combined with the GIL means that OctoPrint stops sending data to the firmware, causing the blobs." ~ paraphrased-Tim

Well, I do agree that it's a real issue. In my experience a wide variety of situations cause this and it goes way beyond just the "start a new browser" setup. It goes beyond OctoPrint and is being discussed on nearly every place where 3D printer hobbyists congregate, for what it's worth.

My tendency is to continue to talk but I should stop at this point and give others a chance to weigh in.

I'm happy to flash my version of Marlin... to what value should I change BUFSIZE...

I'm afraid I'm still running an 8-bit board :frowning: ... but haven't noticed any issues relating to blobs when I restart Octoprint browser (Ender 5, Marlin 2.0.3, RPi3B)...

But maybe I should leave it where it is if I don't have any problems :slight_smile:
A

We've been using the LPC1769 which is a 32bit 120mhz chip. Luckily we have a lot of memory and RAM on there to allow for much larger buffers which is why I am increasing the size.

What I am trying to say is simple. -> If the stream of serial data was never interrupted from (in the time I was testing) loading the UI then the buffer could remain small.

The Marlin firmware defaults to 4 on BUFSIZE because many smaller CPU boards (aka 8 bit or even some low end 32 bit) can't handle a larger size. I'm simply looking into workarounds to make sure the boards have enough data in the event of the serial data stream being interrupted/paused/lagged/etc so the print quality isn't affected.

When testing the firmware I never saw this issue because I start a print and close the browser. We have over 50 machines online each with their own Pi board and the process is the same. Start print, close browser, and wait for print to finish. We're never opening the UI over and over again like some people do. That is why I never noticed the issue on our in house machines.

FWIW printing over USB from a computer even with a BUFSIZE of 4 never shows an issue on a decent PC because it has enough horsepower to multitask properly. If somehow one core (on multi CPU pi boards) could be dedicated to the serial data transmission portion and never touched by other processes that may be something worth doing. I would take a stab at it but the last time I mess with Python was over 10 years ago when I was working on CounterStrike Source server mods. :stuck_out_tongue:

Just curious if you guys did any testing leaving the browser open? I typically leave the page loaded because I'm monitoring the print. I understand this could still be an issue when another browser connects up, ie from mobile phone remotely, etc., but the more information we have the better. For example it only happens when the first connection is established, but additional connections don't have the issue.

Also, curious if you guys are running any plugins on those machines or it a barebone install?

Honestly, what is the use of OctoPrint then? I can exactly do that with a SD-Card in the printer too.
I always have the browser open. And never have issues.

Haven't tested browser left open. We do use a local installation of Overseer to manage the farm but that is making API calls to the Pi setups. I'll add it to my testing to see if leaving it open will affect anything.

When you have 50+ machines between multiple buildings being able to send gcode over the network is convenient.

1 Like

Ok, I can understand that.
But with such a lot of remote printers, how can you control everything is running smoothly with the Web GUI off?

apparently

1 Like

If I were you and my print jobs didn't result in momentary hesitations (especially along the outside edges of your parts) then I probably would leave well-enough alone. You may find in the future that some part with fussy little curves causes problems and you could deal with it then. If it were me, I'd probably double that 4 to an 8 or even more, though.

That's certainly a thought and your point about the GIL is also a good point. If the streaming of gcode is happening in the main thread of Python and some GIL treachery is occurring then pushing that to another thread would help things one could guess.

And yet, people with Repetier also indicate issues with blobbing.

"My works-for-me solution was to change the movement buffer to 64. I'm using Repetier and anet A8, so plenty of free ram because of the microcontroller used: ATmega1284P.
The default value is 16, which it's small for very tiny segments (circles), 32 should work too If not enough ram available."

It feels like we're actually discussing Marlin-with-streaming and Marlin-from-the-SD-card. Any streamed software pushing to Marlin is apt to see this, I'd wager.

"The ASCII buffer for serial input. Individual command line length is set by MAX_CMD_SIZE, and should be long enough to hold a complete G-code line. Set the number of lines with BUFSIZE."
~ Marlin firmware docs

So we're talking about number of lines in that buffer, not 4K or anything like this. Honestly, four frickin' lines is crazy considering the veritable 115200 fire hose that is serial communications these days.

It feels to me like the Marlin crew set defaults that are good for the lowest-common-denominator instead of setting them for the least pain out-of-the-box. Being on the support end of this, I don't love that stance.

Here's my guidance on blobbing and Marlin tweaks.

You might want to take a look at this technique.