I'd hate to think it's my TPLink plugin, as I use it regularly and have not had SerialException errors at all. If it does turn out to be the culprit, I'd need logs and an issue opened on the plugin's github repo.
Edit: It could be that the plugin is doing what it's told to do and powering off your printer on a temperature runaway setting or something similar?
Gina,
It’s not, although is probably is. I did a bit of research on how the BCM2837 does USB. I couldn’t find the actual hardware manual (I put a request into Broadcom, but have not heard back), but I found the hardware peripheral info on the BCM2835 (used in the PI2), so it is probably similar to the one used in the PI3B. The operation of the serial ports (including the USB is similar to the 16550 UART without the DMA.
Anyway, without going into a lot of hardware talk, my suspicions seem to be correct, although I have no idea how Raspbian does it’s I/O. What happens is the serial data is assembled and written into a register. The processor has a limited amount of time to extract that data before the next byte is written into the register. There is a limited time after the serial interrupt when the data has to be retrieved from the UART before new data is written into the register. Timeouts and serial exception mean that when the interrupt occurs, data is not timely read from the register, and so the processor can’t get any the serial data. One can calculate this time, but obviously, Raspbian is not getting the data fast enough before the next byte of data comes in.
The problem mainly occurs when the serial handler is poorly written, and priority is not given to the serial interrupt. Remember, I have no idea how Raspbian does their serial I/O (USB), nor how it is programmed, so I can’t pinpoint where the problem lies. Unfortunately, Gina, I don’t think you can do anything about it, except find a real-time linux that will run on the PI.
What makes the Pi different? On big computers, USB is typically handled by DMA (direct memory access), where the data from the UART is written directly into a memory buffer, and therefore, timing is only critical to keep the buffers switched so the data keeps flowing. X86 architecture is like this, and the 16550 UART works like this. The little machines (like the PI or the Arduino) don’t have enough real estate on board to implement this, and so you have timing issues on high data throughput.
My suspicion on Raspbian is that they ported Debian without rewriting (or minimally rewriting) the I/O handling. These tiny chips require a total rewrite of the I/O handlers when used in a high-throughput or real time capacity. A real-time OS (or a real-time Linux) should have done this already.
When it comes to the snake (Python), how the implementation of the language interpreter and garbage collector interface with Linux may be part of the problem. If they are part of a Linux thread spawn (at the OS level), then all that software can affect the I/O timing because of the software load. You might not even know this from CPU utilization because CPU utilization might not even be calculating this. I typically go by CPU temperature to determine the load on the CPU, not the utilization number. On my PI 2B+, based on the temperature, the CPU utilization Is fairly high because I can see throttling taking place, and the 3D prints show it.
My only comment is to try to explain what is going on, not specifically how to fix it. The real fix is not to use Raspbian as a real-time OS, and to code certain portions of Octopi in C or C++, but I understand this probably isn’t practical.
Take a look at the source code of Marlin, and you will get a fair idea of writing embedded code. The PI is great for some things, but not great for others. Octoprint might have been better on an Arduino, not a PI. The Arduino is a simpler architecture and is easier to code without an OS.
The PI isn’t a big general purpose computer that does multiple things. It’s a tiny machine, that probably does one or two small things well. I had a similar argument in another part of this forum where I was asking questions about shutting off the main power supply of the printer, which also powers Octoprint. I got arguments as to why the PI should be left on all the time because it can do other things. I countered this argument, but I couldn’t convince some folks. Anyway, I figured out how to do what I wanted to do on my own, and now Octoprint nicely shuts down the printer power supply when it shuts down.
Steve
@Steverino as I said though, of this was a case of "the pi is at its limit with 1.5.x and python is just ill suited here" we'd see way more cases of this, across all possible printers. We don't. In fact we only see a handful. That does not indicate a general fundamental problem rooted in OS and/or architecture - that is identical to literally thousands of setups that print problem free. We are trying to identify a pattern here that differs from the setups known to work. Considering the available information, I find "Raspbian combined with Python" to be the cause here somewhat unlikely.
Depends on what is going on inside the processor.
Just giving my opinion.
Just an update for you, printing using 1.5.1 after a few days, I've had no serial connection issues.
I've removed the 'OctoPrint-TuyaSmartplug' and 'GcodeEditor' plugins and I've not had any issues.
Now i know versions of GCodeEditor prior to the latest are included in the plugin blacklist. So that may not be playing nice with OctoPrint.
Also, the TuyaSmartplug plugin was logging errors into the octoprint.log for a while also the install.
Unfortunatly I no longer have those logs.
Intermediary summary:
@Doug_s_Printing:
@Rancorbin:
@Spooky_Wookie:
@drfish:
@Red_Gorilla:
@apainter2:
@AevnsGrandpa:
@fificap:
Legend: exception problem reported and unsolved, exception problem reported and solved, false positive
OctoPrint 1.5.2
Ender 5 Pro
Debian Buster
Creality board 4.2.2
Creality firmware 1.3.1 w/BLTouch with Adapter
Python 3.7.3
Troubleshooting:
Disabled plugins
Changed USB cable power feed is taped off.
Applied OS updates
Changed USB port
Enabled logging
Current PLUGINS:
Bed Visualizer
Detailed Progress
Exclude Region
Navbar Temperature Plugin
PrintTimeGenius Plugin
Printer Dialogs
Printer Notifications
Sidebar Webcam
Simple Emergency Stop
Themeify
Thingiverse Plugin
Virtual Printer
Octolapse
I enabled logging and after the USB port swap and I haven't been able to duplicate since.
I had experienced the Serial Connection error three times deep into a two seperate model prints five to six hours into three 8 hour prints.
To the best of my knowledge my printer was connected and powered on with the upgrade of Octoprint. I haven't duplicated the condition as of yet since changing usb ports
After applying the updates in Debian and changing the usb port it seems to have possilbly resolved my issue. If I can duplicate the error I will publish the serial.log files.
octoprint (2).log (387.6 KB)
Just as a reference changing my usb port seems to have resolved my issues. I have not been able to duplicate the error..
After several days of printing the error has for some reason returned, and is more or less constant.
A long 26 hour print, failed 3 hours in. Nothing on my end has changed, I have attached the octoprint.log file for you. Time index of the failure is around 2020-12-24 18:48:57.
I'm at a loss, so much so I placed an order for a Prusa i3 MK3S+, trying not to simply scrap this printer, would like to get it fixed somehow.
octoprint (8).log (2.1 MB)
Details:
Plugins installed
Software versions
To me it looks like it's at this line...
2020-12-24 13:58:32,642 - octoprint.util.comm - ERROR - Unexpected error while reading from serial port
Traceback (most recent call last):
File "/home/pi/oprint/lib/python3.7/site-packages/octoprint/util/comm.py", line 3831, in _readline
ret = self._serial.readline()
File "/home/pi/oprint/lib/python3.7/site-packages/octoprint/util/comm.py", line 6455, in readline
c = self.read(1)
File "/home/pi/oprint/lib/python3.7/site-packages/serial/serialposix.py", line 596, in read
'device reports readiness to read but returned no data '
serial.serialutil.SerialException: device reports readiness to read but returned no data (device disconnected or multiple access on port?)
2020-12-24 13:58:32,692 - octoprint.util.comm - ERROR - Please see https://faq.octoprint.org/serialerror for possible reasons of this.
Your printer has stopped communicating, with a link to https://faq.octoprint.org/serialerror in the log. I notice you have 2 different creality temp fix plugins installed, might not need more than one? Have you tried minimizing the amount of enabled plugins to just those necessary for your printer to be able to connect. Since it looks like you have Smith 3D firmware I doubt you even need the creality fix plugins at all.
I used sudo apt full-upgrade to update everything on the pi and it fixed my issue.
TL;DR - I can trigger it reliably in my case just by issuing M303 to run PID autotuning. I think there's something fishy going on with the mainboard and firmware on my CR6-SE that's causing this in my case. Using 'minicom' to run the M303 command does complete, but throws its own warning that it lost the serial port connection 30 seconds into the M303 - the same point that Octoprint drops out (minicom just carries on, however - built for the good old days of unreliable serial comms I'm guessing it reconnects automatically).
octoiprint-log-and-terminal-capture.zip (8.0 KB)
I've just hit this. The printer, with Octoprint on the RPi, has been running fine for a good number of weeks since I rebuilt the Pi with the 0.18 Octoprint to try Python 3. It's a Creality CR6 SE, so I have the 5v line on the USB cable isolated.
A print job had finished fine and the printer had been idle for a while (just reporting temperature), and I went to run the PID autotuning from the Terminal. A short time into that, the SerialException was thrown.
In its current state (I haven't power cycled the printer or restarted the Pi or Octoprint yet), it's repeatable - I waited for the PID autotune seqence to finish (the printer carries on doing it after Octoprint disconnects), then reconnected from Octoprint and issued the M303 again. The SerialException report recurred after a similar time into the command.
I switched debug on in Octoprint (I think - added DEBUG for 'octoprint' in the bottom of the logging tab), and did the M303 thing again a couple of times. Octoprint log and terminal capture attached.
I've now restarted Octoprint (not the whole Pi) - same behaviour; I get the SerialException part way into M303.
Now restarted the Pi - same behaviour.
Switched printer off and on - same behaviour.
Installed minicom onto the pi, use it to connect to the printer to run M303 - that worked, however I notice minicom flashed up "Cannot open /dev/ttyACM0" occasionally. In fact the first time it flashed up, is about the same point that Octoprint disconnected (about 30s after the M303 command is issued). I'm guessing that minicom is seeing the same glitch on the serial port that Octoprint is seeing, but is treating it as a temporary issue and carrying on, whereas Octoprint makes it a hard fault. The "Cannot open" message seems to pop up every 30 seconds.
I ran an apt-get update/upgrade on the pi, but that didn't change anything.
I'm using a BTT mainboard (replacement for the Creality CR6 board) with the BTT firmware, which itself is a small delta to the Creality 1.0.3.6 firmware (which we believe branched sometime shortly after Marlin 2.0 - y'all know the drill here).
I'll look to try a more upstream Marlin version and report back here.
Update - I moved the Raspberry Pi to another printer, and lo and behold no problem there with M303. Moved it back and triggered it again with M303.
To state the obvious, this suggests strongly at the moment that my issue is not Octoprint.
The printer I moved it to is a much-modified Ender 3 with a BTT main board and a reasonably current Marlin (BTT do their firmware work on GitHub, and largely keep current with upstream Marlin - 2.0.5.3 I've modified it myself for various reasons and haven't updated for a while).
The CR6-SE is a bit unusual in that BTT have taken the Creality firmware (at least, the version they published the source code for) and tweaked that for their board, so unusually for BTT the firmware isn't current with upstream Marlin.
Difficult to tell from the history on this thread whether there's a common printer component to the reports, as not everyone has reported their precise printer configuration. So what I'm seeing may not be common - however I would say all the current Creality machines share a lot of firmware code, bugs and all, so if there's a firmware component to this issue it would likely affect a lot of current Creality machines.
So I have a couple of things to try. First is to put the Creality mainboard back to see if it's something with the BTT board. Then the firmware, get it closer to Marlin upstream - should be able to use upstream Marlin pretty much unmodified if I disconnect the touchscreen.
Replaced the Creality motherboard (with a firmware fix for the double-temperature-report mess), and no problem issuing M303 - no serial port drop-outs so far.
I had a similar issue here. I fixed the tornado warnings by deleting net gear software I didn't use on my computer but they have an alternate solution here. It didn't solve my issue but it got rid of the tornado warnings. There where a few firmware changes that did on my thread. I am on a skr mini e3 so it might work with your board.
Found what the difference was - with the BTT board, I had chosen to build using their STM32F103RC_bigtree_512K_USB PlatformIO configuration for Marlin which defines USE_USB_COMPOSITE. The Creality configuration didn't do that - and rebuilding with the BTT configuration STM32F103RC_bigtree_512K eliminated the problem.
I don't know what USE_USB_COMPOSITE macro does, but if I had to guess it's sharing the USB interface between serial and something else (debug probe perhaps?) and that sharing is causing glitches whereby it declares it has data available, when actually it doesn't.
So in summary; I had SerialException occur when USE_USB_COMPOSITE was defined for the Marlin build.
We have been following this thread hoping for a solution (sudo update didnt work). We are using a BTT SKR Pro 1.1 on Marlin 2.0. We can find no reference to the USE_USB_Composite macro so am not sure where to add that to try it(conf.h or adv.h ?).
The strange thing is ANY new ISO works fine (no serial exceptions) in our RP 1B+ v1.2, but throws serial exceptions in our new RP 4B.
The 1B+ is slow but it connects just fine.
Is that a clue to what may be happening?
USE_USB_COMPOSITE is set in platformio.ini, it's set for some of the btt environments (set in default_envs in the [platformio] section). Depending on which one you use you may want to try commenting it out.
Hello everyone,
Apologies if this should have been posted as a new topic. I think this appears to be related?
Thank you for this thread, it has been a huge help in my troubleshooting. I am new to 3D Printing and to RPi so am learning as I go.
I have been experiencing this issue and have tried all the suggested fixes. Unfortunately, none have worked for me so far.
My details are as follows:
Printer: Ender 3 V2 with 4.2.7 mainboard
Octoprint Version: Initially 1.5.2, updated to 1.5.3 during troubleshooting
Octopi Version: 0.17.0, updated to 0.18.0 during troubleshooting
System: Raspberry Pi 4 Model B (Pi NOIR Camera module V2 connected)
Browser: Google Chrome Version 88.0.4324.150 (Official Build) (64-bit)
Issue:
Experiencing the SerialError at random during prints longer than 60 minutes.
X,Y and Z stepper motors stop moving, but the hot end and bed stay at their assigned temperature. I have to reconnect to turn off the temps.
Firmware Tested:
Creality V1.0.1
Stable 2.0.7.2
Bugfix 2.0.x.x (15 Feb, 16 Feb and 17 Feb)
| Troubleshooting | Fix? | Actual result |
|---|---|---|
| Moved closer to GPO (removed extension cable) | No | Failed after 60 minutes |
| Swapped USB cable | No | Failed after 60 minutes |
| Printed directly from SD card | No | Caused Screen to lock up, so had to cycle power of the printer |
| Covered 5v pin of USB cable | No | Failed after 60 minutes |
| Ran in Safe mode | No | Failed after 60 minutes |
| Connected Printer to UPS | No | Failed after 40 minutes |
| Printed a shorter print (Less than 60 minutes) | Yes | Successful for 4 different models |
| Rolled back to Creality V1.0.1 | No | Failed after 20 minutes |
I've uploaded the Octoprint.log to my Google drive as it was too large to upload here.
I thank you for ready through this wall of text, I hope I have given enough information.
https://drive.google.com/drive/folders/1rOU31SbAQqe_yaumzlwxFa1xl01XaEc4?usp=sharing
serial (8).log (2.5 MB)
Did you have OctoPrint connected when you printed directly from SD card? If it also failed then, something is up with your hardware.