Communication timeouts causing print freezes

Get Help
1

What is the problem?

Constant communication timeouts resulting in pauses mid-print and marred works.

What did you already try to solve it?

Updated printer stock firmware to marlin ADVi3++, upgraded from a pi zero to a 3B plus, running a 2.5A power supply for the pi, replaced cables (included shielded), run a powered hub, tweaked timeouts up to 30 seconds, dropped print speed to 45ms in GCODE, disabled GCODE visualiser. Tested USB port via PC connection, USB port on printer tested and operates without fault.

Additional information about your setup

Octoprint 1.3.10
Octopi Version 0.15.1, running on Raspberry Pi 3 Model B Plus Rev 1.3 (No webcam)
Printer: Wanhao Duplicator I3+
Firmware: Marlin ADVi3++ 1.1.8

The errors seem to be triggering around the temperature checks, but I've pushed the timing of those right out the wazoo with no discernable change to behaviour.

Serial log (partial):



2019-01-28 05:48:06,357 - Send: N453 G1 F900 X145.906 Y154.108 E55.62963*1
2019-01-28 05:48:06,386 - Recv: ok
2019-01-28 05:48:06,391 - Send: N454 G0 F4800 X145.906 Y153.543*77
2019-01-28 05:48:06,912 - Recv: ok
2019-01-28 05:48:06,915 - Send: N455 G1 F900 X144.940 Y152.577 E55.65235*1
2019-01-28 05:48:06,937 - Recv: ok
2019-01-28 05:48:06,942 - Send: N456 G0 F4800 X145.506 Y152.577*69
2019-01-28 05:48:07,401 - Recv:  T:195.09 /195.00 B:49.67 /50.00 @:52 B@:L���ok
2019-01-28 05:48:08,975 - Recv:  T:194.86 /195.00 B:49.70 /50.00 @:56 B@:127
2019-01-28 05:48:10,972 - Recv:  T:194.52 /195.00 B:50.03 /50.00 @:64 B@:127
2019-01-28 05:48:10,977 - Communication timeout while printing, trying to trigger response from printer. Configure long running commands or increase communication timeout if that happens regularly on specific commands or long moves.


2019-01-28 05:51:16,798 - Send: N1196 G1 F3000 X101.861 Y155.525 E98.63776*6
2019-01-28 05:51:16,966 - Recv: ok
2019-01-28 05:51:16,971 - Send: N1197 G0 F4800 X101.862 Y149.945*116
2019-01-28 05:51:18,775 - Recv: o-� T:195.37 /195.00 B:49.95 /50.00 @:41 B@:0
2019-01-28 05:51:20,774 - Recv:  T:195.37 /195.00 B:49.92 /50.00 @:41 B@:127
2019-01-28 05:51:20,779 - Communication timeout while printing, trying to trigger response from printer. Configure long running commands or increase communication timeout if that happens regularly on specific commands or long moves.


2019-01-28 05:54:54,290 - Send: N2044 G0 F4800 X93.343 Y152.740*75
2019-01-28 05:54:54,550 - Recv:  T:195.26 /195.00 (�r��z��r���34 B@:0
2019-01-28 05:54:54,558 - Recv: ok
2019-01-28 05:54:54,562 - Send: N2045 G1 F3000 X98.193 Y157.590 E160.85232*8
2019-01-28 05:54:54,574 - Recv: ok
2019-01-28 05:54:54,578 - Send: N2046 G0 F4800 X99.795 Y157.591*71
2019-01-28 05:54:54,885 - Recv: ok
2019-01-28 05:54:54,890 - Send: N2047 G1 F3000 X101.861 Y155.525 E160.90090*61
2019-01-28 05:54:55,181 - Recv: ok
2019-01-28 05:54:55,186 - Send: N2048 G0 F4800 X101.862 Y149.945*116
2019-01-28 05:54:56,547 - Recv: o-� T:195.37 /195.00 B:50.31 /50.00 @:34 B@:0
2019-01-28 05:54:58,545 - Recv:  T:195.48 /195.00 B:50.33 /50.00 @:33 B@:0
2019-01-28 05:54:58,554 - Communication timeout while printing, trying to trigger response from printer. Configure long running commands or increase communication timeout if that happens regularly on specific commands or long moves.


2019-01-28 05:55:39,051 - Send: N2218 G0 F4800 X141.544 Y161.097*114
2019-01-28 05:55:39,069 - Recv: ok
2019-01-28 05:55:39,073 - Send: N2219 G1 F3000 X145.905 Y156.735 E173.50922*57
2019-01-28 05:55:39,381 - Recv: ok
2019-01-28 05:55:39,385 - Send: N2220 G0 F4800 X145.906 Y154.391*116
2019-01-28 05:55:39,692 - Recv: ok
2019-01-28 05:55:39,697 - Send: N2221 G1 F3000 X144.092 Y152.577 E173.55188*57
2019-01-28 05:55:40,046 - Recv: ok
2019-01-28 05:55:40,050 - Send: N2222 G0 F4800 X144.407 Y152.577*115
2019-01-28 05:55:40,315 - Recv: ok
2019-01-28 05:55:40,319 - Send: N2223 G1 F3000 X137.386 Y159.598 E173.71700*51
2019-01-28 05:55:40,348 - Recv: ok
2019-01-28 05:55:40,352 - Send: N2224 G0 F4800 X137.386 Y153.941*119
2019-01-28 05:55:40,991 - Recv:  T:194.89 /195.00 B:49.82 /50.00 @:50 B@���Rok
2019-01-28 05:55:42,499 - Recv:  T:195.00 /195.00 B:49.95 /50.00 @:43 B@:127
2019-01-28 05:55:44,497 - Recv:  T:194.55 /195.00 B:50.18 /50.00 @:60 B@:0
2019-01-28 05:55:44,500 - Communication timeout while printing, trying to trigger response from printer. Configure long running commands or increase communication timeout if that happens regularly on specific commands or long moves.
2

Hello @Taleya!

I've noticed these faulty characters in your log. Scrambled data on the the receive path also occur on the the send path. Maybe the USB cable is faulty. try another one as described here: https://discourse.octoprint.org/t/octoprint-randomly-loses-connection-to-the-printer-with-a-serialexception/228

3

Hi Ewald,

I've tried three cables, including ones that were used to firmware flash the machine. Shielded and unshielded. It's worth mentioning the wanhao i have is rebadged as a "cocoon create" (but runs a wanhao board, I've seen it :slight_smile: ) so I was wondering if the issue with the monoprices was still occuring?

4

Is your Pi reporting that it's throttled?

5 
pi@octopi:~/.octoprint/logs $ vcgencmd get_throttled
throttled=0x0
6

Tried yet another cable, a bit better, but still having the same issues:

Send: N12558 G1 X96.549 Y95.291 E273.97350102
Recv: ok
Send: N12559 G1 X96.345 Y95.879 E273.98386101
Recv: ok
Send: N12560 G1 X96.238 Y96.498 E273.99430111
Recv: ok
Send: N12561 G1 X96.224 Y96.865 E274.00041104
Recv: ok
Send: N12562 G1 X96.280 Y97.477 E274.01063106
Recv: ok
Send: N12563 G1 X96.538 Y98.461 E274.02755102
Recv: ok
Send: N12564 G1 X96.558 Y98.651 E274.03072101
Communication timeout while printing, trying to trigger response from printer. Configure long running commands or increase communication timeout if that happens regularly on specific commands or long moves.
Send: N12565 M10534
Recv: ok T:194.80 /195.00 B:49.87 /50.00 @:64 B@:0
Recv: T:194.89 /195.00 B:49.75 /50.00 @:57 B@:127
Communication timeout while printing, trying to trigger response from printer. Configure long running commands or increase communication timeout if that happens regularly on specific commands or long moves.
Send: N12566 M10533
Recv: ok T:194.94 /195.00 B:49.84 /50.00 @:53 B@:127
Send: N12567 G1 X96.559 Y98.710 E274.0317197

I do seriously suspect this is temperature based tbh. It seems to get worse on successive / longer prints where the unit starts seriously heating up.

7

Do we know if the controller board (ATmega1284P) is an 8-bit board?

8

ok new problems evidencing with the new cable, it's actively disconnecting all over the place, and also punching up disconnect errors. Today it actually killed a print with this:

2019-01-30 05:58:43,633 - octoprint.util.comm - ERROR - Something crashed inside the serial connection loop, please report this in OctoPrint's bug tracker:
Traceback (most recent call last):
File "/home/pi/oprint/local/lib/python2.7/site-packages/octoprint/util/comm.py", line 1903, in _monitor
total = int(match.group("total"))
ValueError: invalid literal for int() with base 10: ''
2019-01-30 05:58:43,647 - octoprint.util.comm - INFO - Changing monitoring state from "Printing from SD" to "Offline (Error: See octoprint.log for details)"
2019-01-30 05:58:43,666 - octoprint.plugin - ERROR - Error while calling plugin tracking
Traceback (most recent call last):
File "/home/pi/oprint/local/lib/python2.7/site-packages/octoprint/plugin/init.py", line 230, in call_plugin
result = getattr(plugin, method)(*args, **kwargs)
File "/home/pi/oprint/lib/python2.7/site-packages/octoprint/plugins/tracking/init.py", line 119, in on_event
self._track_printjob_event(event, payload)
File "/home/pi/oprint/lib/python2.7/site-packages/octoprint/plugins/tracking/init.py", line 237, in _track_printjob_event
sha.update(self._settings.get([b"unique_id"]))
TypeError: update() argument 1 must be string or buffer, not None

9

I've tried dropping the speed to 45mm/s in case of this, no change. Dropped it down to 35 even, still had the same issue.

10

Question: Does Octoprint log (or can you enable logging) transfer to SD? I'm setting up a 9 hour print job to feed from the pi to the SD to see how it runs overnight. It will at least help me tighten the diagnostic focus.

11

You can enable logging of everything that goes to or comes from the printer by enabling the serial.log.

12

Hiya

I do have serial log enabled (this is where all the other logging details I've been posting have been originating from) however the serial.log doesn't appear to log SD card uploads. I've had a tail running on serial.log and there's been zero changes.

Octoprint.log shows the file transfer initiating, but nothing beyond that point.

2019-01-30 21:55:18,995 - octoprint.util.comm - INFO - Changing monitoring state from "Operational" to "Sending file to SD"
Waiting for data... (interrupt to abort)

13

ok, I ended up dropping the USB speed via dwc_otg.speed=1 and a reboot, and this seems to have stabilised. Unfortunately we've also had a huge break in our current heatwave, so I can't entirely rule out a temperature issue, but the prints are good across all cables, so the cables can definitely be ruled out. The glitch characters have also ceased. I'm running a bit of a gamut of print jobs to see how we go.

14

I really doubt if the Duplicator's port is USB 1.1 (only). Seriously, you need a short, good-quality, internally-metallic-shielded USB cable without any additional adapters inline to this.

15

that's the one I started with (I actually have a bunch of them, benefits of being a tech). It was still producing errors.

16

And does this cable either have internal metallic shielding or one or two ferrite cores?

1 Like
17

I'm rather getting the impression from your tone that no matter what response I give, you will find fault with it. Regardless, it's an interior braided metal shield. I do have ferrite couplers, but let's be honest the chances of this printer creating enough EM to much up any half decent USB cable are minimal at best.

Doing further testing this is pretty much nailed to be heat triggered. Going from my last post, around the fifth print is where the communication errors started creeping in, as the ambient temp around and inside the printer rose. Trying to determine if it's on the PI or Wanhao side, the possibilities appear to be thermistor arcing up and causing corrupt responses, or potential heat expansion on a slightly dodgy solder job.

18

Apologies, Taleya. I contracted for Robo 3D, a manufacturer of at least a million printers which went out to customers. They routinely shocked me with the lack of common sense and experience they possessed. The manager in charge of software didn't know anything about software. And this is one of the few manufacturers here in the states; you'd think that they knew better. The entire experience left me shaking my head at the state-of-the-art for $700 printers.

Now back up and realize that here I routinely provide free support for people who've purchased a $100-$200 cheap printer which comes equipped with every shortcut which can be made to bring down the price to that level. Color me cynical regarding the hardware out there. And don't get me started on the efforts at bundling firmware.

  • I'm glad you upgraded the Zero to the 3B+ (given that it is the plus version, you might consider doing a sudo apt-get update && sudo apt-get -y upgrade to upgrade the Raspbian firmware)
  • The Wanhao Duplicator might have the fake FTDI chip in it (possibly a dmesg would report this)

Adafruit blog: Why Did My Wanhao i3 Duplicator Catch on Fire?
Thingiverse: Printer not connecting via USB (solved)
Wiki: FTDI controversy
Google groups: Octoprint + RPI2b + Wanhao i3 USB disconnects

This issue would not be between the pi and the computer from which the pi is controlled, thus it does not matter whether Ethernet or WiFi.

This issue is between the Pi and the printer, which is USB/Serial.

Baudrate should be 115200 - no other baud will work (this is what the firmware needs).
Preferably, avoid using a USB hub - keep the cable as short as possible, through as little devices (like hubs or even connectors) as possible - the Pi can usually be mounted very close to the printer.
Use a USB cable with ferite cores (the loops usually close to the ends of the cable) - this should reduce interference.

From that advice, maybe you need to remove the powered hub from the situation (if that's a USB hub). If you think about it, the hub can introduce latency and possibly introduce state-related problems.

Duckduckgo: https://duckduckgo.com/?q=wanhao+intermittent+disconnects&t=osx&ia=web

Amazon comment: with two stars out of five

The machine is OK. The Melzi board is a inferior product. The original came with the printer was defect and was refurbished board. After days of frustration. the replacement came. The USB worked for about a week and failed suddenly. The factory GM and USA tech claimed that USB should not be used and not compatible with computer. They stopped answering after I asked them why USB stopped working after a week if it was not compatible. The factory support is not considered as top notch. I give up on them. So buy with caution and prepare extra money for replacement..

Another two stars out of five:

SO ours lit on fire in the control module after right about a month of owning it. Waiting on Wanhao customer service to help us with that, but yeah ours is totally dead because again it caught fire. We have a 2.1 which isn't supposed to have this issue.

19

Thanks for that - know the feels. Sometimes you just want to jam a squirrel in vendors!

This model actually does have the plastic spacers to prevent the current issue, it was first thing I checked when they issued the alert! It's also an i3 duplicator plus, which is a slightly different model to most of your links (and looks like they took the i3 complaints to heart!)

I don't run it with the powered hub as a matter of course, that was used briefly to determine if there was a powering issue (long shot as I wasn't getting any power dips on the pi, but easy enough to try and rule out) so there's only a direct USBA-USBB shielded cable in place. The current quality of USB cables is shocking, I routinely trash the ones that come with devices and replace them with my own. Dmesg doesn't' have anything particularly interesting, apart from the following, but this was pre-update when the box was freshly built and being tweaked:

[ 3.953906] brcmfmac: brcmf_c_preinit_dcmds: Firmware version = wl0: Feb 27 2 018 03:15:32 version 7.45.154 (r684107 CY) FWID 01-4fbe0b04
[ 3.954486] brcmfmac: brcmf_c_preinit_dcmds: CLM version = API: 12.2 Data: 9. 10.105 Compiler: 1.29.4 ClmImport: 1.36.3 Creation: 2018-03-09 18:56:28

(which got me all excited as I thought I'd nailed it, but the update has eliminated the glitch, but the timeouts still persist)

The FTDI issue appears to be blocking communication entirely from what I've read - I've got communication and it seems to be good until the temp comes up, then I get the timeouts. I've been charting them further (thank you heat wave) and it does look like they're heat generated. I have the printer in a corner, which probably isn't helping come summer, so I've ordered some better fans to replace the stock interiors.

Thanks for the melzi link, I have been side-eyeing that as well as I've noticed the e-stepper gets quite hot. I checked the voltage, but that was good, so I'd attributed it to heatwave / heat creep and replaced the ceramic tape (was due it anyway) and bootied it up, which has helped. Probably worth nabbing a new one, I've been eyeing off Flexions and considering a whole end replacement anyway.

20

Maybe you have to disconnect/reconnect the USB for it to appear in dmesg output, methinks.

Well, if it does disconnect again, quickly look for throttling and throw it back to us if it's anything but zero; one of the reasons could be heat. My own raspi-temp is JavaScript-based but you could go to school on the shell'd operating commands which it runs behind-the-scenes. Presumably /sys/class/thermal/thermal_zone0/temp includes the current cpu temperature.

You might do something like: sudo cat /var/logs/syslog |grep -a "ftdi" to better get a feel of what's in there. Or lsusb. I'm power-reading threads like this, btw. (Don't run sudo rpi-update.)