OctoPrint crashed with 50 layers left on a 3 day print. :(


#1

What is the problem?
I had 50 layers left of a 3 day print and Octo-Print crashed.
I verified this because I was still able to SSH in to Octo-Pi without any issue. I then restarted the octo-print services and was able to log back in.

What did you already try to solve it?
Not sure what to do. I love Octoprint but if I can't trust it, then I can't use it. Not bashing anyone as I know this is the nature of software, but I would sure like to know if there is anything that can be done so that when things like this happen there is a some way to recover from it. Perhaps saving state so that restarting Pi gives you an option to resume the job. Currently once you restart the service, all info is lost.

Additional information about your setup (OctoPrint version, OctoPi version, printer, firmware, octoprint.log, serial.log or output on terminal tab, ...)

Octo Pi : 0.15.0
Octo Print: 1.3.9
CR-10S
Wifi connection.


#2

when things like this happen there is a some way to recover from it.

You can edit out all the G-code at a Z-height below the failure point and
print again.
(Printer G-code is fairly easy this way, the Z coordinate increases
monotonically)

There may be more-automated ways, possibly plug-ins. But you can edit the
G-code without changing settings, re-homing or restarting anything.


#3

This is what I would have loved to do, but when you restart Octoprint it reboots the printer and thus looses all of the homing info, mesh bed level... And my printer homes to the center of the print bed because I have an EZABL. So I'm not sure how what you mentioned could be accomplished.

The only thing I can think of that might work is that Octoprint would need to writ it's location in real time to a file and then on restart assume that the print head is in the last recorded location and continue from there. If the steppers had encoders then this would all be much easier.

Please correct me if I'm wrong. I'm still rather new at this so I'm happy to learn of better ways to do things.

Tony


#4

Not to get too far off the original concern, I've routinely restarted prints. (My Robo 3D printer over its first year has had filament-delivery problems, an inoperative filament-out switch, etc.) So not due to OctoPrint at all, I sometimes get long prints that go bad near the top. Since some of these are expensive parts (carbon fiber), I'm incentivized to restart the job to continue it. Here's an example of a successful restart.

Before:

After:

I developed the Toggle Autolevel plugin to support this. When you're trying to send a second job to the printer—and assuming that you have an IR height sensor—then it's crucial that you disable that. You don't want your nozzle crashing into existing parts.

It really helps if your printer is reasonably level since you'll be operating without any of that mesh bed leveling assistance. On the plus side, PLA sticks really well to PLA. You'll need an accurate measure of the height and some sleuthing skills in the GCODE file.

GetToDahChoppa is what I use to chop up a GCODE file into multiple parts.

I'm pretty sure I used Cura before this to adjust the starting layer to print on a part. And note, you can position a part "off the deck" at the appropriate height.

You'll need to know your startup GCODE in OctoPrint since you don't want it trying to lay down that priming line either; that would be bad.


#5

I'm going to put a pin in your post until my skill level reaches the point where I can properly process what you are saying. My concern is that my printer no longer knew what its current Z height was. So telling it to go to 311 MM off the bed would mean nothing to it. Unless you are saying that there is a way to manually, through gcode, tell the printer what height the nozzle currently is at some how. "Hey printer, that place where your nozzle is right now, thats 311 MM from home."

Am I close to what you were doing? I have a lot to learn about Gcode.

Thanks
Tony


#6

Generally, absolute-mode positioning means that you start with the bed homed in terms of Z and then move the bed up to meet the hotend. G28 Z would tell your printer to do that. All G-related movements after that are done based upon that initial condition.


#7

If it happens again, do a "df -h" when you ssh into the octoprint pi. it is possible that some temporary files could have filled up the SD card, or if you are running the pi 3B+, they have a known issue of crashing in general unless you slow down the processor and memory speed in the /boot/config.txt file.


#8

I have a Pi 3 so this is good news I guess. :slight_smile: Just to clarify, the PI didn't crash, just the OctoPrint service, but you may be talking about general service crashes (I'm hoping you are) so I will go hunting the work around you mention down. Thank you.


#9

Apologies if I've asked this before but does your serial cable either have an internal metallic shielding or does it have ferrite core(s)? Either would be good and absence of both is bad.


#10

Yep, mine was about strictly the 3b+ and system crashes.

If it does it again, before rebooting or restarting octoprint, double check the disk usage with df -h just to verify it didn't fill the file system


#11

I think you meant USB not Serial. It's shielded.


#12

:laugh: And what does the S in USB stand for?

Yeah, do the df suggested earlier. Running out of room on your root drive would cause a crash.

Things we know:

  • Not a 3B+ (emphasis on the plus)
  • Shielded serial cable
  • The OctoPrint service wasn't running at the time of the problem

Things we don't know:

  • 5V 2.5A Raspberry power supply?
  • Firm micro USB connection to the Raspberry Pi for power? There are known problems with some combinations of micro USB plugs versus the connectors.
  • Available space on your root drive (not to be confused with your boot partition)
  • We don't have your octoprint.log from the failed session

#13

LOL... Wow your right... Been throwing the term USB around for so long that I've glazed over what it means. That's embarrassing given my line of work.

I like the way you think.

Add to known things the following:

  • 5V 2.5A provided Raspberry power supply is what is being used here.
  • USB feels appropriately firm in it's connection.

The rest will be provided should the issue arise again.

Thank you all for your time.
:slight_smile:


#14

That "tight connection" also goes for the serial cable to the printer as well as the micro USB for powering the Raspberry.

Try to look in your Settings -> Logging and try to find the octoprint.log version from the crash. Look at the timestamps if you're wondering.


#15

I often ask myself this same question. I still havent solved it, but I have successfully resumed a couple of prints in the past. Most recently something went wrong and I ended up resetting and reprinting.. mostly because grinding the printhead into the print is horrifying, so once it happens once i just surrender.
That said, you might consider figuring out if you have clearances to do each X, Y and Z home individually, and find places on the bed where it will work and not kill anything.
Afteer that it's a case of selectively editing the gcode... chopping out the usual homes, mesh leveling and everything after the heating commands, right up till the layer it choked on.
It's always ugly though. One day I'll be compelled to figure it out better. Even just modding firmware or a chunk of hardware to monitor Z value would be very useful. No doubt the existing marlin/etc power failure code would be well on the way to that kind of thing.


#16

I will say, though, is that everything we learn from these job-restart attempts is useful knowledge. That knowledge about G0/G1-related commands and absolute/relative modes and editing GCODE is something that we might later leverage to create something completely new.

For example, I fully intend to re-tread my Van's. I hope to print a rig to mount on the platform under my print bed, to create a probing tool and some software like they have in the CNC world to "see" the existing stock that's on the platform. Once done, I could then print plastic on the bottoms of my shoes. Can you imagine repairing your shoes with a 3D printer? I can.


#17

I will say, though, is that everything we learn from these job-restart

attempts is useful knowledge.

Some more info and links here:


#18

I used to do CNC work and for safety's sake, there are many times when you just abort a job in the middle because it's starting to go rogue. Given the cost of a 4'x8' sheet of marine-grade plywood as the stock (as much as $500), you want to recover as many of the parts as you can. You get good at interpreting GCODE inline and even adjusting it on-the-fly.

That link you provided is a good one, though. That's essentially the process.


#19

Hey all, I just wanted to resurrect this thread. Ive been waiting a print for a few days now and had the same thing happen. Im not entirely sure what happened but I came home and the printer reset and looks like it just timed out on the layer it was on and reset. So now the layer its on is considered zero. I know the exact layer height that it is on from counting the lines from a refrence point and then back tracking through cura, but I am unable to home the prusa and obtain that datum point for the printer because one of the parts of the model actually crosses the z axis carrige. I thought about modifying the model and sinking it into the bed to the specific layer height within cura, but I have two parts to the model and would not be able to get the exact location correct. Can I use the terminal through octoprint to somehow tell my prusa where its at? or is there a software out there where I can just modify the gcode itself and shift all the layers to this new "zero" point?

Thanks ahead of time!


#20

I have done this many times and always out of desperation (at least with my Robo C2 printer). Most of these resurrections were on carbon fiber—infused parts so I was many dollars into the part so I wanted to make an attempt at least.

It's crucial to determine exactly which layer it died on as part of all this. The fix involves turning off any processing of autoleveling (since that would crash into your parts). It involves being able to home the X/Y axes minimally. So you might need to raise the Z to the max and then manually do the X/Y homing command in the Terminal. It's important to go into absolute positioning mode before all this.

It's fairly involved and you'd have to know the G0/G1 commands intimately for your printer. The approach I take is to chop the gcode file so that what remains is the end of the job. You might need to completely review any startup gcode that's in the file and your own OctoPrint's startup gcode. Priming lines must be removed, for example.