OctoPrint crashed with 50 layers left on a 3 day print. :(

Yep, mine was about strictly the 3b+ and system crashes.

If it does it again, before rebooting or restarting octoprint, double check the disk usage with df -h just to verify it didn't fill the file system

1 Like

I think you meant USB not Serial. It's shielded.

:laugh: And what does the S in USB stand for?

Yeah, do the df suggested earlier. Running out of room on your root drive would cause a crash.

Things we know:

  • Not a 3B+ (emphasis on the plus)
  • Shielded serial cable
  • The OctoPrint service wasn't running at the time of the problem

Things we don't know:

  • 5V 2.5A Raspberry power supply?
  • Firm micro USB connection to the Raspberry Pi for power? There are known problems with some combinations of micro USB plugs versus the connectors.
  • Available space on your root drive (not to be confused with your boot partition)
  • We don't have your octoprint.log from the failed session

LOL... Wow your right... Been throwing the term USB around for so long that I've glazed over what it means. That's embarrassing given my line of work.

I like the way you think.

Add to known things the following:

  • 5V 2.5A provided Raspberry power supply is what is being used here.
  • USB feels appropriately firm in it's connection.

The rest will be provided should the issue arise again.

Thank you all for your time.
:slight_smile:

That "tight connection" also goes for the serial cable to the printer as well as the micro USB for powering the Raspberry.

Try to look in your Settings -> Logging and try to find the octoprint.log version from the crash. Look at the timestamps if you're wondering.

1 Like

I often ask myself this same question. I still havent solved it, but I have successfully resumed a couple of prints in the past. Most recently something went wrong and I ended up resetting and reprinting.. mostly because grinding the printhead into the print is horrifying, so once it happens once i just surrender.
That said, you might consider figuring out if you have clearances to do each X, Y and Z home individually, and find places on the bed where it will work and not kill anything.
Afteer that it's a case of selectively editing the gcode... chopping out the usual homes, mesh leveling and everything after the heating commands, right up till the layer it choked on.
It's always ugly though. One day I'll be compelled to figure it out better. Even just modding firmware or a chunk of hardware to monitor Z value would be very useful. No doubt the existing marlin/etc power failure code would be well on the way to that kind of thing.

I will say, though, is that everything we learn from these job-restart attempts is useful knowledge. That knowledge about G0/G1-related commands and absolute/relative modes and editing GCODE is something that we might later leverage to create something completely new.

For example, I fully intend to re-tread my Van's. I hope to print a rig to mount on the platform under my print bed, to create a probing tool and some software like they have in the CNC world to "see" the existing stock that's on the platform. Once done, I could then print plastic on the bottoms of my shoes. Can you imagine repairing your shoes with a 3D printer? I can.

1 Like

I will say, though, is that everything we learn from these job-restart

attempts is useful knowledge.

Some more info and links here:

2 Likes

I used to do CNC work and for safety's sake, there are many times when you just abort a job in the middle because it's starting to go rogue. Given the cost of a 4'x8' sheet of marine-grade plywood as the stock (as much as $500), you want to recover as many of the parts as you can. You get good at interpreting GCODE inline and even adjusting it on-the-fly.

That link you provided is a good one, though. That's essentially the process.

1 Like

Hey all, I just wanted to resurrect this thread. Ive been waiting a print for a few days now and had the same thing happen. Im not entirely sure what happened but I came home and the printer reset and looks like it just timed out on the layer it was on and reset. So now the layer its on is considered zero. I know the exact layer height that it is on from counting the lines from a refrence point and then back tracking through cura, but I am unable to home the prusa and obtain that datum point for the printer because one of the parts of the model actually crosses the z axis carrige. I thought about modifying the model and sinking it into the bed to the specific layer height within cura, but I have two parts to the model and would not be able to get the exact location correct. Can I use the terminal through octoprint to somehow tell my prusa where its at? or is there a software out there where I can just modify the gcode itself and shift all the layers to this new "zero" point?

Thanks ahead of time!

I have done this many times and always out of desperation (at least with my Robo C2 printer). Most of these resurrections were on carbon fiber—infused parts so I was many dollars into the part so I wanted to make an attempt at least.

It's crucial to determine exactly which layer it died on as part of all this. The fix involves turning off any processing of autoleveling (since that would crash into your parts). It involves being able to home the X/Y axes minimally. So you might need to raise the Z to the max and then manually do the X/Y homing command in the Terminal. It's important to go into absolute positioning mode before all this.

It's fairly involved and you'd have to know the G0/G1 commands intimately for your printer. The approach I take is to chop the gcode file so that what remains is the end of the job. You might need to completely review any startup gcode that's in the file and your own OctoPrint's startup gcode. Priming lines must be removed, for example.

In the spirit of this thread, can I ask if there's a way (log somewhere?) to know which layer last printed before a power failure to my octopi, so that I might edit the gcode to start again at following layer? The xy geometry is very small, but the z is very tall so I'd love to complete this print. Also is it safe to assume whatever didn't complete within the last attempted layer is probably ok to skip at 2mm /layer? And finally do I ignore my z offset when calculating new start z-position? Or do I need to include an allowance?

I know of no easy method since OctoPrint pushes many gcode lines into the printer's receive buffer; you don't know how many were actually printed.

Measure accurately, review the gcode file carefully and look at the layer comments. Look at the gcode at the beginning of a layer and look at the G0 or G1 command which includes a Z value.

When I think I've got the layer that's close, I'll then use the manual jogging commands in the Control tab of OctoPrint (no heat) to move the extruder carefully above this point and in some corner. I'll then jog it closer in the Z direction until I've determined what is likely the exact point where it left off.

Taking this back to the gcode, I'll then chop the file's first part to remove it (leaving some of the initial lines). You'd have to turn off autoleveling and any priming lines and any code which would crash into your part. It may be necessary to home the printer and then add a line which brings the extruder assembly somewhere safely over the middle of the part (within your Gcode Start script).

It's best to try to re-do the entire last layer so that there aren't gaps.

It's all very fussy but you can rescue a part.

Great answer - didnt occur to me but its simple and smart - thanks.

I've been accused of being both simple and smart. :wink:

why it say last z heigh so we run repeat it same height :smile:

Unfortunately, OctoPrint sends commands and the firmware & printer board eventually run those commands. So you never know the last command that was run. You're right, though, recording the last z layer/height would help in this.

i say that maybe it say last layer with log.file and i see this panel and i repeat print this layer or last layer or next layer. thiis not importent. because can be sanded

Doesn't the terminal keep track of exactly where you're at in the print? I'm using OctaPrint connected directly to my Ender 3 with no RasPi in between. I've had two failures on the same print today for completely unrelated reasons. The first time we had a general power failure. Time to get a UPS! The second I'm clueless about. The printer said it reset for no good reason while I was sitting at my laptop two feet away.

So my terminal output at the end is:

------------------------
Send: N271889 G0 F300 X111.565 Y115.872 Z140.6*24
Recv: ok
Send: N271890 G0 F9000 X112.292 Y118.917*126
Recv: ok
Send: N271891 G0 X112.695 Y119.788*26
Recv: ok
Send: N271892 G0 X113.275 Y120.554*27
Recv: ok
Send: N271893 G0 X114.009 Y121.176*17
Recv:  T:240.00 /240.00 B:59.87 /60.00 @:83 B@:127
Recv: ok
Send: N271894 G0 X114.81 Y121.581*42
Recv: ok
Send: N271895 G0 X116.328 Y122.097*24
Recv: ok
Send: N271896 G1 F1009.3 X116.899 Y122.083 E2789.23934*29
Recv: ok
Send: N271897 G1 X117.047 Y122.062 E2789.24431*101
Recv: ok
Send: N271898 G1 X117.24 Y122.041 E2789.25077*89
Recv: ok
Send: N271899 G1 X117.575 Y121.972 E2789.26214*103
Recv: ok
Send: N271900 G1 X117.732 Y121.932 E2789.26753*101
Recv: start
Printer sent 'start' while printing. External reset? Aborting job since printer lost state.
Send: N0 M110 N0*125
Changing monitoring state from "Printing" to "Cancelling"
Recv: echo: External Reset
Recv: Marlistart
Recv: echo: External Reset
Recv: Marlin Marlin 1.1.6
------------------------

My thought was "Trace back to the last OKed command", which, here, is:

-------------
Send: N271899 G1 X117.575 Y121.972 E2789.26214*103
Recv: ok
-------------

If I find that in my .gcode and search backward for Z, I get:

--------------
Send: N271889 G0 F300 X111.565 Y115.872 Z140.6*24
Recv: ok
--------------

Doesn't that tell me the print stopped at Z=140.6?

I then tried various hacks of the .gcode to remove everything before the last successful command except for heat-up. The first time I deleted everything up to the last successful ("Ok") command then except the preheat commands. However I didn't use G92 to reset the initial position, so I had to flip the power switch before the head went off on some kind of extracurricular adventure.

The second time I added:

G92 X117.575 Y121.972 E2789.26214

...right before the command the print failed on, which is supposed to tell the printer "This is where you current position is."

The second time I added that command but forgot the preheat. First it knocked the print off of the base, but it didn't work anyway and started tracking far away again so I had to shut it off.

I'm not exactly sure why those didn't work, because they weren't just off by a few layers, the head was headed for ZMAX before I hit the power switch, but it sounds like this should work. You leave everything exactly where it is, make sure things don't cool down to where you lose adhesion, and continue the print exactly where it left off. Tomorrow I think I will try this deliberately with a simpler model to see if I can figure out where I went wrong. I also got GetToDaChoppa to try using that to do a re-homed restart, but it seems like doing an in-place restart would be more likely to work.

Love and Spaghetti,

Scott

Restarting a failed job can be tricky. If your printer does an autolevel routine normally, you should turn that off on the 2nd job since otherwise your hotend would slam into your part or get inaccurate readings.

You also want to be careful on parts which sit on a raft. Make sure that you calculate the 2nd job properly.

I wrote GetToDahChoppa to both print multi-color parts by layer and to help resurrect failed parts. It essentially does what I used to do manually: remove some of the lines from the gcode file in the middle section but leaving the start/ending.