Updating from 1.3.10 to 1.3.11 through web interface broke octoprint

What certainly strikes me as odd is that:

2019-05-15 10:43:36,083 > /usr/include/arm-linux-gnueabihf/bits/sigset.h:21:1: error: invalid suffix "ypudef" on integer constant
2019-05-15 10:43:36,084 > 4ypudef int __sig_atomib_t;
2019-05-15 10:43:36,084 > ^~~~~~~

Looks like garbage in the sigset.h file causing the compilation error. Here the lines reads like this:

typedef int __sig_atomic_t;

Ever removed power in the past, ran into undervoltage issues or might the SD card or its filesystem become a bit old?

nope.... power is hardwired in from 5VSB line on an ATX psu

pi works fine.... i still have access to it and i have 2 samba shares that i can access

also i did a reboot just before i tried the update and octoprint came up fine

the pip install --force-reinstall https://get.octoprint.org/latest gave a lot of output... how do i capture that to a file so its more easy to attach?

(oprint) pi@octopi:~ $ pip install --force-reinstall https://get.octoprint.org/latest > ~/.octoprint/logs/octoprint_install_manual.log
  Failed building wheel for netifaces
  Failed building wheel for psutil
Command "/home/pi/oprint/bin/python2 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-1WFcOS/netifaces/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-DK8C0Y-record/install-record.txt --single-version-externally-managed --compile --install-headers /home/pi/oprint/include/site/python2.7/netifaces" failed with error code 1 in /tmp/pip-build-1WFcOS/netifaces/
You are using pip version 9.0.3, however version 19.1.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
(oprint) pi@octopi:~ $

octoprint_install_manual.log (47.1 KB)

Apparently it doesn't work as fine as you think it does, otherwise a somewhat important header file wouldn't say 4ypudef where it should say typedef. Or at least that's what your update log claims to be the case here. You can check /usr/include/arm-linux-gnueabihf/bits/sigset.h yourself, issue should be visible on line 21.

And a 5VSB line on an ATX psu doesn't automatically mean "it never lost power without prior shutdown before" either.

So, based on your recent log:

    /usr/include/arm-linux-gnueabihf/bits/sigset.h:22:1: error: invalid suffix "ypudef" on integer constant
     4ypudef int"__sig_atomib_t;
     ^~~~~~~

your header files are corrupt. There's even an " now that doesn't belong. That's not an OctoPrint issue, that's an issue with something on your file system being corrupted or slowly dying for whatever reason. One possible reason is a past powerloss, another is the SD card dying, and there are a multitude of others.

I suggest you make a manual backup, get yourself a new SD card and reimage. If random header files contain bits and bytes they shouldn't, god knows what else is corrupt on that card.

are there ways i can temp fix things without re-image?

i need to get a sd-card reader as i have packed away the only laptop that has one (but it does not have a keyboard or trackpad)

I don't know. If it's only that file, yeah, sure, fix it. It should be this, but better do a diff: Wrong file I think. Seems to be part of libc6-dev, so maybe try apt install --reinstall libc6-dev.

But as I said, it's very likely that this isn't the only things that's corrupt on this image, only the first thing that you ran into, so I can't tell you if that will lead to success or just make you run head first into the next wall.

for sure i need to re-image

but if i can keep it going for 2 weeks more i will just buy a sd card reader and a new sd card and be done with it

time will tell

sudo apt-get install --reinstall libc6-dev and then

sudo service octoprint stop
source ~/oprint/bin/activate
pip install --force-reinstall https://get.octoprint.org/latest
octoprint serve --safe

but yes i'm now in "limb" mode

I ended up just reinstalling my SD card, however I did include the tail of octoprint.log and the update log in the opening post. I saw no errors or anything in them, but now they are lost to the nether.

The card was initially updated from a fresh install - I had literally installed it two nights before the update and never got to use it, so it was an entirely fresh install of octopi with no modifications/updates/etc.

Nothing was run as sudo unless noted, I'm pretty picky about running things as sudo. Perms also looked ok as everything was owned by pi/pi but i ran the chown command to ensure this just to be sure, and it made no difference.

I have not yet updated my reinstall.

I wanted to highlight this.

You didn't get an error about urllib, you got a warning about urllib. It's a fairly strong and serious-looking warning, but .. it's just a warning.

More detail: the boto3 and requests peeps haven't blessed urllib's 1.25.x stuff yet, so the warning says to narrow the requirements to urllib3<1.25,>=1.20. It's a little excessive to do that everywhere.

requests also wants chardet to be narrowly specified.

1 Like

It's difficult for characters in the middle of a text file to get corrupt like this. I would suggest that power-cycling the Raspberry Pi during an upgrade would result in perhaps several related truncated and corrupted text files. Their endings would be trashed rather than the middle.

The underlying ext4 is a journaling file system with checksums. Even if the Raspberry Pi's 5V dropped down to something too low to write to the microSD, it should at least throw a kernel error. (Looking for undervoltage in a case like this is probably a good way of preventing future problems on the same rig.) I would guess that delayed allocation couldn't be blamed here (again, it would be an ending problem). Check /var/log/kern.log, probably looking for the word crc or CRC.

My gut's telling me that this is a network problem during some update/upgrade step which faithfully wrote in-transit corrupted data. But you'd think tcp would prevent that. Apologies, but this sort of thing really shouldn't happen at all.


There's an expression, "one bad apple spoils the whole bushel". If I saw a single corrupted system text file like this, I would start over from scratch. Whatever conditions allowed this corruption in the past likely could have spilled over to other files so it's effectively a ticking time bomb waiting to go off in the future. It's not worth risking your print job(s) to this.

There's definitely a "is it work fixing or just replacing" thing here.

I thought it was "one bad Lisa almost spoils the Apple"?

You can try this:

nano ~/OctoPrint/venv/bin/octoprint
Change, if necessary, 1.3.10 to 1.3.11

pip uninstall urllib3
pip install -I urllib3==1.24.3

sudo reboot

edited to remove the scary-looking URL. Again, I repeat myself: the urllib3 version message is a warning, not an error. It's not the cause of "breaking" Octoprint.

1 Like

Hi tedder42!

It was just a direct link to official website. I don't know why, but in my case "==1.24.3" construction doesn't work.

I did not think about errors or warnings. I just ran this, got errors and made changes to make it work:
~/OctoPrint/venv/bin/octoprint

Traceback (most recent call last):
File "/home/opi/OctoPrint/venv/bin/octoprint", line 6, in <module>
    from pkg_resources import load_entry_point
  File "/home/opi/OctoPrint/venv/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 3019, in <module>
    @_call_aside
  File "/home/opi/OctoPrint/venv/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 3003, in _call_aside
    f(*args, **kwargs)
  File "/home/opi/OctoPrint/venv/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 3032, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
  File "/home/opi/OctoPrint/venv/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 657, in _build_master
    return cls._build_from_requirements(__requires__)
  File "/home/opi/OctoPrint/venv/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 670, in _build_from_requirements
    dists = ws.resolve(reqs, Environment())
  File "/home/opi/OctoPrint/venv/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 854, in resolve
    raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.ContextualVersionConflict: (urllib3 1.25 (/home/opi/.local/lib/python2.7/site-packages), Requirement.parse('urllib3<1.25,>=1.21.1'), set(['requests']))

Ok, I thought because the installer exited here.
I can not reproduce it any more. I have installed Octoprint on the same OS, after cleaning the Octoprint folders and reinstalled urllib3, and at now everything works fine even with the restored backup. If I uninstall urllib3 the octoprint installer install it again and works fine.

If it's any consolation, I just fired up my printer having returned from working away for the last 2 weeks and it all started up fine. There was a nag box on Octoprint telling me an update was available so I installed it, or at least attempted to. It killed it. I've just downloaded the latest version from Octoprint and installed on a spare SD card. It's all working fine again now but there's something in that update that seems to corrupt the install. My Pi was running a pure Octoprint image, nothing else, no plug-ins etc. It could do with fixing or at lease withdrawing until it gets fixed. I'm not sure what it give that the old version didn't but it's not worth the hassle until it's fixed.

I did 10 update scenarios from various OctoPrint+OctoPi installation combinations to 1.3.11. I did another 6 per each RC. Not a single issue. At the time of writing this there are currently over 8000 installations of 1.3.11 reported in the field world wide, with over 34000 hours of successful print jobs shared between them since May 14th. Add to that another ~1000 instances that updated to and ran the RCs and accumulated another 58000 hours of successful printjobs on those.

Given these numbers I think we can safely assume that this isn't a widespread general problem but rather something that seems to affect a small handful of people with what looks like diverse base environments with unknown maintenance state and also very different manifesting symptoms, which makes a common problem that would justify "withdrawing" very unlikely, and also so far doesn't even give a hint that there is something to be fixed on OctoPrint's end in the first place.

That being said I'm happy to give support to people running into update issues - as long as they provide the information needed in order to help them that is, "it's broken" doesn't suffice - and I'll happily push out a hotfix release if during the course of that an actual problem with the update should get identified.

1 Like