Pi Undervoltage/power issues: I had a strange lightning bolt/temperature symbol with an exclamation mark popup in my navbar, what does that mean?

paulbeard · December 31, 2023, 2:11am

Came here because I see a warning on the menubar.
Temp shows as

temp=64.5'C

But…
so this is bad?

vcgencmd  measure_volts
volt=1.3188V

This is the power supply in use, rated as "Output: 5V DC / 2.5A Regulated Input: 100 - 240VAC"

The current job has a few hours yet to run (and maybe wouldn't if it had adequate power?) so I can't do much testing. I wonder if a Micro USB cable and one of the 1000s of iPhone chargers (they are allegedly 5W) would do better.

I do see the backpower issue mentioned above…the Creality Ender 3v2 will light up the screen when powered off from the π's meager power supply. If you're lucky, you'll also get an audible (very audible) alert over a Thermal Runaway. On a powered off machine.

On my other π (a 4B, where the above applies to a 3B), with the machine disconnected, I get

vcgencmd measure_volts
volt=0.8700V

Is the camera drawing that much just being connected?

EDIT: I found the temp issue but it was from 3 days ago:
Dec 27 20:22:35 ender-pi octoprint[416]: !!! FREQUENCY CAPPING DUE TO OVERHEATING REPORTED !!! Improve cooling on the Pi's CPU and GPU.#033[0m
I only see one instance of it but the temp isn't shown.

grep "\!\!\!" syslog
Dec 27 20:22:35 ender-pi octoprint[416]: !!! FREQUENCY CAPPING DUE TO OVERHEATING REPORTED !!! Improve cooling on the Pi's CPU and GPU.#033[0m

I didn't see it til today so maybe it needs to be in red or something…

EDIT 2: the other π I have wasn't doing anything so I disconnected the camera and checked again:

vcgencmd measure_volts
volt=0.8700V

Not sure how useful any of this is, tbh. This is also powered by a Canakit power supply, specifically for the model 4B.

EDIT 3: The model 3B finished its work so I reconnected to an old 5W iPhone charger.

vcgencmd measure_volts
volt=1.2000V

So less than the bespoke charger. Still nowhere near whatever voltage should be there. Starting to think I'm wasting my time. I see there is a warning of insufficient voltage in the UI but the voltage is now 1.3188V, same as the CanaKit unit was delivering.

PrintedWeezl · December 31, 2023, 7:18pm

RPi 4 core voltage scales dynamically with frequency in a range of 0.8V to 1.4V (going above 1.25V requires setting over_voltage in config.txt, and going above 1.35V voids your warranty). For an idle CPU, a value around 0.85V is pretty normal.

paulbeard · December 31, 2023, 7:41pm

Thank you for the clarification. Since the error value wasn't logged and seems to have been transient, I'm not going to care about this issue anymore.

EDIT: a search for the word "backfeed" turned up nothing but that's what I am seeing here. My PSU is fine, I don't see any undervoltage issues on the π — unless the printer (a Creality Ender 3v2) is turned on. I went through some testing last night and the π unit didn't report any voltage issues until I plugged in the USB connection to the powered-off printer.

I rebooted the unit, ran tail -f against syslog, then plugged it in:

Jan 24 21:34:01 rasp-pi kernel: [  435.603596] hwmon hwmon1: Undervoltage detected!

I suspect OctoPrint is searching the log for any instance of logged undervoltage conditions, not monitoring it in real time, so the warning about imminent failed prints may not be accurate. Everytime I check with vcgencmd measure_volts I get volt=1.3250V.

I just tested it again…I plugged in the powered-off printer, and ran the same commands again:

tail -f /var/log/kern.log | grep volt 
Jan 25 05:55:12 rasp-pi kernel: [30506.166209] hwmon hwmon1: Undervoltage detected!
^C
vcgencmd measure_volts
volt=1.3250V

So per this comment everything is within operational range: the Ender is dumb and bad for taking power to make its control screen flash (or stay illuminated if you power it off while connected), but the voltage warnings are not accurate. It would be helpful if the kernel reported the voltage at logging time. It isn't great if OctoPrint throttles itself due to a transient undervoltage surge.

Charlie_Powell · January 31, 2024, 10:11pm

The word more commonly used on these forums is 'backpowering'. It's a huge issue among 3d printers, particularly Creality printers but certainly not exclusively.

OctoPrint checks the results of vcgencmd get_throttled at regular intervals (I can't remember the default from my head, but it's configured in the 'Pi Support' plugin settings). OctoPrint uses this command to work out 4 things, which are given to it:

If there is current undervoltage
If there has been undervoltage since boot
If there is current overheating
If there has been overheating since boot.

vcgencmd measure_volts does not, I believe, accurately represent the throttling situation. I understand that the Pi will report that it's throttling if it thinks the input voltage is lower than ~4.65v, and that's once it's passed the regulators etc. which has a bit of drop on it. At the point that this threshold is reached, get_throttled will report throttling, and the 'Undervoltage detected!' line will be logged.

OctoPrint doesn't throttle itself due to transient undervoltage - it only restricts certain activities (installing plugins & updates sort of things) while vcgencmd get_throttled reports either of the above 'current' states. If it is reporting just that it happened since boot, you will see a warning of this different state.

paulbeard · February 8, 2024, 3:40am

Yeah, I'm not finding this all that helpful. I expect these events were when I reconnected the octoprint device and powered the printer on. Two of these that reverted 30 seconds later are not going to get me to reach for my wallet.

2024-02-07 18:45:31,432 - octoprint.plugins.tracking - INFO - Sent tracking event system_throttled, payload: {'throttled_now': True, 'throttled_past': True, 'throttled_mask': 327685, 'throttled_voltage_now': True, 'throttled_voltage_past': True, 'throttled_overheat_now': False, 'throttled_overheat_past': False}
2024-02-07 18:45:52,178 - octoprint.plugins.tracking - INFO - Sent tracking event system_unthrottled, payload: {'throttled_now': False, 'throttled_past': True, 'throttled_mask': 327680, 'throttled_voltage_now': False, 'throttled_voltage_past': True, 'throttled_overheat_now': False, 'throttled_overheat_past': False}
2024-02-07 18:46:01,696 - octoprint.plugins.tracking - INFO - Sent tracking event system_throttled, payload: {'throttled_now': True, 'throttled_past': True, 'throttled_mask': 327685, 'throttled_voltage_now': True, 'throttled_voltage_past': True, 'throttled_overheat_now': False, 'throttled_overheat_past': False}
2024-02-07 18:46:31,751 - octoprint.plugins.tracking - INFO - Sent tracking event system_unthrottled, payload: {'throttled_now': False, 'throttled_past': True, 'throttled_mask': 327680, 'throttled_voltage_now': False, 'throttled_voltage_past': True, 'throttled_overheat_now': False, 'throttled_overheat_past': False}

Syslog reports more undervoltage conditions:

Feb  7 18:46:32 rasp-pi kernel: [1199993.259285] hwmon hwmon1: Undervoltage detected!
Feb  7 18:46:47 rasp-pi kernel: [1200007.819665] hwmon hwmon1: Undervoltage detected!
Feb  7 19:21:35 rasp-pi kernel: [1202096.185015] hwmon hwmon1: Undervoltage detected!
Feb  7 19:23:33 rasp-pi kernel: [1202214.747644] hwmon hwmon1: Undervoltage detected!

Meanwhile, I get this result all day long:

vcgencmd measure_volts
volt=1.3250V

No idea what input voltage is: that would useful to find in syslog.

This on a π 3B+ with a 3rd party wall wart specific to the 3B+. Sadly, it uses the micro B plug so swapping something else in is not easy. I have various Apple power supplies I could test with, including a 20W, but what will the π do with all that?

Charlie_Powell · February 8, 2024, 7:45am

measure_volts is irrelevant, it doesn't tell you anything about the throttling. If I remember correctly the Pi doesn't actually measure the voltage & then throttle itself using software, undervoltage detection happens with an on/off signal within the chip. I can't remember exactly how it works off the top of my head.

If you have plugged in a usb power supply, then you can measure the voltage with a multimeter across the 5v GPIO pins to see what it is, as this takes in to account the voltage drop of the regulator. Effectively measuring the "other side", this is what the internal 5V line is actually doing.

If it occurs when the printer is plugged in or powered on, then I'd recommend mitigating the back powering issue - Put tape on the 5V pin - Why and how

A 20W phone charger will likely not work, as often they will not output that full power continuously, or they will require some negotiation with the device first.

If you are confident that the undervoltage throttling only happens at specific situations, then you can choose to safely ignore the warning. Instability only comes during throttling events.

paulbeard · February 8, 2024, 2:32pm

I have taken the Ender out of service so the back powering is not an issue now. I replaced it with a (very) used Prusa MK3s which is having some issues, but it's not clear OctoPrint is part of them. What does an undervoltage-induced failure look like? What stability is jeopardized? The spaghetti monster did pay a call last night but the video suggests it was bed adhesion: the pieces were about half complete and then one or more came loose and chaos ensued. I'd like to remove what variables I can so maybe I fly blind for bit or only run jobs when I can be nearby. I'll assume the voltage isn't related as there were no indications about it during the failed job.

It would be useful if there was some indication what caused the throttling. This isn't very helpful.

Feb  7 22:05:55 rasp-pi octoprint[400]: 2024-02-07 22:05:55,569 - octoprint.plugins.tracking - INFO - Sent tracking event system_throttled, payload: {'throttled_now': True, 'throttled_past': True, 'throttled_mask': 327685, 'throttled_voltage_now': True, 'throttled_voltage_past': True, 'throttled_overheat_now': False, 'throttled_overheat_past': False}#033[0m

Charlie_Powell · February 10, 2024, 9:05pm

Undervoltage only impacts the Raspberry Pi directly. The failure process generally looks a bit like this

Undervoltage event occurs, 5V line drops lower than we would like
This is picked up by the SoC on the Pi, which then throttles itself. In the same way that a chip throttles itself when it gets too hot, to avoid a complete brownout processing performance is limited.
As performance is limited, you have all associated instability. Slow UI loads, potentially stuttering on the printer if the Pi cannot keep up. There is no definitive list.
Additionally, voltage drop seems to not be great for the SD card, and can result in writes to the SD card going wonky - this is why installing plugins & updates is not allowed during throttling events, as we've seen a large number of issues with corrupted installs etc.
I suspect there's other components on the RPi board that do not like insufficient power either, but I am not an electrical engineer nor do I know how they are made

In theory, it would be possible to have a print failure. For example, if you opened the OctoPrint UI, and you had a lot of plugins so this took quite a lot of resources. Normally, this might be OK, but because you are throttled now the Pi struggles to both print & load UI at the same time, you get a pause.

There is no other information available. All the Pi knows is that it's voltage dropped, so it is throttling. Undervoltage occurs when your power requirements exceed the power you are giving it. The most power is drawn on startup, so for a weak power supply it can cause voltage drop then. But there is nothing concrete I can tell you other than it's reporting undervoltage.

paulbeard · February 10, 2024, 11:03pm

If I run top(1) on the command line to see how the device running octoprint is doing, It is running at less than 10% of CPU. And these events are transient…the throttling condition lasts less than the 30 second polling cycle or however it's being logged.

So we have a condition that is not sampled in realtime, that has no metrics of any kind, no voltage or specifics, that occurs infrequently and never lasts more than 30 seconds, with no actual link to performance, which we can monitor through top(1) or other command line means and see no relationship between CPU load and the throttling being logged.

I feel pretty confident ignoring it. While I agree it's useful to know that PSU is garbage and is not delivering the wattage your device needs, there seems to be some debate over what voltage it really needs. The data sheet claims it needs 5V/2.5A DC via micro USB connector but testing and anecdotal reports say 5.1V or more is better. The official Rπ PSU delivers 5.1V. Maybe they know something the other PSU makers don't.

I have been doing some testing on a machine that has a π 3B+ with one of these crappy CanaKit PSUs and the print performance seems unrelated to whether octoprint is running, if the job is run from the SD card or via USB. So I am unconvinced this error message is useful as anything than a sign you have a crummy PSU. If anyone has a sample of a series of throttling message that can be tied to a failed print, that would be useful but I haven't seen anything yet. Nor have I read anything about these allegedly underpowered PSUs failing and taking down jobs with them.
The π seems to be coasting, lightly loaded, sending data and taking snapshots on Z-layer changes, and never seems to show any load spikes that I can see. It's not doing the work…the CPU in the printer is doing all that.

foosel · February 12, 2024, 12:48pm

I haven't kept a log over the years, but I've been running this project for more than 11 years now, and during this time I've seen problems reported by users that went away after they switched to a PSU that no longer made their Pi report undervoltage that involved:

Low network bandwidth
Intermittent network failures
Data corruption
Issues in the serial communication with the printer, leading to failed prints
Command corruption on the serial communication leading to constant resends
Parts of the UI failing to load
Failure to write settings to disk

and probably more than that which I now have forgotten.

And let me repeat, those were things that vanished repeatedly once the affected users removed their undervoltage situation and that is why I added the feature to evaluate the reporting of the Pi itself and expose the result on the UI, because all those crappy phone chargers and whatnot were causing a TON of support overhead and bogus bug reports.

And around every 6 months someone comes around and starts up the same discussion again, and don't take this personally please but at this point, after all of the above experience over the course of the past decade this is frankly just plain exhausting.

Take it up with the Raspberry Pi foundation please if you think the Pi is misreporting things here, but understand that the warning, the minor limitations, all of the "in your face" messaging about observed issues are there because we have literally seen some shit happening due to undervoltage, throttling and brownouts.

Hairyloon · February 29, 2024, 8:23pm

Sorry to ask, but can you clarify this:

[quote="foosel, post:72, topic:5256"]if vcgencmd reports an undervoltage event there was an undervoltage event.
[/quote]
"An event" suggests a thing that happened, rather than something which is ongoing. Possibly something happened that caused a momentary dip, and while I can see that that isn't desirable, I would prefer to do a little more diagnosis before I rush out and buy a new PSU.

So, what constitutes an "Event"?
How low does it have to go for how long?
Can the status be reset to see if it happens again?

Charlie_Powell · February 29, 2024, 11:31pm

You will be able to get more specific timings over when undervoltage started & stopped from the syslog/dmesg, I believe it's logged there. What we're referring to as "an event" is any time undervoltage was detected by the Pi. I don't know the exact hardware specs on how quick this can react. The threshold is around 4.65v. OctoPrint itself just regularly checks the current status from vcgencmd, it doesn't actually monitor undervoltage. Questions on specifics like how low and how long should probably go to the RPi foundation since it's their hardware & software, not ours.

OctoPrint does differentiate between "past" conditions and "current" conditions, and if it sees the former it is only going to provide you a warning and nothing more.

Looking at dmesg will tell you more detail about when it happened.

Hairyloon · March 4, 2024, 11:34am

OK, thanks.
Is there some way to reset the warning on Octoprint?

Charlie_Powell · March 4, 2024, 11:50am

There is not - because the warning information is not generated by OctoPrint. If you "reset" it OctoPrint's side, it will just re-appear again as soon as vcgencmd get_throttled is checked, because this has not been reset the status is still the same. OctoPrint is just a messenger here, it doesn't control the warnings.

Ewald_Ikemann · March 11, 2024, 12:57pm

@Will_N
tl.dr ... ...

paulbeard · May 12, 2024, 9:33pm

Very much tired of dealing with this, I don't want to replace the Raspberry π board when it's the PSU that's the issue. Is there a known good 5.1V (or whatever it required to get around this) PSU? Does this official Raspberry π pass muster here?

Hairyloon · May 13, 2024, 12:30pm

Have you tried taping over the USB power line?
I was not convinced by that idea and avoided it for too long, but after I upgraded the PSU again, I decided to try it.
Not seen an undervolting since...
I'll probably get one now, of course.

CmdrCody · May 13, 2024, 2:06pm

I have 4 Pis. One 4, 1 3b+ and 2 2b. So the 2s were swapped out for 3b+s in a application back in 2018. So the 2s are from 2015 and are 9 years old now. One of them will throttle on undervoltage when it boots and about anytime I take it over 75% load. I have rotated all permutations of PSU and only this one will always fail. It's the board. So a little google foo and I found how to disable the check. You can too.
Just be prepared to lose your SD card, your prints and eventually the Pi. Mine sits on my least used printer and the poor thing has certainly paid its' dues.
Ot buy a new 3b+, they are still only $35...

jcassel · May 13, 2024, 3:11pm

I have never had an under voltage issue using a mini PC to run Ubuntu+OctoPrint. They tend to come with the proper power supply in the box. Seems so obviously.

Using a mini PC has completely eliminated the "What's wrong with my Raspberry pi this time routine. Is it the power supply?, Is it the heat?, Is it the SD card? Is it the Wifi? Is it sun spots?"

Some like that churn of fixing then refixing. I find that if it does not require the IO, and even sometimes when it does. It's just easier to use a more finished compute device.

Buy it once set it up once and move on to other more interesting adventures.

hillshum · May 13, 2024, 9:36pm

I've found that the official PSU from the RPi foundation has indeed worked well, in contrast to various units I've tried, some of which were even ostensibly high-quality phone chargers. I'm at a point where I'm getting an official PSU each time I buy a Pi, unless I'm going to put together a custom power supply (like a buck converter connected to a 3D printer PSU).

Also be sure you get a PSU for the correct Pi version.