Unreliable multi webcam setup

Camera model

  • Chinese unbranded "HD" camera
  • Chinese unbranded 5mm endoscope

Both works correctly (tested on other machines)

What is the problem?

I've successfully setup two webcams with the new camera-streamer stack (following the post by foosel) and they works... sometimes. Even without rebooting the system or touching anything the webcams stop sending frames and Octoprint returns the following error response when issuing, for example, a GET /stream to octoprint.local:<port>

HTTP/1.1 500 Server Error
Content-Type: text/plain
Content-Length: 13

Server Error
No frames.

Note that the service is up and running as GET / returns the expected page

HTTP/1.1 200 OK
Content-Type: text/html
Content-Length: 2733
Access-Control-Allow-Origin: *

<!DOCTYPE html>

<html>
...
		<li>
			<a href="stream"><b>/stream</b></a> (MJPEG stream)</b><br>
			<br>
			<ul>
				<li>Get a live stream. Works everywhere, but consumes a ton of bandwidth.</li>
				<li>Uses resolution specified by <i>-camera-stream.height=</i>.</li>
			</ul>
		</li>
...
</body>
</html>

And the system service service is up and running (reporting the endoscope as an example but the same goes with the default service of the HD webcam)

pi@octoprint:~ $ sudo systemctl status camera-streamer-usb@endoscope.service
● camera-streamer-usb@endoscope.service - camera-streamer endoscope
     Loaded: loaded (/etc/systemd/system/camera-streamer-usb@.service; disabled; vendor preset: enabled)
    Drop-In: /etc/systemd/system/camera-streamer-usb@.service.d
             └─override.conf
     Active: active (running) since Thu 2024-03-07 18:37:23 CET; 3h 8min ago
TriggeredBy: ● camera-streamer-usb-endoscope.path
    Process: 1114 ExecCondition=/bin/sh -c [ -e $DEVICE ] (code=exited, status=0/SUCCESS)
    Process: 1115 ExecCondition=/bin/sh -c ! ss -ltn src :$PORT | grep -q LISTEN (code=exited, status=0/SUCCESS)
   Main PID: 1118 (camera-streamer)
      Tasks: 19 (limit: 4915)
        CPU: 25.085s
     CGroup: /system.slice/system-camera\x2dstreamer\x2dusb.slice/camera-streamer-usb@endoscope.service
             └─1118 /usr/bin/camera-streamer --http-port=8081 --camera-type=v4l2 --camera-path=/dev/v4l/by-id/usb-Generic_HD_camera-video-index0 --camera-forma>

Mar 07 21:38:57 octoprint sh[1118]: util/http/http.c: HTTP8081/7: Client disconnected 192.168.1.187.
Mar 07 21:44:07 octoprint sh[1118]: util/http/http.c: HTTP8081/8: Client connected 192.168.1.187 (fd=12).
Mar 07 21:44:07 octoprint sh[1118]: util/http/http.c: HTTP8081/8: Request 'GET' '/stream' ''
Mar 07 21:44:09 octoprint sh[1118]: util/http/http.c: HTTP8081/8: Client disconnected 192.168.1.187.
Mar 07 21:45:06 octoprint sh[1118]: util/http/http.c: HTTP8081/9: Client connected 192.168.1.187 (fd=13).
Mar 07 21:45:06 octoprint sh[1118]: util/http/http.c: HTTP8081/9: Request 'GET' '/stream' ''
Mar 07 21:45:08 octoprint sh[1118]: util/http/http.c: HTTP8081/9: Client disconnected 192.168.1.187.
Mar 07 21:46:02 octoprint sh[1118]: util/http/http.c: HTTP8081/0: Client connected 192.168.1.187 (fd=4).
Mar 07 21:46:02 octoprint sh[1118]: util/http/http.c: HTTP8081/0: Request 'GET' '/stream' ''
Mar 07 21:46:04 octoprint sh[1118]: util/http/http.c: HTTP8081/0: Client disconnected 192.168.1.187.
lines 1-24/24 (END)

The issue does not arise consistently and if no one touch the system, after a while, the cameras might start to work again, after a non defined span of operative time, they randomly stops.
I've successfully printed for hours before they stopped working.

Even when both cams are accessed at the same time the CPU usage is low (under 5% on each core) and the cameras works nicely

I was trying to get a screenshot of it to post along with this but the endoscope frustratingly stopped working while I was about to capture the image -.-

I think this issue is related to Octoprint and the new camera stack and I'm thinking to roll-back to the old stack as this is getting hard to address.
As I didn't really debugged the issue myself I didn't want to open a new Issue on GitHub and was hoping that someone could know why this happens.
I already read about similar issues but the proposed solutions didn't work for me.

It would be nice to have a hint on how to debug this issue as octopi logs are not really helpful this time.

What did you already try to solve it?

  • Changed OPTIONS in config files (currently both have only OPTIONS='--http-listen=0.0.0.0' set)
  • Restarted webcams and camera streamer in various ways (systemctl restart camera-streamer, sudo camera-streamer-control restart, systemctl restart camera-streamer-usb@endoscope.service, etc...
  • Read realtime log journalctl -fu camera-streamer\*
  • Different power supplies for Raspberry (from 15.0W, to 65W)
  • Switching USB ports (currently using USB 3.0 ports for both devices)
  • Restart Octoprint systemctl restart octoprint.service
  • Reboot RPI sudo shutdown now -r
  • Tried switching net interface from wlan0, eth0. Currently using connection trough Ethernet cable

Have you tried running in safe mode?

Yes, multiple times

Did running in safe mode solve the problem?

No, it didn't. The issue persist without any difference in behaviour

Systeminfo Bundle

octoprint-systeminfo-20240308004634.zip (67.6 KB)

Additional information about your setup

OctoPrint version, OctoPi version, printer, firmware, browser, operating system, ... as much data as possible

  • OctoPrint 1.9.3
  • Python 3.9.2
  • OctoPi* 1.0.0cam (build 2023.10.09.154319)
  • Original Prusa Mini
  • Official FW v5.1.2+13478
  • Mozilla Firefox Developer Edition for Arch Linux - 124.0b6 (64-bit)

Does restarting camera-streamer fix the issue? If so it might be yet another case of

which is the current blocker for making this the default streamer.

I see a lot of

Mar 07 19:01:08 octoprint sh[645]: device/links.c: SNAPSHOT:capture:mplane:buf0: Capture image is outdated. Skipped. Now: 1567108381, vs 1447081231.

in your camera-streamer journal entries in the system info bundle, so maybe that's indeed the same issue with a different error message but the same outcome.

No, it doesn't fix it

pi@octoprint:~ $ sudo camera-streamer-control restart
Running restart for camera-streamer-libcamera.service...
... done.
Running restart for camera-streamer-usb-default.path...
... done.
Running restart for camera-streamer-usb@default.service...
... done.
Running restart for camera-streamer-usb-endoscope.path...
... done.
Running restart for camera-streamer-usb@endoscope.service...
... done.
pi@octoprint:~ $ systemctl restart camera-streamer
==== AUTHENTICATING FOR org.freedesktop.systemd1.manage-units ===
Authentication is required to restart 'camera-streamer.service'.
Authenticating as: ,,, (pi)
Password:
==== AUTHENTICATION COMPLETE ===
pi@octoprint:~ $ systemctl status camera-streamer
● camera-streamer.service - camera-streamer
     Loaded: loaded (/etc/systemd/system/camera-streamer.service; enabled; vendor preset: enabled)
    Drop-In: /etc/systemd/system/camera-streamer.service.d
             └─override.conf
     Active: active (exited) since Tue 2024-03-12 12:33:45 CET; 12s ago
    Process: 9184 ExecStart=/usr/bin/camera-streamer-control start (code=exited, status=0/SUCCESS)
   Main PID: 9184 (code=exited, status=0/SUCCESS)
        CPU: 72ms

Mar 12 12:33:44 octoprint camera-streamer-control[9184]: Running start for camera-streamer-libcamera.service...
Mar 12 12:33:39 octoprint systemd[1]: camera-streamer.service: Succeeded.
Mar 12 12:33:39 octoprint systemd[1]: Stopped camera-streamer.
Mar 12 12:33:40 octoprint systemd[1]: Starting camera-streamer...
Mar 12 12:33:44 octoprint camera-streamer-control[9184]: ... done.
Mar 12 12:33:44 octoprint camera-streamer-control[9184]: Running start for camera-streamer-usb-default.path...
Mar 12 12:33:44 octoprint camera-streamer-control[9184]: ... done.
Mar 12 12:33:44 octoprint camera-streamer-control[9184]: Running start for camera-streamer-usb-endoscope.path...
Mar 12 12:33:45 octoprint camera-streamer-control[9184]: ... done.
Mar 12 12:33:45 octoprint systemd[1]: Finished camera-streamer.
pi@octoprint:~ $

The http://octoprint.local:8081/stream still return 500 error.

Empirically I can say that the default camera works better. Sometimes it had crashed too but currently only the endoscope, on port 8081, isn't working correctly and returns 500. As I making this tests the endoscope does not work at all (even across reboot and restarts), I've tried everything I listed before, including physically reconnect the USB. No frames are received.

Do you suggest to switch to mainline latest release with the "old" camera stack or are there other fixes that I can try?

If it's that unstable for you, yeah, moving back to the old stack is probably better to get you up and running. Though long term it's of course more helpful if we can figure out what's going wrong here, though it is definitely out of my hands as it is an issue within camera-streamer or the camera driver or even camera itself from the sound of it.

I'm testing the endoscope on another PC with Ubuntu 23.10 (Linux 6.5.0-21-generic) right now and it works flawlessly.

As you already said, it probably is more of a camera-streamer related problem... but I am curious to know why this won't work and, if there is anything I can do to help in the debug process, feel free to ask.
I'll provide requested information and, in case no solution is found, switch to old setup

I suggest you open a ticket on the camera-streamer repository, maybe the author has some ideas on what to try to further debut this: Issues · ayufan/camera-streamer · GitHub

As suggested, I have opened a new issue on the camera-streamer repo