New camera stack - one of 2 cameras not active

Camera model

Logitech HD920
Logitech B525HD
Raspi Cam V2 (Working)

What is the problem?

I've setup 3 cameras using the new camera stack. Raspi Cam and 2 Logitech via USB 3.0.
Both are working - but not at the same time. What do I mean? After a restart of the system or reloading the camera stack (systemctl restart) one of them is working and the other only shows "server error" on the web. The camera-streamer endpoint is available.

What did you already try to solve it?

Removed the configs, added cameras again. Rebooted, tried different usb ports.

Have you tried running in safe mode?

Yes, without any effect.

Did running in safe mode solve the problem?

No.

Systeminfo Bundle

systeminfo.zip (139.1 KB)

Additional information

When accessing the web on url: http://192.168.1.129:8082 the endpoint from camera streamer can be open. When then accessing the /steam I only get "Server Error".

The installation is new, running on the current Octoprint (1.9.3) and OctoPi (1.0). The Hardware is a Raspberry Pi 4 with 2GB.

Conf Logitech B525HD
florian@octoprint:~ $ cat /boot/camera-streamer/usb-B525.conf
### Options for USB based cameras

# The port on which the webcam server for the camera should listen on.
PORT=8082

# The path to the camera device.
DEVICE=/dev/v4l/by-id/usb-046d_B525_HD_Webcam_61432820-video-index0

# The image format of the camera.
FORMAT=YUYV

# The resolution to set on the camera. Defaults to 1280x720.
WIDTH=1280
HEIGHT=720

# The framerate to set on the camera. Defaults to 15fps.
FRAMERATE=15

# Additional options.
OPTIONS="--http-listen=0.0.0.0"
Conf Logitech HD920
florian@octoprint:~ $ cat /boot/camera-streamer/usb-HD920.conf
### Options for USB based cameras

# The port on which the webcam server for the camera should listen on.
PORT=8081

# The path to the camera device.
DEVICE=/dev/v4l/by-id/usb-046d_HD_Pro_Webcam_C920_8D15191F-video-index0

# The image format of the camera.
FORMAT=YUYV

# The resolution to set on the camera. Defaults to 1280x720.
WIDTH=1920
HEIGHT=1080

# The framerate to set on the camera. Defaults to 15fps.
FRAMERATE=30

# Additional options.
OPTIONS="--http-listen=0.0.0.0"
general camera stuff
florian@octoprint:~ $ lsusb
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 005: ID 2c99:0002 Prusa Original Prusa i3 MK3
Bus 001 Device 004: ID 046d:0836 Logitech, Inc. B525 HD Webcam
Bus 001 Device 003: ID 046d:0892 Logitech, Inc. OrbiCam
Bus 001 Device 002: ID 2109:3431 VIA Labs, Inc. Hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

florian@octoprint:~ $ v4l2-ctl --list-devices
bcm2835-codec-decode (platform:bcm2835-codec):
        /dev/video10
        /dev/video11
        /dev/video12
        /dev/video18
        /dev/video31
        /dev/media3

bcm2835-isp (platform:bcm2835-isp):
        /dev/video13
        /dev/video14
        /dev/video15
        /dev/video16
        /dev/video20
        /dev/video21
        /dev/video22
        /dev/video23
        /dev/media1
        /dev/media2

unicam (platform:fe801000.csi):
        /dev/video4
        /dev/video5
        /dev/media4

rpivid (platform:rpivid):
        /dev/video19
        /dev/media0

HD Pro Webcam C920 (usb-0000:01:00.0-1.1):
        /dev/video0
        /dev/video1
        /dev/media5

B525 HD Webcam (usb-0000:01:00.0-1.2):
        /dev/video2
        /dev/video3
        /dev/media6

Does http://192.168.1.129:8081 also work?

How about http://192.168.1.129:8081/webcam/?action=stream

and also http://192.168.1.129:8082/webcam/?action=stream

Does http://192.168.1.129:8081 also work?

-> Yes.

How about http://192.168.1.129:8081/webcam/?action=stream

-> yes, either this or the other.

and also http://192.168.1.129:8082/webcam/?action=stream

-> Server Error is the only thing I see.

Try this.

Type in

systemctl | grep camera

That will result in a list. Look for the entries which end in ".service".
Those are the service threads the main camera-streamer launcher service spawns off.

For instance if you named your second camera "hd920" you will have a service called "camera-streamer-usb@h920.service"

Once you know the name of the child service type this...

systemctl status camera-streamer-usb@h920.service

It should report back if the streaming service is actually running and the port it is using.

Thanks for the reply.

So, cameras are registered as services:

florian@octoprint:~ $ systemctl | grep camera
  camera-streamer-usb-B525.path                                                                                                              loaded active running   camera-streamer B525 autolaunch
  camera-streamer-usb-HD920.path                                                                                                             loaded active running   camera-streamer HD920 autolaunch
  camera-streamer-libcamera.service                                                                                                          loaded active running   camera-streamer libcamera
  camera-streamer-usb@B525.service                                                                                                           loaded active running   camera-streamer B525
  camera-streamer-usb@HD920.service                                                                                                          loaded active running   camera-streamer HD920
  camera-streamer.service                                                                                                                    loaded active exited    camera-streamer
  system-camera\x2dstreamer\x2dusb.slice                                                                                                     loaded active active    system-camera\x2dstreamer\x2dusb.slice

The status of the "problem" camera:

florian@octoprint:~ $ systemctl status camera-streamer-usb@B525.service
â—Ź camera-streamer-usb@B525.service - camera-streamer B525
     Loaded: loaded (/etc/systemd/system/camera-streamer-usb@.service; disabled; vendor preset: enabled)
    Drop-In: /etc/systemd/system/camera-streamer-usb@.service.d
             └─override.conf
     Active: active (running) since Wed 2024-02-14 11:00:41 CET; 22h ago
TriggeredBy: â—Ź camera-streamer-usb-B525.path
    Process: 2707 ExecCondition=/bin/sh -c [ -e $DEVICE ] (code=exited, status=0/SUCCESS)
    Process: 2713 ExecCondition=/bin/sh -c ! ss -ltn src :$PORT | grep -q LISTEN (code=exited, status=0/SUCCESS)
   Main PID: 2720 (camera-streamer)
      Tasks: 11 (limit: 3135)
        CPU: 2min 34.721s
     CGroup: /system.slice/system-camera\x2dstreamer\x2dusb.slice/camera-streamer-usb@B525.service
             └─2720 /usr/bin/camera-streamer --http-port=8082 --camera-type=v4l2 --camera-path=/dev/v4l/by-id/usb-046d_B525_HD_Webcam_61432820-video-index0 --camera-format=YUYV --camera-width=1280 --camera-height=720 --camera-fps=15 --camera-nbufs=2>

Feb 15 08:26:03 octoprint sh[2720]: util/http/http.c: HTTP8082/9: Client disconnected 192.168.1.128.
Feb 15 08:26:09 octoprint sh[2720]: util/http/http.c: HTTP8082/0: Client connected 192.168.1.128 (fd=4).
Feb 15 08:26:09 octoprint sh[2720]: util/http/http.c: HTTP8082/0: Request 'GET' '/webcam/' 'action=stream'
Feb 15 08:26:10 octoprint sh[2720]: util/http/http.c: HTTP8082/1: Client connected 192.168.1.128 (fd=5).
Feb 15 08:26:10 octoprint sh[2720]: util/http/http.c: HTTP8082/1: Request 'GET' '/webcam/' 'action=stream'
Feb 15 08:26:11 octoprint sh[2720]: util/http/http.c: HTTP8082/0: Client disconnected 192.168.1.128.
Feb 15 08:26:12 octoprint sh[2720]: util/http/http.c: HTTP8082/1: Client disconnected 192.168.1.128.
Feb 15 08:29:11 octoprint sh[2720]: util/http/http.c: HTTP8082/2: Client connected 192.168.1.128 (fd=6).
Feb 15 08:29:11 octoprint sh[2720]: util/http/http.c: HTTP8082/2: Request 'GET' '/webcam/' 'action=stream'
Feb 15 08:29:13 octoprint sh[2720]: util/http/http.c: HTTP8082/2: Client disconnected 192.168.1.128.

The same for the working one:

florian@octoprint:~ $ systemctl status camera-streamer-usb@HD920.service
â—Ź camera-streamer-usb@HD920.service - camera-streamer HD920
     Loaded: loaded (/etc/systemd/system/camera-streamer-usb@.service; disabled; vendor preset: enabled)
    Drop-In: /etc/systemd/system/camera-streamer-usb@.service.d
             └─override.conf
     Active: active (running) since Wed 2024-02-14 11:00:41 CET; 22h ago
TriggeredBy: â—Ź camera-streamer-usb-HD920.path
    Process: 2708 ExecCondition=/bin/sh -c [ -e $DEVICE ] (code=exited, status=0/SUCCESS)
    Process: 2714 ExecCondition=/bin/sh -c ! ss -ltn src :$PORT | grep -q LISTEN (code=exited, status=0/SUCCESS)
   Main PID: 2721 (camera-streamer)
      Tasks: 11 (limit: 3135)
        CPU: 2min 34.450s
     CGroup: /system.slice/system-camera\x2dstreamer\x2dusb.slice/camera-streamer-usb@HD920.service
             └─2721 /usr/bin/camera-streamer --http-port=8081 --camera-type=v4l2 --camera-path=/dev/v4l/by-id/usb-046d_HD_Pro_Webcam_C920_8D15191F-video-index0 --camera-format=YUYV --camera-width=1920 --camera-height=1080 --camera-fps=30 --camera-nb>

Feb 15 08:32:25 octoprint sh[2721]: device/buffer_queue.c: SNAPSHOT:output:mplane: The dma_buf (CAMERA:capture:buf1) is too long: 4147200 vs space=4055040
Feb 15 08:32:25 octoprint sh[2721]: device/buffer_queue.c: SNAPSHOT:output:mplane: The dma_buf (CAMERA:capture:buf0) is too long: 4147200 vs space=4055040
Feb 15 08:32:25 octoprint sh[2721]: device/buffer_queue.c: SNAPSHOT:output:mplane: The dma_buf (CAMERA:capture:buf1) is too long: 4147200 vs space=4055040
Feb 15 08:32:26 octoprint sh[2721]: device/buffer_queue.c: SNAPSHOT:output:mplane: The dma_buf (CAMERA:capture:buf0) is too long: 4147200 vs space=4055040
Feb 15 08:32:26 octoprint sh[2721]: device/buffer_queue.c: SNAPSHOT:output:mplane: The dma_buf (CAMERA:capture:buf1) is too long: 4147200 vs space=4055040
Feb 15 08:32:26 octoprint sh[2721]: device/buffer_queue.c: SNAPSHOT:output:mplane: The dma_buf (CAMERA:capture:buf0) is too long: 4147200 vs space=4055040
Feb 15 08:32:26 octoprint sh[2721]: device/buffer_queue.c: SNAPSHOT:output:mplane: The dma_buf (CAMERA:capture:buf1) is too long: 4147200 vs space=4055040
Feb 15 08:32:26 octoprint sh[2721]: device/buffer_queue.c: SNAPSHOT:output:mplane: The dma_buf (CAMERA:capture:buf0) is too long: 4147200 vs space=4055040
Feb 15 08:32:27 octoprint sh[2721]: device/buffer_queue.c: SNAPSHOT:output:mplane: The dma_buf (CAMERA:capture:buf1) is too long: 4147200 vs space=4055040
Feb 15 08:32:27 octoprint sh[2721]: util/http/http.c: HTTP8081/4: Client disconnected 192.168.1.128.

Update:

I just found out, that after trying to switch cameras in Octoprint and reloading the urls on the web, the other 920HD stopped working and the B825 is working know. Without any restart or systemctl command.

It looks like both of your USB cameras are using YUYV instread of jpeg streaming.
This puts a bigger load on the USB interface and the Pi in general.

You should modify the configuration files to enable jpeg streaming instead and increase the number of buffers while you are at it.

The streams could be timing out because of the buffer issue.

# The image format of the camera.
FORMAT=JPEG

#Number of buffers to use. Defaults to 2
NBUFS=10

Then reboot.

I had this problem until I made the above changes.

Thanks for the answer. I've made the changes and tried to restart the systemctl.
This does not solve the issue right away.

However, as I'm printing currently a 28h print, I cannot restart the pi :smiley:
I will come back tomorrow the mark the answer as solved or to keep you updated.

Many thanks!

You could restart each stream individually while the PI is printing.

systemctl restart camera-streamer-usb@HD920.service

systemctl restart camera-streamer-usb@B525.service

Then issue the status command again to make sure they both came up.

OK, I restarted each service, but still getting the Server error on the B525HD

florian@octoprint:~ $ sudo systemctl restart camera-streamer-usb@B525.service
florian@octoprint:~ $ systemctl status camera-streamer-usb@B525.service
â—Ź camera-streamer-usb@B525.service - camera-streamer B525
     Loaded: loaded (/etc/systemd/system/camera-streamer-usb@.service; disabled; vendor preset: enabled)
    Drop-In: /etc/systemd/system/camera-streamer-usb@.service.d
             └─override.conf
     Active: active (running) since Thu 2024-02-15 10:15:08 CET; 3s ago
TriggeredBy: â—Ź camera-streamer-usb-B525.path
    Process: 15980 ExecCondition=/bin/sh -c [ -e $DEVICE ] (code=exited, status=0/SUCCESS)
    Process: 15982 ExecCondition=/bin/sh -c ! ss -ltn src :$PORT | grep -q LISTEN (code=exited, status=0/SUCCESS)
   Main PID: 15985 (camera-streamer)
      Tasks: 11 (limit: 3135)
        CPU: 280ms
     CGroup: /system.slice/system-camera\x2dstreamer\x2dusb.slice/camera-streamer-usb@B525.service
             └─15985 /usr/bin/camera-streamer --http-port=8082 --camera-type=v4l2 --camera-path=/dev/v4l/by-id/usb-046d_B525_HD_Webcam_61432820-video-index0 --camera-format=JPEG --camera-width=1280 --camera-height=720 --camera-fps=24 --camera-nbufs=>

Feb 15 10:15:08 octoprint sh[15985]: device/v4l2/device_options.c: VIDEO: Configuring option 'H264 Minimum QP Value' (00990a61) = 16
Feb 15 10:15:08 octoprint sh[15985]: device/v4l2/device_options.c: VIDEO: Configuring option 'H264 Maximum QP Value' (00990a62) = 32
Feb 15 10:15:08 octoprint sh[15985]: device/links.c: ?: Link 0: CAMERA:capture[1280x720/MJPG/10] => [DECODER:output:mplane[1280x720/MJPG/10], SNAPSHOT-CAPTURE, STREAM-CAPTURE]
Feb 15 10:15:08 octoprint sh[15985]: device/links.c: ?: Link 1: DECODER:capture:mplane[1280x720/NV12/10] => [VIDEO:output:mplane[1280x720/NV12/10]]
Feb 15 10:15:08 octoprint sh[15985]: device/links.c: ?: Link 2: VIDEO:capture:mplane[1280x720/H264/10] => [VIDEO-CAPTURE]
Feb 15 10:15:08 octoprint sh[15985]: device/buffer_list.c: CAMERA:capture: Streaming started... Was 0 of 10 enqueud
Feb 15 10:15:08 octoprint sh[15985]: device/buffer_list.c: DECODER:output:mplane: Streaming started... Was 0 of 10 enqueud
Feb 15 10:15:08 octoprint sh[15985]: device/buffer_list.c: DECODER:capture:mplane: Streaming started... Was 0 of 10 enqueud
Feb 15 10:15:08 octoprint sh[15985]: device/buffer_list.c: VIDEO:output:mplane: Streaming started... Was 0 of 10 enqueud
Feb 15 10:15:08 octoprint sh[15985]: device/buffer_list.c: VIDEO:capture:mplane: Streaming started... Was 0 of 10 enqueud

Strange, the service is running according to what you posted.

Haproxy configuration changes are not needed if you made the correct --http-listen=0.0.0.0 entry in the camera's configuration.

I wonder if something is stepping on the port (if that is possible?).

Try

sudo apt install lsof

Then

lsof -i:8082

You should only get one line back reading...

camera-st 920 63846 3u IPv4 17435 0t0 TCP *:8082 (LISTEN)

Also cold boot (power off) after your print job is done and check again.

Actually, I get nothing back

florian@octoprint:~ $ lsof -i:8082
florian@octoprint:~ $

I also got a server error, after changing the port to 8083 on the B525HD.

The "server error" is an issue inside camera-streamer. It's definitely running, and you're able to access it fine. It's returning this response as an internal server error.

We're not completely sure but it seems camera-streamer can't recover properly if it has an issue with the stream going stable - there's an open bug report somewhere on the repo. It's the reason the new stack is not the default because this issue exists.

Sorry I forgot the sudo in front of lsof.

Try lsof -i:8082

Regarding what Charlie_Powell says. I recall fighting with this where the stream would seem to disappear and ended up reducing the resolution of the streams until this stopped, then working backwards.

You may want to set the stream resolution very low, say 640x480 at 10 FPS, make sure you can see the streams and then change the resolution back up in steps until it appears to be stable.

florian@octoprint:/boot/camera-streamer $ sudo lsof -i:8083
lsof: no pwd entry for UID 63846
COMMAND     PID     USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
lsof: no pwd entry for UID 63846
camera-st 16579    63846    3u  IPv4 544301      0t0  TCP *:8083 (LISTEN)

After setting the settings to 3fps and 480p I don't get a server error, but instead the "missing image" picture/icon.

Looks like the bug @Charlie_Powell mentioned seams the problem here.

I will wait for any updates and just use 2 cams for the moment.

Try FORMAT=MJPG for all cameras. The usb hub inside the pi could be flooded.
Perhaps also try different combinations of the usb 2 and 3 ports. Just an idea to try if it helps.

Nope. As soon as I switch the B525 to MJPG I'll get a server error. Running it on 480p in YUYV its fine.
I also tried all possible combinations of ports, without any effect.

Are there any news about this? Have been fighting this issue since yesterday and can't get it to work.

I'm also getting the 502 Bad Gateway message from my usaually working camera from time to time when I restart my raspberry