Camera-streamer + pi cam 2 + full frame capture

msroest · August 8, 2023, 9:35pm

Camera model

Pi Camera 2

What is the problem?

I'm trying to move from the old camera mjpg streamer to the new camera-streamer framework. And trying to get the Cam 2 to get a un-cropped image out of the pi cam2 sensor. When setting the camera-stream sensor parameters to get a full frame 3280x2464 or 1640x1232 I get a image that is half filled with green pixels

The same setup on the same pi with my older SD card using the old framework works fine

If I set the height/width back to 1920x1080 the green pixels are gone but I get the annoying cropped FOV.

What did you already try to solve it?

I've tried multiple different setting to get full frame.
Switched back to the old SD card to validate that it's not hardware
Searched to find other reports of similar issues

Have you tried running in safe mode?

No

Did running in safe mode solve the problem?

N/A

Systeminfo Bundle

You can download this in OctoPrint's System Information dialog ... no bundle, no support!)

octoprint-systeminfo-20230808153340.zip (121.9 KB)

Additional information about your setup

OctoPrint version, OctoPi version, printer, firmware, browser, operating system, ... as much data as possible

Raspberry Pi 4b 8GB
picam 2
Old SD:

------------------------------------------------------------------------------
OctoPrint version : 1.9.2
OctoPi version    : 0.18.0
------------------------------------------------------------------------------

New SD:

------------------------------------------------------------------------------
OctoPrint version : 1.9.2
OctoPi version    : 1.0.0
------------------------------------------------------------------------------

PrintedWeezl · August 8, 2023, 10:08pm

Could you try to open the stream with vlc? Just to rule out the browser

msroest · August 8, 2023, 10:34pm

So some more interesting data. It looks like I'm only getting 1 partial frame and then something is crashing/failing as I need to restart the camera-streamer-libcamera service each time after loading the stream.

The web shows the one frame with the green rest of the scene. If I do it with VLC it momentarily opens a video window then outputs

Your input can't be opened
VLC is unable to open the MRL 'http://octopi.internal.roe.st/webcam/?action=stream'. Check the log for details.

into the errors and warnings window.
I turned on debug logging and see this in the VLC log

mjpeg debug: MIME boundary not found in 40960 bytes of data
main debug: Buffering 0%
mjpeg warning: no more data
mjpeg error: no EOL
main debug: EOF reached
main debug: looking for text renderer module matching "any": 3 candidates
main debug: Stream buffering done (0 ms in 0 ms)
main debug: using text renderer module "freetype"
main debug: looking for video converter module matching "any": 16 candidates
swscale debug: 32x32 (32x32) chroma: YUVA -> 16x16 (16x16) chroma: RGBA with scaling using Bicubic (good quality)
main debug: using video converter module "swscale"
main debug: looking for video converter module matching "any": 16 candidates
yuvp debug: YUVP to YUVA converter
main debug: using video converter module "yuvp"
main debug: Deinterlacing available
main debug: deinterlace -1, mode auto, is_needed 0
main debug: looking for vout window module matching "any": 1 candidates
macosx debug: Opening video window
macosx debug: Releasing old IOKit other assertion (39295)
macosx debug: Activated assertion NoDisplaySleepAssertion through IOKit (39297)
macosx debug: toggle playlist from state: removed splitview 0, minimized view 0. Event 2
macosx debug: toggle playlist to state: removed splitview 0, minimized view 0
macosx debug: returning videoview with proposed position x=0, y=0, width=992, height=736

the camera-streamer-libcamera log on the machine has

Aug 08 16:30:43 octopi sh[1572]: util/http/http.c: ?: HTTP listening on 127.0.0.1:8080.
Aug 08 16:30:43 octopi sh[1572]: /usr/bin/camera-streamer Version: 0.2.5 (41d8dfd)
Aug 08 16:30:43 octopi sh[1572]: [0:09:29.054433885] [1572]  INFO Camera camera_manager.cpp:297 libcamera v0.0.5+82-2783c8d8
Aug 08 16:30:43 octopi sh[1572]: [0:09:29.086842640] [1591]  WARN RPI vc4.cpp:383 Mismatch between Unicam and CamHelper for embedded data usage!
Aug 08 16:30:43 octopi sh[1572]: [0:09:29.087674592] [1591]  INFO RPI vc4.cpp:437 Registered camera /base/soc/i2c0mux/i2c@1/imx219@10 to Unicam device /dev/media4 and ISP device /dev/med>
Aug 08 16:30:43 octopi sh[1572]: device/libcamera/device.cc: CAMERA: Device path=/base/soc/i2c0mux/i2c@1/imx219@10 opened
Aug 08 16:30:43 octopi sh[1572]: [0:09:29.088380046] [1572]  INFO Camera camera.cpp:1033 configuring streams: (0) 1437x1080-YUYV
Aug 08 16:30:43 octopi sh[1572]: [0:09:29.088870798] [1591]  INFO RPI vc4.cpp:565 Sensor: /base/soc/i2c0mux/i2c@1/imx219@10 - Selected sensor format: 1640x1232-SBGGR10_1X10 - Selected un>
Aug 08 16:30:43 octopi sh[1572]: [0:09:29.090551387] [1572]  INFO Camera camera.cpp:1033 configuring streams: (0) 1437x1080-YUYV (1) 3280x2464-SBGGR10_CSI2P
Aug 08 16:30:43 octopi sh[1572]: [0:09:29.090987048] [1591]  INFO RPI vc4.cpp:565 Sensor: /base/soc/i2c0mux/i2c@1/imx219@10 - Selected sensor format: 3280x2464-SBGGR10_1X10 - Selected un>
Aug 08 16:30:43 octopi sh[1572]: device/buffer_list.c: CAMERA:capture: Using: 1437x1080/YUYV, buffers=2, bytesperline=2880, sizeimage=0.0MiB
Aug 08 16:30:43 octopi sh[1572]: device/buffer_list.c: CAMERA:capture: Opened 2 buffers. Memory used: 5.9 MiB
Aug 08 16:30:43 octopi sh[1572]: device/buffer_list.c: CAMERA:capture:1: Using: 3280x2464/BG10, buffers=2, bytesperline=4128, sizeimage=0.0MiB
Aug 08 16:30:43 octopi sh[1572]: device/buffer_list.c: CAMERA:capture:1: Opened 2 buffers. Memory used: 19.4 MiB
Aug 08 16:30:43 octopi sh[1572]: device/v4l2/device.c: SNAPSHOT: Device path=/dev/video31 fd=37 opened
Aug 08 16:30:43 octopi sh[1572]: device/v4l2/buffer_list.c: SNAPSHOT:output:mplane: Requested resolution=1437x1080 is unavailable. Got 1437x1088.
Aug 08 16:30:43 octopi sh[1572]: device/buffer_list.c: SNAPSHOT:output:mplane: Using: 1408x1056/YUYV, buffers=2, bytesperline=2880, sizeimage=2.9MiB
Aug 08 16:30:43 octopi sh[1572]: device/buffer_list.c: SNAPSHOT:output:mplane: Opened 2 buffers. Memory used: 0.0 MiB
Aug 08 16:30:43 octopi sh[1572]: device/buffer_list.c: SNAPSHOT:capture:mplane: Using: 1408x1056/JPEG, buffers=2, bytesperline=0, sizeimage=4.0MiB
Aug 08 16:30:43 octopi sh[1572]: device/buffer_list.c: SNAPSHOT:capture:mplane: Opened 2 buffers. Memory used: 8.0 MiB
Aug 08 16:30:43 octopi sh[1572]: device/v4l2/device.c: RESCALLER:STREAM: Device path=/dev/video12 fd=40 opened
Aug 08 16:30:43 octopi sh[1572]: device/buffer_list.c: RESCALLER:STREAM:output:mplane: Using: 1437x1080/YUYV, buffers=2, bytesperline=2880, sizeimage=3.0MiB
Aug 08 16:30:43 octopi sh[1572]: device/buffer_list.c: RESCALLER:STREAM:output:mplane: Opened 2 buffers. Memory used: 0.0 MiB
Aug 08 16:30:43 octopi sh[1572]: device/buffer_list.c: RESCALLER:STREAM:capture:mplane: Using: 992x736/YUYV, buffers=2, bytesperline=1984, sizeimage=1.4MiB
Aug 08 16:30:43 octopi sh[1572]: device/buffer_list.c: RESCALLER:STREAM:capture:mplane: Opened 2 buffers. Memory used: 2.8 MiB
Aug 08 16:30:43 octopi sh[1572]: device/v4l2/device.c: STREAM: Device path=/dev/video31 fd=43 opened
Aug 08 16:30:43 octopi sh[1572]: device/buffer_list.c: STREAM:output:mplane: Using: 992x736/YUYV, buffers=2, bytesperline=1984, sizeimage=1.4MiB
Aug 08 16:30:43 octopi sh[1572]: device/buffer_list.c: STREAM:output:mplane: Opened 2 buffers. Memory used: 0.0 MiB
Aug 08 16:30:43 octopi sh[1572]: device/buffer_list.c: STREAM:capture:mplane: Using: 992x736/JPEG, buffers=2, bytesperline=0, sizeimage=4.0MiB
Aug 08 16:30:43 octopi sh[1572]: device/buffer_list.c: STREAM:capture:mplane: Opened 2 buffers. Memory used: 8.0 MiB
Aug 08 16:30:43 octopi sh[1572]: device/v4l2/device.c: VIDEO: Device path=/dev/video11 fd=46 opened
Aug 08 16:30:43 octopi sh[1572]: device/buffer_list.c: VIDEO:output:mplane: Using: 992x736/YUYV, buffers=2, bytesperline=1984, sizeimage=1.4MiB
Aug 08 16:30:43 octopi sh[1572]: device/buffer_list.c: VIDEO:output:mplane: Opened 2 buffers. Memory used: 0.0 MiB
Aug 08 16:30:43 octopi sh[1572]: device/buffer_list.c: VIDEO:capture:mplane: Using: 992x736/H264, buffers=2, bytesperline=0, sizeimage=0.5MiB
Aug 08 16:30:43 octopi sh[1572]: device/buffer_list.c: VIDEO:capture:mplane: Opened 2 buffers. Memory used: 1.0 MiB
Aug 08 16:30:43 octopi sh[1572]: device/device.c: CAMERA: Setting frame interval_us=0 for FPS=15
Aug 08 16:30:43 octopi sh[1572]: device/v4l2/device_options.c: SNAPSHOT: Configuring option 'Compression Quality' (009d0903) = 80
Aug 08 16:30:43 octopi sh[1572]: device/v4l2/device_options.c: STREAM: Configuring option 'Compression Quality' (009d0903) = 80
Aug 08 16:30:43 octopi sh[1572]: device/v4l2/device_options.c: VIDEO: Configuring option 'Repeat Sequence Header' (009909e2) = 1
Aug 08 16:30:43 octopi sh[1572]: device/v4l2/device_options.c: VIDEO: Configuring option 'Video Bitrate Mode' (009909ce) = 0
Aug 08 16:30:43 octopi sh[1572]: device/v4l2/device_options.c: VIDEO: Configuring option 'Video Bitrate' (009909cf) = 2000000
Aug 08 16:30:43 octopi sh[1572]: device/v4l2/device_options.c: VIDEO: Configuring option 'Repeat Sequence Header' (009909e2) = 5000000
Aug 08 16:30:43 octopi sh[1572]: device/v4l2/device_options.c: VIDEO: Configuring option 'H264 I-Frame Period' (00990a66) = 30
Aug 08 16:30:43 octopi sh[1572]: device/v4l2/device_options.c: VIDEO: Configuring option 'H264 Level' (00990a67) = 11
Aug 08 16:30:43 octopi sh[1572]: device/v4l2/device_options.c: VIDEO: Configuring option 'H264 Profile' (00990a6b) = 4
Aug 08 16:30:43 octopi sh[1572]: device/v4l2/device_options.c: VIDEO: Configuring option 'H264 Minimum QP Value' (00990a61) = 16
Aug 08 16:30:43 octopi sh[1572]: device/v4l2/device_options.c: VIDEO: Configuring option 'H264 Maximum QP Value' (00990a62) = 32
Aug 08 16:30:43 octopi sh[1572]: device/links.c: ?: Link 0: CAMERA:capture[1437x1080/YUYV/2] => [SNAPSHOT:output:mplane[1408x1056/YUYV/2], RESCALLER:STREAM:output:mplane[1437x1080/YUYV/2>
Aug 08 16:30:43 octopi sh[1572]: device/links.c: ?: Link 1: SNAPSHOT:capture:mplane[1408x1056/JPEG/2] => [SNAPSHOT-CAPTURE]
Aug 08 16:30:43 octopi sh[1572]: device/links.c: ?: Link 2: RESCALLER:STREAM:capture:mplane[992x736/YUYV/2] => [STREAM:output:mplane[992x736/YUYV/2], VIDEO:output:mplane[992x736/YUYV/2]]
Aug 08 16:30:43 octopi sh[1572]: device/links.c: ?: Link 3: STREAM:capture:mplane[992x736/JPEG/2] => [STREAM-CAPTURE]
Aug 08 16:30:43 octopi sh[1572]: device/links.c: ?: Link 4: VIDEO:capture:mplane[992x736/H264/2] => [VIDEO-CAPTURE]
Aug 08 16:30:43 octopi sh[1572]: device/buffer_list.c: CAMERA:capture: Streaming started... Was 0 of 2 enqueud
Aug 08 16:30:43 octopi sh[1572]: device/buffer_list.c: SNAPSHOT:output:mplane: Streaming started... Was 0 of 2 enqueud
Aug 08 16:30:43 octopi sh[1572]: device/buffer_list.c: RESCALLER:STREAM:output:mplane: Streaming started... Was 0 of 2 enqueud
Aug 08 16:30:43 octopi sh[1572]: device/buffer_list.c: SNAPSHOT:capture:mplane: Streaming started... Was 0 of 2 enqueud
Aug 08 16:30:43 octopi sh[1572]: device/buffer_list.c: RESCALLER:STREAM:capture:mplane: Streaming started... Was 0 of 2 enqueud
Aug 08 16:30:43 octopi sh[1572]: device/buffer_list.c: STREAM:output:mplane: Streaming started... Was 0 of 2 enqueud
Aug 08 16:30:43 octopi sh[1572]: device/buffer_list.c: VIDEO:output:mplane: Streaming started... Was 0 of 2 enqueud
Aug 08 16:30:43 octopi sh[1572]: device/buffer_list.c: STREAM:capture:mplane: Streaming started... Was 0 of 2 enqueud
Aug 08 16:30:43 octopi sh[1572]: device/buffer_list.c: VIDEO:capture:mplane: Streaming started... Was 0 of 2 enqueud
Aug 08 16:30:47 octopi sh[1572]: util/http/http.c: HTTP8080/0: Client connected 127.0.0.1 (fd=4).
Aug 08 16:30:47 octopi sh[1572]: util/http/http.c: HTTP8080/0: Request 'GET' '/' 'action=stream'
Aug 08 16:30:48 octopi sh[1572]: device/links.c: RESCALLER:STREAM:capture:mplane:buf0: Capture image is outdated. Skipped. Now: 574633528, vs 573586792.
Aug 08 16:30:49 octopi sh[1572]: device/links.c: RESCALLER:STREAM:capture:mplane:buf0: Capture image is outdated. Skipped. Now: 575233536, vs 574186673.
Aug 08 16:30:49 octopi sh[1572]: device/links.c: RESCALLER:STREAM:capture:mplane:buf0: Capture image is outdated. Skipped. Now: 575763525, vs 574719902.
Aug 08 16:30:50 octopi sh[1572]: util/http/http.c: HTTP8080/0: Client disconnected 127.0.0.1.
Aug 08 16:30:50 octopi sh[1572]: device/links.c: RESCALLER:STREAM:capture:mplane:buf0: Capture image is outdated. Skipped. Now: 576363512, vs 575319784.
Aug 08 16:32:10 octopi sh[1572]: util/http/http.c: HTTP8080/1: Client connected 127.0.0.1 (fd=5).
Aug 08 16:32:10 octopi sh[1572]: util/http/http.c: HTTP8080/1: Request 'GET' '/' 'action=stream'
Aug 08 16:32:11 octopi sh[1572]: device/links.c: RESCALLER:STREAM:capture:mplane:buf0: Capture image is outdated. Skipped. Now: 657482963, vs 656437161.
Aug 08 16:32:12 octopi sh[1572]: device/links.c: RESCALLER:STREAM:capture:mplane:buf0: Capture image is outdated. Skipped. Now: 657982994, vs 656503809.
Aug 08 16:32:12 octopi sh[1572]: util/http/http.c: HTTP8080/1: Client disconnected 127.0.0.1.
Aug 08 16:32:12 octopi sh[1572]: device/links.c: RESCALLER:STREAM:capture:mplane:buf0: Capture image is outdated. Skipped. Now: 658613048, vs 657570273

PrintedWeezl · August 8, 2023, 11:36pm

shot in the blue: increase the vram

since you got plenty of ram just try 512 MB

msroest · August 9, 2023, 12:57pm

No dice with 512 gpu_mem.
I also did a full update to make sure kernel etc were all up to date same behaviour.

Double checked my older SD it has 128 set for gpu mem.

msroest · August 14, 2023, 3:05pm

Well here's a fun little update. I did a apt update && apt full upgrade on my old SD and now the old camera streamer is also not working anymore. However I can still pull images from the camera using raspistill.

Also another piece of interesting data with the new SD card even after the streamer seems to crash with the partial image it's still holding the camera device open if I try to use the libcamera-* apps to access the camera. If I stop the streamer service I can then use the libcamera-* apps to access the sensor.

Seems like it must be something with camera-streamer so I've opened Full Frame capture with PiCam2 · Issue #95 · ayufan/camera-streamer · GitHub

msroest · August 17, 2023, 1:04pm

Just updated to the most recent 0.2.6 release (from 0.2.5) of camera-streamer and still the same behaviour.

msroest · August 18, 2023, 2:34pm

Someone on the github issue posted the solution

github.com/ayufan/camera-streamer

Full Frame capture with PiCam2

opened 03:05PM - 14 Aug 23 UTC

msroest

I'm attempting to move my octoprint to the new version using camera-streamer and… I've come across a very strange issue. When trying to get full frame capture from my PiCam2. I've setup the my libcamera.conf as follows based on this issue https://github.com/ayufan/camera-streamer/issues/3 ``` ### Options for libcamera based cameras (PiCam, Arducam, ...) # The port on which the webcam server for the camera should listen on. If you have only # one camera, leave at 8080. If you have more, change to 8081, 8082, etc. The primary # camera will be considered the one with 8080. PORT=8080 # The resolution to request on the camera sensor. Defaults to 1280x720. WIDTH=3280 HEIGHT=2464 # The height to use for the video stream. Defaults to 720. VIDEO_HEIGHT=720 # The height to use for the snapshots. Defaults to 1080. SNAPSHOT_HEIGHT=1080 # The framerate to set on the camera. Defaults to 15fps. FRAMERATE=15 # Additional options. By default enables continuous auto focus (if possible). #OPTIONS='--camera-options="AfMode=2" --camera-options="AfRange=2"' OPTIONS='-camera-nbufs=2 --camera-high_res_factor=2 --camera-low_res_factor=4' ``` In this mode the camera-streamer returns a half a frame with the rest being green <img width="589" alt="Screenshot 2023-08-08 at 15 07 10" src="https://github.com/ayufan/camera-streamer/assets/91028/15a32c98-0138-46dd-b350-00ef7a730843"> If I update the config to use 1920x1080 capture the streamer works fine but the view is garbage cause of the crop. If I stop camera-streamer and use `libcamera-jpeg -o image.jpg --nopreview` it correctly pulls the whole frame: ![rotatednopreview](https://github.com/ayufan/camera-streamer/assets/91028/db5d179e-c1fe-4018-ad08-e671e62a5a3d)

SNAPSHOT_HEIGHT=2464

Has to be set.