Low-Latency H264 Streaming Support w/ WebRTC

johnboiles · September 11, 2021, 10:46pm

The good thing is that it seems like like the h264_omx transcoded 640x480 stream works great out of the box and is lower bandwidth and comparable latency to the mjpegstreamer stream! Seems promising that using 640x480 transcoded h264 streams could be a good OctoPrint default in the future.

Maybe to keep this moving forward we just make that the default for the plugin and advanced users can try to get raw 1080p forwarded if they want.

Charlie_Powell · September 11, 2021, 11:05pm

The default streaming resolution in OctoPi is 640x480, 10fps so it wouldn't be unreasonable to have the same defaults there.

johnboiles · September 11, 2021, 11:06pm

Oh cool. Yeah 640x480 @ 30fps seems to be working just fine with WebRTC so there's some improvement right there then! The biggest bandwidth improvement in h264 comes from motion frames so might as well take advantage of them!

jneilliii · September 11, 2021, 11:28pm

I can confirm that v4l2-ctl commands do work for rotation. I used the camera settings plugin to set it.

johnboiles · September 13, 2021, 2:04am

FYI I just pushed a pretty big performance improvement to my aiortc branch when using h264 passthrough. See https://github.com/aiortc/aiortc/pull/564 for more info.

johnboiles · September 13, 2021, 6:32am

@jneilliii I think I have a solution for the Pi camera with h264 passthrough. I needed to enable inline PPS/SPS headers in v4l2 via the barely documented repeat_sequence_header control. Make sure to pull my latest octoprint-webrtc-support branch in aiortc to get the aforementioned performance improvement first. Then try this:

v4l2-ctl -d /dev/video0 -v pixelformat=H264,width=1920,height=1080
v4l2-ctl -d /dev/video0 -c video_bitrate_mode=1,video_bitrate=4000000,h264_profile=4,h264_i_frame_period=60,h264_level=11,repeat_sequence_header=1
v4l2-ctl -d /dev/video0 -p 30

/home/pi/aiortc/venv/bin/python /home/pi/aiortc/examples/webcam/webcam.py --no-transcode --preferred-codec=video/H264 --video-options='{"video_size": "1920x1080", "framerate": "30", "input_format": "h264", "pixelformat": "H264"}'

I'm getting 720p streaming with the pi camera and it's only using ~20% CPU and 85mb ram. At 1080p it uses about the same CPU and 123mb ram. So maybe there's hope for it working on the Pi Zero (though I did also set gpu_mem=265 in /boot/config.txt; not sure if that's needed for the Pi Zero or not)

You can also try scaling the resolution down to 1280x720 @ 2000000bps which should lower RAM requirements a bit.

johnboiles · September 13, 2021, 5:57pm

I did some slightly more scientific tests and I'm seeing ~450ms latency with my Logitech c920 and a mere < 200ms on the Pi camera both at 1080p!! As a baseline, mjpg-streamer with a Pi camera is ~300ms. So it looks like WebRTC has mjpeg-streamer beat on latency, bitrate/bandwidth, resolution and framerate . (Just maybe not in simplicity because video codecs are hard )

Logitech + WebRTC

Pi Camera + WebRTC

Pi Camera + Mjpegstreamer

Charlie_Powell · September 13, 2021, 8:15pm

That's a smart way of measuring the latency, I'll have to keep that in mind! Great results.

b-morgan · September 13, 2021, 8:48pm

@Charlie_Powell may have figured it out but I haven't... How are you measuring the latency?

johnboiles · September 13, 2021, 9:01pm

Setup: Open both the webcam stream and a stopwatch on my computer. Point the streaming camera at my computer screen. Take a picture or screenshot of the computer. The difference in timestamps between the stopwatch on the screen and what's shown in the stream is the end-to-end latency.

I hooked everything up to ethernet so I wasn't also measuring my wifi performance

jneilliii · September 16, 2021, 7:52pm

We've opened a specific channel #dev-octopi in the Discord server at the request of @Quinninator to allow better real-time collaboration on your struggles with prepping the octopi side changes.

Damian_Britton · November 27, 2021, 2:56pm

Got everything running through Janus and accessible via direct ip access in a web browser but when I try to point Octoprint webcam stream url to it I get "Bad status line 'Invalid method encountered". Any ideas on whats going on here? Set up is Pi 3B running 0.18.1 - 1.7.2 with PiCam H264 transcoded

ERROR:aiohttp.server:Error handling request
Traceback (most recent call last):
  File "/home/pi/aiortc/venv/lib/python3.7/site-packages/aiohttp/web_protocol.py", line 334, in data_received
    messages, upgraded, tail = self._request_parser.feed_data(data)
  File "aiohttp/_http_parser.pyx", line 551, in aiohttp._http_parser.HttpParser.feed_data
aiohttp.http_exceptions.BadStatusLine: 400, message="Bad status line 'Invalid method encountered'"
INFO:aiohttp.access:192.168.1.63 [27/Nov/2021:14:54:20 +0000] "UNKNOWN / HTTP/1.0" 400 204 "-" "-"

acca_rien · December 13, 2021, 9:59am

Hello, i have c920 webcam, i'd like to use h.264
i use octopi with raspberry pi 3b

i have octoprint 1.7.2, does it support WebRTC ?

If i select maintenance RC branch, there is no upper version proposed. Are maintenance and maintenance RC the same ?

Thanks

Charlie_Powell · December 13, 2021, 2:47pm

There's currently no maintenance release candidate available newer than 1.7.2, only manually installing from GitHub on the maintenance branch has support for webRTC.

acca_rien · December 13, 2021, 7:37pm

Thanks, i added it manually
I have performed those steps :

To update to a version of OctoPrint with WebRTC support, checkout the maintenance branch (PR4225 has been merged into maintenance and will be released with 1.8.0 ).

To set up a WebRTC server on your Pi, see this post (see also this post if you have a Pi camera)

i can see my camera image but image is strange :

What did i do wrong please ?

In addition, can someone detail the procedure to make the server start automatically please ?

Thanks

Greg_Shaw · January 6, 2022, 8:43pm

@johnboiles I was wondering will this new approach work for the new Logitech C920 cameras that no longer have the hardware h.264 built in?

ytwytw · February 17, 2022, 8:20pm

I have multiple cameras and it seems its only able to open /dev/video0

Is there any way to change it?

Grant_Williamson · December 1, 2022, 5:27am

Hi john, Novice user here,

Following along with you line for line,

when i hit

"Use STUN server"

then followed by

START i get a error,

"

======== Running on http://0.0.0.0:8080 ========
(Press CTRL+C to quit)
INFO:aiohttp.access:192.168.1.157 [01/Dec/2022:05:23:36 +0000] "GET / HTTP/1.1" 200 979 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.0.0 Safari/537.36"
INFO:aiohttp.access:192.168.1.157 [01/Dec/2022:05:23:36 +0000] "GET /client.js HTTP/1.1" 200 2426 "http://octopi.local:8080/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.0.0 Safari/537.36"
[video4linux2,v4l2 @ 0x2494030] ioctl(VIDIOC_STREAMON): Invalid argument
ERROR:aiohttp.server:Error handling request
Traceback (most recent call last):
File "/home/pi/aiortc/venv/lib/python3.7/site-packages/aiohttp/web_protocol.py", line 433, in _handle_request
resp = await request_handler(request)
File "/home/pi/aiortc/venv/lib/python3.7/site-packages/aiohttp/web_app.py", line 504, in _handle
resp = await handler(request)
File "/home/pi/aiortc/examples/webcam/webcam.py", line 93, in offer
audio, video = create_local_tracks(args.play_from, transcode=args.transcode, options=args.video_options)
File "/home/pi/aiortc/examples/webcam/webcam.py", line 46, in create_local_tracks
webcam = MediaPlayer("/dev/video0", format="v4l2", transcode=transcode, options=options)
File "/home/pi/aiortc/venv/lib/python3.7/site-packages/aiortc/contrib/media.py", line 309, in init
self.__container = av.open(file=file, format=format, mode="r", options=options)
File "av/container/core.pyx", line 354, in av.container.core.open
File "av/container/core.pyx", line 225, in av.container.core.Container.cinit
File "av/container/core.pyx", line 257, in av.container.core.Container.err_check
File "av/error.pyx", line 336, in av.error.err_check
av.error.ValueError: [Errno 22] Invalid argument: '/dev/video0'
INFO:aiohttp.access:192.168.1.157 [01/Dec/2022:05:23:37 +0000] "POST /offer HTTP/1.1" 500 244 "http://octopi.local:8080/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.0.0 Safari/537.36"
INFO:aiohttp.access:127.0.0.1 [01/Dec/2022:05:23:41 +0000] "GET /?action=stream HTTP/1.1" 200 998 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.0.0 Safari/537.36"

"

Any ideas? cheers!

fischer-felix · December 2, 2022, 9:53pm

If you are using a pi camera, you could try setting it to legacy mode:
sudo raspi-config > 3 Interface Options > I1 Legacy Camera

PDB_maker · February 13, 2023, 3:57pm

I also get an error once I click "Start"

The web page (in Firefox) shows:
SyntaxError: JSON.parse: unexpected non-whitespace character after JSON data at line 1 column 5 of the JSON data

Also, I get errors in the terminal on my Pi:

$ sudo systemctl stop webcamd
$ ./venv/bin/python examples/webcam/webcam.py --no-transcode --preferred-codec=video/H264 --video-options='{"video_size": "1920x1080", "framerate": "30", "input_format": "h264"}'
======== Running on http://0.0.0.0:8080 ========
(Press CTRL+C to quit)

Once I connect to the webpage at 192.168.1.6:8080,

INFO:aiohttp.access:192.168.1.130 [13/Feb/2023:15:40:41 +0000] "GET / HTTP/1.1" 200 979 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/109.0"
INFO:aiohttp.access:192.168.1.130 [13/Feb/2023:15:40:41 +0000] "GET /client.js HTTP/1.1" 200 2426 "http://192.168.1.6:8080/" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/109.0"
INFO:aiohttp.access:192.168.1.130 [13/Feb/2023:15:40:41 +0000] "GET /favicon.ico HTTP/1.1" 404 172 "http://192.168.1.6:8080/" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/109.0"

After I check "Use STUN Server" and click "Start" I get the SyntaxError: JSON.parse: unexpected non-whitespace character after JSON data at line 1 column 5 of the JSON data and the following in the terminal on my Pi:

ERROR:aiohttp.server:Error handling request
Traceback (most recent call last):
  File "/home/bpeyser/aiortc/venv/lib/python3.9/site-packages/aiohttp/web_protocol.py", line 433, in _handle_request
    resp = await request_handler(request)
  File "/home/bpeyser/aiortc/venv/lib/python3.9/site-packages/aiohttp/web_app.py", line 504, in _handle
    resp = await handler(request)
  File "/home/bpeyser/aiortc/examples/webcam/webcam.py", line 96, in offer
    pc.addTrack(video)
  File "/home/bpeyser/aiortc/src/aiortc/rtcpeerconnection.py", line 428, in addTrack
    transceiver = self.__createTransceiver(
  File "/home/bpeyser/aiortc/src/aiortc/rtcpeerconnection.py", line 1055, in __createTransceiver
    dtlsTransport = self.__createDtlsTransport()
  File "/home/bpeyser/aiortc/src/aiortc/rtcpeerconnection.py", line 1032, in __createDtlsTransport
    dtlsTransport = RTCDtlsTransport(iceTransport, self.__certificates)
  File "/home/bpeyser/aiortc/src/aiortc/rtcdtlstransport.py", line 375, in __init__
    self.__ctx = certificate._create_ssl_context()
  File "/home/bpeyser/aiortc/src/aiortc/rtcdtlstransport.py", line 211, in _create_ssl_context
    _openssl_assert(lib.SSL_CTX_use_certificate(ctx, self._cert._x509) == 1)  # type: ignore
AttributeError: 'builtins.Certificate' object has no attribute '_x509'

It looks like an error with SSL, but I'm not sure why it's trying to use a certificate for SSL. I thought that would be skipped. I looked briefly at the aiortc source, but I just don't know enough about it to figure out how to ensure it doesn't try SSL. Maybe I can give is a self-signed certificate?

I run this on a Pi 4B/2GB running PiOS and OctoPrint, not OctoPi. Also, you'll see this runs python3.9 instead of python3.7. Otherwise, I'm using the instructions provided in Low-Latency H264 Streaming Support w/ WebRTC - #8 by johnboiles.

Any suggestions?

Also, FYI webcam info:

Summary

$ v4l2-ctl --list-formats
ioctl: VIDIOC_ENUM_FMT
	Type: Video Capture

	[0]: 'MJPG' (Motion-JPEG, compressed)
	[1]: 'H264' (H.264, compressed)
	[2]: 'YUYV' (YUYV 4:2:2)

$ v4l2-ctl --all
Driver Info:
	Driver name      : uvcvideo
	Card type        : T6 Webcam
	Bus info         : usb-0000:01:00.0-1.3
	Driver version   : 5.15.84
	Capabilities     : 0x84a00001
		Video Capture
		Metadata Capture
		Streaming
		Extended Pix Format
		Device Capabilities
	Device Caps      : 0x04200001
		Video Capture
		Streaming
		Extended Pix Format
Media Driver Info:
	Driver name      : uvcvideo
	Model            : T6 Webcam
	Serial           : 985100000000100
	Bus info         : usb-0000:01:00.0-1.3
	Media version    : 5.15.84
	Hardware revision: 0x00000010 (16)
	Driver version   : 5.15.84
Interface Info:
	ID               : 0x03000002
	Type             : V4L Video
Entity Info:
	ID               : 0x00000001 (1)
	Name             : T6 Webcam
	Function         : V4L2 I/O
	Flags         : default
	Pad 0x01000007   : 0: Sink
	  Link 0x02000013: from remote pad 0x100000a of entity 'Processing 4': Data, Enabled, Immutable
Priority: 2
Video input : 0 (Camera 1: ok)
Format Video Capture:
	Width/Height      : 1920/1080
	Pixel Format      : 'MJPG' (Motion-JPEG)
	Field             : None
	Bytes per Line    : 0
	Size Image        : 3617600
	Colorspace        : Default
	Transfer Function : Default (maps to Rec. 709)
	YCbCr/HSV Encoding: Default (maps to ITU-R 601)
	Quantization      : Default (maps to Full Range)
	Flags             : 
Crop Capability Video Capture:
	Bounds      : Left 0, Top 0, Width 1920, Height 1080
	Default     : Left 0, Top 0, Width 1920, Height 1080
	Pixel Aspect: 1/1
Selection Video Capture: crop_default, Left 0, Top 0, Width 1920, Height 1080, Flags: 
Selection Video Capture: crop_bounds, Left 0, Top 0, Width 1920, Height 1080, Flags: 
Streaming Parameters Video Capture:
	Capabilities     : timeperframe
	Frames per second: 30.000 (30/1)
	Read buffers     : 0
                     brightness 0x00980900 (int)    : min=0 max=200 step=1 default=100 value=140
                       contrast 0x00980901 (int)    : min=0 max=200 step=1 default=100 value=150
                     saturation 0x00980902 (int)    : min=0 max=200 step=1 default=100 value=150
                            hue 0x00980903 (int)    : min=0 max=360 step=1 default=0 value=0