Low-Latency H264 Streaming Support w/ WebRTC

johnboiles · September 13, 2021, 6:32am

@jneilliii I think I have a solution for the Pi camera with h264 passthrough. I needed to enable inline PPS/SPS headers in v4l2 via the barely documented repeat_sequence_header control. Make sure to pull my latest octoprint-webrtc-support branch in aiortc to get the aforementioned performance improvement first. Then try this:

v4l2-ctl -d /dev/video0 -v pixelformat=H264,width=1920,height=1080
v4l2-ctl -d /dev/video0 -c video_bitrate_mode=1,video_bitrate=4000000,h264_profile=4,h264_i_frame_period=60,h264_level=11,repeat_sequence_header=1
v4l2-ctl -d /dev/video0 -p 30

/home/pi/aiortc/venv/bin/python /home/pi/aiortc/examples/webcam/webcam.py --no-transcode --preferred-codec=video/H264 --video-options='{"video_size": "1920x1080", "framerate": "30", "input_format": "h264", "pixelformat": "H264"}'

I'm getting 720p streaming with the pi camera and it's only using ~20% CPU and 85mb ram. At 1080p it uses about the same CPU and 123mb ram. So maybe there's hope for it working on the Pi Zero (though I did also set gpu_mem=265 in /boot/config.txt; not sure if that's needed for the Pi Zero or not)

You can also try scaling the resolution down to 1280x720 @ 2000000bps which should lower RAM requirements a bit.

johnboiles · September 13, 2021, 5:57pm

I did some slightly more scientific tests and I'm seeing ~450ms latency with my Logitech c920 and a mere < 200ms on the Pi camera both at 1080p!! As a baseline, mjpg-streamer with a Pi camera is ~300ms. So it looks like WebRTC has mjpeg-streamer beat on latency, bitrate/bandwidth, resolution and framerate . (Just maybe not in simplicity because video codecs are hard )

Logitech + WebRTC

Pi Camera + WebRTC

Pi Camera + Mjpegstreamer

Charlie_Powell · September 13, 2021, 8:15pm

That's a smart way of measuring the latency, I'll have to keep that in mind! Great results.

b-morgan · September 13, 2021, 8:48pm

@Charlie_Powell may have figured it out but I haven't... How are you measuring the latency?

johnboiles · September 13, 2021, 9:01pm

Setup: Open both the webcam stream and a stopwatch on my computer. Point the streaming camera at my computer screen. Take a picture or screenshot of the computer. The difference in timestamps between the stopwatch on the screen and what's shown in the stream is the end-to-end latency.

I hooked everything up to ethernet so I wasn't also measuring my wifi performance

jneilliii · September 16, 2021, 7:52pm

We've opened a specific channel #dev-octopi in the Discord server at the request of @Quinninator to allow better real-time collaboration on your struggles with prepping the octopi side changes.

Damian_Britton · November 27, 2021, 2:56pm

Got everything running through Janus and accessible via direct ip access in a web browser but when I try to point Octoprint webcam stream url to it I get "Bad status line 'Invalid method encountered". Any ideas on whats going on here? Set up is Pi 3B running 0.18.1 - 1.7.2 with PiCam H264 transcoded

ERROR:aiohttp.server:Error handling request
Traceback (most recent call last):
  File "/home/pi/aiortc/venv/lib/python3.7/site-packages/aiohttp/web_protocol.py", line 334, in data_received
    messages, upgraded, tail = self._request_parser.feed_data(data)
  File "aiohttp/_http_parser.pyx", line 551, in aiohttp._http_parser.HttpParser.feed_data
aiohttp.http_exceptions.BadStatusLine: 400, message="Bad status line 'Invalid method encountered'"
INFO:aiohttp.access:192.168.1.63 [27/Nov/2021:14:54:20 +0000] "UNKNOWN / HTTP/1.0" 400 204 "-" "-"

acca_rien · December 13, 2021, 9:59am

Hello, i have c920 webcam, i'd like to use h.264
i use octopi with raspberry pi 3b

i have octoprint 1.7.2, does it support WebRTC ?

If i select maintenance RC branch, there is no upper version proposed. Are maintenance and maintenance RC the same ?

Thanks

Charlie_Powell · December 13, 2021, 2:47pm

There's currently no maintenance release candidate available newer than 1.7.2, only manually installing from GitHub on the maintenance branch has support for webRTC.

acca_rien · December 13, 2021, 7:37pm

Thanks, i added it manually
I have performed those steps :

To update to a version of OctoPrint with WebRTC support, checkout the maintenance branch (PR4225 has been merged into maintenance and will be released with 1.8.0 ).

To set up a WebRTC server on your Pi, see this post (see also this post if you have a Pi camera)

i can see my camera image but image is strange :

What did i do wrong please ?

In addition, can someone detail the procedure to make the server start automatically please ?

Thanks

Greg_Shaw · January 6, 2022, 8:43pm

@johnboiles I was wondering will this new approach work for the new Logitech C920 cameras that no longer have the hardware h.264 built in?

ytwytw · February 17, 2022, 8:20pm

I have multiple cameras and it seems its only able to open /dev/video0

Is there any way to change it?

Grant_Williamson · December 1, 2022, 5:27am

Hi john, Novice user here,

Following along with you line for line,

when i hit

"Use STUN server"

then followed by

START i get a error,

"

======== Running on http://0.0.0.0:8080 ========
(Press CTRL+C to quit)
INFO:aiohttp.access:192.168.1.157 [01/Dec/2022:05:23:36 +0000] "GET / HTTP/1.1" 200 979 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.0.0 Safari/537.36"
INFO:aiohttp.access:192.168.1.157 [01/Dec/2022:05:23:36 +0000] "GET /client.js HTTP/1.1" 200 2426 "http://octopi.local:8080/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.0.0 Safari/537.36"
[video4linux2,v4l2 @ 0x2494030] ioctl(VIDIOC_STREAMON): Invalid argument
ERROR:aiohttp.server:Error handling request
Traceback (most recent call last):
File "/home/pi/aiortc/venv/lib/python3.7/site-packages/aiohttp/web_protocol.py", line 433, in _handle_request
resp = await request_handler(request)
File "/home/pi/aiortc/venv/lib/python3.7/site-packages/aiohttp/web_app.py", line 504, in _handle
resp = await handler(request)
File "/home/pi/aiortc/examples/webcam/webcam.py", line 93, in offer
audio, video = create_local_tracks(args.play_from, transcode=args.transcode, options=args.video_options)
File "/home/pi/aiortc/examples/webcam/webcam.py", line 46, in create_local_tracks
webcam = MediaPlayer("/dev/video0", format="v4l2", transcode=transcode, options=options)
File "/home/pi/aiortc/venv/lib/python3.7/site-packages/aiortc/contrib/media.py", line 309, in init
self.__container = av.open(file=file, format=format, mode="r", options=options)
File "av/container/core.pyx", line 354, in av.container.core.open
File "av/container/core.pyx", line 225, in av.container.core.Container.cinit
File "av/container/core.pyx", line 257, in av.container.core.Container.err_check
File "av/error.pyx", line 336, in av.error.err_check
av.error.ValueError: [Errno 22] Invalid argument: '/dev/video0'
INFO:aiohttp.access:192.168.1.157 [01/Dec/2022:05:23:37 +0000] "POST /offer HTTP/1.1" 500 244 "http://octopi.local:8080/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.0.0 Safari/537.36"
INFO:aiohttp.access:127.0.0.1 [01/Dec/2022:05:23:41 +0000] "GET /?action=stream HTTP/1.1" 200 998 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.0.0 Safari/537.36"

"

Any ideas? cheers!

fischer-felix · December 2, 2022, 9:53pm

If you are using a pi camera, you could try setting it to legacy mode:
sudo raspi-config > 3 Interface Options > I1 Legacy Camera

PDB_maker · February 13, 2023, 3:57pm

I also get an error once I click "Start"

The web page (in Firefox) shows:
SyntaxError: JSON.parse: unexpected non-whitespace character after JSON data at line 1 column 5 of the JSON data

Also, I get errors in the terminal on my Pi:

$ sudo systemctl stop webcamd
$ ./venv/bin/python examples/webcam/webcam.py --no-transcode --preferred-codec=video/H264 --video-options='{"video_size": "1920x1080", "framerate": "30", "input_format": "h264"}'
======== Running on http://0.0.0.0:8080 ========
(Press CTRL+C to quit)

Once I connect to the webpage at 192.168.1.6:8080,

INFO:aiohttp.access:192.168.1.130 [13/Feb/2023:15:40:41 +0000] "GET / HTTP/1.1" 200 979 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/109.0"
INFO:aiohttp.access:192.168.1.130 [13/Feb/2023:15:40:41 +0000] "GET /client.js HTTP/1.1" 200 2426 "http://192.168.1.6:8080/" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/109.0"
INFO:aiohttp.access:192.168.1.130 [13/Feb/2023:15:40:41 +0000] "GET /favicon.ico HTTP/1.1" 404 172 "http://192.168.1.6:8080/" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/109.0"

After I check "Use STUN Server" and click "Start" I get the SyntaxError: JSON.parse: unexpected non-whitespace character after JSON data at line 1 column 5 of the JSON data and the following in the terminal on my Pi:

ERROR:aiohttp.server:Error handling request
Traceback (most recent call last):
  File "/home/bpeyser/aiortc/venv/lib/python3.9/site-packages/aiohttp/web_protocol.py", line 433, in _handle_request
    resp = await request_handler(request)
  File "/home/bpeyser/aiortc/venv/lib/python3.9/site-packages/aiohttp/web_app.py", line 504, in _handle
    resp = await handler(request)
  File "/home/bpeyser/aiortc/examples/webcam/webcam.py", line 96, in offer
    pc.addTrack(video)
  File "/home/bpeyser/aiortc/src/aiortc/rtcpeerconnection.py", line 428, in addTrack
    transceiver = self.__createTransceiver(
  File "/home/bpeyser/aiortc/src/aiortc/rtcpeerconnection.py", line 1055, in __createTransceiver
    dtlsTransport = self.__createDtlsTransport()
  File "/home/bpeyser/aiortc/src/aiortc/rtcpeerconnection.py", line 1032, in __createDtlsTransport
    dtlsTransport = RTCDtlsTransport(iceTransport, self.__certificates)
  File "/home/bpeyser/aiortc/src/aiortc/rtcdtlstransport.py", line 375, in __init__
    self.__ctx = certificate._create_ssl_context()
  File "/home/bpeyser/aiortc/src/aiortc/rtcdtlstransport.py", line 211, in _create_ssl_context
    _openssl_assert(lib.SSL_CTX_use_certificate(ctx, self._cert._x509) == 1)  # type: ignore
AttributeError: 'builtins.Certificate' object has no attribute '_x509'

It looks like an error with SSL, but I'm not sure why it's trying to use a certificate for SSL. I thought that would be skipped. I looked briefly at the aiortc source, but I just don't know enough about it to figure out how to ensure it doesn't try SSL. Maybe I can give is a self-signed certificate?

I run this on a Pi 4B/2GB running PiOS and OctoPrint, not OctoPi. Also, you'll see this runs python3.9 instead of python3.7. Otherwise, I'm using the instructions provided in Low-Latency H264 Streaming Support w/ WebRTC - #8 by johnboiles.

Any suggestions?

Also, FYI webcam info:

Summary

$ v4l2-ctl --list-formats
ioctl: VIDIOC_ENUM_FMT
	Type: Video Capture

	[0]: 'MJPG' (Motion-JPEG, compressed)
	[1]: 'H264' (H.264, compressed)
	[2]: 'YUYV' (YUYV 4:2:2)

$ v4l2-ctl --all
Driver Info:
	Driver name      : uvcvideo
	Card type        : T6 Webcam
	Bus info         : usb-0000:01:00.0-1.3
	Driver version   : 5.15.84
	Capabilities     : 0x84a00001
		Video Capture
		Metadata Capture
		Streaming
		Extended Pix Format
		Device Capabilities
	Device Caps      : 0x04200001
		Video Capture
		Streaming
		Extended Pix Format
Media Driver Info:
	Driver name      : uvcvideo
	Model            : T6 Webcam
	Serial           : 985100000000100
	Bus info         : usb-0000:01:00.0-1.3
	Media version    : 5.15.84
	Hardware revision: 0x00000010 (16)
	Driver version   : 5.15.84
Interface Info:
	ID               : 0x03000002
	Type             : V4L Video
Entity Info:
	ID               : 0x00000001 (1)
	Name             : T6 Webcam
	Function         : V4L2 I/O
	Flags         : default
	Pad 0x01000007   : 0: Sink
	  Link 0x02000013: from remote pad 0x100000a of entity 'Processing 4': Data, Enabled, Immutable
Priority: 2
Video input : 0 (Camera 1: ok)
Format Video Capture:
	Width/Height      : 1920/1080
	Pixel Format      : 'MJPG' (Motion-JPEG)
	Field             : None
	Bytes per Line    : 0
	Size Image        : 3617600
	Colorspace        : Default
	Transfer Function : Default (maps to Rec. 709)
	YCbCr/HSV Encoding: Default (maps to ITU-R 601)
	Quantization      : Default (maps to Full Range)
	Flags             : 
Crop Capability Video Capture:
	Bounds      : Left 0, Top 0, Width 1920, Height 1080
	Default     : Left 0, Top 0, Width 1920, Height 1080
	Pixel Aspect: 1/1
Selection Video Capture: crop_default, Left 0, Top 0, Width 1920, Height 1080, Flags: 
Selection Video Capture: crop_bounds, Left 0, Top 0, Width 1920, Height 1080, Flags: 
Streaming Parameters Video Capture:
	Capabilities     : timeperframe
	Frames per second: 30.000 (30/1)
	Read buffers     : 0
                     brightness 0x00980900 (int)    : min=0 max=200 step=1 default=100 value=140
                       contrast 0x00980901 (int)    : min=0 max=200 step=1 default=100 value=150
                     saturation 0x00980902 (int)    : min=0 max=200 step=1 default=100 value=150
                            hue 0x00980903 (int)    : min=0 max=360 step=1 default=0 value=0

jneilliii · February 13, 2023, 7:38pm

you might want to check out using go2rtc instead of the webrtc server implementation described here.

PDB_maker · February 14, 2023, 6:39pm

I finally got go2rtc to work in my browser, using the following streams setting in go2rtc.yaml:

streams:
  stream1: exec:ffmpeg -f v4l2 -input_format h264 -video_size 1920x1080 -framerate 30 -i /dev/video0 -c:v copy -an -rtsp_transport tcp -f rtsp {output}

I can open http://192.168.1.6:1984/stream.html?src=stream1 and see the webcam in Firefox. I think it is using h264 encoding directly from my webcam, without re-encoding (-c:v copy). Some other links shown in http://192.168.1.6:1984/links.html?src=stream1 also work. However, I can't open that in VLC, only in Firefox (or Chrome). I can open the RTSP link in VLC, and it is using h264, 30fps, probably directly from the webcam.

However, when I put http://192.168.1.6:1984/stream.html?src=stream1 in the Octoprint webcam stream URL setting, I don't see anything in the Stream test. I also tried http://192.168.1.6:8555, which is the WebRTC listen port, as well as the other ports. Not that I thought those would work. None of the other links that are OK in my browser work, either, such as http://192.168.1.6:1984/webrtc.html?src=stream1.

I also tried using webrtc://192.168.1.6:8555 and webrtc://192.168.1.6:8555/stream1 in Octoprint, and those didn't work, either. I feel like I'm really close to this, but I'm missing something.

Maybe one more hint?

Thanks!

jneilliii · February 14, 2023, 7:26pm

@kantlivelong mentioned the other day he's using my webcam iframe plugin to work with go2rtc's intelligent delivery mechanism.

PDB_maker · February 15, 2023, 1:42am

That did work!

I set the webcam stream to http://192.168.1.6:1984/stream.html?src=stream1 and it worked fine in the iframe.

Thanks for the suggestion.

Unfortunately, I couldn't figure out how to get a snapshot URL for timelapse.

kantlivelong · February 15, 2023, 6:08pm

go2rtc does offer a snapshot feature but it's not in jpeg format and instead a single frame mp4. I don't think there's any way in OP to handle this right now but you could make a small service to convert it and provide a consumable URL.