I'm happy to say that at least the plugin is loading properly it seems.
Interesting. H264 is definitely there. Maybe there's some PyAV flag different than input_format you need to specify to get it to do the right thing. LMK if you figure it out; I might give it a try later.
Awesome! Yeah from my testing if you keep the resolution low the Pi can handle transcoding just fine since aiortc uses the hardware h264 encoder (h264_omx).
You don't by chance know if there are video-options for rotation/mirroring?
There definitely aren't for the Logitech c920 -- I looked into this and ended up just remounting my camera so that it's right side up. I think I remember seeing some potential options for the Pi camera though, but I'm not 100% sure.
You can always do it in CPU with PyAV or ffmpeg but I'm not sure how CPU intensive that would be.
Edit: maybe look at v4l2-ctrl --list-ctrls-menus and see if there's anything about rotation
Oh good point. I might be able to use the camera settings plugin to set something.
I can say if I include the command lines from your instructions it chokes pretty bad on the zero. I'm assuming causing some kind of system error potentially and killing the os in some way.
I bet you're running out of RAM. There's only 512mb on the Zero looks like where I have 1gb on the 3B. Python isn't particularly RAM efficient and probably between running OctoPrint and buffering 1080p h264 you could hit that limit pretty quickly. You could try my command and just scale down the resolution and see if it helps.
Yeah, I figured something along those lines. Tried this command and it doesn't seem to be streaming at all. Maybe dropping the fps too would be a good idea?
(venv) pi@pizero:~/aiortc/examples/webcam $ python webcam.py --no-transcode --preferred-codec=video/H264 --video-options='{"video_size": "640x480", "framerate": "30", "input_format": "h264"}'
======== Running on http://0.0.0.0:8080 ========
(Press CTRL+C to quit)
INFO:aiohttp.access:192.168.0.134 [11/Sep/2021:21:38:47 +0000] "GET / HTTP/1.1" 200 985 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.63 Safari/537.36"
INFO:aiohttp.access:192.168.0.134 [11/Sep/2021:21:38:47 +0000] "GET /client.js HTTP/1.1" 200 2432 "http://192.168.0.101:8080/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.63 Safari/537.36"
INFO:aiohttp.access:192.168.0.134 [11/Sep/2021:21:38:48 +0000] "GET /favicon.ico HTTP/1.1" 404 178 "http://192.168.0.101:8080/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.63 Safari/537.36"
INFO:aioice.ice:Connection(0) Remote candidate "19646e72-d2a1-4f5a-b0a4-8011a35ab5fa.local" could not be resolved
INFO:aiohttp.access:192.168.0.134 [11/Sep/2021:21:38:52 +0000] "POST /offer HTTP/1.1" 200 2479 "http://192.168.0.101:8080/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.63 Safari/537.36"
INFO:aioice.ice:Connection(0) Check CandidatePair(('192.168.0.101', 57735) -> ('73.122.180.167', 59030)) State.FROZEN -> State.WAITING
INFO:aioice.ice:Connection(0) Discovered peer reflexive candidate Candidate(cUeMt7hpIq 1 udp 1845501695 192.168.0.134 59030 typ prflx)
Connection state is connecting
INFO:aioice.ice:Connection(0) Check CandidatePair(('192.168.0.101', 57735) -> ('73.122.180.167', 59030)) State.WAITING -> State.IN_PROGRESS
INFO:aioice.ice:Connection(0) Check CandidatePair(('192.168.0.101', 57735) -> ('192.168.0.134', 59030)) State.WAITING -> State.IN_PROGRESS
INFO:aioice.ice:Connection(0) Check CandidatePair(('192.168.0.101', 57735) -> ('192.168.0.134', 59030)) State.IN_PROGRESS -> State.SUCCEEDED
INFO:aioice.ice:Connection(0) ICE completed
Connection state is connected
WARNING:aiortc.contrib.media:MediaPlayer(/dev/video0) Skipping video packet with no pts
I just got to this point with my Pi camera and I'm seeing the same thing -- I think the --video-options might need to be different for the pi camera. I think we're not getting an h264 stream.
cool, at lest we know it's consistent between two different environments.
Yeah I'm def thankful it's consistent 
I'm able to get it to work by using ffmpeg to get the video stream then piping it in to webcam.py. Not sure why we can't get it to work directly in webcam.py. But this works for me:
sudo mkdir -p /run/webcam
sudo mkfifo /run/webcam/tmp.fifo
sudo chmod -R 0777 /run/webcam/tmp.fifo
# It might be possible to work these into the ffmpeg command somehow
v4l2-ctl -d /dev/video0 -c video_bitrate_mode=1,video_bitrate=2000000,h264_profile=0,h264_i_frame_period=60
v4l2-ctl -d /dev/video0 -v pixelformat=H264,width=1920,height=1080
v4l2-ctl -d /dev/video0 -p 30
ffmpeg \
\
-vcodec h264 \
-pix_fmt h264 \
-video_size 1920x1080 \
-i /dev/video0 \
\
-c:v copy \
-f mpegts \
pipe:1 > /run/webcam/tmp.fifo
# In a separate ssh session:
python examples/webcam/webcam.py --play-from /run/webcam/tmp.fifo --no-transcode
The framerate is a little uneven (perhaps something about how ffmpeg writes to pipes?) but the latency overall is low. The CPU/RAM usage is low for 1080p too:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
809 pi 20 0 220660 65808 35956 R 57.3 8.6 0:51.74 python
817 pi 20 0 212256 95616 89712 S 1.7 12.5 0:01.08 ffmpeg
I had to set gpu_mem=256 in /boot/config.txt to get it to work though. That's probably a dealbreaker for the Pi Zero.
The good thing is that it seems like like the h264_omx transcoded 640x480 stream works great out of the box and is lower bandwidth and comparable latency to the mjpegstreamer stream! Seems promising that using 640x480 transcoded h264 streams could be a good OctoPrint default in the future.
Maybe to keep this moving forward we just make that the default for the plugin and advanced users can try to get raw 1080p forwarded if they want.
The default streaming resolution in OctoPi is 640x480, 10fps so it wouldn't be unreasonable to have the same defaults there.
1 Like
Oh cool. Yeah 640x480 @ 30fps seems to be working just fine with WebRTC so there's some improvement right there then! The biggest bandwidth improvement in h264 comes from motion frames so might as well take advantage of them!
I can confirm that v4l2-ctl commands do work for rotation. I used the camera settings plugin to set it.
1 Like
FYI I just pushed a pretty big performance improvement to my aiortc branch when using h264 passthrough. See https://github.com/aiortc/aiortc/pull/564 for more info.
@jneilliii I think I have a solution for the Pi camera with h264 passthrough. I needed to enable inline PPS/SPS headers in v4l2 via the barely documented repeat_sequence_header control. Make sure to pull my latest octoprint-webrtc-support branch in aiortc to get the aforementioned performance improvement first. Then try this:
v4l2-ctl -d /dev/video0 -v pixelformat=H264,width=1920,height=1080
v4l2-ctl -d /dev/video0 -c video_bitrate_mode=1,video_bitrate=4000000,h264_profile=4,h264_i_frame_period=60,h264_level=11,repeat_sequence_header=1
v4l2-ctl -d /dev/video0 -p 30
/home/pi/aiortc/venv/bin/python /home/pi/aiortc/examples/webcam/webcam.py --no-transcode --preferred-codec=video/H264 --video-options='{"video_size": "1920x1080", "framerate": "30", "input_format": "h264", "pixelformat": "H264"}'
I'm getting 720p streaming with the pi camera and it's only using ~20% CPU and 85mb ram. At 1080p it uses about the same CPU and 123mb ram. So maybe there's hope for it working on the Pi Zero (though I did also set gpu_mem=265 in /boot/config.txt; not sure if that's needed for the Pi Zero or not)
You can also try scaling the resolution down to 1280x720 @ 2000000bps which should lower RAM requirements a bit.
I did some slightly more scientific tests and I'm seeing ~450ms latency with my Logitech c920 and a mere < 200ms on the Pi camera both at 1080p!! As a baseline, mjpg-streamer with a Pi camera is ~300ms. So it looks like WebRTC has mjpeg-streamer beat on latency, bitrate/bandwidth, resolution and framerate 
. (Just maybe not in simplicity because video codecs are hard 
)
Logitech + WebRTC
Pi Camera + WebRTC
Pi Camera + Mjpegstreamer
That's a smart way of measuring the latency, I'll have to keep that in mind! Great results.
1 Like