Low-Latency H264 Streaming Support w/ WebRTC

Development
1

TL;DR: WebRTC streaming from OctoPrint is working pretty well! You can get 1080p 30fps video with as low as 200ms latency and 4mbps video stream (higher resolution, lower bandwidth AND lower latency than mjpeg-streamer).

  • To update to a version of OctoPrint with WebRTC support, checkout the maintenance branch (PR4225 has been merged into maintenance and will be released with 1.8.0 :tada:).
  • To set up a WebRTC server on your Pi, see this post (see also this post if you have a Pi camera)
  • To get the server to run automatically at startup, take a look at my work-in-progress OctoPi changes.

Testing, feedback, and code contributions are all appreciated!!

Original post:

I tried out @Chudsaviet's Experimental HLS support this weekend and I'm loving the 1080p video from my OctoPrint instance and lower bandwidth compared to mjpegstreamer. But, HLS is inherently a high-latency video protocol, and my video lag is about 22s.

So I'm looking into alternate video protocols that have low latency. Some of the options:

Of these, WebRTC is the most standardized and widely supported, so that's where I'm starting. It's also intentionally optimized for as-low-latency-as-possible, which is helpful.

I first took a look at this Balena project for a WebRTC camera. It uses aiortc to implement camera streaming with WebRTC, but it also uses mjpeg for the video stream. Then I found that aiortc has a h264 webcam example. I was able to get it running out of the box at 640x480 and <1s latency on my OctoPi instance using hardware h264 (omx) encoding (although the colors were messed up -- probably a pixel format thing). Here's what I did:

git clone git@github.com:aiortc/aiortc.git
sudo apt update
sudo apt install python3-venv libsrtp2-dev
python3 -m venv venv
source venv/bin/activate
pip install google-crc32c==1.1.2
pip install aiohttp aiortc opencv-python
sudo systemctl stop ffmpeg_hls.service
python webcam.py

Next steps:

  • Fix the pixel format issue messing up my colors
  • Try with the raspi camera (I'm currently using a Logitech c920)
  • Try to figure out how to change video settings in the aiortc webcam example (using the PyAV APIs) OR figure out how to just pipe in output from an FFMPEG command into a WebRTC session.

If anyone wants to pitch in on this, I'd appreciate the help and testing!

4 Likes
2

Looking into the PyAV APIs a bit more they're not so easy to understand. I'm thinking the simplest and most adaptable solution here will involve using aiortc to handle the WebRTC stuff, and then using a ffmpeg command to format the video. This will make it easy for people with various webcam setups to adapt the system to their own needs without writing Python code (and instead changing the ffmpeg command). Here's what I've tried so far, but it seems like aiortc is still transcoding my input even though I'm giving it an h264 steam.

sudo mkdir -p /run/webcam
sudo mkfifo /run/webcam/tmp.fifo
sudo chmod -R 0755 /run/webcam/tmp.fifo
ffmpeg \
    \
    -vcodec h264 \
    -pix_fmt h264 \
    -video_size 640x480 \
    -i /dev/video0 \
    \
    -c:v copy \
    -f mpegts \
    pipe:1 > /run/webcam/tmp.fifo

# In a separate ssh session:
python webcam.py --play-from /run/webcam/tmp.fifo

This uses the native h264 stream from my (older model) Logitech c920 and copies it straight into the tmp.fifo FIFO. The webcam streamer then reads from tmp.fifo and streams it out over RTC. It works, but colors are messed up, and also my video seems to be (unnecessarily) transcoded (which is probably why my colors are still messed up). At least it's using the Pi's hardware h264_omx encoder to re-encode!

1 Like
3

I'm no Python wizard, but isn't it possible to get the ffmpeg stream directly into Python without creating another command somewhere ?

Will definitely follow your work though, very interested in the outcome

1 Like
4

you are correct, you could just run the ffmpeg command to generate the stream and push it to twitch or YouTube, but without a server to give those streams out to consumers there's nowhere for it to go. I'm pretty sure that is how this aiortc implementation is handling it, and almost positive the HLS support in OctoPi is also doing this utilizing nginx proxy as the middle man between stream and consumer.

1 Like
5

It's definitely possible to execute ffmpeg from inside python using subprocess. I'm just thinking end users might be more comfortable modifying a ffmpeg command then modifying python code.

Ooh this gives me an idea -- I wonder if instead of ingesting the ffmpeg video stream into the Python code with a pipe, I could just set up a WebRTC server with aiortc and then stream into it over rtp to localhost. I'll look into that as an alternate implementation.

6

Have Python load configuration options from a file, because having people launch ffmpeg and then another Python program isn't going to be easy as well...

7

These PRs into aiortc are looking like promising places to start. One might work out of the box with a Pi Camera (without transcoding), but I don't have a Pi Camera set up to test it.

8

Ok I got that first PR working with my c920 after a few changes! Latency is ~700ms even with 1080p!

# Checkout my branch of aiortc
git clone https://github.com/johnboiles/aiortc.git
cd aiortc
git checkout octoprint-webrtc-support

# Get some apt dependencies
sudo apt update
sudo apt install python3-venv libavdevice-dev libavfilter-dev libopus-dev libvpx-dev pkg-config libsrtp2-dev

# Create a python virtual environment
python3 -m venv venv
source venv/bin/activate

# Upgrade pip because why not
pip install --upgrade pip

# Get dependencies specific to webcam.py
pip install aiohttp crc32c

# Install aiortc from the local source
pip install .

# [Optional] If you want to modify the python code, point to the source files for python and copy in a few compiled files
export  PYTHONPATH=$PWD/src
cp ./venv/lib/python3.7/site-packages/aiortc/codecs/* src/aiortc/codecs/

# Stop anything using the webcam right now
sudo systemctl stop ffmpeg_hls
sudo systemctl stop webcamd

# Run the example
cd examples/webcam
python webcam.py --preferred-codec=video/H264

# Or if you're adventurous and want to try passing forward raw h264 data (works on old Logitcech c920s and maybe others)
python webcam.py --no-transcode --preferred-codec=video/H264 --video-options='{"video_size": "1920x1080", "framerate": "30", "input_format": "h264"}'

Now open your octopi instance on port 8080 (possibly: http://octopi.local:8080), check the 'Use STUN server' option and hit 'Start' (I can get it to work without STUN sometimes but I usually have to refresh the page and hit 'Start' a second time).

It might need some changes to work with the Pi camera --I'm not sure if the v4l2 API works the same for it. It'd be great if someone could try that.

Update: Chrome, Firefox & Safari all working now. Updated the code and instructions above.

1 Like
9

My OctoPrint PR is up! https://github.com/OctoPrint/OctoPrint/pull/4225

I also got started on some of the required OctoPi changes: https://github.com/guysoft/OctoPi/compare/devel...johnboiles:webrtc-support?expand=1

I still need to edit the OctoPi scripts to install all the aiortc dependencies, but there's no rush on that since it depends on the OctoPrint PR getting merged. Also it might be simpler if some of the aiortc PRs I'm using in my branch (#559 #564 #498) get merged into aiortc since then I could just pip install aiortc.

With my current aiortc server implementation, I'm not saving jpg snapshots, but that should be a solvable problem. Worst case we can grab video data from ffmpeg and use a -vf "select=eq(pict_type\,I)" filter to only save I-frames to jpg before passing the raw h264 stream on to the WebRTC server. But I might be able to implement something similar in python in a more simple way.

10

sudo systemctl stop webcamd

Ran into a few errors when prepping to test the plugin version of your changes on an octopi 0.18 install. Figured out that pip needed to be upgraded in the venv using pip install --upgrade pip, but that didn't seem to resolve the setup process completely. Had to install rust using curl https://sh.rustup.rs -sSf | sh and pip install setuptools_rust before python setup.py install in your instructions.

Installed /home/pi/aiortc/venv/lib/python3.7/site-packages/aiortc-1.2.1-py3.7-linux-armv6l.egg
Processing dependencies for aiortc==1.2.1
Searching for cryptography>=2.2
Reading https://pypi.org/simple/cryptography/
Downloading https://files.pythonhosted.org/packages/cc/98/8a258ab4787e6f835d350639792527d2eb7946ff9fc0caca9c3f4cf5dcfe/cryptography-3.4.8.tar.gz#sha256=94cc5ed4ceaefcbe5bf38c8fba6a21fc1d365bb8fb826ea1688e3370b2e24a1c
Best match: cryptography 3.4.8
Processing cryptography-3.4.8.tar.gz
Writing /tmp/easy_install-_neis07t/cryptography-3.4.8/setup.cfg
Running cryptography-3.4.8/setup.py -q bdist_egg --dist-dir /tmp/easy_install-_neis07t/cryptography-3.4.8/egg-dist-tmp-40ehcvdg

        =============================DEBUG ASSISTANCE==========================
        If you are seeing an error here please try the following to
        successfully install cryptography:

        Upgrade to the latest pip and try again. This will fix errors for most
        users. See: https://pip.pypa.io/en/stable/installing/#upgrading-pip
        =============================DEBUG ASSISTANCE==========================

Traceback (most recent call last):
  File "/home/pi/aiortc/venv/lib/python3.7/site-packages/setuptools/sandbox.py", line 154, in save_modules
    yield saved
  File "/home/pi/aiortc/venv/lib/python3.7/site-packages/setuptools/sandbox.py", line 195, in setup_context
    yield
  File "/home/pi/aiortc/venv/lib/python3.7/site-packages/setuptools/sandbox.py", line 250, in run_setup
    _execfile(setup_script, ns)
  File "/home/pi/aiortc/venv/lib/python3.7/site-packages/setuptools/sandbox.py", line 45, in _execfile
    exec(code, globals, locals)
  File "/tmp/easy_install-_neis07t/cryptography-3.4.8/setup.py", line 14, in <module>
    long_description = f.read()
ModuleNotFoundError: No module named 'setuptools_rust'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "setup.py", line 71, in <module>
    extras_require=extras_require,
  File "/home/pi/aiortc/venv/lib/python3.7/site-packages/setuptools/__init__.py", line 145, in setup
    return distutils.core.setup(**attrs)
  File "/usr/lib/python3.7/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/usr/lib/python3.7/distutils/dist.py", line 966, in run_commands
    self.run_command(cmd)
  File "/usr/lib/python3.7/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/home/pi/aiortc/venv/lib/python3.7/site-packages/setuptools/command/install.py", line 67, in run
    self.do_egg_install()
  File "/home/pi/aiortc/venv/lib/python3.7/site-packages/setuptools/command/install.py", line 117, in do_egg_install
    cmd.run()
  File "/home/pi/aiortc/venv/lib/python3.7/site-packages/setuptools/command/easy_install.py", line 443, in run
    self.easy_install(spec, not self.no_deps)
  File "/home/pi/aiortc/venv/lib/python3.7/site-packages/setuptools/command/easy_install.py", line 685, in easy_install
    return self.install_item(None, spec, tmpdir, deps, True)
  File "/home/pi/aiortc/venv/lib/python3.7/site-packages/setuptools/command/easy_install.py", line 732, in install_item
    self.process_distribution(spec, dist, deps)
  File "/home/pi/aiortc/venv/lib/python3.7/site-packages/setuptools/command/easy_install.py", line 777, in process_distribution
    [requirement], self.local_index, self.easy_install
  File "/home/pi/aiortc/venv/lib/python3.7/site-packages/pkg_resources/__init__.py", line 782, in resolve
    replace_conflicting=replace_conflicting
  File "/home/pi/aiortc/venv/lib/python3.7/site-packages/pkg_resources/__init__.py", line 1065, in best_match
    return self.obtain(req, installer)
  File "/home/pi/aiortc/venv/lib/python3.7/site-packages/pkg_resources/__init__.py", line 1077, in obtain
    return installer(requirement)
  File "/home/pi/aiortc/venv/lib/python3.7/site-packages/setuptools/command/easy_install.py", line 704, in easy_install
    return self.install_item(spec, dist.location, tmpdir, deps)
  File "/home/pi/aiortc/venv/lib/python3.7/site-packages/setuptools/command/easy_install.py", line 730, in install_item
    dists = self.install_eggs(spec, download, tmpdir)
  File "/home/pi/aiortc/venv/lib/python3.7/site-packages/setuptools/command/easy_install.py", line 915, in install_eggs
    return self.build_and_install(setup_script, setup_base)
  File "/home/pi/aiortc/venv/lib/python3.7/site-packages/setuptools/command/easy_install.py", line 1183, in build_and_install
    self.run_setup(setup_script, setup_base, args)
  File "/home/pi/aiortc/venv/lib/python3.7/site-packages/setuptools/command/easy_install.py", line 1169, in run_setup
    run_setup(setup_script, args)
  File "/home/pi/aiortc/venv/lib/python3.7/site-packages/setuptools/sandbox.py", line 253, in run_setup
    raise
  File "/usr/lib/python3.7/contextlib.py", line 130, in __exit__
    self.gen.throw(type, value, traceback)
  File "/home/pi/aiortc/venv/lib/python3.7/site-packages/setuptools/sandbox.py", line 195, in setup_context
    yield
  File "/usr/lib/python3.7/contextlib.py", line 130, in __exit__
    self.gen.throw(type, value, traceback)
  File "/home/pi/aiortc/venv/lib/python3.7/site-packages/setuptools/sandbox.py", line 166, in save_modules
    saved_exc.resume()
  File "/home/pi/aiortc/venv/lib/python3.7/site-packages/setuptools/sandbox.py", line 141, in resume
    six.reraise(type, exc, self._tb)
  File "/home/pi/aiortc/venv/lib/python3.7/site-packages/setuptools/_vendor/six.py", line 685, in reraise
    raise value.with_traceback(tb)
  File "/home/pi/aiortc/venv/lib/python3.7/site-packages/setuptools/sandbox.py", line 154, in save_modules
    yield saved
  File "/home/pi/aiortc/venv/lib/python3.7/site-packages/setuptools/sandbox.py", line 195, in setup_context
    yield
  File "/home/pi/aiortc/venv/lib/python3.7/site-packages/setuptools/sandbox.py", line 250, in run_setup
    _execfile(setup_script, ns)
  File "/home/pi/aiortc/venv/lib/python3.7/site-packages/setuptools/sandbox.py", line 45, in _execfile
    exec(code, globals, locals)
  File "/tmp/easy_install-_neis07t/cryptography-3.4.8/setup.py", line 14, in <module>
    long_description = f.read()
ModuleNotFoundError: No module named 'setuptools_rust'

The setup install process is still running on my pi zero w. I know not ideal, but that's what I have a raspicam on that supports h264. It is giving loads of cyan/magenta messages during the setup install (example below) but it's not erroring out, so I haven't been able to fully test the plugin version of the OctoPrint changes, but it definitely didn't break the control tab with mjpgstreamer.

https://github.com/jneilliii/OctoPrint-Webrtc

In file included from /usr/include/openssl/e_os2.h:13,
                 from build/temp.linux-armv6l-3.7/_openssl.c:589:
/usr/include/openssl/x509.h:728:1: note: declared here
 DEPRECATEDIN_1_1_0(ASN1_TIME *X509_CRL_get_nextUpdate(X509_CRL *crl))
 ^~~~~~~~~~~~~~~~~~
build/temp.linux-armv6l-3.7/_openssl.c: At top level:
build/temp.linux-armv6l-3.7/_openssl.c:777:13: warning: ‘ERR_load_Cryptography_OSRandom_strings’ declared ‘static’ but never defined [-Wunused-function]
 static void ERR_load_Cryptography_OSRandom_strings(void);
1 Like
11

Thanks!! I updated my instructions here and in the PR.

Very weird! I didn't have to do this. I bet there aren't python wheels for some packages for arm6l so python is building them from source. I just read yesterday about some Python packages causing a stir for requiring Rust. Interesting.

Oooh awesome, I'm eager to hear if it just works or if there needs to be some aiortc changes.

12

So it didn't appear to work after the compile was successful. I'll try to get some more info for you over the weekend, but the server did start and I could check the box for stun and start, but the video was not showing up in test. Not completely sure if it's my patches to make it a plugin or the underlying stream not working. More to come after I get home from work tonight.

1 Like
13

Ok thanks for the update! I have a Pi camera in a box somewhere I can try out as well on my side sometime soon.

I'm also noticing that if I tab away from the Control tab and back too many times, it seems to open concurrent connections to the server and pegs one cpu core at 100% on the Pi. After 3 or 4 quick iterations of this, the video will get slow and sometimes freeze. Eventually the old connections expire and things recover but I probably need to do two things: limit the number of open connections on the server and see if I can reuse or gracefully close the WebRTC connection in the browser when leaving the control tab.

Update: I added code to my aiortc octoprint-webrtc-support branch to limit the number of connections (it just disconnects the oldest connection when a new client joins). I also updated the JS code in the Octoprint PR so that it reuses the open connection if available!

14

Interestingly, I just deleted my aiortc venv just now because I accidentally installed all the octoprint dependencies in it, and this time it forced me to install rustc.

Update: I figured out a better way. If you just pip install . from the aiortc directory it downloads the wheels instead of trying to recompile all the things! I updated my instructions accordingly.

15

curious if you have any guidance on this message?

/home/pi/aiortc/venv/lib/python3.7/site-packages/google_crc32c/__init__.py:29: RuntimeWarning: As the c extension couldn't be imported, `google-crc32c` is using a pure python implementation that is significantly slower. If possible, please configure a c build environment and compile the extension
16

I looked into it briefly but couldn't figure out how to get rid of it. If you're able to figure it out, that'd be helpful! I did some profiling and anything mentioning crc was wayyyyy far down in the list of functions taking up execution time, so I don't think it's a particularly pressing issue.

At one point I had trouble installing the latest google-crc32c, and that's why I had the specific instruction to pip install google-crc32c==1.1.2. But I didn't run into that on the latest rebuild of my venv so I removed it from my instructions (perhaps upgrading pip solved it?).

17

think I figured it out, pip install crc32c seems to have made that message go away.

1 Like
18

unfortunately, it seems h264 import as an option doesn't work, but running webcam.py without any options the feed does load in the examples page under media.

19

Confirmed that gets rid of the message! Wow they should really just say that in the error message :slight_smile:

So v4l2 can't stream h264 directly from the Pi camera? Could you run /usr/bin/v4l2-ctl --device=/dev/video0 --list-formats to see what formats it can output to v4l2?

20 
ioctl: VIDIOC_ENUM_FMT
        Type: Video Capture

        [0]: 'YU12' (Planar YUV 4:2:0)
        [1]: 'YUYV' (YUYV 4:2:2)
        [2]: 'RGB3' (24-bit RGB 8-8-8)
        [3]: 'JPEG' (JFIF JPEG, compressed)
        [4]: 'H264' (H.264, compressed)
        [5]: 'MJPG' (Motion-JPEG, compressed)
        [6]: 'YVYU' (YVYU 4:2:2)
        [7]: 'VYUY' (VYUY 4:2:2)
        [8]: 'UYVY' (UYVY 4:2:2)
        [9]: 'NV12' (Y/CbCr 4:2:0)
        [10]: 'BGR3' (24-bit BGR 8-8-8)
        [11]: 'YV12' (Planar YVU 4:2:0)
        [12]: 'NV21' (Y/CrCb 4:2:0)
        [13]: 'RX24' (32-bit XBGR 8-8-8-8)

ioctl: VIDIOC_ENUM_FMT
        Type: Video Capture

        [0]: 'YU12' (Planar YUV 4:2:0)
                Size: Stepwise 32x32 - 2592x1944 with step 2/2
        [1]: 'YUYV' (YUYV 4:2:2)
                Size: Stepwise 32x32 - 2592x1944 with step 2/2
        [2]: 'RGB3' (24-bit RGB 8-8-8)
                Size: Stepwise 32x32 - 2592x1944 with step 2/2
        [3]: 'JPEG' (JFIF JPEG, compressed)
                Size: Stepwise 32x32 - 2592x1944 with step 2/2
        [4]: 'H264' (H.264, compressed)
                Size: Stepwise 32x32 - 2592x1944 with step 2/2
        [5]: 'MJPG' (Motion-JPEG, compressed)
                Size: Stepwise 32x32 - 2592x1944 with step 2/2
        [6]: 'YVYU' (YVYU 4:2:2)
                Size: Stepwise 32x32 - 2592x1944 with step 2/2
        [7]: 'VYUY' (VYUY 4:2:2)
                Size: Stepwise 32x32 - 2592x1944 with step 2/2
        [8]: 'UYVY' (UYVY 4:2:2)
                Size: Stepwise 32x32 - 2592x1944 with step 2/2
        [9]: 'NV12' (Y/CbCr 4:2:0)
                Size: Stepwise 32x32 - 2592x1944 with step 2/2
        [10]: 'BGR3' (24-bit BGR 8-8-8)
                Size: Stepwise 32x32 - 2592x1944 with step 2/2
        [11]: 'YV12' (Planar YVU 4:2:0)
                Size: Stepwise 32x32 - 2592x1944 with step 2/2
        [12]: 'NV21' (Y/CrCb 4:2:0)
                Size: Stepwise 32x32 - 2592x1944 with step 2/2
        [13]: 'RX24' (32-bit XBGR 8-8-8-8)
                Size: Stepwise 32x32 - 2592x1944 with step 2/2

EDIT: Aruducam Focus camera....