I'm no Python wizard, but isn't it possible to get the ffmpeg stream directly into Python without creating another command somewhere ?
Will definitely follow your work though, very interested in the outcome
1 Like
you are correct, you could just run the ffmpeg command to generate the stream and push it to twitch or YouTube, but without a server to give those streams out to consumers there's nowhere for it to go. I'm pretty sure that is how this aiortc implementation is handling it, and almost positive the HLS support in OctoPi is also doing this utilizing nginx proxy as the middle man between stream and consumer.
1 Like
It's definitely possible to execute ffmpeg from inside python using subprocess. I'm just thinking end users might be more comfortable modifying a ffmpeg command then modifying python code.
Ooh this gives me an idea -- I wonder if instead of ingesting the ffmpeg video stream into the Python code with a pipe, I could just set up a WebRTC server with aiortc and then stream into it over rtp to localhost. I'll look into that as an alternate implementation.
Have Python load configuration options from a file, because having people launch ffmpeg and then another Python program isn't going to be easy as well...
These PRs into aiortc are looking like promising places to start. One might work out of the box with a Pi Camera (without transcoding), but I don't have a Pi Camera set up to test it.
Ok I got that first PR working with my c920 after a few changes! Latency is ~700ms even with 1080p!
# Checkout my branch of aiortc
git clone https://github.com/johnboiles/aiortc.git
cd aiortc
git checkout octoprint-webrtc-support
# Get some apt dependencies
sudo apt update
sudo apt install python3-venv libavdevice-dev libavfilter-dev libopus-dev libvpx-dev pkg-config libsrtp2-dev
# Create a python virtual environment
python3 -m venv venv
source venv/bin/activate
# Upgrade pip because why not
pip install --upgrade pip
# Get dependencies specific to webcam.py
pip install aiohttp crc32c
# Install aiortc from the local source
pip install .
# [Optional] If you want to modify the python code, point to the source files for python and copy in a few compiled files
export PYTHONPATH=$PWD/src
cp ./venv/lib/python3.7/site-packages/aiortc/codecs/* src/aiortc/codecs/
# Stop anything using the webcam right now
sudo systemctl stop ffmpeg_hls
sudo systemctl stop webcamd
# Run the example
cd examples/webcam
python webcam.py --preferred-codec=video/H264
# Or if you're adventurous and want to try passing forward raw h264 data (works on old Logitcech c920s and maybe others)
python webcam.py --no-transcode --preferred-codec=video/H264 --video-options='{"video_size": "1920x1080", "framerate": "30", "input_format": "h264"}'
Now open your octopi instance on port 8080 (possibly: http://octopi.local:8080), check the 'Use STUN server' option and hit 'Start' (I can get it to work without STUN sometimes but I usually have to refresh the page and hit 'Start' a second time).
It might need some changes to work with the Pi camera --I'm not sure if the v4l2 API works the same for it. It'd be great if someone could try that.
Update: Chrome, Firefox & Safari all working now. Updated the code and instructions above.
1 Like
My OctoPrint PR is up! https://github.com/OctoPrint/OctoPrint/pull/4225
I also got started on some of the required OctoPi changes: https://github.com/guysoft/OctoPi/compare/devel...johnboiles:webrtc-support?expand=1
I still need to edit the OctoPi scripts to install all the aiortc dependencies, but there's no rush on that since it depends on the OctoPrint PR getting merged. Also it might be simpler if some of the aiortc PRs I'm using in my branch (#559 #564 #498) get merged into aiortc since then I could just pip install aiortc.
With my current aiortc server implementation, I'm not saving jpg snapshots, but that should be a solvable problem. Worst case we can grab video data from ffmpeg and use a -vf "select=eq(pict_type\,I)" filter to only save I-frames to jpg before passing the raw h264 stream on to the WebRTC server. But I might be able to implement something similar in python in a more simple way.
sudo systemctl stop webcamd
Ran into a few errors when prepping to test the plugin version of your changes on an octopi 0.18 install. Figured out that pip needed to be upgraded in the venv using pip install --upgrade pip, but that didn't seem to resolve the setup process completely. Had to install rust using curl https://sh.rustup.rs -sSf | sh and pip install setuptools_rust before python setup.py install in your instructions.
Installed /home/pi/aiortc/venv/lib/python3.7/site-packages/aiortc-1.2.1-py3.7-linux-armv6l.egg
Processing dependencies for aiortc==1.2.1
Searching for cryptography>=2.2
Reading https://pypi.org/simple/cryptography/
Downloading https://files.pythonhosted.org/packages/cc/98/8a258ab4787e6f835d350639792527d2eb7946ff9fc0caca9c3f4cf5dcfe/cryptography-3.4.8.tar.gz#sha256=94cc5ed4ceaefcbe5bf38c8fba6a21fc1d365bb8fb826ea1688e3370b2e24a1c
Best match: cryptography 3.4.8
Processing cryptography-3.4.8.tar.gz
Writing /tmp/easy_install-_neis07t/cryptography-3.4.8/setup.cfg
Running cryptography-3.4.8/setup.py -q bdist_egg --dist-dir /tmp/easy_install-_neis07t/cryptography-3.4.8/egg-dist-tmp-40ehcvdg
=============================DEBUG ASSISTANCE==========================
If you are seeing an error here please try the following to
successfully install cryptography:
Upgrade to the latest pip and try again. This will fix errors for most
users. See: https://pip.pypa.io/en/stable/installing/#upgrading-pip
=============================DEBUG ASSISTANCE==========================
Traceback (most recent call last):
File "/home/pi/aiortc/venv/lib/python3.7/site-packages/setuptools/sandbox.py", line 154, in save_modules
yield saved
File "/home/pi/aiortc/venv/lib/python3.7/site-packages/setuptools/sandbox.py", line 195, in setup_context
yield
File "/home/pi/aiortc/venv/lib/python3.7/site-packages/setuptools/sandbox.py", line 250, in run_setup
_execfile(setup_script, ns)
File "/home/pi/aiortc/venv/lib/python3.7/site-packages/setuptools/sandbox.py", line 45, in _execfile
exec(code, globals, locals)
File "/tmp/easy_install-_neis07t/cryptography-3.4.8/setup.py", line 14, in <module>
long_description = f.read()
ModuleNotFoundError: No module named 'setuptools_rust'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "setup.py", line 71, in <module>
extras_require=extras_require,
File "/home/pi/aiortc/venv/lib/python3.7/site-packages/setuptools/__init__.py", line 145, in setup
return distutils.core.setup(**attrs)
File "/usr/lib/python3.7/distutils/core.py", line 148, in setup
dist.run_commands()
File "/usr/lib/python3.7/distutils/dist.py", line 966, in run_commands
self.run_command(cmd)
File "/usr/lib/python3.7/distutils/dist.py", line 985, in run_command
cmd_obj.run()
File "/home/pi/aiortc/venv/lib/python3.7/site-packages/setuptools/command/install.py", line 67, in run
self.do_egg_install()
File "/home/pi/aiortc/venv/lib/python3.7/site-packages/setuptools/command/install.py", line 117, in do_egg_install
cmd.run()
File "/home/pi/aiortc/venv/lib/python3.7/site-packages/setuptools/command/easy_install.py", line 443, in run
self.easy_install(spec, not self.no_deps)
File "/home/pi/aiortc/venv/lib/python3.7/site-packages/setuptools/command/easy_install.py", line 685, in easy_install
return self.install_item(None, spec, tmpdir, deps, True)
File "/home/pi/aiortc/venv/lib/python3.7/site-packages/setuptools/command/easy_install.py", line 732, in install_item
self.process_distribution(spec, dist, deps)
File "/home/pi/aiortc/venv/lib/python3.7/site-packages/setuptools/command/easy_install.py", line 777, in process_distribution
[requirement], self.local_index, self.easy_install
File "/home/pi/aiortc/venv/lib/python3.7/site-packages/pkg_resources/__init__.py", line 782, in resolve
replace_conflicting=replace_conflicting
File "/home/pi/aiortc/venv/lib/python3.7/site-packages/pkg_resources/__init__.py", line 1065, in best_match
return self.obtain(req, installer)
File "/home/pi/aiortc/venv/lib/python3.7/site-packages/pkg_resources/__init__.py", line 1077, in obtain
return installer(requirement)
File "/home/pi/aiortc/venv/lib/python3.7/site-packages/setuptools/command/easy_install.py", line 704, in easy_install
return self.install_item(spec, dist.location, tmpdir, deps)
File "/home/pi/aiortc/venv/lib/python3.7/site-packages/setuptools/command/easy_install.py", line 730, in install_item
dists = self.install_eggs(spec, download, tmpdir)
File "/home/pi/aiortc/venv/lib/python3.7/site-packages/setuptools/command/easy_install.py", line 915, in install_eggs
return self.build_and_install(setup_script, setup_base)
File "/home/pi/aiortc/venv/lib/python3.7/site-packages/setuptools/command/easy_install.py", line 1183, in build_and_install
self.run_setup(setup_script, setup_base, args)
File "/home/pi/aiortc/venv/lib/python3.7/site-packages/setuptools/command/easy_install.py", line 1169, in run_setup
run_setup(setup_script, args)
File "/home/pi/aiortc/venv/lib/python3.7/site-packages/setuptools/sandbox.py", line 253, in run_setup
raise
File "/usr/lib/python3.7/contextlib.py", line 130, in __exit__
self.gen.throw(type, value, traceback)
File "/home/pi/aiortc/venv/lib/python3.7/site-packages/setuptools/sandbox.py", line 195, in setup_context
yield
File "/usr/lib/python3.7/contextlib.py", line 130, in __exit__
self.gen.throw(type, value, traceback)
File "/home/pi/aiortc/venv/lib/python3.7/site-packages/setuptools/sandbox.py", line 166, in save_modules
saved_exc.resume()
File "/home/pi/aiortc/venv/lib/python3.7/site-packages/setuptools/sandbox.py", line 141, in resume
six.reraise(type, exc, self._tb)
File "/home/pi/aiortc/venv/lib/python3.7/site-packages/setuptools/_vendor/six.py", line 685, in reraise
raise value.with_traceback(tb)
File "/home/pi/aiortc/venv/lib/python3.7/site-packages/setuptools/sandbox.py", line 154, in save_modules
yield saved
File "/home/pi/aiortc/venv/lib/python3.7/site-packages/setuptools/sandbox.py", line 195, in setup_context
yield
File "/home/pi/aiortc/venv/lib/python3.7/site-packages/setuptools/sandbox.py", line 250, in run_setup
_execfile(setup_script, ns)
File "/home/pi/aiortc/venv/lib/python3.7/site-packages/setuptools/sandbox.py", line 45, in _execfile
exec(code, globals, locals)
File "/tmp/easy_install-_neis07t/cryptography-3.4.8/setup.py", line 14, in <module>
long_description = f.read()
ModuleNotFoundError: No module named 'setuptools_rust'
The setup install process is still running on my pi zero w. I know not ideal, but that's what I have a raspicam on that supports h264. It is giving loads of cyan/magenta messages during the setup install (example below) but it's not erroring out, so I haven't been able to fully test the plugin version of the OctoPrint changes, but it definitely didn't break the control tab with mjpgstreamer.
https://github.com/jneilliii/OctoPrint-Webrtc
In file included from /usr/include/openssl/e_os2.h:13,
from build/temp.linux-armv6l-3.7/_openssl.c:589:
/usr/include/openssl/x509.h:728:1: note: declared here
DEPRECATEDIN_1_1_0(ASN1_TIME *X509_CRL_get_nextUpdate(X509_CRL *crl))
^~~~~~~~~~~~~~~~~~
build/temp.linux-armv6l-3.7/_openssl.c: At top level:
build/temp.linux-armv6l-3.7/_openssl.c:777:13: warning: ‘ERR_load_Cryptography_OSRandom_strings’ declared ‘static’ but never defined [-Wunused-function]
static void ERR_load_Cryptography_OSRandom_strings(void);
1 Like
Thanks!! I updated my instructions here and in the PR.
Very weird! I didn't have to do this. I bet there aren't python wheels for some packages for arm6l so python is building them from source. I just read yesterday about some Python packages causing a stir for requiring Rust. Interesting.
Oooh awesome, I'm eager to hear if it just works or if there needs to be some aiortc changes.
So it didn't appear to work after the compile was successful. I'll try to get some more info for you over the weekend, but the server did start and I could check the box for stun and start, but the video was not showing up in test. Not completely sure if it's my patches to make it a plugin or the underlying stream not working. More to come after I get home from work tonight.
1 Like
Ok thanks for the update! I have a Pi camera in a box somewhere I can try out as well on my side sometime soon.
I'm also noticing that if I tab away from the Control tab and back too many times, it seems to open concurrent connections to the server and pegs one cpu core at 100% on the Pi. After 3 or 4 quick iterations of this, the video will get slow and sometimes freeze. Eventually the old connections expire and things recover but I probably need to do two things: limit the number of open connections on the server and see if I can reuse or gracefully close the WebRTC connection in the browser when leaving the control tab.
Update: I added code to my aiortc octoprint-webrtc-support branch to limit the number of connections (it just disconnects the oldest connection when a new client joins). I also updated the JS code in the Octoprint PR so that it reuses the open connection if available!
Interestingly, I just deleted my aiortc venv just now because I accidentally installed all the octoprint dependencies in it, and this time it forced me to install rustc.
Update: I figured out a better way. If you just pip install . from the aiortc directory it downloads the wheels instead of trying to recompile all the things! I updated my instructions accordingly.
curious if you have any guidance on this message?
/home/pi/aiortc/venv/lib/python3.7/site-packages/google_crc32c/__init__.py:29: RuntimeWarning: As the c extension couldn't be imported, `google-crc32c` is using a pure python implementation that is significantly slower. If possible, please configure a c build environment and compile the extension
I looked into it briefly but couldn't figure out how to get rid of it. If you're able to figure it out, that'd be helpful! I did some profiling and anything mentioning crc was wayyyyy far down in the list of functions taking up execution time, so I don't think it's a particularly pressing issue.
At one point I had trouble installing the latest google-crc32c, and that's why I had the specific instruction to pip install google-crc32c==1.1.2. But I didn't run into that on the latest rebuild of my venv so I removed it from my instructions (perhaps upgrading pip solved it?).
think I figured it out, pip install crc32c seems to have made that message go away.
1 Like
unfortunately, it seems h264 import as an option doesn't work, but running webcam.py without any options the feed does load in the examples page under media.
Confirmed that gets rid of the message! Wow they should really just say that in the error message 
So v4l2 can't stream h264 directly from the Pi camera? Could you run /usr/bin/v4l2-ctl --device=/dev/video0 --list-formats to see what formats it can output to v4l2?
ioctl: VIDIOC_ENUM_FMT
Type: Video Capture
[0]: 'YU12' (Planar YUV 4:2:0)
[1]: 'YUYV' (YUYV 4:2:2)
[2]: 'RGB3' (24-bit RGB 8-8-8)
[3]: 'JPEG' (JFIF JPEG, compressed)
[4]: 'H264' (H.264, compressed)
[5]: 'MJPG' (Motion-JPEG, compressed)
[6]: 'YVYU' (YVYU 4:2:2)
[7]: 'VYUY' (VYUY 4:2:2)
[8]: 'UYVY' (UYVY 4:2:2)
[9]: 'NV12' (Y/CbCr 4:2:0)
[10]: 'BGR3' (24-bit BGR 8-8-8)
[11]: 'YV12' (Planar YVU 4:2:0)
[12]: 'NV21' (Y/CrCb 4:2:0)
[13]: 'RX24' (32-bit XBGR 8-8-8-8)
ioctl: VIDIOC_ENUM_FMT
Type: Video Capture
[0]: 'YU12' (Planar YUV 4:2:0)
Size: Stepwise 32x32 - 2592x1944 with step 2/2
[1]: 'YUYV' (YUYV 4:2:2)
Size: Stepwise 32x32 - 2592x1944 with step 2/2
[2]: 'RGB3' (24-bit RGB 8-8-8)
Size: Stepwise 32x32 - 2592x1944 with step 2/2
[3]: 'JPEG' (JFIF JPEG, compressed)
Size: Stepwise 32x32 - 2592x1944 with step 2/2
[4]: 'H264' (H.264, compressed)
Size: Stepwise 32x32 - 2592x1944 with step 2/2
[5]: 'MJPG' (Motion-JPEG, compressed)
Size: Stepwise 32x32 - 2592x1944 with step 2/2
[6]: 'YVYU' (YVYU 4:2:2)
Size: Stepwise 32x32 - 2592x1944 with step 2/2
[7]: 'VYUY' (VYUY 4:2:2)
Size: Stepwise 32x32 - 2592x1944 with step 2/2
[8]: 'UYVY' (UYVY 4:2:2)
Size: Stepwise 32x32 - 2592x1944 with step 2/2
[9]: 'NV12' (Y/CbCr 4:2:0)
Size: Stepwise 32x32 - 2592x1944 with step 2/2
[10]: 'BGR3' (24-bit BGR 8-8-8)
Size: Stepwise 32x32 - 2592x1944 with step 2/2
[11]: 'YV12' (Planar YVU 4:2:0)
Size: Stepwise 32x32 - 2592x1944 with step 2/2
[12]: 'NV21' (Y/CrCb 4:2:0)
Size: Stepwise 32x32 - 2592x1944 with step 2/2
[13]: 'RX24' (32-bit XBGR 8-8-8-8)
Size: Stepwise 32x32 - 2592x1944 with step 2/2
EDIT: Aruducam Focus camera....
Interesting. H264 is definitely there. Maybe there's some PyAV flag different than input_format you need to specify to get it to do the right thing. LMK if you figure it out; I might give it a try later.
Awesome! Yeah from my testing if you keep the resolution low the Pi can handle transcoding just fine since aiortc uses the hardware h264 encoder (h264_omx).