OctoPrint on PyPy3...on the Raspberry Pi

Showcase
1

I've noticed that the OctoPrint instance on my Hypercube 300 (which was running on CPython 3.7) was a bit more responsive than the OctoPrint instance on my AM8 (which was running on CPython 2.7). I've since upgraded the AM8's OctoPrint so that it's on CPython 3.7 as well, but then I decided to try for a bit more speed and switch from CPython to PyPy.

There's just one snag: PyPy3 that's available in Raspbian is v7.0, which is compatible with CPython 3.5. OctoPrint wants 3.6 as a minimum. This is supported in newer versions of PyPy (7.3.1 is current), but building PyPy3 on a Raspberry Pi is an exercise in frustration. It won't build on my RPi 3s at all because they don't have enough RAM.

Solution: build PyPy3 for Raspbian on another host.

I installed qemu on my home workstation: a Ryzen 7 3800X with 32 GB RAM and a 512 GB NVMe SSD, running Gentoo. Pretty decent instructions for what follows are available at https://wiki.gentoo.org/wiki/Raspberry_Pi#Compiling_using_chroot; this is a summary.

Emerge qemu with at least these USE flags:

  • static-user
  • qemu_user_targets_arm
  • qemu_user_targets_aarch64

Also, include --buildpkg among the options to emerge. We'll need a binary package to install into the Raspbian image later.

Download the latest Raspbian Lite image. Extract the two partitions from the image, and resize the root filesystem to at least 16 GB so you'll have room to install tools and build PyPy3. Loopback-mount the root filesystem someplace convenient (we'll use /mnt/rpi as an example), and do the same with the boot filesystem within it. Bind-mount /proc, /sys, and /dev at the appropriate points, and copy /etc/resolv.conf into the root filesystem.

Now, install the qemu binary package into Raspbian:

ROOT=/mnt/rpi sudo emerge --usepkgonly --oneshot --nodeps qemu

Make sure qemu binfmts are loaded:

sudo /etc/init.d/qemu-binfmt start

and chroot into Raspbian:

sudo chroot /mnt/rpi /bin/bash -l

At this point, you're running Raspbian on your desktop in emulation. Download the latest PyPy3, unpack it, and compile it according to the instructions at https://doc.pypy.org/en/latest/build.html. I don't know how to package software to integrate with apt-get, but the build process can produce a tarball that can be unpacked into /opt, and that works well enough for me. Building PyPy3 7.3.1 took somewhere around three hours, and I noticed that it never used more than 4 cores (of the 8 I have available).

Speaking of tarballs, this is the one I produced if you don't want to go to the trouble of making one yourself:

However you come by it, unpack it into /opt. You can then follow the OctoPrint installation instructions, summarized as follows:

cd ~
export PATH=$PATH:/opt/pypy3-7.3.1-raspbian/bin
virtualenv -p pypy3 octoprint-pypy3
octoprint-pypy3/bin/pip install OctoPrint

and then edit /etc/systemd/system/octoprint.service to point to ~/octoprint-pypy3 instead of wherever it's currently set to go. (Before doing that, you may want to back up and download your OctoPrint config so you can restore it in the new PyPy3-based instance.)

1 Like
2

So, did it result in noticeably more speed?

3

Giving it the Bed Level Visualizer install test now. Let's see how well it does with numpy...

Edit: Didn't work out for me, and user experience felt slower than the default interpreter included with octopi for initial page loads.

4

It seems to take longer to start up, but once it's going, it seems more responsive. I've had some other issues with the printer lately that I need to resolve before I can do some more comprehensive testing.

There was one plugin I had installed that won't work with PyPy: OctoPrint-DetailedProgress.

5

Giving it the Bed Level Visualizer install test now. Let's see how well it does with numpy...

I thought I already had it installed, but didn't. Tried installing it, but ran into an error...checked the log, and it would appear that the deactivated OctoPrint-DetailedProgress plugin is interfering with it:

2020-08-01 14:48:03,806 - octoprint.plugin.core - ERROR - Invalid syntax in /home/salfter/octoprint-pypy3/site-packages/octoprint_detailedprogress/__init__.py for plugin detailedprogress
Traceback (most recent call last):
  File "/home/salfter/octoprint-pypy3/site-packages/octoprint/plugin/core.py", line 599, in _parse_metadata
    root = ast.parse(f.read(), filename=path)
  File "/opt/pypy3-7.3.1-raspbian/lib-python/3/ast.py", line 35, in parse
    return compile(source, filename, mode, PyCF_ONLY_AST)
  File "<string>", line 45
    currentData = { "progress": { "completion": 100, "printTimeLeft": 0 } }
    ^
TabError: inconsistent use of tabs and spaces in indentation
2020-08-01 14:48:03,967 - octoprint.plugin.core - WARNING - Plugin OctoPrint-DetailedProgress (0.2.5) can't be compiled under Python 3.6.9 due to invalid syntax

I then went to uninstall OctoPrint-DetailedProgress...now it's saying it's still building a wheel for numpy. Once that finished, a little bit later it said it was done installing Bed Visualizer and was now going about uninstalling OctoPrint-DetailedProgress. Having finished that, it's now restarting OctoPrint.

After all that, I can confirm that the Bed Visualizer does work under PyPy3. Building numpy takes a fair bit of time, but it eventually succeeds, and the build time should only be a one-time hit.

(Judging by the results it returned, I need to make some shims to level out the bed. One more task toward getting this printer dialed in. :-) )

6

Does PyPy have it's own wheels somewhere that can be registered? I've recently gotten the numpy install down to under a minute with Python 3 and piwheels, but I'm assuming the cp37m means cpython?

https://www.piwheels.org/project/numpy/

7

I don't know. I'd kinda prefer building from source, myself...even had OctoPrint running that way for a while on Gentoo (64-bit userspace on the RPi!), but there are some dependencies with current versions I've not managed to fully sort out yet.