Towards Python 3 and OctoPrint 1.4.0

foosel · September 30, 2019, 3:30pm

I figured I'd go on the record somewhere now due to stuff like

making this seem a tad urgent.

As said in various OctoPrint on Air episodes, OctoPrint 1.4.0 will support Python 2 and 3.

Plugin authors are advised to test their plugins for compatibility with Python 3 against ~~the devel branch of OctoPrint (1.4.x)~~ the latest 1.4.0 Release Candidate of OctoPrint (currently 1.4.0rc5) or the rc/maintenance branch. To do that install Python 3, install OctoPrint in a Python 3 virtualenv, set the control property __plugin_pythoncompat__ of your plugin to >=2.7,<4 (the default is >=2.7,<3) and install it into the same Python 3 virtualenv. Test it thoroughly.

If your plugin works as expected, everything is fine and you can keep the control property and also file a PR against the plugin repository to update the listing of your plugin by setting compatibility.python in your plugin's registration to >=2.7,<4 as well:

compatibility:
  python: ">=2.7,<4"

That way it will also show as compatible in OctoPrint 1.4.x's plugin manager.

If your plugin doesn't work under Python 3, you probably want to take a look at this cheat sheet to help you fix it.

If you run into any kind of OctoPrint issues while testing, please open proper tickets for them!

Going forward, your plugins should (ideally) support both python 2 and 3 until OctoPrint drops support for Python 2 (which by current planning it won't do until version 2.0.0 since it will mean backwards incompatibility). This also means keeping an eye on third party dependencies you are using!

Now for some bad news. The new comm layer has made huge steps forward, it prints reliably for me and feature completion is almost there. But with it pretty much now being October and 1.4.0 having to be released until 2020 due to the looming Python 2 EOL date it won't make it into 1.4.0 but will have to wait until the next feature release.

I've thought long and hard about this decision, and it honestly sucks, but it is simply too risky to push it out now since there won't be enough time left until January 1st 2020 to give it the overall testing by me and the community that such a crucial part of the software needs.

The new plan is to prepare a first RC of 1.4.0 ASAP and then merge the new comm layer into the new 1.5.0.dev version of the devel branch that will then get created - which btw might also end up as being rebranded as 2.0.0. That way those people running the devel branch will be able to test it and provide feedback and I don't have to worry that some issue in it nukes the pre-2020 release of 1.4.0.

From a risk management point of view that simply makes more sense, even though I can tell you I really would have loved to finally put an end to this chapter (especially since as long as it isn't merged to stable I get to do everything relating to the comm layer twice, in the current and the new one).

That'll be all.

PrintedWeezl · September 30, 2019, 4:08pm

com-optimize

jneilliii · September 30, 2019, 6:52pm

Looks like my next couple of weekends are full now. Thanks for the heads up @foosel. I'm going to be testing the config.yaml option for forcing compatibility as you described in the On-Air #26 like below.

plugins:
  _forcedCompatible:
  - plugin_identifier_1
  - plugin_identifier_2
  - plugin_identifier_3

OutsourcedGuru · September 30, 2019, 10:49pm

That's fine about the new com layer. It's better to have it safe than now.

I've recently spent at least two weeks working on just a single plugin (it's huge of course).

Needed to mark the plugin so that OctoPrint thinks I've made it Python 3—compatible
Needed to visit all relative imports to adjust them

Before

from screen_jobreports import JobHistoryTile

After

from .screen_jobreports import JobHistoryTile

Wrapped filter() calls with list() as in...

Before

photoFiles = filter(lambda x: x.find('.png') != -1, allFilesAndFolders)

After

photoFiles = list(filter(lambda x: x.find('.png') != -1, allFilesAndFolders))

Wrapped returns from serial reads with str() as in:

Before

   line = self.board.read(self.board.in_waiting)

After

   line = str(self.board.read(self.board.in_waiting))

There is still (likely) more work to do with respect to threading.thread(), subprocess.check_output() -> popen(), etc.

I found that the octoprint.server.userManager.getAllUsers() output in v1.4 seems to be rather different from 1.3.11, so much so that I still haven't fixed that yet. It now has a very granulated rights situation going on but I'm used to the simpler version from earlier. I'll tackle that soon enough.

I'm in a balancing act between trying to get any sort of Raspberry-compatible kiosk-style operating system to behave itself on the Pi4B hardware platform with respect to Kivy... and getting a Python 3—happy version of OctoPrint + my plugin + Kivy 1.11.1 to show a full-screen window and to simply work as before. I've tried Raspbian, ArchLinux, Ubuntu64-bit so far and can't get the userspace to present a Kivy-compatible device stack. This will eventually sort itself out. For now, it's a pain in my butt.

jneilliii · October 1, 2019, 5:40pm

I went through the following steps to set-up my python3 environment and all seems well, except OctoPrint is complaining of being throttled and I can't install plugins. I didn't even know that throttling occurred when on windows and that was part of the pi plugin?

Install Python 3.7.4 in c:\Python3 making sure it was added to path during install.
In command prompt do the following.

mkdir c:\OctoPrint
cd \OctoPrint
pip install virtualenv
venv\Scripts\activate.bat
pip install --upgrade pip
pip install https://github.com/foosel/OctoPrint/archive/devel.zip
octoprint serve

What am I missing?

Browser developer console shows this:

OutsourcedGuru · October 1, 2019, 6:46pm

This seems wrong to me. For example, you haven't created a virtual environment for Python 3.

Install virtualenv globally as you've done
pip install pip --upgrade # upgrade pip globally
Change to someplace where you can git clone OctoPrint. This might be your home folder.
git clone https://github.com/foosel/OctoPrint.git
virtualenv -p python3 oprint # Create a Python 3—based virtual environment
source oprint/bin/activate # Activate it
cd OctoPrint
git checkout devel # Set things to the development branch where the 1.4.x work is happening
python setup.py clean
pip install .
octoprint serve

foosel · October 1, 2019, 8:38pm

Looks more like a caching issue to me to be honest, at least the js console.

jneilliii · October 2, 2019, 7:48pm

So nothing seems to be able to make this go away for me. I've tried reinstalling Python, re-creating the virtualenv, going to the site in incognito mode, clearing browsing data, etc. and the same browser console error displays. Version installed seems to be right displaying OctoPrint 1.4.0.dev2369+g49fdac75.Here's my octoprint.log file, which doesn't seem to show any errors whatsoever.

octoprint.log (20.5 KB)

foosel · October 2, 2019, 8:16pm

Ah nevermind... probably a merge bug, I'm seeing it too. Will see that this gets fixed ASAP.

For the record, here's how I've setup a (concurrent) Python 3.4 install under Windows:

Install from the executable installer package, do not add it to the PATH
Open a shell in OctoPrint's source folder, using the full path to the Python 3.4 python executable install virtualenv (.../python.exe -m pip install virtualenv) and then create a python 3 virtualenv (.../python.exe -m virtualenv venv3)
Activate the virtualenv
Install dependencies: pip install -e .[develop]

Note that for Linux there's also already updated documentation.

edit Yep, merge error.

edit2 1.4.0.dev2370+g0035e7c3 should be fixed

jneilliii · October 2, 2019, 9:03pm

Thanks @foosel, it's working fine now.

jneilliii · October 3, 2019, 5:32pm

Quick question in regards to dependencies. It seems that one of my dependencies (urllib2) used for several of my projects were split into multiple libraries. There is a link at the python documentation page to 2to3, which appears to help you translate your code from one base version to another, but my question is, how would one be able to support both versions from a dependency aspect?

foosel · October 3, 2019, 5:38pm

https://python-future.org/compatible_idioms.html#urllib-module

jneilliii · October 3, 2019, 5:40pm

Sorry, I was just looking at that...thanks.

jneilliii · October 3, 2019, 9:30pm

Might need some deeper help @foosel with https://github.com/jneilliii/OctoPrint-ConsolidateTempControl, which was a fork of your example. When enabled in Python 3 environment it is causing errors in core code, but no pointer to within the plugin's code of what is causing the error. I think it is probably the global __plugin_settings_overlay__ part causing the error since that's really the only thing in Python side that is not normal compared to my other plugins. Any thoughts?

2019-10-03 17:22:41,508 - octoprint.server.preemptive_cache - INFO - Preemptively caching / (ui _default) for {'base_url': 'http://localhost:5000/', 'path': '/', 'query_string': 'l10n=en'}
[2019-10-03 17:22:41,748] ERROR in app: Exception on / [GET]
Traceback (most recent call last):
  File "c:\octoprint\venv\lib\site-packages\flask\app.py", line 1982, in wsgi_app
    response = self.full_dispatch_request()
  File "c:\octoprint\venv\lib\site-packages\flask\app.py", line 1614, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "c:\octoprint\venv\lib\site-packages\flask\app.py", line 1517, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "c:\octoprint\venv\lib\site-packages\flask\_compat.py", line 33, in reraise
    raise value
  File "c:\octoprint\venv\lib\site-packages\flask\app.py", line 1612, in full_dispatch_request
    rv = self.dispatch_request()
  File "c:\octoprint\venv\lib\site-packages\flask\app.py", line 1598, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "c:\users\jim neill\documents\octoprint\src\octoprint\server\views.py", line 181, in index
    fetch_template_data(refresh=force_refresh)
  File "c:\users\jim neill\documents\octoprint\src\octoprint\server\views.py", line 755, in fetch_template_data
    sorted_missing = sorted(missing_in_order, key=key_func)
TypeError: '<' not supported between instances of 'str' and 'NoneType'
[2019-10-03 17:22:41,973] ERROR in app: Exception on / [GET]
Traceback (most recent call last):
  File "c:\octoprint\venv\lib\site-packages\flask\app.py", line 1982, in wsgi_app
    response = self.full_dispatch_request()
  File "c:\octoprint\venv\lib\site-packages\flask\app.py", line 1614, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "c:\octoprint\venv\lib\site-packages\flask\app.py", line 1517, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "c:\octoprint\venv\lib\site-packages\flask\_compat.py", line 33, in reraise
    raise value
  File "c:\octoprint\venv\lib\site-packages\flask\app.py", line 1612, in full_dispatch_request
    rv = self.dispatch_request()
  File "c:\octoprint\venv\lib\site-packages\flask\app.py", line 1598, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "c:\users\jim neill\documents\octoprint\src\octoprint\server\views.py", line 181, in index
    fetch_template_data(refresh=force_refresh)
  File "c:\users\jim neill\documents\octoprint\src\octoprint\server\views.py", line 755, in fetch_template_data
    sorted_missing = sorted(missing_in_order, key=key_func)
TypeError: '<' not supported between instances of 'str' and 'NoneType'
[2019-10-03 17:24:46,443] ERROR in app: Exception on / [GET]
Traceback (most recent call last):
  File "c:\octoprint\venv\lib\site-packages\flask\app.py", line 1982, in wsgi_app
    response = self.full_dispatch_request()
  File "c:\octoprint\venv\lib\site-packages\flask\app.py", line 1614, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "c:\octoprint\venv\lib\site-packages\flask\app.py", line 1517, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "c:\octoprint\venv\lib\site-packages\flask\_compat.py", line 33, in reraise
    raise value
  File "c:\octoprint\venv\lib\site-packages\flask\app.py", line 1612, in full_dispatch_request
    rv = self.dispatch_request()
  File "c:\octoprint\venv\lib\site-packages\flask\app.py", line 1598, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "c:\users\jim neill\documents\octoprint\src\octoprint\server\views.py", line 181, in index
    fetch_template_data(refresh=force_refresh)
  File "c:\users\jim neill\documents\octoprint\src\octoprint\server\views.py", line 755, in fetch_template_data
    sorted_missing = sorted(missing_in_order, key=key_func)
TypeError: '<' not supported between instances of 'str' and 'NoneType'

OutsourcedGuru · October 3, 2019, 10:13pm

Sounds like this sort of thing, to be honest. Presumably there's a comparison of something (like a version number) against a None value from some lookup.

jneilliii · October 6, 2019, 5:02am

So I have narrowed down the issue a little more with that __plugin_settings_overlay__ related issue. I have confirmed first that is the issue by commenting it out and the plugin loads fine. It also seems to work fine when it's the only plugin loaded with a tab template. Once another plugin that also has a tab template the error happens and the server is unreachable with the error above. This also only happens when the config.yaml doesn't contain any ordering in the appearance section. If the tabs are ordered in config.yaml everything also loads up fine.

Stefan_Cohen · October 15, 2019, 9:59am

A few notes for setting up a python 3 development env on a clean Ubuntu 18.04 host in addition to the guides/steps provided by @OutsourcedGuru and @foosel above:

I needed to run apt install python3-dev in the virtual environment in order for psutil and regex to build during `pip install -e .[develop,plugins]. The rest of the requirements were already met.

I got a runtime error when running octoprint serve: RuntimeError: Click will abort further execution because Python 3 was configured to use ASCII as encoding for the environment. That can be solved by running export LC_ALL=C.UTF-8 and export LANG=C.UTF-8 (or add them to your .bashrc)

Running octoprint as root is quite useful during development and can be done by starting using: octoprint serve --iknowwhatimdoing (assuming that you know what you are doing )

OutsourcedGuru · October 15, 2019, 5:17pm

I wouldn't run octoprint as root. No way, Jose.

Yeah, you're right on the python3-dev. There's kind of a hand-wave gesture that goes along with "well, they should know how to do this..." attitude, it almost feels like. We're expected to read something like this...

Python2:

sudo apt-get install python-setuptools python-pyqt5 python-pip

...know that we're in Python3 and then just translate this into...

Python3:

sudo apt-get install python3-setuptools python3-pyqt5 python3-pip

...for any install. And then we're supposed to cross our fingers and hope that every author's code actually works and is fully-tested in both Python versions.

For what it's worth, I ALWAYS set the default language when I'm setting the language in sudo raspi-config (it's that second prompt). The second prompt has None as the default and this seems to break a few things out there.

Stefan_Cohen · October 15, 2019, 6:50pm

Cool. I've been able to run both the Dashboard and DisplayLayerProgress (with a very minor fix) under Python3 today. It looks good so far and I haven't found any other bugs yet.

I'm not sure I got that _forcedCompatible: mode to work as expected because I still couldn't browse any plugins after that. Installing via URL worked so I used that. Was that the expected behaviour?

This Is my config:

plugins:
    _forcedCompatible:
    - dashboard
    - DisplayLayerProgress

OutsourcedGuru · October 15, 2019, 8:24pm

I'm not sure if the formatting of your config file example is correct, to be honest. Maybe that's correct. Hm...

Visiting the repository and searching for _forcedCompatible I'm not seeing it. Changed to the devel branch, still not seeing it. It's not showing up in rc/devel. Color me confused.