Migrating plugins to Python 2 & 3 compatibility - experiences?

I've been meaning to put out a proper blog post for the whole Python 3 migration situation for a while now and the recent LineProcessorStream situation just put that back up high on my stack of top priorities.

I'd be interested to hear from the plugin developer community active here who's already migrated a bunch of plugins what the most common pitfalls were so I can compile that into the blog post. I figure stuff like str vs bytes and the necessity to decode into unicode strings on data coming in from streams and the different behaviour of map, filter and friends which no longer return collections but iterators and thus need wrapping are two of the biggest issues, but maybe there are other things not on my radar that y'all have run into and I'd be keen to learn about them.

Explicitly pinging @jneilliii, @FormerLurker, @OllisGit since I know they've been quite busy with that kinda stuff, but everyone is welcome to contribute!

Here are some of the ones I included earlier. Here's another.

In setting things up and in following advice/tutorials from the Internet, you often have to make adjustments. For example, the Internet advice gives you a list of things to install and you have to decorate some for the Python 3 version, in other words. If you're not in the (Python 3) virtual environment and there's a mention of running python or pip you're expected to substitute python3 and pip3, respectively.

It's best to keep the new Python 3 virtual environment folder as ~/oprint so that you don't have to adjust /etc/default/octoprint.

1 Like

I'll start at the beginning: Getting a Python 3.7 virtual environment running.

Since I want to test both on python 2.7 and 3.7+, I created two virtual environments: venv and venv37. There are plenty of tutorials for getting Octoprint running on 2.7, so I will skip that. I am assuming that OctoPrint 1.4.0 is already setup and running on c:\Plugin\OctoPrint, which includes a virtual environment for python 2.7 at c:\Plugin\OctoPrint\venv. If not, follow these steps and get that working.
What I did is the following:

  1. Install Python 3.7 and make a note of the installation folder. On my machine it was installed in C:\Users\USER_NAME_HERE\AppData\Local\Programs\Python\Python37. Install and reboot.
  2. Adjust your environment variables for Python 3. There are other ways of doing this, but I found this to be the easiest. You can edit the variables from System->Advanced->Environment Variables, or just search for Environment Variables. Edit your path variable and add the following two paths (adjusted for your installation folder from step 1):
    C:\Users\USER_NAME_HERE\AppData\Local\Programs\Python\Python37\
    C:\Users\USER_NAME_HERE\AppData\Local\Programs\Python\Python37\Scripts\
  3. Add or edit a new system variable called PY_HOME with the following value:
    C:\Users\USER_NAME_HERE\AppData\Local\Programs\Python\Python37\
  4. Open GitBash (if it was already open, close it and reopen it else your environment variable changes might not be used) and check the python version via:
    python --version
    You should get a python 3 version similar to this:
    Python 3.7.3
    If you see Python 2.7.x, something is wrong with your environment variables. Double check your paths in steps 2 and 3, and maybe reboot.
  5. Install virtualenv with the following command:
    pip install virtualenv
  6. Navigate to your current Octoprint installation directory (assuming it is at c:\plugin\OctoPrint) via:
    cd c:\plugin\OctoPrint
  7. Now create a virtual environment for python 3.7 by entering:
    virtualenv venv37
  8. Activate the virtual environment via:
    source venv37/Scripts/activate
    You should now see (venv37)
  9. It never hurts to upgrade pip via pip install --upgrade pip, so I suggest you try that now.
  10. Install OctoPrint including its regular and development and plugin development dependencies via:
    pip install -e .[develop,plugins]
    That will take a while.
  11. Once that is finished you will have two virtual environments, one for python 2.7 (venv), and another for python 3.7 (venv37). The next step is making both of these environments work in your IDE. I will add steps for this as soon as I can.
1 Like

Has anyone here used 2to3? I haven't tried it out yet, but it looks interesting. Might be a good place to start.

1 Like

I have used 2to3. Since I'm fairly leery about other people's transpilers I tend to go the route of:

  • clone my code into another working directory
  • run 2to3 there
  • use Visual Studio Code's Source Control tab to review the before/after changes that made
  • go to school on their changes
1 Like

Took me way too long to get back to my own topic thanks to the new RC - sorry for that :confused:

You can actually skip a ton of these steps - or at least it worked fine on my end to do that... virtualenv has an argument --python with which you can specify the Python interpreter with which to create the venv. So it boils down to installing both 2.7 and 3.7 somewhere and then telling virtualenv which interpreter to use:

virtualenv --python=C:/Python27/python.exe venv27
virtualenv --python=C:/Python37/python.exe venv37

Then just activate whichever one you need, or - that's how I do it - configure both interpreters in your IDE and then switch there. The only bad thing about the latter is that at least my (slightly outdated) version of PyCharm always needs an extra minute to get back on track after that, and I have to make sure to close all active run/debug panels relating to the old Python environment or it doesn't properly switch :woman_shrugging:

Does that support 2 and 3 concurrently? I always took it to be more a case of "once that's run the code won't run under Py2 anymore", which at least for now would be counter productive.

What I've made good use of is this cheat sheet by the Python-Future project:

https://python-future.org/compatible_idioms.html

And in general it seems to be a good idea to turn on Py3 behaviour as far as possible in your source files through various from __future__ imports. E.g. (from OctoPrint's code base):

# -*- coding: utf-8 -*-
from __future__ import absolute_import, division, print_function, unicode_literals

I also have a post-it note attached to my main screen for more than a year now that currently says

  • except Exception:
  • to_bytes & to_unicode
  • io.open
  • list(<generator>)
    • map
    • filter
    • ...
  • __metaclass__

Explanation follows :wink:

except Exception:

Instead of just except: for broad catch-all blocks. Not Python 3 specific, but something we switched to in the code base with the Py3 changes.

to_bytes & to_unicode

Two helper methods built into OctoPrint (octoprint.util package) that should hopefully make it easier to ensure the expected data type. Will keep their hands off of anything that's already in the correct type.

io.open

Instead of open if I remember correctly, though I cannot for the life of me right now remember what exactly was special about that. Pretty sure it had something to do with encoding though (because that's the worst about this whole migration).

list(<generator>)

This is what @OutsourcedGuru already mentioned, a ton of the functional programming kinda methods in Python 3 are now returning generators instead of ready data, so if you want to do stuff with that later where it needs to be a list you'll need to convert yourself. Has kicked me in the teeth a couple of times now, one of the fixes that went into RC6 was actually another one of these.

__metaclass__

Meta classes work differently in Py2 (__metaclass__ field) and Py3 (metaclass= argument in the class declaration). In an incompatible way of course. So yeah, this works best:

from future.utils import with_metaclass
class Foo(with_metaclass(FooType, BarType)):
    pass

And then there's this result of searching for except ImportError: in OctoPrint's code base:

  • Queue vs queue:
    try:
        # noinspection PyCompatibility
        import queue # Py3
    except ImportError:
        # noinspection PyCompatibility
        import Queue as queue # Py2
    
  • Chainmap:
    try:
        # noinspection PyCompatibility
        from collections.abc import ChainMap # Py3, built-in
    except ImportError:
        # noinspection PyCompatibility
        from chainmap import Chainmap # Py2, external dependency
    
  • KeysView:
    try:
        # noinspection PyCompatibility
        from collections.abc import KeysView # Py3
    except ImportError:
        # noinspection PyCompatibility
        from collections import KeysView # Py2
    
  • scandir & walk:
    try:
        # noinspection PyCompatibility
        from os import scandir, walk # Py3, built-in
    except ImportError:
        # noinspection PyCompatibility
        from scandir import scandir, walk # Py2, external dependency
    
  • HTMLParser
    try:
        # noinspection PyCompatibility
        from html.parser import HTMLParser # Py3
    except ImportError:
        # noinspection PyCompatibility
        from HTMLParser import HTMLParser # Py2
    
  • HTTPResponse
    try:
        # noinspection PyCompatibility
        from http.client import HTTPResponse # Py3
    except ImportError:
        # noinspection PyCompatibility
        from httplib import HTTPResponse # Py2
    
  • HTTPServer & BaseHTTPRequestHandler:
    try:
        # noinspection PyCompatibility
        from http.server import HTTPServer, BaseHTTPRequestHandler # Py3
    except ImportError:
        # noinspection PyCompatibility
        from BaseHTTPServer import HTTPServer, BaseHTTPRequestHandler # Py2
    
  • urlencode
    try:
        # noinspection PyCompatibility
        from urllib.parse import urlencode # Py3
    except ImportError:
        # noinspection PyCompatibility
        from urllib import urlencode # Py2
    
  • urllib.unquote vs urllib.parse.unquote (same goes for urlencode):
    try:
        # noinspection PyCompatibility
        from urllib.parse import unquote # Py3
    except:
        # noinspection PyCompatibility
        from urllib import unquote # Py2
    
  • urlparse :crazy_face:
    try:
        # noinspection PyCompatibility
        from urllib import parse as urlparse # Py3
    except ImportError:
        # noinspection PyCompatibility
        import urlparse # Py2
    
  • collections.Iterable and Python 3.8 (!):
    try:
        from collections import Iterable
    except ImportError:
        # Python >= 3.8
        from collections.abc import Iterable
    

Now ain't that fun?

2 Likes

Wow, look at you. I'm reminded of the ski instructor from South Park. I should write a CLI called ski_instructor which takes a Py2 file as an argument, looks for those and then reports back warnings in his transcript.

I need to study this thread more closely but with only one 3D printer, I need to understand how to switch back and forth between Py2 and Py3 as some of the plugins I have installed are "required" for successful printing but I need a Py3 environment to migrate, test, and debug the plugin(s) I have written.

Another thing I think would be desirable is an enhancement to the Plugin Manager that can display the Py2/Py3 status of each plugin. This would make it easier for me to determine when my "required" subset of plugins will work in the Py3 environment (or if I need to "adopt" some to move things along).

1 Like

The technique I'd use would be to iteratively visit the plugin's repository and look for the tell-tale signature by searching their entire repository for: __plugin_pythoncompat__ and look for...

__plugin_pythoncompat__ = ">=2.7,<4"

Even better:

https://plugins.octoprint.org/plugins.json

Look for compatibility -> python.

Py3

Of 190 plugins in the .json, 38 are compatible with Python3.

I just wrote something. If you have NodeJS installed on your workstation...

cd sites # wherever you store projects
mkdir testplugins
cd testplugins
npm init
npm install
npm install --save octo-client
# edit node_modules/octo-client/config.js with your API key!
touch index.js
# --- Contents follow
const https = require('https');
const OctoPrint = require('octo-client');

String.prototype.isNotInList = function() {
  let value = this.valueOf();
  for (let i = 0, l = arguments[0].length; i < l; i += 1) {
    if (arguments[0][i] === value) return false;
  }
  return true;
}

function fetchPluginJSON(callback) {
  var url = 'https://plugins.octoprint.org/plugins.json'
  // id and compatibility.python
  https.get(url, function(resp) {
    let data = '';
    resp.on('data', (chunk) => {
      data += chunk;
    });
    resp.on('end', function() {
      callback(true, data);
    });
  }).on("error", function(err) {
    callback(false, err.message);
  });
}

OctoPrint.settings(function(opResponse){
  var listBundled = [
    'action_command_prompt',
    'announcements',
    'discovery',
    'errortracking',
    'pi_support',
    'pluginmanager',
    'softwareupdate',
    'tracking'
  ];
  var jsonUrlPlugins = {}

  fetchPluginJSON(function(wasSuccessful, urlJsonDataOrError) {
    if (wasSuccessful) {
      jsonUrlPlugins = urlJsonDataOrError;
      jsonOpList = opResponse.plugins;
      var listOpNames = Object.keys(jsonOpList);
      var installed3rdPartyPlugins = listOpNames.filter(function(name) {
        return name.isNotInList(listBundled);
      });
      var specificItem = {};
      var version = '';
      installed3rdPartyPlugins.forEach(function(name) {
        specificItem = {};
        version = '';
        specificItem = JSON.parse(jsonUrlPlugins).filter(function(entry) {return entry.id == name;});
        if (specificItem.length == 0) {
          console.log(name + ': (?)' + ' '.repeat(30 - name.length) + ' not in public list (unknown status)');
        } else {
          version = specificItem[0].compatibility.python;
          if (version.indexOf('<4') > -1) {
            console.log(name + ': (YES)' +  ' '.repeat(28 - name.length) + ' Python 3 compatible [' + version + ']');
          } else {
            version = specificItem[0].compatibility.python;
            console.log(name + ': (NO)' +  ' '.repeat(29 - name.length) + ' not Python 3 compatible [' + version + ']');
          }
        }
      });
    } else {
      console.log('Plugin lookup error: ' + urlJsonDataOrError);
    }
  });
});

# --- End of index.js contents

node . # Run the report

unpublishedplugin: (?)              not in public list (unknown status)
stlviewer: (YES)                    Python 3 compatible [>=2.7,<4]
screensquish: (NO)                  not Python 3 compatible [>=2.7,<3]

A little bit more context about your program:

DisplayLayerProgress: (NO)          not Python 3 compatible [>=2.7,<3]
PrintJobHistory: (?)                not in public list (unknown status)

HINT: You need to enter your OctoPrint-Connection-Settings to node_modules/octo-client/config.js before executing.

My DLP-Plugin is listed as not compatible, this is not true!
Reason: I didn't know/realised that, beside

  • this statement __plugin_pythoncompat__ = ">=2.7,<4",
  • syntactically python3-sugar,
  • I also need to add this info into the plugin-description for the plugin-respository:
compatibility:
  python: ">=2.7,<4"

Btw. beside your already mentioned python 3 - statements on top of this thread, there is one thing missing myDict.hasKey(something) https://docs.python.org/3.1/whatsnew/3.0.html#builtins

I created a wiki-page about that stuff (at the moment it doesn't include much content): https://github.com/OllisGit/OctoPrint-KnowledgeBase/wiki/PluginPython23
@FormerLurker I added a short section about your gcode_parser.py, because I use this in my DryRun-Plugin..maybe it helps.

1 Like

Noted in the instructions, thanks.

And yeah, I think OctoPrint 1.4 under a Py3 environment would need the plugin repository to have that compatibility information for others to install it.

Yeah, without the compatibility setting in the plugin repository, the plugin manager won't show the plugins listed to install under a Py3 environment. They get filtered out.

Sorry about that, working on a lengthy write up as we speak to make all this clearer.

Needed to do it that way, otherwise the only way to have done this would have been either

  1. Offer incompatible plugins to the user for installation and have them run into errors during install - really bad user experience.
  2. Try to fetch that information from the net from every single OctoPrint instance on every single repository listing - completely unfeasible.

So just like the other compatibility information, it lives in there now. And I had to do the additional flag in the code base because otherwise stuff might explode on plugin load thanks to syntactic changes.

I've got a first version of a migration guide online here:

https://docs.octoprint.org/en/staging-maintenance/plugins/python3_migration.html

In the end I decided against a blog post and rather made this part of the documentation. Let me know what you think please!

I'm thinking about releasing 1.4.0 this Wednesday (March 4th), if things continue to look as well as they do right now. I want the migration guide to go out along with it, so the current timeline means I'll be able to incorporate any changes that come out of this thread until tomorrow ~18:00 CET

2 Likes

Sweet. I wrote yesterday that I thought you'd release this week. :slight_smile:

Don't forget /etc/default/octoprint, btw.

What about that? That's an OctoPi detail, not generic OctoPrint.

Hi @foosel,
IMHO the approach to include this important topic to the official documentation is the right way!
Now it is also possible to raise Pull-Request to improve the content (e.g typos or setting up PyCharm with two envs ;-)).

I like the idea that you include the setup of the two environments and also the checklist at the end.

But from my point of view one important thing is missing:

  • OctoPrint API-Breaking changes (maybe not the right words)

From a plugin developer perspective, of course I need to know/learn the differences between Pyt2/3, but I also need to know which OctoPrint-API/Hook/Payload behaves differently.

Currently I only know the OctoPrint LineProcessorStream-Hook, but maybe there is more.
My suggestion is to put such findings also to this document in a separate section.

What do you think?
Olli

BTW. I like the release date!!!

I would say that most of us begin with an OctoPi-imaged rig and modify it for Py3. So when you sudo service octoprint restart I'm guessing that it uses the indicated file for that configuration information and that points to the old ~/oprint location rather than the one you've described in the newly-created documentation.