Migrating plugins to Python 2 & 3 compatibility - experiences?

foosel · February 28, 2020, 2:10pm

Took me way too long to get back to my own topic thanks to the new RC - sorry for that

You can actually skip a ton of these steps - or at least it worked fine on my end to do that... virtualenv has an argument --python with which you can specify the Python interpreter with which to create the venv. So it boils down to installing both 2.7 and 3.7 somewhere and then telling virtualenv which interpreter to use:

virtualenv --python=C:/Python27/python.exe venv27
virtualenv --python=C:/Python37/python.exe venv37

Then just activate whichever one you need, or - that's how I do it - configure both interpreters in your IDE and then switch there. The only bad thing about the latter is that at least my (slightly outdated) version of PyCharm always needs an extra minute to get back on track after that, and I have to make sure to close all active run/debug panels relating to the old Python environment or it doesn't properly switch

Does that support 2 and 3 concurrently? I always took it to be more a case of "once that's run the code won't run under Py2 anymore", which at least for now would be counter productive.

What I've made good use of is this cheat sheet by the Python-Future project:

https://python-future.org/compatible_idioms.html

And in general it seems to be a good idea to turn on Py3 behaviour as far as possible in your source files through various from __future__ imports. E.g. (from OctoPrint's code base):

# -*- coding: utf-8 -*-
from __future__ import absolute_import, division, print_function, unicode_literals

I also have a post-it note attached to my main screen for more than a year now that currently says

except Exception:

to_bytes & to_unicode

io.open

list(<generator>)

map

filter

...

__metaclass__

Explanation follows

except Exception:

Instead of just except: for broad catch-all blocks. Not Python 3 specific, but something we switched to in the code base with the Py3 changes.

to_bytes & to_unicode

Two helper methods built into OctoPrint (octoprint.util package) that should hopefully make it easier to ensure the expected data type. Will keep their hands off of anything that's already in the correct type.

io.open

Instead of open if I remember correctly, though I cannot for the life of me right now remember what exactly was special about that. Pretty sure it had something to do with encoding though (because that's the worst about this whole migration).

list(<generator>)

This is what @OutsourcedGuru already mentioned, a ton of the functional programming kinda methods in Python 3 are now returning generators instead of ready data, so if you want to do stuff with that later where it needs to be a list you'll need to convert yourself. Has kicked me in the teeth a couple of times now, one of the fixes that went into RC6 was actually another one of these.

__metaclass__

Meta classes work differently in Py2 (__metaclass__ field) and Py3 (metaclass= argument in the class declaration). In an incompatible way of course. So yeah, this works best:

from future.utils import with_metaclass
class Foo(with_metaclass(FooType, BarType)):
    pass

And then there's this result of searching for except ImportError: in OctoPrint's code base:

Queue vs queue:

try:
    # noinspection PyCompatibility
    import queue # Py3
except ImportError:
    # noinspection PyCompatibility
    import Queue as queue # Py2

Chainmap:

try:
    # noinspection PyCompatibility
    from collections.abc import ChainMap # Py3, built-in
except ImportError:
    # noinspection PyCompatibility
    from chainmap import Chainmap # Py2, external dependency

KeysView:

try:
    # noinspection PyCompatibility
    from collections.abc import KeysView # Py3
except ImportError:
    # noinspection PyCompatibility
    from collections import KeysView # Py2

scandir & walk:

try:
    # noinspection PyCompatibility
    from os import scandir, walk # Py3, built-in
except ImportError:
    # noinspection PyCompatibility
    from scandir import scandir, walk # Py2, external dependency

HTMLParser

try:
    # noinspection PyCompatibility
    from html.parser import HTMLParser # Py3
except ImportError:
    # noinspection PyCompatibility
    from HTMLParser import HTMLParser # Py2

HTTPResponse

try:
    # noinspection PyCompatibility
    from http.client import HTTPResponse # Py3
except ImportError:
    # noinspection PyCompatibility
    from httplib import HTTPResponse # Py2

HTTPServer & BaseHTTPRequestHandler:

try:
    # noinspection PyCompatibility
    from http.server import HTTPServer, BaseHTTPRequestHandler # Py3
except ImportError:
    # noinspection PyCompatibility
    from BaseHTTPServer import HTTPServer, BaseHTTPRequestHandler # Py2

urlencode

try:
    # noinspection PyCompatibility
    from urllib.parse import urlencode # Py3
except ImportError:
    # noinspection PyCompatibility
    from urllib import urlencode # Py2

urllib.unquote vs urllib.parse.unquote (same goes for urlencode):

try:
    # noinspection PyCompatibility
    from urllib.parse import unquote # Py3
except:
    # noinspection PyCompatibility
    from urllib import unquote # Py2

urlparse

try:
    # noinspection PyCompatibility
    from urllib import parse as urlparse # Py3
except ImportError:
    # noinspection PyCompatibility
    import urlparse # Py2

collections.Iterable and Python 3.8 (!):

try:
    from collections import Iterable
except ImportError:
    # Python >= 3.8
    from collections.abc import Iterable

Now ain't that fun?

OutsourcedGuru · February 28, 2020, 4:22pm

Wow, look at you. I'm reminded of the ski instructor from South Park. I should write a CLI called ski_instructor which takes a Py2 file as an argument, looks for those and then reports back warnings in his transcript.

b-morgan · February 28, 2020, 6:54pm

I need to study this thread more closely but with only one 3D printer, I need to understand how to switch back and forth between Py2 and Py3 as some of the plugins I have installed are "required" for successful printing but I need a Py3 environment to migrate, test, and debug the plugin(s) I have written.

Another thing I think would be desirable is an enhancement to the Plugin Manager that can display the Py2/Py3 status of each plugin. This would make it easier for me to determine when my "required" subset of plugins will work in the Py3 environment (or if I need to "adopt" some to move things along).

OutsourcedGuru · February 28, 2020, 7:27pm

The technique I'd use would be to iteratively visit the plugin's repository and look for the tell-tale signature by searching their entire repository for: __plugin_pythoncompat__ and look for...

__plugin_pythoncompat__ = ">=2.7,<4"

Even better:

https://plugins.octoprint.org/plugins.json

Look for compatibility -> python.

Py3

b-morgan · February 28, 2020, 7:43pm

Of 190 plugins in the .json, 38 are compatible with Python3.

OutsourcedGuru · February 28, 2020, 9:51pm

I just wrote something. If you have NodeJS installed on your workstation...

cd sites # wherever you store projects
mkdir testplugins
cd testplugins
npm init
npm install
npm install --save octo-client
# edit node_modules/octo-client/config.js with your API key!
touch index.js
# --- Contents follow
const https = require('https');
const OctoPrint = require('octo-client');

String.prototype.isNotInList = function() {
  let value = this.valueOf();
  for (let i = 0, l = arguments[0].length; i < l; i += 1) {
    if (arguments[0][i] === value) return false;
  }
  return true;
}

function fetchPluginJSON(callback) {
  var url = 'https://plugins.octoprint.org/plugins.json'
  // id and compatibility.python
  https.get(url, function(resp) {
    let data = '';
    resp.on('data', (chunk) => {
      data += chunk;
    });
    resp.on('end', function() {
      callback(true, data);
    });
  }).on("error", function(err) {
    callback(false, err.message);
  });
}

OctoPrint.settings(function(opResponse){
  var listBundled = [
    'action_command_prompt',
    'announcements',
    'discovery',
    'errortracking',
    'pi_support',
    'pluginmanager',
    'softwareupdate',
    'tracking'
  ];
  var jsonUrlPlugins = {}

  fetchPluginJSON(function(wasSuccessful, urlJsonDataOrError) {
    if (wasSuccessful) {
      jsonUrlPlugins = urlJsonDataOrError;
      jsonOpList = opResponse.plugins;
      var listOpNames = Object.keys(jsonOpList);
      var installed3rdPartyPlugins = listOpNames.filter(function(name) {
        return name.isNotInList(listBundled);
      });
      var specificItem = {};
      var version = '';
      installed3rdPartyPlugins.forEach(function(name) {
        specificItem = {};
        version = '';
        specificItem = JSON.parse(jsonUrlPlugins).filter(function(entry) {return entry.id == name;});
        if (specificItem.length == 0) {
          console.log(name + ': (?)' + ' '.repeat(30 - name.length) + ' not in public list (unknown status)');
        } else {
          version = specificItem[0].compatibility.python;
          if (version.indexOf('<4') > -1) {
            console.log(name + ': (YES)' +  ' '.repeat(28 - name.length) + ' Python 3 compatible [' + version + ']');
          } else {
            version = specificItem[0].compatibility.python;
            console.log(name + ': (NO)' +  ' '.repeat(29 - name.length) + ' not Python 3 compatible [' + version + ']');
          }
        }
      });
    } else {
      console.log('Plugin lookup error: ' + urlJsonDataOrError);
    }
  });
});

# --- End of index.js contents

node . # Run the report

unpublishedplugin: (?)              not in public list (unknown status)
stlviewer: (YES)                    Python 3 compatible [>=2.7,<4]
screensquish: (NO)                  not Python 3 compatible [>=2.7,<3]

OllisGit · February 29, 2020, 10:36am

A little bit more context about your program:

loading of https://plugins.octoprint.org/plugins.json
loads the list of your "local" installation
compares each list and output the result

DisplayLayerProgress: (NO)          not Python 3 compatible [>=2.7,<3]
PrintJobHistory: (?)                not in public list (unknown status)

HINT: You need to enter your OctoPrint-Connection-Settings to node_modules/octo-client/config.js before executing.

My DLP-Plugin is listed as not compatible, this is not true!
Reason: I didn't know/realised that, beside

this statement __plugin_pythoncompat__ = ">=2.7,<4",
syntactically python3-sugar,
I also need to add this info into the plugin-description for the plugin-respository:

compatibility:
  python: ">=2.7,<4"

Btw. beside your already mentioned python 3 - statements on top of this thread, there is one thing missing myDict.hasKey(something) https://docs.python.org/3.1/whatsnew/3.0.html#builtins

I created a wiki-page about that stuff (at the moment it doesn't include much content): https://github.com/OllisGit/OctoPrint-KnowledgeBase/wiki/PluginPython23
@FormerLurker I added a short section about your gcode_parser.py, because I use this in my DryRun-Plugin..maybe it helps.

OutsourcedGuru · February 29, 2020, 4:42pm

Noted in the instructions, thanks.

And yeah, I think OctoPrint 1.4 under a Py3 environment would need the plugin repository to have that compatibility information for others to install it.

jneilliii · February 29, 2020, 5:11pm

Yeah, without the compatibility setting in the plugin repository, the plugin manager won't show the plugins listed to install under a Py3 environment. They get filtered out.

foosel · March 2, 2020, 9:59am

Sorry about that, working on a lengthy write up as we speak to make all this clearer.

Needed to do it that way, otherwise the only way to have done this would have been either

Offer incompatible plugins to the user for installation and have them run into errors during install - really bad user experience.
Try to fetch that information from the net from every single OctoPrint instance on every single repository listing - completely unfeasible.

So just like the other compatibility information, it lives in there now. And I had to do the additional flag in the code base because otherwise stuff might explode on plugin load thanks to syntactic changes.

foosel · March 2, 2020, 12:17pm

I've got a first version of a migration guide online here:

https://docs.octoprint.org/en/staging-maintenance/plugins/python3_migration.html

In the end I decided against a blog post and rather made this part of the documentation. Let me know what you think please!

I'm thinking about releasing 1.4.0 this Wednesday (March 4th), if things continue to look as well as they do right now. I want the migration guide to go out along with it, so the current timeline means I'll be able to incorporate any changes that come out of this thread until tomorrow ~18:00 CET

OutsourcedGuru · March 2, 2020, 4:51pm

Sweet. I wrote yesterday that I thought you'd release this week.

Don't forget /etc/default/octoprint, btw.

foosel · March 2, 2020, 5:59pm

What about that? That's an OctoPi detail, not generic OctoPrint.

OllisGit · March 2, 2020, 6:31pm

Hi @foosel,
IMHO the approach to include this important topic to the official documentation is the right way!
Now it is also possible to raise Pull-Request to improve the content (e.g typos or setting up PyCharm with two envs ;-)).

I like the idea that you include the setup of the two environments and also the checklist at the end.

But from my point of view one important thing is missing:

OctoPrint API-Breaking changes (maybe not the right words)

From a plugin developer perspective, of course I need to know/learn the differences between Pyt2/3, but I also need to know which OctoPrint-API/Hook/Payload behaves differently.

Currently I only know the OctoPrint LineProcessorStream-Hook, but maybe there is more.
My suggestion is to put such findings also to this document in a separate section.

What do you think?
Olli

BTW. I like the release date!!!

OutsourcedGuru · March 2, 2020, 7:32pm

I would say that most of us begin with an OctoPi-imaged rig and modify it for Py3. So when you sudo service octoprint restart I'm guessing that it uses the indicated file for that configuration information and that points to the old ~/oprint location rather than the one you've described in the newly-created documentation.

foosel · March 3, 2020, 10:57am

The thing is, I made it a conscious effort not to have any breaking changes on the API. What you are referring to here isn't a breaking change in OctoPrint, it's a breaking change in Python 3 that affects a helper class provided by OctoPrint as that inherits from a core Python class. I'll make a mention of it in the bytes vs string section though to give it some more visibility. It's also the only thing that I'm aware of at present.

edit Added:

I sincerely hope that that's not how most plugin developers start given that the tutorial explicitly walks people through setting up a proper local development environment. Developing directly on a Pi is just painful.

I also make it a point to start OctoPrint manually from shell in the migration guide, in the corresponding venv, via the octoprint executable. I'd rather not want to confuse people and start with OctoPi details there, especially not since the guide isn't about how to migrate your OctoPi image to using Python 3, but rather how to migrate your OctoPrint plugins to Python 3... I'll think about adding a footnote though.

edit Added, and not as a footnote:

BillyBlaze · March 8, 2020, 3:26pm

Things I had todo for TouchUI.

First off; migration docs where helpfull!

Second, this is what I had to fix:

basestring no longer supported, update crossdomain decorator (/w backwards compatibly) with the following

try:
  basestring
except NameError:
  basestring = str

Hashing now requires encoding:
Old: hashlib.md5(contentFile.read())
New: hashlib.md5(contentFile.read().encode('utf-8'))

That was it for me.

b-morgan · March 30, 2020, 10:38pm

@foosel, I just tried to follow your Migrating to Python 3 and I'm still a little confused...

I have a working OctoPi 0.17.0 / OctoPrint 1.4.0 with installed plugins. I believe this "virtual environment is "~/oprint". I have created a venv3, activated it, and installed OctoPrint.

Maybe I'm too old to read between the lines but the next step is, "create an editable install of your plugin, start the server and start testing" and for me, that leaves out a step (or more) before like "shutdown your current OctoPrint environment" and maybe a step after "to return to your original OctoPrint environment".

I think maybe the second note (If you want to migrate your existing OctoPrint install) might be more relevant to what I am trying to accomplish, have both environments available and switch between them as needed. Adding the "switch" commands would be very useful, IMO.

OllisGit · April 8, 2020, 8:00pm

For my new Version of DLP I need to handle different import-statements. My solution looks like this:

import sys
is_py2 = sys.version[0] == '2'
if is_py2:
    import Queue as queue
else:
    import queue as queue

@foosel Maybe it could be a part of the offical-doc...MergeRequest against this branch, right? https://github.com/OctoPrint/OctoPrint/blob/40a0913fa440e8c9618184729d27145f13a3f846/docs/plugins/python3_migration.rst

foosel · April 9, 2020, 9:48am

Personally I'd rather recommend this pattern:

try:
    import queue
except ImportError:
    import Queue as queue