Migrating plugins to Python 2 & 3 compatibility - experiences?

Wow, look at you. I'm reminded of the ski instructor from South Park. I should write a CLI called ski_instructor which takes a Py2 file as an argument, looks for those and then reports back warnings in his transcript.

I need to study this thread more closely but with only one 3D printer, I need to understand how to switch back and forth between Py2 and Py3 as some of the plugins I have installed are "required" for successful printing but I need a Py3 environment to migrate, test, and debug the plugin(s) I have written.

Another thing I think would be desirable is an enhancement to the Plugin Manager that can display the Py2/Py3 status of each plugin. This would make it easier for me to determine when my "required" subset of plugins will work in the Py3 environment (or if I need to "adopt" some to move things along).

1 Like

The technique I'd use would be to iteratively visit the plugin's repository and look for the tell-tale signature by searching their entire repository for: __plugin_pythoncompat__ and look for...

__plugin_pythoncompat__ = ">=2.7,<4"

Even better:

https://plugins.octoprint.org/plugins.json

Look for compatibility -> python.

Py3

Of 190 plugins in the .json, 38 are compatible with Python3.

I just wrote something. If you have NodeJS installed on your workstation...

cd sites # wherever you store projects
mkdir testplugins
cd testplugins
npm init
npm install
npm install --save octo-client
# edit node_modules/octo-client/config.js with your API key!
touch index.js
# --- Contents follow
const https = require('https');
const OctoPrint = require('octo-client');

String.prototype.isNotInList = function() {
  let value = this.valueOf();
  for (let i = 0, l = arguments[0].length; i < l; i += 1) {
    if (arguments[0][i] === value) return false;
  }
  return true;
}

function fetchPluginJSON(callback) {
  var url = 'https://plugins.octoprint.org/plugins.json'
  // id and compatibility.python
  https.get(url, function(resp) {
    let data = '';
    resp.on('data', (chunk) => {
      data += chunk;
    });
    resp.on('end', function() {
      callback(true, data);
    });
  }).on("error", function(err) {
    callback(false, err.message);
  });
}

OctoPrint.settings(function(opResponse){
  var listBundled = [
    'action_command_prompt',
    'announcements',
    'discovery',
    'errortracking',
    'pi_support',
    'pluginmanager',
    'softwareupdate',
    'tracking'
  ];
  var jsonUrlPlugins = {}

  fetchPluginJSON(function(wasSuccessful, urlJsonDataOrError) {
    if (wasSuccessful) {
      jsonUrlPlugins = urlJsonDataOrError;
      jsonOpList = opResponse.plugins;
      var listOpNames = Object.keys(jsonOpList);
      var installed3rdPartyPlugins = listOpNames.filter(function(name) {
        return name.isNotInList(listBundled);
      });
      var specificItem = {};
      var version = '';
      installed3rdPartyPlugins.forEach(function(name) {
        specificItem = {};
        version = '';
        specificItem = JSON.parse(jsonUrlPlugins).filter(function(entry) {return entry.id == name;});
        if (specificItem.length == 0) {
          console.log(name + ': (?)' + ' '.repeat(30 - name.length) + ' not in public list (unknown status)');
        } else {
          version = specificItem[0].compatibility.python;
          if (version.indexOf('<4') > -1) {
            console.log(name + ': (YES)' +  ' '.repeat(28 - name.length) + ' Python 3 compatible [' + version + ']');
          } else {
            version = specificItem[0].compatibility.python;
            console.log(name + ': (NO)' +  ' '.repeat(29 - name.length) + ' not Python 3 compatible [' + version + ']');
          }
        }
      });
    } else {
      console.log('Plugin lookup error: ' + urlJsonDataOrError);
    }
  });
});

# --- End of index.js contents

node . # Run the report

unpublishedplugin: (?)              not in public list (unknown status)
stlviewer: (YES)                    Python 3 compatible [>=2.7,<4]
screensquish: (NO)                  not Python 3 compatible [>=2.7,<3]

A little bit more context about your program:

DisplayLayerProgress: (NO)          not Python 3 compatible [>=2.7,<3]
PrintJobHistory: (?)                not in public list (unknown status)

HINT: You need to enter your OctoPrint-Connection-Settings to node_modules/octo-client/config.js before executing.

My DLP-Plugin is listed as not compatible, this is not true!
Reason: I didn't know/realised that, beside

  • this statement __plugin_pythoncompat__ = ">=2.7,<4",
  • syntactically python3-sugar,
  • I also need to add this info into the plugin-description for the plugin-respository:
compatibility:
  python: ">=2.7,<4"

Btw. beside your already mentioned python 3 - statements on top of this thread, there is one thing missing myDict.hasKey(something) https://docs.python.org/3.1/whatsnew/3.0.html#builtins

I created a wiki-page about that stuff (at the moment it doesn't include much content): https://github.com/OllisGit/OctoPrint-KnowledgeBase/wiki/PluginPython23
@FormerLurker I added a short section about your gcode_parser.py, because I use this in my DryRun-Plugin..maybe it helps.

1 Like

Noted in the instructions, thanks.

And yeah, I think OctoPrint 1.4 under a Py3 environment would need the plugin repository to have that compatibility information for others to install it.

Yeah, without the compatibility setting in the plugin repository, the plugin manager won't show the plugins listed to install under a Py3 environment. They get filtered out.

Sorry about that, working on a lengthy write up as we speak to make all this clearer.

Needed to do it that way, otherwise the only way to have done this would have been either

  1. Offer incompatible plugins to the user for installation and have them run into errors during install - really bad user experience.
  2. Try to fetch that information from the net from every single OctoPrint instance on every single repository listing - completely unfeasible.

So just like the other compatibility information, it lives in there now. And I had to do the additional flag in the code base because otherwise stuff might explode on plugin load thanks to syntactic changes.

I've got a first version of a migration guide online here:

https://docs.octoprint.org/en/staging-maintenance/plugins/python3_migration.html

In the end I decided against a blog post and rather made this part of the documentation. Let me know what you think please!

I'm thinking about releasing 1.4.0 this Wednesday (March 4th), if things continue to look as well as they do right now. I want the migration guide to go out along with it, so the current timeline means I'll be able to incorporate any changes that come out of this thread until tomorrow ~18:00 CET

2 Likes

Sweet. I wrote yesterday that I thought you'd release this week. :slight_smile:

Don't forget /etc/default/octoprint, btw.

What about that? That's an OctoPi detail, not generic OctoPrint.

Hi @foosel,
IMHO the approach to include this important topic to the official documentation is the right way!
Now it is also possible to raise Pull-Request to improve the content (e.g typos or setting up PyCharm with two envs ;-)).

I like the idea that you include the setup of the two environments and also the checklist at the end.

But from my point of view one important thing is missing:

  • OctoPrint API-Breaking changes (maybe not the right words)

From a plugin developer perspective, of course I need to know/learn the differences between Pyt2/3, but I also need to know which OctoPrint-API/Hook/Payload behaves differently.

Currently I only know the OctoPrint LineProcessorStream-Hook, but maybe there is more.
My suggestion is to put such findings also to this document in a separate section.

What do you think?
Olli

BTW. I like the release date!!!

I would say that most of us begin with an OctoPi-imaged rig and modify it for Py3. So when you sudo service octoprint restart I'm guessing that it uses the indicated file for that configuration information and that points to the old ~/oprint location rather than the one you've described in the newly-created documentation.

The thing is, I made it a conscious effort not to have any breaking changes on the API. What you are referring to here isn't a breaking change in OctoPrint, it's a breaking change in Python 3 that affects a helper class provided by OctoPrint as that inherits from a core Python class. I'll make a mention of it in the bytes vs string section though to give it some more visibility. It's also the only thing that I'm aware of at present.

edit Added:

I sincerely hope that that's not how most plugin developers start given that the tutorial explicitly walks people through setting up a proper local development environment. Developing directly on a Pi is just painful.

I also make it a point to start OctoPrint manually from shell in the migration guide, in the corresponding venv, via the octoprint executable. I'd rather not want to confuse people and start with OctoPi details there, especially not since the guide isn't about how to migrate your OctoPi image to using Python 3, but rather how to migrate your OctoPrint plugins to Python 3... I'll think about adding a footnote though.

edit Added, and not as a footnote:

2 Likes

Things I had todo for TouchUI.

First off; migration docs where helpfull!

Second, this is what I had to fix:

  1. basestring no longer supported, update crossdomain decorator (/w backwards compatibly) with the following
try:
  basestring
except NameError:
  basestring = str
  1. Hashing now requires encoding:
    Old: hashlib.md5(contentFile.read())
    New: hashlib.md5(contentFile.read().encode('utf-8'))

That was it for me.

1 Like

@foosel, I just tried to follow your Migrating to Python 3 and I'm still a little confused...

I have a working OctoPi 0.17.0 / OctoPrint 1.4.0 with installed plugins. I believe this "virtual environment is "~/oprint". I have created a venv3, activated it, and installed OctoPrint.

Maybe I'm too old to read between the lines but the next step is, "create an editable install of your plugin, start the server and start testing" and for me, that leaves out a step (or more) before like "shutdown your current OctoPrint environment" and maybe a step after "to return to your original OctoPrint environment".

I think maybe the second note (If you want to migrate your existing OctoPrint install) might be more relevant to what I am trying to accomplish, have both environments available and switch between them as needed. Adding the "switch" commands would be very useful, IMO.

1 Like

For my new Version of DLP I need to handle different import-statements. My solution looks like this:

import sys
is_py2 = sys.version[0] == '2'
if is_py2:
    import Queue as queue
else:
    import queue as queue

@foosel Maybe it could be a part of the offical-doc...MergeRequest against this branch, right? https://github.com/OctoPrint/OctoPrint/blob/40a0913fa440e8c9618184729d27145f13a3f846/docs/plugins/python3_migration.rst

Personally I'd rather recommend this pattern:

try:
    import queue
except ImportError:
    import Queue as queue

See also

But yeah, it might make sense to add an example of that to the guide. maintenance branch, path was correct.

Dear Santa,

For Christmas I would like both Python 2 and Python 3 to have identical imports for queue.

Thanks,
OG