How to retain plugins when updating/recreateing docker image?


#1

What is the problem?
I am currently switching to Docker to host my various in-house services and started working on OctoPrint. My testing includes making sure that relevant data is retained on the data volume when the image changes.

My issue is that OctoPrint seems to keep it's configuration, but loses all plugins. They're just "gone", but you can still see traces of them in the config.yaml (update check) and some folders they create in /data.

What did you already try to solve it?
Try both ocroprint/octoprint and nunofgs/octoprint and searched forums to no avil.

Additional information about your setup

Current 1.3.9 release


#2

Before running your updates, you might try running the following from the Raspberry:

cat ~/.octoprint/logs/octoprint.log |grep "|" | grep "= /home/pi/" | grep -v bundled | sort | uniq | sed 's|\|||g'> ~/.octoprint/current-plugins.txt

...followed by this from your workstation:

scp pi@octopi.local:~/.octoprint/current-plugins.txt ~/.

It should result in a list of your plugins (to be reinstalled after the upgrade).


#3

While I appreciate the workaround I think it would be much more beneficial for all Docker users to have the plugins on a persistent volume and not do this chore every update.

Are so few ppl using Docker that his never popped up?

If I find the time I'll try to understand where the lost data is saved and post a solution here.


#4

I would venture to say that most people just flash the octopi image to a sd card and aren't using docker. I would love a solution to backup/restore my installed plugins for those times when I do end up re-flashing my sd card. @OutsourcedGuru, I though you were building something along those lines?


#5

@jneilliii Yes, it's still there in OctoPi-upgrade-helper which is where I just cribbed the commands from a moment ago. I use it. It works for me.


#6

@TheFitzZZ You could be one of perhaps two people that I know of who are using Docker. I must admit that I'm late to the Docker fan craze, having only recently added it to my MacBook. It seems to want more of my attention that I have available at the moment in order to fully appreciate it.

But my take on it is that it essentially works in a way which is the equivalent of a VM. If you need to persist a portion of a Docker image then you might consider pushing the ~/.octoprint folder over to a USB drive and then removing it from the original, then creating an equivalent symlink where it was that points to the USB drive. Potentially, this might work.


#7

I as well just started using docker and am replacing my Raspis with an old W520 workstation laptop. With docker I can host a wide variety of my home automation and now printing services.

The Container also has /home/octoprint/.octoprint on a persistent volume - hence my remark that the configuration itself is .. well persistent :slight_smile: I'll try to find out what's missing and fix it for the two ppl using OctoPrint on docker haha :wink:

Thanks for your input guys!


#8

Small update: using docker diff I was able to see changed data compared to the original. And there it was:

A /opt/octoprint/venv/lib/python2.7/site-packages/octoprint_bedlevelvisualizer
A /opt/octoprint/venv/lib/python2.7/site-packages/octoprint_bedlevelvisualizer/static
A /opt/octoprint/venv/lib/python2.7/site-packages/octoprint_bedlevelvisualizer/static/img
A /opt/octoprint/venv/lib/python2.7/site-packages/octoprint_bedlevelvisualizer/static/img/settings_general.png
A /opt/octoprint/venv/lib/python2.7/site-packages/octoprint_bedlevelvisualizer/static/js
A /opt/octoprint/venv/lib/python2.7/site-packages/octoprint_bedlevelvisualizer/static/js/bedlevelvisualizer.js
A /opt/octoprint/venv/lib/python2.7/site-packages/octoprint_bedlevelvisualizer/static/js/plotly-latest.min.js
A /opt/octoprint/venv/lib/python2.7/site-packages/octoprint_bedlevelvisualizer/templates
A /opt/octoprint/venv/lib/python2.7/site-packages/octoprint_bedlevelvisualizer/templates/bedlevelvisualizer_settings.jinja2
A /opt/octoprint/venv/lib/python2.7/site-packages/octoprint_bedlevelvisualizer/templates/bedlevelvisualizer_tab.jinja2
A /opt/octoprint/venv/lib/python2.7/site-packages/octoprint_bedlevelvisualizer/templates/bedlevelvisualizer_wizard.jinja2
A /opt/octoprint/venv/lib/python2.7/site-packages/octoprint_bedlevelvisualizer/__init__.py
A /opt/octoprint/venv/lib/python2.7/site-packages/octoprint_bedlevelvisualizer/__init__.pyc

So I need to find a smooth way to reinstall the missing python packages when the container is replaced/updated.

Will keep you posted.


#9

Nice, one of my plugins!


#10

While the IT guy in me want's a more advanced fix using a dynamically created requirements.txt that is used to reinstall all missing packages on a container update/redeployment, I opted for the next best thing: having the site packages on a persistent volume:

octoprint-packages:/opt/octoprint/venv/lib/python2.7/site-packages

As long as I keep the config and package volume together, things should be fine.

Good enough for me, cheers guys :slight_smile:


#11

Not sure exactly how all that works but ~/.octoprint seems to be the location of all the things you reasonably do during a session of OctoPrint. Most critically, the config.yaml under this folder is important since it's the sum of all your settings.


#12

As I already stated two times, that directory and the configuration were mapped to a volume by default (as defined by the docker file) - but that is not enough. When it doesn't find the missing packages, they don't magically reinstall themselves.

So, as outlined in my last post, either one maps the site-packages to a persistent volume or the current docker implementation is extended in a proper way to handle this.


#13

Just to raise my hand as another person using Docker with Octoprint, in my case basically to simplify running multiple Octoprint instances on my server for all of the various printers. I'm pretty much using a docker volume work-around as well at the moment.