I have a few instances of Octoprint (latest) each running on a separate Win10 machine
(Yeah, W10... Ouch! please stop beating me!)
I want all those Octoprint instances to always be in sync.
More specifically, I wish to sync stuff like:
Some custom UI Controls frequently added - via config.yaml
Logging configuration modifications done via logging.yaml
Some GCode scripts (located at %AppData%\Octoprint\scripts\gcode)
Some Printer Profiles frequently modified (located at %AppData%\Octoprint\printerProfiles)
The latest files (.yaml and other) are located and updated at a private Github repo.
I thought I should write a simple plugin that compares (upon server startup) the installed vs. latest files and if needed fetches the latest and finally restarts the server.
Anyone encountered a similar need?
Maybe I am missing a much simpler solution?
Maybe someone knows of a similar plugin that might more easily be adapted to this kind of task?
There are other ways you could do it. You could have a windows serves (Mac and Linux user myself) as your repository as you say. Or you could store config in something like a git repository and then use a tool like Jenkins to push it out automatically when updated.
Or another approach would be to Docker, and Push out a new container image.
But if no knowledge of any of these the rsync would be easiest.
I have some familiarity with the all tools you mentioned.
All approaches sound feasible to me.
I think they all target this task as a “File system sync task” and not as a “Octoprint config sync task”.
(Which I agreed might indeed be a better approach)
But this approach means that the Octoprint server is oblivious to the underlying config file change.
Which means the following would probably need to be taken care of somehow:
Octoprint Server restart needs to take place for the new configs to take effect.
This is not so straightforward, as some of my Octoprint instances might actually be running (and printing…) I obviously cannot kill and restart a running instance.
Just counting on the next occasional server restart is obviously not a good idea.
If you were to create a plugin to sync the config, it could popup an announcement easily. Are you looking at syncing everything or just settings? If it was installed plugins it may be harder to sync, as they need to be actually installed in the environment.
My idea for this kind of thing would be a plugin that looked to a repository somewhere, and notified you if there were changes in an announcement, and you could update config & restart when you wanted. Maybe look at a plugin like this (OctoPrint-GitFiles) for inspiration?
The possibilities are limitless. You could create a Docker image that basically upon start pulled all its files from a repository, be it git, a file share of whatever, and updates could be flagged as safe, safe if idle, install and restart, manual etc.
Does something like Octofarm already do all this? Might be worth having a look.
I am aware that if I wrote a "config sync plugin" it could also easily handle the announcements and restarts issues.
(This is why at this moment writing a plugin still looks like an appealing approach)
For my specific needs I was not interested in syncing everything.
Although Syncing Plugins would be "nice to have" also in my case, it seemed to me like too much hassle (as you mentioned)
Will nevertheless look at OctoPrint-GitFiles
Looks relevant in either case.
Wow! will surely have a look at OctoFarm!
Although from first glance I couldn't spot any "migrate config" or similar. (Maybe missed it)
Anyhow, this gives a new direction to me.
Even if OctoFarm itself doesn't answer my needs, maybe building a simple REST API Octoprint client would do the trick with really minimal effort?
OctoFarm is being actively developed and I think there has been discussion of this type of config type import/export tools, but the concept of a plugin/git update/announcement scenario is very intriguing to me. It would be fairly easy to create a plugin that reacts to the onSettingsUpdated event and pushes a commit to a "master" github repo.of the config.yaml file with a version tag. Then in github create an action workflow that on commit of a tag it automatically creates a release, which would then trigger a plugin update on all the machines. The plugin update would then respond by deploying the new settingsl.
The only problem I see here is the uniqueness of configs between instances, because there is a salt/id generated during the initial setup wizard for hashing password and stuff, and if you push those changes to github and syn between instances there could be a security concern. You could just push specific non-unique parts of the settings to the github, and encrypt it in some way.
You'd have to figure out the logic to not get in an update loop too. Something like during update have a flag that ignores settings updates.
The specific plugin related stuff to look at in the docs are listed below....
get_settings_version: make this match the plugin version so on every update you can use the next item.
on_settings_migrate: use this to compare current installed version versus update version. This would do the work updating the settings.
So the way I do updates on all my Pi's is to use Ansible, you can get a free version from Red Hat.
I write a single YAML file which explains to Ansible what I want it to do, then run it against each machine in turn.
I have bespoke settings for proxies, which actually needs tweaking on Raspbian in about five places, DNS is ALWAYS wrong in nsswitch.conf and so on.
Oh yeah disk mount points as well.
I hadn't looked at installing plug-ins that way - but that would make life MUCH simpler.
Although having a fully automated process is surely the coolest thing, a manual process will do the job - for a small number of nodes and low frequency of changes - and I think might allow for more user control:
I now think of a plugin-based solution, something along these lines:
(Wonder if it would make sense to others. Please let me know…)
On each Octoprint instance
1.1. Install the “Conf Sync” Plugin.
1.2. Configure the plugin it with the repo URL and credentials.
Have one “master” instance
2.1. On the master instance configure the plugin as “master”
2.2. Only the master allows “pushing” (“publish”?) the config to the repo.
Note that I prefer it done “on demand” by a manual user operation, not by an onSettingsUpdated event.
Doing it manually allows, if wished, investigating the config changes on the master node before publishing it.
The “Slave” instances
3.1. Configure the plugin on each slave instance as “slave”
3.2. The slave plugin will routinely check for config updates.
3.3. A “slave” check policy could be (for example)
On each server startup, on each end-of-print, while idle every xxx minutes.
And there should also be a “check now” manual option, available via the Plugin UI.
3.4. If a newer config is identified by the check process – Announce.
User can now (again, manually) perform the “Update config now” operation whenever she likes to, or silence the announcement.
3.5. The Plugin UI tab should indicate things like:
Last check time, last known repo version, currently installed version.
3.6. The plugin version itself should also be synchronized, that is, if differs from the masters version, announce.
3.7. Maybe a special care should be taken wrt the first installation of the plugin.
Regarding the concerns you raised about the unique identifiers, keys etc., well, I didn't think of it. Sure looks like a problem.
Not sure what’s the best way to deal with it.
Maybe the “Update” functionality should skip all the yaml regions that deal with node identification.
Unfortunately I don't have an RPi4 8GB running the 64-bit OS to test it on, and AFAIK the 64-bit OS is still only in beta, but I imagine there would probably be some library impacts.
If you have one at your disposal, by all means give it a go and let me know how you get on. More than happy to update the Ansible role if there's a demand for Octoprint on 64-bit.
