GitFiles plugin (v1.1.2 available)

Development
1

I've just created what I hope will be a useful plugin. It's called OctoPrint-GitFiles. For those who want to maintain the latest sliced copy of their files, this will basically do a git pull behind-the-scenes.

Note that it probably won't work with Windows since it uses a symlink (at the moment). This feels like a cleaner approach than to run a git init directly in ~/.octoprint/uploads itself.

GitFiles (currently v1.0.4)

  1. Repository: Follow the instructions on the README.md found there. This version doesn't yet work for OctoPrint installed on a Windows server so hold off on that platform. I've tested on OSX/Raspbian and it's behaving nicely so far.
  2. You'll need to create a repository on Github and to place at least one gcode file in it. I wouldn't suggest putting anything other than a README.md, a .gitignore and possibly a LICENSE file in your repository other than gcode files themselves.

You can always tail the octoprint.log if you want to see anything happening when you press the new button.

tail -f .octoprint/logs/octoprint.log

For foosel, earlier conversation:

  1. I wanted to have you review the approach which is adding an ~/.octoprint/gitfiles folder and then creating a symlink as ~/.octoprint/uploads/github. It then looks like this in the Files side panel:

30%20AM

I don't think that OctoPrint upgrades later should do any harm to the uploads folder so it should be safe enough.

  1. Secondly, I need some help with the code. Here on line 12 I had to remove self. before the function name so that I could find it in the same file on line 49 where I'm binding it to the button. I'm not sure what's going on with the ViewModel, obviously. It works in this version; it just feels like a hack to throw the function into a higher scope.

Be kind—it's my first published plugin on OctoPrint.

1 Like
2

Sweet, I'll check it out soon.

3

Nice idea. I think it would be smart to make it bi-directional. You mention on the readme that if you upload a file via octoprint it will cause issues pulling, by making it bidirectional it would eliminate that and also give the added benefit of backing up your locally uploaded files. I think there are hooks for file upload that you could use for triggering the push.

4

Regarding your view model issue, the problem is that the button you are inserting there is bound to the FilesViewModel (since you are injecting it into an area controlled by that).

In cases where you are inserting buttons or other interactive elements into parts of the UI that are already under the control by another model I'd actually suggest to go with vanilla JS - so a simply click handler defined via jquery. Makes it way less of a hassle to make that button do what you want and you can declare the target either on the view model or just make it a private function within the model's scope.

In your backend code, instead of grabbing settings() you can also just use self._settings.global_get_basefolder - you implemented the SettingsPlugin mixin, so you have _settings injected and that is a PluginSettings instance.

For the file creation stuff, I'd suggest to use Python core functions as much as possible instead of spawning shell commands - take a look at the os module, especially the "File and Directories" methods, those should cover you and only leave the two git commands to handle via sarge or subprocess. Btw - wouldn't it be easier to just run one clone command instead of initializing an empty repo and setting the remote?

Now... in general this approach is a bit on the hacky side due to the file processing and metadata management that OctoPrint does in the uploads folder - it's really not prepared for any external modification of the stuff that lies there, even though I have tried my best to make it resilient as much as possible against something like that happening.

It might happen that it will rename files (e.g. spaces get replaced with _). It will also want to analyse any uploaded files, and if you don't automatically trigger that after a pull I think it will only detect any new files on the next server startup and then possibly have a huge backlog of stuff to analyse. I might be mistaken here though, not sure right now how intelligent I made the file list creation.

5

Thanks for the feedback. I included a delayed push of the Files side panel refresh button to hopefully kick off any analysis. It's good that you mentioned the renaming; I wouldn't have known about that and will include a warning in the README.md.

Regarding "why not just clone?", it's probably a result of my baby-step approach of just trying to see activity happening from this JavaScript->Python pipeline that's a little foreign to me. I've surprised myself that I finally was able to get something server-side to listen to my attempts.

I'll spend some time this weekend modifying it with your suggestions. Thanks, again.

Question: If I switch to the os.symlink() version, will this work on Windows?

6

No, sadly not, os.symlink is only available on UNIX systems, at least according to the docs.

7

Can you do a quick test with mklink to see if it will work as expected on Windows?

mklink /D sourceDirectory destinationLink
8

I teach software development students. (One that's just graduated is a former plumber-turned-JavaScript-coder now by going through the curriculum/program.) All this git push/pull stuff is perhaps the hardest thing for them to grasp and especially in the areas of merging.

Your average OctoPrint user doesn't do development. Your average person running a 3D printer farm or production shop who could use this... probably does just enough coding to understand the basic flow patterns of git for managing files. I guess I'm trying to make this easy with a one-way flow valve for them.

Anyone who's savvy enough to think bidirection git is probably clever enough to have jumped to the command prompt and run the raw git commands themselves, I think I'd argue at this point.

What you're describing would have some merit in a 3D printer manufacturer's design/testing departments. Each person probably has one or more printers which they test prototypes on and then, when approved, they might push to the public repository of available files for their own customers. So here, I'd bring back up that 80/20 rule for open source: offer the part of the code which makes 80% of the people happy (for free) and then charge a contractor's wages to build the other 20% to a corporation which would need that level of support.

9

So, to break this down for the dummies, of which I am going to proclaim myself the king, what we would be doing, is when you upload a file to print, we already have the option of saving it to SD, or to a folder, but, now we would also be able to upload to a github repository, and then be able to select a file FROM the github to be printed at a later date

My question, as self proclaimed king of the dummies, is, would this be a public repo, or would we each have to create our own ?

And then, how would that programming work with the > path to file ?

It would seem that a public repo would be easier to program, although ridiculously insecure, while a private one, while being more secure, might be programatically... difficult

Unless, of course, you're not using octoprint to do the original uploading, which would solve one security issue, the other would be that, I'm not sure I want all my prints available to the public. I mean, isn't github basically public ?

10

Imagine that you're in a shop and you have orders to fulfill. Let's say that you have a cool Raspi case project for a client and there are two parts to print for it.

Over time, you probably go through revisions with the client. They want the lettering to be this high, let's say. You change the stl, you slice to a new gcode. You push to the printer.

You now get a second printer (same type) and you're now printing off 300 cases per month as fast as you can. You want to always make sure that your two printers have the very latest copy of anything.

Fast-forward a few months and you have lots of projects, each with their respective parts and it's too difficult to manage the versioning.

.-------------------------------------------------.
| Local git-controlled folder on your workstation |
`-------------------------------------------------'

:arrow_down: git add . + git commit + git push

.-------------------.
| github repository |
`-------------------'

:arrow_down: git pull (as implemented in this plugin by just pressing the button)

.-----------.
| OctoPrint |
`-----------'

Check out the simple layout in my own repository for an example.

As for the public aspect of this, you could always either have a private github repository or run your own github-compatible server. I'm running a Gogs server at home (unlimited private repositories obviously) but I'll likely push it to Linode.com for safety's sake at some point. When I move this to the cloud I'll probably change over to Gitea instead since it's better maintained.

Let me know if you're interested in paying github for a private repository and we could talk about the adjustments necessary to your ~/.octoprint/gitfiles/.git/config file for this.

11

GitFiles v1.1.1

I decided to move the location of the local repository on the Raspberry since OctoPrint doesn't seem to like it if it's under the ~/.octoprint umbrella, complaining in the octoprint.log file. It's now in the pi user's home directory.

I've now published it and did the PR and that's passed inspection. It should be available soon enough after the merge.

12

GitFiles v1.1.2

Unfortunately, I implemented the earlier suggestions from foosel but I noticed an ugly side-effect: Switching to self._settings... did not result in persisting the changes to config.yaml as my earlier approach had done. So I had to make a quick-and-dirty upgrade to compensate for this before too many people had installed this.

Secondly, the safety approach of trying to move the repository to the user's home directory didn't ultimately work since some of the OctoPrint code doesn't want to walk through that symlink. I'm thinking that the analyzer code wasn't happy. So this version moves the repository under uploads and gets rid of the symlink approach completely; it should now be compatible with a Windows server, on the plus side of things.

If you installed the v1.1.1 version let me know. Thanks.

13

Ok, I am going to take Spyder's place in the "one more time for the dummies" on how to drive this...

I have set up a github repository on github.com

On my pi, I can see recent gcode files in ~/.octoprint/uploads

Here is where I come a little "unstuck" in the first note in this thread you say.

which implies that one shouldn't do a git init in that directory.

Yet going to the "workflow" section where it says (I presume manually, to get things started) to do a git add . - it says that that directory is not a git repository...

Is there a setup step that I have missed, or did I just misinterpret the comment about avoiding git inits...

14

The Settings page includes the workflow that I'd guessing you're quoting here and I'll describe it in real-scenario terms for you.

I have a MacBook and most of my github-related work in in the ~/sites folder area. Since I want to use git to control my 3D file, I create a folder under here for that purpose:

  • mkdir ~/sites/3d-files

I copy the first draft of my favorite gcode files into this folder. I'd probably do this with Finder. Next, I have to tell the folder that it's to be tracked with git so I initialize it:

  • cd ~/sites/3d-files
  • git init

This will create a hidden database. You can see a bit of what's going on by printing one of those files:

  • cat .git/config

Note how it looks now. It will change after I create a github repository and connect to that. But first, I haven't staged those files nor have I committed them. Until I do that, they won't be pushed to a remote repository:

  • git add .
  • git commit -m "Initial push"

I'll need a github repository so that OctoPrint can see it. In github, I log into my account and select the button to create a new repository. I name it "3d-files", give it a description, make sure that it's marked public and press OK. Ignore the other stuff. Github then shows me a screen of information; at the bottom of that are two commands and I'll need to press the button which allows me to copy those. Then—making sure that you're still in that same folder in the command console—I need to paste those two commands, noting that your version would have your github ID instead of mine.

  • git remote add origin https://github.com/OutsourcedGuru/3d-files.git
  • git push -u origin master

Now, it would be necessary to install the plugin, visit the Settings page and to enter the repository's URL, making sure that I press the Save button:

"https://github.com/OutsourcedGuru/3d-files.git"

I then return to the home page, go to the Files side panel widget and press the Octocat button. After a few seconds, a gitfiles folder should show up and it should have those gcode files in it.

So now, that Workflow section merely talks about what you do when you make edits to those gcode files:

  1. Go to that folder again on your workstation
  2. Copy in the new version(s) of those files
  3. git add .
  4. git commit -m "Fixed whatever part"
  5. git push
  6. In OctoPrint, push the button again