Reduce the size of the uploads directory with overlay and squashfs

Showcase
1

Hi everyone, I like to keep the gcode files but my "uploads" directory grew quickly, so I decided to compress it and mount it as an overlay. For me was 237M vs 1,2G, so almost a gig of space recovered.

Here's a "quick and dirty" howto in case someone find this useful:

We need to install squashfs-tools.

sudo apt udpate
sudo apt install squashfs-tools

Then stop octoprint (maybe not necessary, but it does not hurt) and create the compressed file-system.

cd ~/.octoprint
mksquashfs uploads uploads.sfs -b 65536 -comp xz -Xdict-size 100%

This will take a while depending of the size of uploads directory.
Now create a directory to put all the stuff, move the uploads.sfs file there and create the directories needed for the overlay.

mkdir -p ~/.overlay/{upper,lower,work}
mv ~/.octoprint/uploads.sfs ~/.overlay/

Now it's time to edit /etc/fstab (change the "pi" user with whatever you have).

sudo nano -w /etc/fstab

Then add:

/home/pi/.overlay/uploads.sfs   /home/pi/.overlay/lower   squashfs   loop,ro   0 0

overlay /home/pi/.octoprint/uploads overlay x-systemd.requires=/home/pi/.overlay/lower,lowerdir=/home/pi/.overlay/lower,upperdir=/home/pi/.overlay/upper,workdir=/home/pi/.overlay/work 0 0

"x-systemd.requires=/home/pi/.overlay/lower" it's required because systemd don't respect the order in fstab, funny uh?
Now test the thing:

sudo mount /home/pi/.overlay/lower
sudo mount /home/pi/.octoprint/uploads

checking with "df -h" you should see something like this:

/dev/loop0       237M   237M     0 100% /home/pi/.overlay/lower
overlay           30G   4,0G   25G  14% /home/pi/.octoprint/uploads

Check if the files are there and then umount again the uploads directory. Erase the original files and gain the space recovered:

sudo umount /home/pi/.octoprint/uploads
rm /home/pi/.octoprint/uploads/* -rf
rm  /home/pi/.octoprint/uploads/.* -rf
sudo mount /home/pi/.octoprint/uploads

Files should be there again (hopefully :sweat_smile:).

Now from time to time or if the upper directory grow to much, we need to update the squash file. This can be done via bash script ie:

#!/bin/bash
USER="pi"

sudo /bin/systemctl stop octoprint.service

/usr/bin/mksquashfs /home/${USER}/.octoprint/uploads \
/tmp/uploads_tmp.sfs -b 65536 -comp xz -Xdict-size 100%

sudo /bin/umount /home/${USER}/.octoprint/uploads
sudo /bin/umount /home/${USER}/.overlay/lower

/bin/rm /home/${USER}/.overlay/upper/* -rf
/bin/rm /home/${USER}/.overlay/upper/.* -rf
/bin/mv /tmp/uploads_tmp.sfs /home/${USER}/.overlay/uploads.sfs

sudo /bin/mount /home/${USER}/.overlay/lower
sudo /bin/mount /home/${USER}/.octoprint/uploads

sudo /bin/systemctl start octoprint.service

Sorry for my poor English.
Salud.

1 Like
2

You might want to verify if this now works with the new Backup & Restore plugin that's been introduced to v1.3.10 of OctoPrint. I'm guessing that a backup might possibly break if the default checkboxes are used... and the restore might also break if that's the case.

3

I just tested the backup part and everything looks good. From the point of view of Octoprint is just a normal directory, I would be surprised if it failed for some reason, but it is better to try.

Thanks for pointing that.
Salud.

1 Like
4

Well done.

(I mentioned that because I'd created a plugin that earlier was using a symlink and then something else didn't want to follow that. Sometimes it's better to test than to assume.)

1 Like