Setting up system backups using restic

To back up all my configs, time lapses, filament databases, etc, I've set up a package called restic to back up the contents of the /home/pi directory on my octoprint instances to cloud storage (a backblaze[*] b2 bucket in my case, but an AWS s3 bucket works just as well, or a NFS or Samba share mount, or just an external USB key that gets plugged into the octoprint instance would work as well.)

[*] Full disclosure, I work for backblaze, but this doesn't need to be backblaze specific.)

First, this guide assumes B2 bucket storage and a PI as the octoprint instance, but can be adapted to other storage backends and computers fairly easily. I'm assuming that you're comfortable with using the command line and can ssh into the octoprint instance.

Setting up Restic

You can always refer to the restic docs online, However, I'm going to link to the specific sections that deal with this setup.

The first decision is which version of restic you want to use. I've gone with the latest stable version:

pi@octopi-b1:~ $ restic version
restic 0.14.0 compiled with go1.19 on linux/arm

You could install whatever version your package manager has, but last time I tried, the version in the debian repos was very old (so old it didn't support b2), so I grabbed the linux/arm version from the binary releases page. Direct link here. You can download it directly onto the Pi by logging in and running:

$ wget https://github.com/restic/restic/releases/download/v0.14.0/restic_0.14.0_linux_arm.bz2

You now need to uncompress it

$ bzip2 -d restic_0.14.0_linux_arm.bz2

and move the resulting file someplace you can find it. I recommend /usr/local/bin

$ mv restic_0.14.0_linux_arm /usr/local/bin/restic
$ chmod a+rx /usr/local/bin/restic

Now see if you can run it:

$ restic version
restic 0.14.0 compiled with go1.19 on linux/arm

If you get an error about not finding it, then you might need to add /usr/local/bin to your path:

$ echo 'export PATH="${PATH}":/usr/local/bin' >> ~/.bashrc
$ source ~/.bashrc

Configuing and initializing the repository
The first thing we need to do is configure the repository (repo) directory/location where the backups will be stored. As I said before, I followed the directions for using Backblaze B2, but you can just as easily follow the directions for S3 or some other choice.

Once you have that set up, I like to start creating a place where all the restic specific configuration will live. I put it in a hidden directory in the user's home dir: /home/pi/.restic and have a script there that will contain the environment variables (environment.sh) needed for restic to find the repo.

$ mkdir -p /home/pi/.restic
$ cat > /home/pi/.restic/environment.sh << EOF
> export B2_ACCOUNT_ID=<B2 ACCOUNT ID>
> export B2_ACCOUNT_KEY=<B2 ACCOUNT KEY>
> export RESTIC_REPOSITORY=b2:<B2 BUCKET NAME>:restic
> export RESTIC_PASSWORD_FILE=/home/pi/.restic/repo.pw
> EOF

(Substitute appropriate values of course)
You'll notice that we refer to a non-existent file repo.pw which needs to contain the password used to secure the contents of your repository. I generate mine randomly using
$ gpg -a --gen-random 0 35 > /home/pi/.restic/repo.pw
But you can create your own if you like. You'll need to back up the values for the environment variables and the password file somewhere safe, or you won't be able to restore your backups when you need to.

Now let's initialize the repository:

$ source ~/.restic/environment.sh
$ restic init

This should only take a few seconds. If it doesn't, then make sure that you've properly set up your back-end storage.

Backing up your data
Now we're ready to back up the data for the first time

$ restic backup /home/pi

This may take a little while the first time if you have a lot of stuff in your home directory, but subsequent runs will only need to copy the information that has changed, and so will typically only take a few seconds.

Check that it created a snapshot of your data like so:

$ restic snapshots
repository ccff70e1 opened (repository version 2) successfully, password is correct
ID        Time                 Host            Tags        Paths
-----------------------------------------------------------------------
706f74ac  2023-01-03 18:45:01  octopi                      /home/pi
-----------------------------------------------------------------------
1 snapshot

Congratulations, you've created a backup of your data. That weird identifier at the beginning of the line (actually a hexadecimal number), is the snapshot ID, and you can look at the individual files backed up like so:

$ restic ls 706f74ac

Automating the backups
A single point-in-time snapshot of your data isn't all that useful, so it makes sense to keep making new snapshots. As I said earlier, subsequent backups only copy the data that has changed, and as such will only take up a little more space each time. I make backups of my octoprint instances every hour, and it takes up relatively little space.

I created a script to run all the necessary commands, and then automated running that script every hour using the cron facility.

First let's create a place for the script to live:

$ mkdir -p /home/pi/bin

My script looks like this:

$ nano ~/bin/backup
#!/bin/bash

source /home/pi/.restic/environment.sh

echo Starting backup at $(date)
/usr/local/bin/restic backup /home/pi
echo Finished backup at $(date)

Now we can make the script executable

$ chmod a+rx /home/pi/bin/backup

now run it

$ /home/pi/bin/backup
pi@octopi-b1:~ $ /home/pi/bin/backup
Starting backup at Tue 3 Jan 19:45:01 PST 2023
using parent snapshot 706f74ac

Files:           2 new,     1 changed, 10843 unmodified
Dirs:            0 new,     5 changed,  2225 unmodified
Added to the repository: 47.203 KiB (9.075 KiB stored)

processed 10846 files, 1.549 GiB in 0:36
snapshot 53d9a24b saved
Finished backup at Tue 3 Jan 19:45:41 PST 2023

Now that we've got a script to do the dirty work, let's run it on a schedule. I choose to use cron, which is a very old unix/linux utility, and slightly more cryptic than most tools these days, but it's very powerful.

You schedule cron to run things using the crontab command:

$ crontab -e

that will bring up an editor (nano by default) and allow you to schedule jobs. To run the backup every hour at 45 minutes past the hour, add the following line at the bottom of the file and save it out:

45 * * * * /home/pi/bin/backup >> /tmp/restic.log 2>&1

You can monitor the file /tmp/restic.log to check to see that stuff is running properly.

Cleaning out old snapshots

After a while, you'll have lots of old snapshots that you probably don't need any more

$ restic snapshots
repository ccff70e1 opened (repository version 2) successfully, password is correct
ID        Time                 Host        Tags        Paths
---------------------------------------------------------------
[....]
e2714de3  2023-01-03 15:45:01  octopi                  /home/pi 
aeff5b40  2023-01-03 16:45:01  octopi                  /home/pi
5479fd84  2023-01-03 17:45:01  octopi                  /home/pi
706f74ac  2023-01-03 18:45:01  octopi                  /home/pi
53d9a24b  2023-01-03 19:45:01  octopi                  /home/pi
---------------------------------------------------------------
170 snapshots

Now you can delete individual snapshots like so:

$ restic forget <snapshot_id>
[...]
$ restic prune

The prune command causes restic to actually free up the space in the repository that it no longer referencing files in current snapshots. However, deleting them manually really doesn't scale well.... it's too cumbersome to delete 2 dozen snapshots every day, so instead we can give it rules about what it should keep

$ restic forget --keep-last 10 --prune

This will cause it keep only the 10 most recent snapshots, and automatically prune or recover unused space.

I use a slightly more sophisticated version like so:

$ restic forget --keep-within 1d --keep-within-daily 30d --keep-within-weekly 1y --prune

this instructs it to keep every snapshot for the last day, a daily snapshot for the last 30 days, and a weekly snapshot for the last year. The output looks something like:

repository ccff70e1 opened (repository version 2) successfully, password is correct
Applying Policy: keep daily snapshots within 30d, weekly snapshots within 1y and all snapshots within 1d of the newest
keep 55 snapshots:
ID        Time                 Host        Tags        Reasons           Paths
---------------------------------------------------------------------------------
bc301182  2022-11-27 23:45:01  octopi-b1               weekly within 1y  /home/pi
d55079b7  2022-12-04 23:45:01  octopi-b1               daily within 30d  /home/pi
                                                       weekly within 1y
a2386c38  2022-12-05 23:45:01  octopi-b1               daily within 30d  /home/pi
7a69b78d  2022-12-06 23:45:01  octopi-b1               daily within 30d  /home/pi
a6d6698c  2022-12-07 23:45:01  octopi-b1               daily within 30d  /home/pi
d03754c4  2022-12-08 23:45:01  octopi-b1               daily within 30d  /home/pi
3b676494  2022-12-09 23:45:01  octopi-b1               daily within 30d  /home/pi
d777d991  2022-12-10 23:45:01  octopi-b1               daily within 30d  /home/pi
d513dada  2022-12-11 23:45:01  octopi-b1               daily within 30d  /home/pi
                                                       weekly within 1y
675287c5  2022-12-12 23:45:02  octopi-b1               daily within 30d  /home/pi
a8d8c748  2022-12-13 23:45:01  octopi-b1               daily within 30d  /home/pi
ebadcb11  2022-12-14 23:45:01  octopi-b1               daily within 30d  /home/pi
52ba5370  2022-12-15 23:45:01  octopi-b1               daily within 30d  /home/pi
31d9ef98  2022-12-16 23:45:01  octopi-b1               daily within 30d  /home/pi
d036be7f  2022-12-17 23:45:01  octopi-b1               daily within 30d  /home/pi
295fbaf4  2022-12-18 23:45:01  octopi-b1               daily within 30d  /home/pi
                                                       weekly within 1y
95c2acdc  2022-12-19 23:45:01  octopi-b1               daily within 30d  /home/pi
234b6c01  2022-12-20 23:45:01  octopi-b1               daily within 30d  /home/pi
fce0e694  2022-12-21 23:45:01  octopi-b1               daily within 30d  /home/pi
2bb6a4e9  2022-12-22 23:45:01  octopi-b1               daily within 30d  /home/pi
9ff4e50c  2022-12-23 23:45:01  octopi-b1               daily within 30d  /home/pi
0321d6e2  2022-12-24 23:45:01  octopi-b1               daily within 30d  /home/pi
3ef73526  2022-12-25 23:45:01  octopi-b1               daily within 30d  /home/pi
                                                       weekly within 1y
137be2ba  2022-12-26 23:45:01  octopi-b1               daily within 30d  /home/pi
ca913560  2022-12-27 23:45:01  octopi-b1               daily within 30d  /home/pi
f4c70b9d  2022-12-28 23:45:01  octopi-b1               daily within 30d  /home/pi
6fdcaddf  2022-12-29 23:45:01  octopi-b1               daily within 30d  /home/pi
5f47166d  2022-12-30 23:45:01  octopi-b1               daily within 30d  /home/pi
50a4a44c  2022-12-31 23:45:01  octopi-b1               daily within 30d  /home/pi
fa55d5c1  2023-01-01 23:45:02  octopi-b1               daily within 30d  /home/pi
                                                       weekly within 1y
19956be3  2023-01-02 20:45:01  octopi-b1               within 1d         /home/pi
8c6e4b75  2023-01-02 21:45:01  octopi-b1               within 1d         /home/pi
f171ade4  2023-01-02 22:45:01  octopi-b1               within 1d         /home/pi
41180c0b  2023-01-02 23:45:01  octopi-b1               within 1d         /home/pi
                                                       daily within 30d
a0f5ad2f  2023-01-03 00:45:01  octopi-b1               within 1d         /home/pi
4853c815  2023-01-03 01:45:01  octopi-b1               within 1d         /home/pi
eaeb7249  2023-01-03 02:45:01  octopi-b1               within 1d         /home/pi
dd3b2e30  2023-01-03 03:45:01  octopi-b1               within 1d         /home/pi
dcd1de77  2023-01-03 04:45:01  octopi-b1               within 1d         /home/pi
34a6d6bc  2023-01-03 05:45:01  octopi-b1               within 1d         /home/pi
dd41ff98  2023-01-03 06:45:01  octopi-b1               within 1d         /home/pi
6201e9de  2023-01-03 07:45:02  octopi-b1               within 1d         /home/pi
c23418a6  2023-01-03 08:45:01  octopi-b1               within 1d         /home/pi
f4cc119f  2023-01-03 09:45:01  octopi-b1               within 1d         /home/pi
69d9b8ab  2023-01-03 10:45:01  octopi-b1               within 1d         /home/pi
04d15b40  2023-01-03 11:45:01  octopi-b1               within 1d         /home/pi
50320b5a  2023-01-03 12:45:01  octopi-b1               within 1d         /home/pi
ebd7d6ee  2023-01-03 13:45:01  octopi-b1               within 1d         /home/pi
0d853737  2023-01-03 14:45:01  octopi-b1               within 1d         /home/pi
e2714de3  2023-01-03 15:45:01  octopi-b1               within 1d         /home/pi
aeff5b40  2023-01-03 16:45:01  octopi-b1               within 1d         /home/pi
5479fd84  2023-01-03 17:45:01  octopi-b1               within 1d         /home/pi
706f74ac  2023-01-03 18:45:01  octopi-b1               within 1d         /home/pi
53d9a24b  2023-01-03 19:45:01  octopi-b1               within 1d         /home/pi
97091f7c  2023-01-03 19:45:41  octopi-b1               within 1d         /home/pi
                                                       daily within 30d
                                                       weekly within 1y
---------------------------------------------------------------------------------
55 snapshots

remove 116 snapshots:
ID        Time                 Host        Tags        Paths
---------------------------------------------------------------
d93273eb  2022-11-30 23:45:01  octopi-b1               /home/pi
c41753be  2022-12-01 23:45:01  octopi-b1               /home/pi
85cad9f9  2022-12-02 23:45:01  octopi-b1               /home/pi
329b535b  2022-12-03 23:45:01  octopi-b1               /home/pi
f9c37f5b  2022-12-29 00:45:01  octopi-b1               /home/pi
281e2d22  2022-12-29 01:45:01  octopi-b1               /home/pi
939faf10  2022-12-29 02:45:01  octopi-b1               /home/pi
dee3b939  2022-12-29 03:45:01  octopi-b1               /home/pi
13371466  2022-12-29 04:45:01  octopi-b1               /home/pi
b6edc3aa  2022-12-29 05:45:02  octopi-b1               /home/pi
f731d67c  2022-12-29 06:45:02  octopi-b1               /home/pi
459bae7e  2022-12-29 07:45:01  octopi-b1               /home/pi
2cf68f76  2022-12-29 08:45:01  octopi-b1               /home/pi
fd83ec35  2022-12-29 09:45:01  octopi-b1               /home/pi
5c567cb5  2022-12-29 10:45:01  octopi-b1               /home/pi
d2bdbd4a  2022-12-29 11:45:01  octopi-b1               /home/pi
c0b60b1f  2022-12-29 12:45:01  octopi-b1               /home/pi
e852c82d  2022-12-29 13:45:01  octopi-b1               /home/pi
83935235  2022-12-29 14:45:01  octopi-b1               /home/pi
d4b9718a  2022-12-29 15:45:01  octopi-b1               /home/pi
35536f21  2022-12-29 16:45:01  octopi-b1               /home/pi
e6c4c82a  2022-12-29 17:45:01  octopi-b1               /home/pi
64be62c2  2022-12-29 18:45:01  octopi-b1               /home/pi
47bb29bc  2022-12-29 19:45:01  octopi-b1               /home/pi
f73e2f84  2022-12-29 20:45:01  octopi-b1               /home/pi
adff5f08  2022-12-29 21:45:02  octopi-b1               /home/pi
1f647bdc  2022-12-29 22:45:01  octopi-b1               /home/pi
f18da850  2022-12-30 00:45:01  octopi-b1               /home/pi
ee41dabb  2022-12-30 01:45:01  octopi-b1               /home/pi
063e561e  2022-12-30 02:45:01  octopi-b1               /home/pi
6c43ee77  2022-12-30 03:45:02  octopi-b1               /home/pi
5967bd00  2022-12-30 04:45:01  octopi-b1               /home/pi
5b97abc0  2022-12-30 05:45:01  octopi-b1               /home/pi
b642cf8e  2022-12-30 06:45:01  octopi-b1               /home/pi
03aad30a  2022-12-30 07:45:01  octopi-b1               /home/pi
00088559  2022-12-30 08:45:01  octopi-b1               /home/pi
868b532d  2022-12-30 09:45:01  octopi-b1               /home/pi
daf7416b  2022-12-30 10:45:01  octopi-b1               /home/pi
461339ed  2022-12-30 11:45:01  octopi-b1               /home/pi
d0abcc5b  2022-12-30 12:45:01  octopi-b1               /home/pi
febc08be  2022-12-30 13:45:01  octopi-b1               /home/pi
82337059  2022-12-30 14:45:01  octopi-b1               /home/pi
3aba444c  2022-12-30 15:45:01  octopi-b1               /home/pi
83d39387  2022-12-30 16:45:01  octopi-b1               /home/pi
7bbe4ac6  2022-12-30 17:45:01  octopi-b1               /home/pi
712dca9d  2022-12-30 18:45:01  octopi-b1               /home/pi
ddeec78c  2022-12-30 19:45:01  octopi-b1               /home/pi
7d023178  2022-12-30 20:45:01  octopi-b1               /home/pi
212be964  2022-12-30 21:45:01  octopi-b1               /home/pi
69697da7  2022-12-30 22:45:01  octopi-b1               /home/pi
6ff170e4  2022-12-31 00:45:01  octopi-b1               /home/pi
077b2bd9  2022-12-31 01:45:01  octopi-b1               /home/pi
d031315d  2022-12-31 02:45:01  octopi-b1               /home/pi
09a1c296  2022-12-31 03:45:01  octopi-b1               /home/pi
8b159d8b  2022-12-31 04:45:01  octopi-b1               /home/pi
eb004144  2022-12-31 05:45:01  octopi-b1               /home/pi
286a2b66  2022-12-31 06:45:01  octopi-b1               /home/pi
248e831b  2022-12-31 07:45:01  octopi-b1               /home/pi
c324aab2  2022-12-31 08:45:01  octopi-b1               /home/pi
380e7566  2022-12-31 09:45:01  octopi-b1               /home/pi
4ba8568d  2022-12-31 10:45:01  octopi-b1               /home/pi
91caf3be  2022-12-31 11:45:01  octopi-b1               /home/pi
213fd232  2022-12-31 12:45:01  octopi-b1               /home/pi
7a8d7de5  2022-12-31 13:45:01  octopi-b1               /home/pi
43f6da4c  2022-12-31 14:45:01  octopi-b1               /home/pi
49a9a3be  2022-12-31 15:45:01  octopi-b1               /home/pi
1af816cd  2022-12-31 16:45:01  octopi-b1               /home/pi
1e50584c  2022-12-31 17:45:01  octopi-b1               /home/pi
a53d431e  2022-12-31 18:45:01  octopi-b1               /home/pi
1b1e0795  2022-12-31 19:45:01  octopi-b1               /home/pi
7bcd3996  2022-12-31 20:45:01  octopi-b1               /home/pi
1542bd9d  2022-12-31 21:45:01  octopi-b1               /home/pi
e45f0a17  2022-12-31 22:45:01  octopi-b1               /home/pi
3ec2578f  2023-01-01 00:45:01  octopi-b1               /home/pi
c1ac6d30  2023-01-01 01:45:01  octopi-b1               /home/pi
52d6b535  2023-01-01 02:45:01  octopi-b1               /home/pi
60987619  2023-01-01 03:45:01  octopi-b1               /home/pi
b176a46a  2023-01-01 04:45:01  octopi-b1               /home/pi
08e58b8d  2023-01-01 05:45:01  octopi-b1               /home/pi
db075305  2023-01-01 06:45:01  octopi-b1               /home/pi
b077f1d1  2023-01-01 07:45:01  octopi-b1               /home/pi
7453f62d  2023-01-01 08:45:01  octopi-b1               /home/pi
42b7654e  2023-01-01 09:45:01  octopi-b1               /home/pi
a9ae67a5  2023-01-01 10:45:01  octopi-b1               /home/pi
da32f883  2023-01-01 11:45:01  octopi-b1               /home/pi
a9ebfc4c  2023-01-01 12:45:01  octopi-b1               /home/pi
c6905e1c  2023-01-01 13:45:01  octopi-b1               /home/pi
9f6a6baa  2023-01-01 14:45:01  octopi-b1               /home/pi
716c2cea  2023-01-01 15:45:01  octopi-b1               /home/pi
b5a120a1  2023-01-01 16:45:01  octopi-b1               /home/pi
ecba02bd  2023-01-01 17:45:01  octopi-b1               /home/pi
3ef2ca8c  2023-01-01 18:45:01  octopi-b1               /home/pi
f3039c58  2023-01-01 19:45:01  octopi-b1               /home/pi
99d4ea01  2023-01-01 20:45:01  octopi-b1               /home/pi
548a3d83  2023-01-01 21:45:02  octopi-b1               /home/pi
9de2959b  2023-01-01 22:45:01  octopi-b1               /home/pi
98944292  2023-01-02 00:45:01  octopi-b1               /home/pi
683135fd  2023-01-02 01:45:02  octopi-b1               /home/pi
ccd08b94  2023-01-02 02:45:02  octopi-b1               /home/pi
230c1fdf  2023-01-02 03:45:01  octopi-b1               /home/pi
5499ad9f  2023-01-02 04:45:01  octopi-b1               /home/pi
41e40927  2023-01-02 05:45:01  octopi-b1               /home/pi
899ac841  2023-01-02 06:45:01  octopi-b1               /home/pi
ae6fee5f  2023-01-02 07:45:02  octopi-b1               /home/pi
85bed7e5  2023-01-02 08:45:01  octopi-b1               /home/pi
b46e6ce0  2023-01-02 09:45:01  octopi-b1               /home/pi
3ff36509  2023-01-02 10:45:01  octopi-b1               /home/pi
deec8f55  2023-01-02 11:45:01  octopi-b1               /home/pi
990cd8e7  2023-01-02 12:45:01  octopi-b1               /home/pi
3e01debb  2023-01-02 13:45:01  octopi-b1               /home/pi
333c5db2  2023-01-02 14:45:01  octopi-b1               /home/pi
d2cf7ebe  2023-01-02 15:45:02  octopi-b1               /home/pi
94125754  2023-01-02 16:45:01  octopi-b1               /home/pi
baeecf08  2023-01-02 17:45:01  octopi-b1               /home/pi
e9b285ef  2023-01-02 18:45:01  octopi-b1               /home/pi
270f28a9  2023-01-02 19:45:01  octopi-b1               /home/pi
---------------------------------------------------------------
116 snapshots
[... prune info ...]

So in this case it kept 55 snapshots and deleted 116, and then cleaned up. I created a prune script similar to backup script:

$ cat ~/bin/prune
#!/bin/bash

source ${HOME}/.restic/environment.sh

/usr/local/bin/restic forget --keep-within 1d --keep-within-daily 30d --keep-within-weekly 1y --prune

then created a crontab entry to prune one a day:

59 23 * * * /home/mitch/bin/prune >> /tmp/restic.log 2>&1

Read up on how to restore from your snapshots as well.