7.5 KiB
.. title: Automating Borg .. date: 2020-02-02 .. slug: automating-borg .. updated: 2020-02-02 .. status: published .. tags: backup, borgbackup, borg, borgmatic .. category: backup .. authors: Elia el Lazkani .. description: We've had a look at Borg before, let's find out how to automate it. .. type: text
In the previous blog post entitle "{{% doc %}}borgbackup{{% /doc %}}, I talked about borg. If you read that post, you would've noticed that borg has a lot of features. With a lot of features come a lot of automation.
If you were thinking about using borg, you should either make a simple cron or you're gonna have to write an elaborate script to take care of all the different steps.
What if I told you there's another way ? An easier way ! The Borgmatic way… What would you say ?
Borgmatic
Borgmatic is defined on their website as follows.
borgmatic is simple, configuration-driven backup software for servers and workstations. Protect your files with client-side encryption. Backup your databases too. Monitor it all with integrated third-party services.
If you go down to it, borgmatic uses borg's API to automate a list of configurable tasks. This way, it saves you the trouble of writing your own scripts to automate these steps.
Borgmatic uses a YAML configuration file. Let's configure a few tasks.
Location
First, let's start by configuring the locations that borg is going to be working with.
location:
source_directories:
- /home/
repositories:
- user@backupserver:sourcehostname.borg
one_file_system: true
exclude_patterns:
- /home/*/.cache
- '*.pyc'
This tells borg that we need to backup our /home directories excluding a few patterns.
Let's not forget that we told borg where the repository is located at.
Storage
We need to configure the storage next.
storage:
# Recommended
# encryption_passcommand: secret-tool lookup borg-repository repo-name
encryption_passphrase: "ReallyStrongPassphrase"
compression: zstd,15
ssh_command: ssh -i /path/to/private/key
borg_security_directory: /path/to/base/config/security
archive_name_format: 'borgmatic-{hostname}-{now}'In this section, we tell borg a little big of information about our repository. What are the credentials, where it can find them, etc.
The easy way is to go with a passphrase, but I recommend using an encryption_passcommand instead.
I also use zstd for encryption instead of lz4, you better do your research before you change the default.
I also recommend, just as they do, the use of a security directory as well.
Retention
We can configure a retention for our backups, if we like.
retention:
keep_hourly: 7
keep_daily: 7
keep_weekly: 4
keep_monthly: 6
keep_yearly: 2
prefix: "borgmatic-"
The part of what to keep from hourly to daily is self explanatory.
I would like to point out the prefix part as it is important.
This is the prefix that borgmatic uses to consider backups for pruning.
warning
Watch out for the retention prefix
Consistency
After the updates, we'd like to check our backups.
consistency:
checks:
- repository
- archives
check_last: 3
prefix: "borgmatic-"warning
Watch out, again, for the consistency prefix
Hooks
Finally, hooks.
I'm going to talk about hooks a bit. Hooks can be used to backup MySQL, PostgreSQL or MariaDB.
They can also be hooks for on_error, before_backup, after_backup, before_everything and after_everything.
You can also hook to third party services which you can check on their webpage.
I deployed my own, so I configured my own.
Borgmatic Configuration
Let's put everything together now.
location:
source_directories:
- /home/
repositories:
- user@backupserver:sourcehostname.borg
one_file_system: true
exclude_patterns:
- /home/*/.cache
- '*.pyc'
storage:
# Recommended
# encryption_passcommand: secret-tool lookup borg-repository repo-name
encryption_passphrase: "ReallyStrongPassphrase"
compression: zstd,15
ssh_command: ssh -i /path/to/private/key
borg_security_directory: /path/to/base/config/security
archive_name_format: 'borgmatic-{hostname}-{now}'
retention:
keep_hourly: 7
keep_daily: 7
keep_weekly: 4
keep_monthly: 6
keep_yearly: 2
prefix: "borgmatic-"
consistency:
checks:
- repository
- archives
check_last: 3
prefix: "borgmatic-"
Now that we have everything together, let's save it in /etc/borgmatic.d/home.yaml.
Usage
If you have borg and borgmatic already installed on your system and the borgmatic configuration file in place, you can test it out.
You can create the repository.
# borgmatic init -v 2
You can list the backups for the repository.
# borgmatic list --last 5 borgmatic-home-2020-01-30T22:01:30 Thu, 2020-01-30 22:01:42 [0000000000000000000000000000000000000000000000000000000000000000] borgmatic-home-2020-01-31T22:02:12 Fri, 2020-01-31 22:02:24 [0000000000000000000000000000000000000000000000000000000000000000] borgmatic-home-2020-02-01T22:01:34 Sat, 2020-02-01 22:01:45 [0000000000000000000000000000000000000000000000000000000000000000] borgmatic-home-2020-02-02T16:01:22 Sun, 2020-02-02 16:01:32 [0000000000000000000000000000000000000000000000000000000000000000] borgmatic-home-2020-02-02T18:01:36 Sun, 2020-02-02 18:01:47 [0000000000000000000000000000000000000000000000000000000000000000]
You could run a check.
# borgmatic check -v 1 /etc/borgmatic.d/home.yaml: Pinging Healthchecks start /borg/home: Running consistency checks Remote: Starting repository check Remote: Starting repository index check Remote: Completed repository check, no problems found. Starting archive consistency check... Analyzing archive borgmatic-home-2020-02-01T22:01:34 (1/3) Analyzing archive borgmatic-home-2020-02-02T16:01:22 (2/3) Analyzing archive borgmatic-home-2020-02-02T18:01:36 (3/3) Orphaned objects check skipped (needs all archives checked). Archive consistency check complete, no problems found. summary: /etc/borgmatic.d/home.yaml: Successfully ran configuration file
But most of all, if you simply run borgmatic without any parameters, it will run through the whole configuration and apply all the steps.
At this point, you can simply add the borgmatic command in a cron to run on an interval.
The other options would be to configure a systemd timer and service to run this on an interval.
The latter is usually provided to you if you used your package manager to install borgmatic.
Conclusion
If you've checked borg and found it too much work to script, give borgmatic a try. I've been using borgmatic for few weeks now with no issues at all. I recently hooked it to a monitoring system so I will have a better view on when it runs, how much time each run takes. Also, if any of my backups fail I get notified by email. I hope you enjoy borg and borgmatic as much as I am.