This repository has been archived on 2023-06-11. You can view files and clone it, but cannot push or open issues or pull requests.
blog.lazkani.io-20200902-hi.../posts/backup/automating-borg.org

237 lines
7.5 KiB
Org Mode

#+BEGIN_COMMENT
.. title: Automating Borg
.. date: 2020-02-02
.. slug: automating-borg
.. updated: 2020-02-02
.. status: published
.. tags: backup, borgbackup, borg, borgmatic
.. category: backup
.. authors: Elia el Lazkani
.. description: We've had a look at **Borg** before, let's find out how to automate it.
.. type: text
#+END_COMMENT
In the previous blog post entitle "{{% doc %}}borgbackup{{% /doc %}}, I talked about *borg*.
If you read that post, you would've noticed that *borg* has a lot of features.
With a lot of features come a lot of automation.
If you were thinking about using *borg*, you should either make a /simple cron/ or you're gonna have to write an elaborate script to take care of all the different steps.
What if I told you there's another way ? An easier way ! The *Borgmatic* way... What would you say ?
* Borgmatic
*Borgmatic* is defined on their [[https://torsion.org/borgmatic/][website]] as follows.
#+BEGIN_QUOTE
borgmatic is simple, configuration-driven backup software for servers
and workstations. Protect your files with client-side encryption.
Backup your databases too. Monitor it all with integrated third-party
services.
#+END_QUOTE
If you go down to it, *borgmatic* uses *borg*'s /API/ to automate a list of configurable /tasks/.
This way, it saves you the trouble of writing your own scripts to automate these steps.
*Borgmatic* uses a /YAML/ configuration file. Let's configure a few tasks.
* Location
First, let's start by configuring the locations that *borg* is going to be working with.
#+BEGIN_SRC yaml
location:
source_directories:
- /home/
repositories:
- user@backupserver:sourcehostname.borg
one_file_system: true
exclude_patterns:
- /home/*/.cache
- '*.pyc'
#+END_SRC
This tells *borg* that we need to backup our =/home= directories excluding a few patterns.
Let's not forget that we told *borg* where the repository is located at.
* Storage
We need to configure the storage next.
#+BEGIN_SRC yaml
storage:
# Recommended
# encryption_passcommand: secret-tool lookup borg-repository repo-name
encryption_passphrase: "ReallyStrongPassphrase"
compression: zstd,15
ssh_command: ssh -i /path/to/private/key
borg_security_directory: /path/to/base/config/security
archive_name_format: 'borgmatic-{hostname}-{now}'
#+END_SRC
In this section, we tell borg a little big of information about our repository.
What are the credentials, where it can find them, etc.
The easy way is to go with a =passphrase=, but I recommend using an =encryption_passcommand= instead.
I also use =zstd= for encryption instead of =lz4=, you better do your research before you change the default.
I also recommend, just as they do, the use of a security directory as well.
* Retention
We can configure a retention for our backups, if we like.
#+BEGIN_SRC yaml
retention:
keep_hourly: 7
keep_daily: 7
keep_weekly: 4
keep_monthly: 6
keep_yearly: 2
prefix: "borgmatic-"
#+END_SRC
The part of what to keep from /hourly/ to /daily/ is self explanatory.
I would like to point out the =prefix= part as it is important.
This is the /prefix/ that *borgmatic* uses to consider backups for *pruning*.
#+BEGIN_EXPORT html
<div class="admonition warning">
<p class="admonition-title">warning</p>
#+END_EXPORT
Watch out for the retention =prefix=
#+BEGIN_EXPORT html
</div>
#+END_EXPORT
* Consistency
After the updates, we'd like to check our backups.
#+BEGIN_SRC yaml
consistency:
checks:
- repository
- archives
check_last: 3
prefix: "borgmatic-"
#+END_SRC
#+BEGIN_EXPORT html
<div class="admonition warning">
<p class="admonition-title">warning</p>
#+END_EXPORT
Watch out, again, for the consistency =prefix=
#+BEGIN_EXPORT html
</div>
#+END_EXPORT
* Hooks
Finally, hooks.
I'm going to talk about hooks a bit. Hooks can be used to backup *MySQL*, *PostgreSQL* or *MariaDB*.
They can also be hooks for =on_error=, =before_backup=, =after_backup=, =before_everything= and =after_everything=.
You can also hook to third party services which you can check on their webpage.
I deployed my own, so I configured my own.
* Borgmatic Configuration
Let's put everything together now.
#+BEGIN_SRC yaml
location:
source_directories:
- /home/
repositories:
- user@backupserver:sourcehostname.borg
one_file_system: true
exclude_patterns:
- /home/*/.cache
- '*.pyc'
storage:
# Recommended
# encryption_passcommand: secret-tool lookup borg-repository repo-name
encryption_passphrase: "ReallyStrongPassphrase"
compression: zstd,15
ssh_command: ssh -i /path/to/private/key
borg_security_directory: /path/to/base/config/security
archive_name_format: 'borgmatic-{hostname}-{now}'
retention:
keep_hourly: 7
keep_daily: 7
keep_weekly: 4
keep_monthly: 6
keep_yearly: 2
prefix: "borgmatic-"
consistency:
checks:
- repository
- archives
check_last: 3
prefix: "borgmatic-"
#+END_SRC
Now that we have everything together, let's save it in =/etc/borgmatic.d/home.yaml=.
* Usage
If you have *borg* and *borgmatic* already installed on your system and the *borgmatic* configuration file in place, you can test it out.
You can create the repository.
#+BEGIN_EXAMPLE
# borgmatic init -v 2
#+END_EXAMPLE
You can list the backups for the repository.
#+BEGIN_EXAMPLE
# borgmatic list --last 5
borgmatic-home-2020-01-30T22:01:30 Thu, 2020-01-30 22:01:42 [0000000000000000000000000000000000000000000000000000000000000000]
borgmatic-home-2020-01-31T22:02:12 Fri, 2020-01-31 22:02:24 [0000000000000000000000000000000000000000000000000000000000000000]
borgmatic-home-2020-02-01T22:01:34 Sat, 2020-02-01 22:01:45 [0000000000000000000000000000000000000000000000000000000000000000]
borgmatic-home-2020-02-02T16:01:22 Sun, 2020-02-02 16:01:32 [0000000000000000000000000000000000000000000000000000000000000000]
borgmatic-home-2020-02-02T18:01:36 Sun, 2020-02-02 18:01:47 [0000000000000000000000000000000000000000000000000000000000000000]
#+END_EXAMPLE
You could run a check.
#+BEGIN_EXAMPLE
# borgmatic check -v 1
/etc/borgmatic.d/home.yaml: Pinging Healthchecks start
/borg/home: Running consistency checks
Remote: Starting repository check
Remote: Starting repository index check
Remote: Completed repository check, no problems found.
Starting archive consistency check...
Analyzing archive borgmatic-home-2020-02-01T22:01:34 (1/3)
Analyzing archive borgmatic-home-2020-02-02T16:01:22 (2/3)
Analyzing archive borgmatic-home-2020-02-02T18:01:36 (3/3)
Orphaned objects check skipped (needs all archives checked).
Archive consistency check complete, no problems found.
summary:
/etc/borgmatic.d/home.yaml: Successfully ran configuration file
#+END_EXAMPLE
But most of all, if you simply run =borgmatic= without any parameters, it will run through the whole configuration and apply all the steps.
At this point, you can simply add the =borgmatic= command in a *cron* to run on an interval.
The other options would be to configure a =systemd= *timer* and *service* to run this on an interval.
The latter is usually provided to you if you used your *package manager* to install *borgmatic*.
* Conclusion
If you've checked *borg* and found it too much work to script, give *borgmatic* a try.
I've been using borgmatic for few weeks now with no issues at all.
I recently hooked it to a monitoring system so I will have a better view on when it runs, how much time each run takes.
Also, if any of my backups fail I get notified by email. I hope you enjoy *borg* and *borgmatic* as much as I am.