New post about Borgmatic
This commit is contained in:
parent
99a5aedd74
commit
409cc3117b
1 changed files with 247 additions and 0 deletions
247
posts/backup/automating-borg.rst
Normal file
247
posts/backup/automating-borg.rst
Normal file
|
|
@ -0,0 +1,247 @@
|
|||
.. title: Automating Borg
|
||||
.. date: 2020-02-02
|
||||
.. slug: automating-borg
|
||||
.. updated: 2020-02-02
|
||||
.. status: published
|
||||
.. tags: backup, borgbackup, borg, borgmatic
|
||||
.. category: backup
|
||||
.. authors: Elia el Lazkani
|
||||
.. description: We've had a look at **Borg** before, let's find out how to automate it.
|
||||
.. type: text
|
||||
|
||||
|
||||
In the previous :doc:`previous blog post <borgbackup>`, I talked about **borg**.
|
||||
If you read that post, you would've noticed that **borg** has a lot of features.
|
||||
With a lot a of features come a lot of automation.
|
||||
|
||||
If you were thinking about using **borg**, you should either make a *simple cron*
|
||||
or you're gonna have to write an elaborate script to take care of all the different steps.
|
||||
|
||||
What if I told you there's another way ? An easier way ! The **Borgmatic** way...
|
||||
What would you say ?
|
||||
|
||||
.. TEASER_END
|
||||
|
||||
Borgmatic
|
||||
=========
|
||||
|
||||
**Borgmatic** is defined on their `website <https://torsion.org/borgmatic/>`_ as follows.
|
||||
|
||||
borgmatic is simple, configuration-driven backup software for servers and workstations.
|
||||
Protect your files with client-side encryption. Backup your databases too.
|
||||
Monitor it all with integrated third-party services.
|
||||
|
||||
If you go down to it, **borgmatic** uses **borg**'s *API* to automate a list of configurable *tasks*.
|
||||
This way, it saves you the trouble of writing your own scripts to automate these steps.
|
||||
|
||||
**Borgmatic** uses a *YAML* configuration file. Let's configure a few tasks.
|
||||
|
||||
Location
|
||||
========
|
||||
|
||||
First, let's start by configuring the locations that **borg** is going to be working with.
|
||||
|
||||
.. code:: yaml
|
||||
|
||||
location:
|
||||
source_directories:
|
||||
- /home/
|
||||
|
||||
repositories:
|
||||
- user@backupserver:sourcehostname.borg
|
||||
|
||||
one_file_system: true
|
||||
|
||||
exclude_patterns:
|
||||
- /home/*/.cache
|
||||
- '*.pyc'
|
||||
|
||||
This tells **borg** that we need to backup our ``/home`` directories excluding a few patterns.
|
||||
Let's not forget that we told **borg** where the repository is located at.
|
||||
|
||||
Storage
|
||||
=======
|
||||
|
||||
We need to configure the storage next.
|
||||
|
||||
.. code:: yaml
|
||||
|
||||
storage:
|
||||
# Recommended
|
||||
# encryption_passcommand: secret-tool lookup borg-repository repo-name
|
||||
|
||||
encryption_passphrase: "ReallyStrongPassphrase"
|
||||
compression: zstd,15
|
||||
ssh_command: ssh -i /path/to/private/key
|
||||
borg_security_directory: /path/to/base/config/security
|
||||
archive_name_format: 'borgmatic-{hostname}-{now}'
|
||||
|
||||
In this section, we tell borg a little big of information about our repository.
|
||||
What are the credentials, where it can find them, etc.
|
||||
|
||||
The easy way is to go with a ``passphrase``, but I recommend using an ``encryption_passcommand`` instead.
|
||||
I also use ``zstd`` for encryption instead of ``lz4``, you better do your research before you change the default.
|
||||
I also recommend, just as they do, the use of a security directory as well.
|
||||
|
||||
Retention
|
||||
=========
|
||||
|
||||
We can configure a retention for our backups, if we like.
|
||||
|
||||
.. code:: yaml
|
||||
|
||||
retention:
|
||||
keep_hourly: 7
|
||||
keep_daily: 7
|
||||
keep_weekly: 4
|
||||
keep_monthly: 6
|
||||
keep_yearly: 2
|
||||
|
||||
prefix: "borgmatic-"
|
||||
|
||||
The part of what to keep from *hourly* to *daily* is self explanatory.
|
||||
I would like to point out the ``prefix`` part as it is important.
|
||||
This is the *prefix* that **borgmatic** uses to consider backups for **pruning**.
|
||||
|
||||
.. warning::
|
||||
|
||||
Watch out for the retention ``prefix``
|
||||
|
||||
Consistency
|
||||
===========
|
||||
|
||||
After the updates, we'd like to check our backups.
|
||||
|
||||
.. code:: yaml
|
||||
|
||||
consistency:
|
||||
checks:
|
||||
- repository
|
||||
- archives
|
||||
|
||||
check_last: 3
|
||||
|
||||
prefix: "borgmatic-"
|
||||
|
||||
.. warning::
|
||||
|
||||
Watch out, again, for the consistency ``prefix``
|
||||
|
||||
Hooks
|
||||
=====
|
||||
|
||||
Finally, hooks.
|
||||
|
||||
I'm going to talk about hooks a bit. Hooks can be used to backup **MySQL**, **PostgreSQL** or **MariaDB**.
|
||||
They can also be hooks for ``on_error``, ``before_backup``, ``after_backup``, ``before_everything`` and ``after_everything``.
|
||||
You can also hook to third party services which you can check on their webpage.
|
||||
|
||||
I deployed my own, so I configured my own.
|
||||
|
||||
Borgmatic Configuration
|
||||
=======================
|
||||
|
||||
Let's put everything together now.
|
||||
|
||||
.. code:: yaml
|
||||
|
||||
location:
|
||||
source_directories:
|
||||
- /home/
|
||||
|
||||
repositories:
|
||||
- user@backupserver:sourcehostname.borg
|
||||
|
||||
one_file_system: true
|
||||
|
||||
exclude_patterns:
|
||||
- /home/*/.cache
|
||||
- '*.pyc'
|
||||
|
||||
storage:
|
||||
# Recommended
|
||||
# encryption_passcommand: secret-tool lookup borg-repository repo-name
|
||||
|
||||
encryption_passphrase: "ReallyStrongPassphrase"
|
||||
compression: zstd,15
|
||||
ssh_command: ssh -i /path/to/private/key
|
||||
borg_security_directory: /path/to/base/config/security
|
||||
archive_name_format: 'borgmatic-{hostname}-{now}'
|
||||
|
||||
retention:
|
||||
keep_hourly: 7
|
||||
keep_daily: 7
|
||||
keep_weekly: 4
|
||||
keep_monthly: 6
|
||||
keep_yearly: 2
|
||||
|
||||
prefix: "borgmatic-"
|
||||
|
||||
consistency:
|
||||
checks:
|
||||
- repository
|
||||
- archives
|
||||
|
||||
check_last: 3
|
||||
|
||||
prefix: "borgmatic-"
|
||||
|
||||
|
||||
Now that we have everything together, let's save it in ``/etc/borgmatic.d/home.yaml``.
|
||||
|
||||
Usage
|
||||
=====
|
||||
|
||||
If you have **borg** and **borgmatic** already installed on your system and the **borgmatic** configuration file in place,
|
||||
you can test it out.
|
||||
|
||||
You can create the repository.
|
||||
|
||||
.. code:: text
|
||||
|
||||
# borgmatic init -v 2
|
||||
|
||||
You can list the backups for the repository.
|
||||
|
||||
.. code:: text
|
||||
|
||||
# borgmatic list --last 5
|
||||
borgmatic-home-2020-01-30T22:01:30 Thu, 2020-01-30 22:01:42 [0000000000000000000000000000000000000000000000000000000000000000]
|
||||
borgmatic-home-2020-01-31T22:02:12 Fri, 2020-01-31 22:02:24 [0000000000000000000000000000000000000000000000000000000000000000]
|
||||
borgmatic-home-2020-02-01T22:01:34 Sat, 2020-02-01 22:01:45 [0000000000000000000000000000000000000000000000000000000000000000]
|
||||
borgmatic-home-2020-02-02T16:01:22 Sun, 2020-02-02 16:01:32 [0000000000000000000000000000000000000000000000000000000000000000]
|
||||
borgmatic-home-2020-02-02T18:01:36 Sun, 2020-02-02 18:01:47 [0000000000000000000000000000000000000000000000000000000000000000]
|
||||
|
||||
You could run a check.
|
||||
|
||||
.. code:: text
|
||||
|
||||
# borgmatic check -v 1
|
||||
/etc/borgmatic.d/home.yaml: Pinging Healthchecks start
|
||||
/borg/home: Running consistency checks
|
||||
Remote: Starting repository check
|
||||
Remote: Starting repository index check
|
||||
Remote: Completed repository check, no problems found.
|
||||
Starting archive consistency check...
|
||||
Analyzing archive borgmatic-home-2020-02-01T22:01:34 (1/3)
|
||||
Analyzing archive borgmatic-home-2020-02-02T16:01:22 (2/3)
|
||||
Analyzing archive borgmatic-home-2020-02-02T18:01:36 (3/3)
|
||||
Orphaned objects check skipped (needs all archives checked).
|
||||
Archive consistency check complete, no problems found.
|
||||
|
||||
summary:
|
||||
/etc/borgmatic.d/home.yaml: Successfully ran configuration file
|
||||
|
||||
But most of all, if you simply run ``borgmatic`` without any parameters, it will run through the whole configuration and apply all the steps.
|
||||
|
||||
At this point, you can simply add the ``borgmatic`` command in a **cron** to run on an interval.
|
||||
The other options would be to configure a ``systemd`` **timer** and **service** to run this on an interval.
|
||||
The latter is usually provided to you if you used your **package manager** to install **borgmatic**.
|
||||
|
||||
Conclusion
|
||||
==========
|
||||
|
||||
If you've checked **borg** and found it too much work to script, give **borgmatic** a try.
|
||||
I've been using borgmatic for few weeks now with no issues at all.
|
||||
I recently hooked it to a monitoring system so I will have a better view on when it runs, how much time each run takes.
|
||||
Also, if any of my backups fail I get notified by email. I hope you enjoy **borg** and **borgmatic** as much as I am.
|
||||
Reference in a new issue