blog.lazkani.io/content/posts/automating-borg.md

+++
title = "Automating Borg"
author = ["Elia el Lazkani"]
date = 2020-02-02T21:00:00+01:00
lastmod = 2021-06-28T00:00:27+02:00
tags = ["borgmatic", "borgbackup", "borg"]
categories = ["backup"]
draft = false
+++

In the previous blog post entitle [BorgBackup]({{< relref "borgbackup" >}}), I talked about **borg**.
If you read that post, you would've noticed that **borg** has a lot of features.
With a lot of features come a lot of automation.

If you were thinking about using **borg**, you should either make a _simple cron_ or you're gonna have to write an elaborate script to take care of all the different steps.

What if I told you there's another way ? An easier way ! The **Borgmatic** way... What would you say ?

<!--more-->


## Borgmatic {#borgmatic}

**Borgmatic** is defined on their [website](https://torsion.org/borgmatic/) as follows.

> borgmatic is simple, configuration-driven backup software for servers
> and workstations. Protect your files with client-side encryption.
> Backup your databases too. Monitor it all with integrated third-party
> services.

If you go down to it, **borgmatic** uses **borg**'s _API_ to automate a list of configurable _tasks_.
This way, it saves you the trouble of writing your own scripts to automate these steps.

**Borgmatic** uses a _YAML_ configuration file. Let's configure a few tasks.


## Location {#location}

First, let's start by configuring the locations that **borg** is going to be working with.

```yaml
location:
    source_directories:
        - /home/

    repositories:
        - user@backupserver:sourcehostname.borg

    one_file_system: true

    exclude_patterns:
        - /home/*/.cache
        - '*.pyc'
```

This tells **borg** that we need to backup our `/home` directories excluding a few patterns.
Let's not forget that we told **borg** where the repository is located at.


## Storage {#storage}

We need to configure the storage next.

```yaml
storage:
    # Recommended
    # encryption_passcommand: secret-tool lookup borg-repository repo-name

    encryption_passphrase: "ReallyStrongPassphrase"
    compression: zstd,15
    ssh_command: ssh -i /path/to/private/key
    borg_security_directory: /path/to/base/config/security
    archive_name_format: 'borgmatic-{hostname}-{now}'
```

In this section, we tell borg a little big of information about our repository.
What are the credentials, where it can find them, etc.

The easy way is to go with a `passphrase`, but I recommend using an `encryption_passcommand` instead.
I also use `zstd` for encryption instead of `lz4`, you better do your research before you change the default.
I also recommend, just as they do, the use of a security directory as well.


## Retention {#retention}

We can configure a retention for our backups, if we like.

```yaml
retention:
    keep_hourly: 7
    keep_daily: 7
    keep_weekly: 4
    keep_monthly: 6
    keep_yearly: 2

    prefix: "borgmatic-"
```

The part of what to keep from _hourly_ to _daily_ is self explanatory.
I would like to point out the `prefix` part as it is important.
This is the _prefix_ that **borgmatic** uses to consider backups for **pruning**.

<div class="admonition warning">
<p class="admonition-title">warning</p>

Watch out for the retention `prefix`

</div>


## Consistency {#consistency}

After the updates, we'd like to check our backups.

```yaml
consistency:
    checks:
        - repository
        - archives

    check_last: 3

    prefix: "borgmatic-"
```

<div class="admonition warning">
<p class="admonition-title">warning</p>

Watch out, again, for the consistency `prefix`

</div>


## Hooks {#hooks}

Finally, hooks.

I'm going to talk about hooks a bit. Hooks can be used to backup **MySQL**, **PostgreSQL** or **MariaDB**.
They can also be hooks for `on_error`, `before_backup`, `after_backup`, `before_everything` and `after_everything`.
You can also hook to third party services which you can check on their webpage.

I deployed my own, so I configured my own.


## Borgmatic Configuration {#borgmatic-configuration}

Let's put everything together now.

```yaml
location:
    source_directories:
        - /home/

    repositories:
        - user@backupserver:sourcehostname.borg

    one_file_system: true

    exclude_patterns:
        - /home/*/.cache
        - '*.pyc'

storage:
    # Recommended
    # encryption_passcommand: secret-tool lookup borg-repository repo-name

    encryption_passphrase: "ReallyStrongPassphrase"
    compression: zstd,15
    ssh_command: ssh -i /path/to/private/key
    borg_security_directory: /path/to/base/config/security
    archive_name_format: 'borgmatic-{hostname}-{now}'

retention:
    keep_hourly: 7
    keep_daily: 7
    keep_weekly: 4
    keep_monthly: 6
    keep_yearly: 2

    prefix: "borgmatic-"

consistency:
    checks:
        - repository
        - archives

    check_last: 3

    prefix: "borgmatic-"
```

Now that we have everything together, let's save it in `/etc/borgmatic.d/home.yaml`.


## Usage {#usage}

If you have **borg** and **borgmatic** already installed on your system and the **borgmatic** configuration file in place, you can test it out.

You can create the repository.

```text
 # borgmatic init -v 2
```

You can list the backups for the repository.

```text
 # borgmatic list --last 5
 borgmatic-home-2020-01-30T22:01:30 Thu, 2020-01-30 22:01:42 [0000000000000000000000000000000000000000000000000000000000000000]
 borgmatic-home-2020-01-31T22:02:12 Fri, 2020-01-31 22:02:24 [0000000000000000000000000000000000000000000000000000000000000000]
 borgmatic-home-2020-02-01T22:01:34 Sat, 2020-02-01 22:01:45 [0000000000000000000000000000000000000000000000000000000000000000]
 borgmatic-home-2020-02-02T16:01:22 Sun, 2020-02-02 16:01:32 [0000000000000000000000000000000000000000000000000000000000000000]
 borgmatic-home-2020-02-02T18:01:36 Sun, 2020-02-02 18:01:47 [0000000000000000000000000000000000000000000000000000000000000000]
```

You could run a check.

```text
 # borgmatic check -v 1
 /etc/borgmatic.d/home.yaml: Pinging Healthchecks start
 /borg/home: Running consistency checks
 Remote: Starting repository check
 Remote: Starting repository index check
 Remote: Completed repository check, no problems found.
 Starting archive consistency check...
 Analyzing archive borgmatic-home-2020-02-01T22:01:34 (1/3)
 Analyzing archive borgmatic-home-2020-02-02T16:01:22 (2/3)
 Analyzing archive borgmatic-home-2020-02-02T18:01:36 (3/3)
 Orphaned objects check skipped (needs all archives checked).
 Archive consistency check complete, no problems found.

 summary:
 /etc/borgmatic.d/home.yaml: Successfully ran configuration file
```

But most of all, if you simply run `borgmatic` without any parameters, it will run through the whole configuration and apply all the steps.

At this point, you can simply add the `borgmatic` command in a **cron** to run on an interval.
The other options would be to configure a `systemd` **timer** and **service** to run this on an interval.
The latter is usually provided to you if you used your **package manager** to install **borgmatic**.


## Conclusion {#conclusion}

If you've checked **borg** and found it too much work to script, give **borgmatic** a try.
I've been using borgmatic for few weeks now with no issues at all.
I recently hooked it to a monitoring system so I will have a better view on when it runs, how much time each run takes.
Also, if any of my backups fail I get notified by email. I hope you enjoy **borg** and **borgmatic** as much as I am.