blog.lazkani.io/content/posts/simple-cron-monitoring-with-healthchecks.md

+++ title = "Simple cron monitoring with HealthChecks" author = ["Elia el Lazkani"] date = 2020-02-09 lastmod = 2020-02-09 tags = ["healthchecks", "cron"] categories = ["monitoring"] draft = false +++

In a previous post entitled "[Automating Borg]({{< relref "automating-borg" >}})", I showed you how you can automate your borg backups with borgmatic.

After I started using borgmatic for my backups and hooked it to a cron running every 2 hours, I got interested into knowing what's happening to my backups at all times.

My experience comes handy in here, I know I need a monitoring system. I also know that traditional monitoring systems are too complex for my use case.

I need something simple. I need something I can deploy myself.

Choosing a monitoring system

I already know I don't want a traditional monitoring system like nagios or sensu or prometheus. It is not needed, it's an overkill.

I went through the list of hooks that borgmatic offers out of the box and checked each project.

I came across HealthChecks.

HealthChecks

The HealthChecks project works in a simple manner. It simply offers syou an endpoint which you need to ping within a certain period, otherwise you get paged.

It has a lot of integrations from simple emails to other third party services that will call or message you or even trigger push notifications to your phone.

In my case, a simple email is enough. After all, they are simply backups and if they failed now, they will work when cron runs again in 2 hours.

Deploy

Let's create a docker-compose service configuration that looks like the following:

healthchecks:
  container_name: healthchecks
  image: linuxserver/healthchecks:v1.12.0-ls48
  restart: unless-stopped
  ports:
    - "127.0.0.1:8000:8000"
  volumes:
    - "./healthchecks/data:/config"
  environment:
    PUID: "5000"
    PGID: "5000"
    SECRET_KEY: "super-secret-key"
    ALLOWED_HOSTS: '["*"]'
    DEBUG: "False"
    DEFAULT_FROM_EMAIL: "noreply@healthchecks.example.com"
    USE_PAYMENTS: "False"
    REGISTRATION_OPEN: "False"
    EMAIL_HOST: "smtp.example.com"
    EMAIL_PORT: "587"
    EMAIL_HOST_USER: "smtp@healthchecks.example.com"
    EMAIL_HOST_PASSWORD: "super-secret-password"
    EMAIL_USE_TLS: "True"
    SITE_ROOT: "https://healthchecks.example.com"
    SITE_NAME: "HealthChecks"
    MASTER_BADGE_LABEL: "HealthChecks"
    PING_ENDPOINT: "https://healthchecks.example.com/ping/"
    PING_EMAIL_DOMAIN: "healthchecks.example.com"
    TWILIO_ACCOUNT: "None"
    TWILIO_AUTH: "None"
    TWILIO_FROM: "None"
    PD_VENDOR_KEY: "None"
    TRELLO_APP_KEY: "None"

This will create a docker container exposing it locally on 127.0.0.1:8000. Let's point nginx to it and expose it using something similar to the following.

server {
    listen   443 ssl;
    server_name  healthchecks.example.com;

    ssl_certificate /path/to/the/fullchain.pem;
    ssl_certificate_key /path/to/the/privkey.pem;

    location / {
        proxy_pass http://127.0.0.1:8000;

        add_header X-Frame-Options SAMEORIGIN;
        add_header X-XSS-Protection "1; mode=block";
        proxy_redirect off;
        proxy_buffering off;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-Forwarded-Port $server_port;
        proxy_read_timeout 90;
    }

}

This should do nicely.

Usage

Now it's a simple matter of creating a checks.

{{< figure src="/ox-hugo/borgbackup-healthchecks.png" caption="Figure 1: HealthChecks monitoring for BorgBackup" target="_blank" link="/ox-hugo/borgbackup-healthchecks.png" >}}

This will give you a link that looks like the following

https://healthchecks.example.com/ping/84b2a834-02f5-524f-4c27-a2f24562b219

Let's feed it to borgmatic.

hooks:
  healthchecks: https://healthchecks.example.com/ping/84b2a834-02f5-524f-4c27-a2f24562b219

After you configure the borgmatic hook to keep HealthChecks in the know of what's going on. We can take a look at the log to see what happened and when.

{{< figure src="/ox-hugo/borgbackup-healthchecks-logs.png" caption="Figure 2: HealthChecks monitoring for BorgBackup" target="_blank" link="/ox-hugo/borgbackup-healthchecks-logs.png" >}}

Conclusion

As we saw in the blog post, now I am always in the know about my backups. If my backup fails, I get an email to notify me of a failure. I can also monitor how much time it takes my backups to run. This is a very important feature for me to have.

The question of deploying one's own monitoring system is a personal choice. After all, one can use free third party services if they would like. The correct answer though is to always monitor.