154 lines
5.4 KiB
Org Mode
154 lines
5.4 KiB
Org Mode
#+BEGIN_COMMENT
|
|
.. title: Simple cron monitoring with HealthChecks
|
|
.. date: 2020-02-09
|
|
.. slug: simple-cron-monitoring-with-healthchecks
|
|
.. updated: 2020-02-09
|
|
.. status: published
|
|
.. tags: monitoring, healthchecks, cron
|
|
.. category: monitoring
|
|
.. authors: Elia el Lazkani
|
|
.. description: Ever needed to monitor simple things ? Well, HealthChecks seems perfect for that.
|
|
.. type: text
|
|
#+END_COMMENT
|
|
|
|
In a previous post entitled "{{% doc %}}automating-borg{{% /doc %}}", I showed you how you can automate your *borg* backups with *borgmatic*.
|
|
|
|
After I started using *borgmatic* for my backups and hooked it to a /cron/ running every 2 hours, I got interested into knowing what's happening to my backups at all times.
|
|
|
|
My experience comes handy in here, I know I need a monitoring system. I also know that traditional monitoring systems are too complex for my use case.
|
|
|
|
I need something simple. I need something I can deploy myself.
|
|
|
|
{{{TEASER_END}}}
|
|
|
|
* Choosing a monitoring system
|
|
I already know I don't want a traditional monitoring system like /nagios/ or /sensu/ or /prometheus/. It is not needed, it's an overkill.
|
|
|
|
I went through the list of hooks that *borgmatic* offers out of the box and checked each project.
|
|
|
|
I came across [[https://healthchecks.io/][HealthChecks]].
|
|
|
|
* HealthChecks
|
|
The [[https://healthchecks.io/][HealthChecks]] project works in a simple manner.
|
|
It simply offers syou an endpoint which you need to ping within a certain period, otherwise you get paged.
|
|
|
|
It has a lot of integrations from simple emails to other third party services that will call or message you or even trigger push notifications to your phone.
|
|
|
|
In my case, a simple email is enough. After all, they are simply backups and if they failed now, they will work when cron runs again in 2 hours.
|
|
|
|
* Deploy
|
|
Let's create a docker-compose service configuration that looks like the
|
|
following:
|
|
|
|
#+BEGIN_SRC yaml
|
|
healthchecks:
|
|
container_name: healthchecks
|
|
image: linuxserver/healthchecks:v1.12.0-ls48
|
|
restart: unless-stopped
|
|
ports:
|
|
- "127.0.0.1:8000:8000"
|
|
volumes:
|
|
- "./healthchecks/data:/config"
|
|
environment:
|
|
PUID: "5000"
|
|
PGID: "5000"
|
|
SECRET_KEY: "super-secret-key"
|
|
ALLOWED_HOSTS: '["*"]'
|
|
DEBUG: "False"
|
|
DEFAULT_FROM_EMAIL: "noreply@healthchecks.example.com"
|
|
USE_PAYMENTS: "False"
|
|
REGISTRATION_OPEN: "False"
|
|
EMAIL_HOST: "smtp.example.com"
|
|
EMAIL_PORT: "587"
|
|
EMAIL_HOST_USER: "smtp@healthchecks.example.com"
|
|
EMAIL_HOST_PASSWORD: "super-secret-password"
|
|
EMAIL_USE_TLS: "True"
|
|
SITE_ROOT: "https://healthchecks.example.com"
|
|
SITE_NAME: "HealthChecks"
|
|
MASTER_BADGE_LABEL: "HealthChecks"
|
|
PING_ENDPOINT: "https://healthchecks.example.com/ping/"
|
|
PING_EMAIL_DOMAIN: "healthchecks.example.com"
|
|
TWILIO_ACCOUNT: "None"
|
|
TWILIO_AUTH: "None"
|
|
TWILIO_FROM: "None"
|
|
PD_VENDOR_KEY: "None"
|
|
TRELLO_APP_KEY: "None"
|
|
#+END_SRC
|
|
|
|
This will create a docker container exposing it locally on =127.0.0.1:8000=.
|
|
Let's point nginx to it and expose it using something similar to the following.
|
|
|
|
#+BEGIN_EXAMPLE
|
|
server {
|
|
listen 443 ssl;
|
|
server_name healthchecks.example.com;
|
|
|
|
ssl_certificate /path/to/the/fullchain.pem;
|
|
ssl_certificate_key /path/to/the/privkey.pem;
|
|
|
|
location / {
|
|
proxy_pass http://127.0.0.1:8000;
|
|
|
|
add_header X-Frame-Options SAMEORIGIN;
|
|
add_header X-XSS-Protection "1; mode=block";
|
|
proxy_redirect off;
|
|
proxy_buffering off;
|
|
proxy_set_header Host $host;
|
|
proxy_set_header X-Real-IP $remote_addr;
|
|
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
|
proxy_set_header X-Forwarded-Proto $scheme;
|
|
proxy_set_header X-Forwarded-Port $server_port;
|
|
proxy_read_timeout 90;
|
|
}
|
|
|
|
}
|
|
#+END_EXAMPLE
|
|
|
|
This should do nicely.
|
|
|
|
* Usage
|
|
Now it's a simple matter of creating a checks.
|
|
|
|
|
|
#+BEGIN_EXPORT html
|
|
<a class="reference" href="/images/simple-cron-monitoring-with-healthchecks/borgbackup-healthchecks.png" alt="HealthChecks monitoring for BorgBackup" algin="center">
|
|
<img src="/images/simple-cron-monitoring-with-healthchecks/borgbackup-healthchecks.thumbnail.png">
|
|
#+END_EXPORT
|
|
#+BEGIN_EXPORT html
|
|
</a>
|
|
#+END_EXPORT
|
|
|
|
This will give you a link that looks like the following
|
|
|
|
#+BEGIN_EXAMPLE
|
|
https://healthchecks.example.com/ping/84b2a834-02f5-524f-4c27-a2f24562b219
|
|
#+END_EXAMPLE
|
|
|
|
Let's feed it to *borgmatic*.
|
|
|
|
#+BEGIN_SRC yaml
|
|
hooks:
|
|
healthchecks: https://healthchecks.example.com/ping/84b2a834-02f5-524f-4c27-a2f24562b219
|
|
#+END_SRC
|
|
|
|
After you configure the *borgmatic* hook to keep /HealthChecks/ in the know of what's going on.
|
|
We can take a look at the log to see what happened and when.
|
|
|
|
#+BEGIN_EXPORT html
|
|
<a class="reference" href="/images/simple-cron-monitoring-with-healthchecks/borgbackup-healthchecks-logs.png" alt="HealthChecks monitoring log for BorgBackup" algin="center">
|
|
<img src="/images/simple-cron-monitoring-with-healthchecks/borgbackup-healthchecks-logs.thumbnail.png">
|
|
#+END_EXPORT
|
|
#+BEGIN_EXPORT html
|
|
</a>
|
|
#+END_EXPORT
|
|
|
|
|
|
* Conclusion
|
|
As we saw in the blog post, now I am always in the know about my backups.
|
|
If my backup fails, I get an email to notify me of a failure.
|
|
I can also monitor how much time it takes my backups to run.
|
|
This is a very important feature for me to have.
|
|
|
|
The question of deploying one's own monitoring system is a personal choice.
|
|
After all, one can use free third party services if they would like.
|
|
The correct answer though is to always monitor.
|