5.4 KiB
.. title: Simple cron monitoring with HealthChecks .. date: 2020-02-09 .. slug: simple-cron-monitoring-with-healthchecks .. updated: 2020-02-09 .. status: published .. tags: monitoring, healthchecks, cron .. category: monitoring .. authors: Elia el Lazkani .. description: Ever needed to monitor simple things ? Well, HealthChecks seems perfect for that. .. type: text
In a previous post entitled "{{% doc %}}automating-borg{{% /doc %}}", I showed you how you can automate your borg backups with borgmatic.
After I started using borgmatic for my backups and hooked it to a cron running every 2 hours, I got interested into knowing what's happening to my backups at all times.
My experience comes handy in here, I know I need a monitoring system. I also know that traditional monitoring systems are too complex for my use case.
I need something simple. I need something I can deploy myself.
{{{TEASER_END}}}
Choosing a monitoring system
I already know I don't want a traditional monitoring system like nagios or sensu or prometheus. It is not needed, it's an overkill.
I went through the list of hooks that borgmatic offers out of the box and checked each project.
I came across HealthChecks.
HealthChecks
The HealthChecks project works in a simple manner. It simply offers syou an endpoint which you need to ping within a certain period, otherwise you get paged.
It has a lot of integrations from simple emails to other third party services that will call or message you or even trigger push notifications to your phone.
In my case, a simple email is enough. After all, they are simply backups and if they failed now, they will work when cron runs again in 2 hours.
Deploy
Let's create a docker-compose service configuration that looks like the following:
healthchecks:
container_name: healthchecks
image: linuxserver/healthchecks:v1.12.0-ls48
restart: unless-stopped
ports:
- "127.0.0.1:8000:8000"
volumes:
- "./healthchecks/data:/config"
environment:
PUID: "5000"
PGID: "5000"
SECRET_KEY: "super-secret-key"
ALLOWED_HOSTS: '["*"]'
DEBUG: "False"
DEFAULT_FROM_EMAIL: "noreply@healthchecks.example.com"
USE_PAYMENTS: "False"
REGISTRATION_OPEN: "False"
EMAIL_HOST: "smtp.example.com"
EMAIL_PORT: "587"
EMAIL_HOST_USER: "smtp@healthchecks.example.com"
EMAIL_HOST_PASSWORD: "super-secret-password"
EMAIL_USE_TLS: "True"
SITE_ROOT: "https://healthchecks.example.com"
SITE_NAME: "HealthChecks"
MASTER_BADGE_LABEL: "HealthChecks"
PING_ENDPOINT: "https://healthchecks.example.com/ping/"
PING_EMAIL_DOMAIN: "healthchecks.example.com"
TWILIO_ACCOUNT: "None"
TWILIO_AUTH: "None"
TWILIO_FROM: "None"
PD_VENDOR_KEY: "None"
TRELLO_APP_KEY: "None"
This will create a docker container exposing it locally on 127.0.0.1:8000
.
Let's point nginx to it and expose it using something similar to the following.
server { listen 443 ssl; server_name healthchecks.example.com; ssl_certificate /path/to/the/fullchain.pem; ssl_certificate_key /path/to/the/privkey.pem; location / { proxy_pass http://127.0.0.1:8000; add_header X-Frame-Options SAMEORIGIN; add_header X-XSS-Protection "1; mode=block"; proxy_redirect off; proxy_buffering off; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; proxy_set_header X-Forwarded-Port $server_port; proxy_read_timeout 90; } }
This should do nicely.
Usage
Now it's a simple matter of creating a checks.
This will give you a link that looks like the following
https://healthchecks.example.com/ping/84b2a834-02f5-524f-4c27-a2f24562b219
Let's feed it to borgmatic.
hooks:
healthchecks: https://healthchecks.example.com/ping/84b2a834-02f5-524f-4c27-a2f24562b219
After you configure the borgmatic hook to keep HealthChecks in the know of what's going on. We can take a look at the log to see what happened and when.
Conclusion
As we saw in the blog post, now I am always in the know about my backups. If my backup fails, I get an email to notify me of a failure. I can also monitor how much time it takes my backups to run. This is a very important feature for me to have.
The question of deploying one's own monitoring system is a personal choice. After all, one can use free third party services if they would like. The correct answer though is to always monitor.