diff --git a/images/simple_cron_monitoring_with_healthchecks/borgbackup_healthchecks.png b/images/simple_cron_monitoring_with_healthchecks/borgbackup_healthchecks.png new file mode 100644 index 0000000..1b4e940 Binary files /dev/null and b/images/simple_cron_monitoring_with_healthchecks/borgbackup_healthchecks.png differ diff --git a/images/simple_cron_monitoring_with_healthchecks/borgbackup_healthchecks_logs.png b/images/simple_cron_monitoring_with_healthchecks/borgbackup_healthchecks_logs.png new file mode 100644 index 0000000..4637b14 Binary files /dev/null and b/images/simple_cron_monitoring_with_healthchecks/borgbackup_healthchecks_logs.png differ diff --git a/posts/monitoring/simple_cron_monitoring_with_healthchecks.rst b/posts/monitoring/simple_cron_monitoring_with_healthchecks.rst new file mode 100644 index 0000000..8a368a0 --- /dev/null +++ b/posts/monitoring/simple_cron_monitoring_with_healthchecks.rst @@ -0,0 +1,154 @@ +.. title: Simple cron monitoring with HealthChecks +.. date: 2020-02-09 +.. slug: simple_cron_monitoring_with_healthchecks +.. updated: 2020-02-09 +.. status: published +.. tags: monitoring, healthchecks, cron +.. category: monitoring +.. authors: Elia el Lazkani +.. description: Ever needed to monitor simple things ? Well, HealthChecks seems perfect for that. +.. type: text + +In a :doc:`previous post `, I showed you how you can automate your **borg** backups with **borgmatic**. + +After I started using **borgmatic** for my backups and hooked it to a *cron* running every 2 hours, I got interested into knowing what's happening to my backups at all times. + +My experience comes handy in here, I know I need a monitoring system. +I also know that traditional monitoring systems are too complex for my use case. + +I need something simple. I need something I can deploy myself. + +.. TEASER_END + +Choosing a monitoring system +============================ + +I already know I don't want a traditional monitoring system like *nagios* or *sensu* or *prometheus*. +It is not needed, it's an overkill. + +I went through the list of hooks that **borgmatic** offers out of the box and checked each project. + +I came across `HealthChecks `_. + +HealthChecks +============ + +The `HealthChecks `_ project works in a simple manner. +It simply offers syou an endpoint which you need to ping within a certain period, otherwise you get paged. + +It has a lot of integrations from simple emails to other third party services that will call or message you or even trigger push notifications to your phone. + +In my case, a simple email is enough. After all, they are simply backups and if they failed now, they will work when cron runs again in 2 hours. + +Deploy +====== + +Let's create a `docker-compose` service configuration that looks like the following: + +.. code:: yaml + + healthchecks: + container_name: healthchecks + image: linuxserver/healthchecks:v1.12.0-ls48 + restart: unless-stopped + ports: + - "127.0.0.1:8000:8000" + volumes: + - "./healthchecks/data:/config" + environment: + PUID: "5000" + PGID: "5000" + SECRET_KEY: "super-secret-key" + ALLOWED_HOSTS: '["*"]' + DEBUG: "False" + DEFAULT_FROM_EMAIL: "noreply@healthchecks.example.com" + USE_PAYMENTS: "False" + REGISTRATION_OPEN: "False" + EMAIL_HOST: "smtp.example.com" + EMAIL_PORT: "587" + EMAIL_HOST_USER: "smtp@healthchecks.example.com" + EMAIL_HOST_PASSWORD: "super-secret-password" + EMAIL_USE_TLS: "True" + SITE_ROOT: "https://healthchecks.example.com" + SITE_NAME: "HealthChecks" + MASTER_BADGE_LABEL: "HealthChecks" + PING_ENDPOINT: "https://healthchecks.example.com/ping/" + PING_EMAIL_DOMAIN: "healthchecks.example.com" + TWILIO_ACCOUNT: "None" + TWILIO_AUTH: "None" + TWILIO_FROM: "None" + PD_VENDOR_KEY: "None" + TRELLO_APP_KEY: "None" + +This will create a docker container exposing it locally on ``127.0.0.1:8000``. +Let's point nginx to it and expose it using something similar to the following. + +.. code:: text + + server { + listen 443 ssl; + server_name healthchecks.example.com; + + ssl_certificate /path/to/the/fullchain.pem; + ssl_certificate_key /path/to/the/privkey.pem; + + location / { + proxy_pass http://127.0.0.1:8000; + + add_header X-Frame-Options SAMEORIGIN; + add_header X-XSS-Protection "1; mode=block"; + proxy_redirect off; + proxy_buffering off; + proxy_set_header Host $host; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Forwarded-Proto $scheme; + proxy_set_header X-Forwarded-Port $server_port; + proxy_read_timeout 90; + } + + } + +This should do nicely. + + +Usage +===== + +Now it's a simple matter of creating a checks. + +.. thumbnail:: /images/simple_cron_monitoring_with_healthchecks/borgbackup_healthchecks.png + :align: center + :alt: HealthChecks monitoring for BorgBackup + + +This will give you a link that looks like the following ``https://healthchecks.example.com/ping/84b2a834-02f5-524f-4c27-a2f24562b219`` + +Let's feed it to **borgmatic**. + +.. code:: yaml + + hooks: + healthchecks: https://healthchecks.example.com/ping/84b2a834-02f5-524f-4c27-a2f24562b219 + + +After you configure the **borgmatic** hook to keep *HealthChecks* in the know of what's going on. +We can take a look at the log to see what happened and when. + +.. thumbnail:: /images/simple_cron_monitoring_with_healthchecks/borgbackup_healthchecks_logs.png + :align: center + :alt: HealthChecks monitoring log for BorgBackup + + + +Conclusion +========== + +As we saw in the blog post, now I am always in the know about my backups. +If my backup fails, I get an email to notify me of a failure. +I can also monitor how much time it takes my backups to run. +This is a very important feature for me to have. + +The question of deploying one's own monitoring system is a personal choice. +After all, one can use free third party services if they would like. +The correct answer though is to always monitor.