+++ title = "Raspberry Pi, Container Orchestration and Swarm right at home" author = ["Elia el Lazkani"] date = 2022-08-24 lastmod = 2022-08-24 tags = ["docker", "linux", "arm", "swarm", "traefik", "raspberry-pi"] categories = ["container"] draft = false +++ When I started looking into solutions for my home container orchestration, I wanted a solution that runs on my 2 Raspberry Pis. These beasts have 4 virtual CPUs and a whoping 1GB of memory each. In other words, not a lot of resources to go around. What can I run on these? I wonder! ## Consideration {#consideration} If we look at the state of _container orchestration_ today, we see that _Kubernetes_ domates the space. _Kubernetes_ is awesome, but will it run on my Pis ? I doubt it. Fret not ! There are other, _more lightweight_, solutions out there. Let's discuss them briefly. ### K3s {#k3s} I have experience with _K3s_. I even wrote a blog [post]({{< relref "building-k3s-on-a-pi" >}}) on it. Unfortunately, I found that _K3s_ uses almost half of the memory resources of the Pis to run. That's too much overhead lost. ### MicroK8s {#microk8s} _MicroK8s_ is a Canonical project. It has similarities to _K3s_ in the way of easy deployment and lightweight focus. The end result is also extremly similar to _K3s_ in resource usage. ### Nomad {#nomad} _Nomad_ is a _HashiCorp_ product and just all their other products, it is very well designed, very robust and extremly versatile. Running it on the Pis was a breeze, it barely used any resources. It shoulds great so far, doesn't it ? Well, sort of. The deployment and configuration of _Nomad_ is a bit tricky and requires a bit of moving components. Those can be automated with _Ansible_ eventually. Aside that, _Nomad_ requires extra configuration to install and enable CNI and service discovery. Finally, it has a steep learning curve to deploy containers in the cluster and you have HCL to deal with. ### Swarm {#swarm} I was surprised to find that not only _Docker Swarm_ is still alive, it also became a mode which comes preshipped with _docker_ since a few years ago. I also found out that _Swarm_ has great _Ansible_ integration, for both initializing and creating the cluster and deploying _stacks_ and _services_ into it. After all, if you are already familiar with _docker-compose_, you'll feel right at home. ## Setting up a Swarm cluster {#setting-up-a-swarm-cluster} I set up to deploy my _Swarm Cluster_ and manage it using _Ansible_. I didn't want to do the work again in the future and I wanted to go the IaC (_Infrastructure as Code_) route, as should you. At this stage, I have to take a few assumptions. I assume that you already have at least 2 machines with a Linux Distribution installed on them. I, also, assume that _docker_ is already installed and running on both machines. Finally, all the dependencies required to run _Ansible_ on both hosts (`python3-docker` and `python3-jsondiff` on _Ubuntu_). There are **two** types of _nodes_ in a _Swarm_ cluster; `manager` and `worker`. The **first** node used to initialize the cluster is the _leader_ node which is also a `manager` node. ### Leader {#leader} For the `leader` node, our tasks are going to be initializing the cluster. Before we do so, let's create our _quick and dirty_ **Ansible** `inventory` file. ```yaml --- all: hosts: children: leader: hosts: node001: ansible_host: 192.168.0.100 ansible_user: user ansible_port: 22 ansible_become: yes ansible_become_method: sudo manager: worker: hosts: node002: ansible_host: 192.168.0.101 ansible_user: user ansible_port: 22 ansible_become: yes ansible_become_method: sudo ```

warning

This isn't meant to be deployed in **production** in a _professional_ setting. It goes without saying, the `leader` is static, not highly available and prone to failure. The `manager` and `worker` node tasks are, also, dependent on the successful run of the initialization task on the `leader`.
Now that we've taken care of categorizing the nodes and writing the _Ansible_ `inventory`, let's initialize a _Swarm_ cluster. ```yaml --- - name: Init a new swarm cluster community.docker.docker_swarm: state: present advertise_addr: "{{ hostvars[inventory_hostname]['ansible_default_ipv4']['address'] }}" register: clustering_swarm_cluster ```

Note

We use `hostvars[inventory_hostname]['ansible_default_ipv4']['address']` which returns the IP address of the node itself. This is the IP adress used to advertise.

Note

We use `register` to save the returned response from the cluster initialization into a new variable we called `clustering_swarm_cluster`. This will come handy later.
This should take care of initializing a new _Swarm_ cluster. You can verify if _Swarm_ is running. ```shell $ docker system info 2>&1 | grep Swarm Swarm: active ``` ### Manager {#manager} If you have a larger number of nodes, you might require more than one `manager` node. To join more _managers_ to the cluster, we can use the power of _Ansible_ again. ```yaml --- - name: Add manager node to Swarm cluster community.docker.docker_swarm: state: join advertise_addr: "{{ hostvars[inventory_hostname]['ansible_default_ipv4']['address'] }}" join_token: "{{ hostvars[groups['leader'][0]]['clustering_swarm_cluster']['swarm_facts']['JoinTokens']['Manager'] }}" remote_addrs: [ "{{ hostvars[groups['leader'][0]]['ansible_default_ipv4']['address'] }}:2377" ] ```

Note

We access the token we saved earlier on the `leader` to join a `manager` to the cluster using `hostvars[groups['leader'][0]]['clustering_swarm_cluster']['swarm_facts']['JoinTokens']['Manager']`.

Note

If we can get a hostvar from a different node, we can also get the IP of such node with `hostvars[groups['leader'][0]]['ansible_default_ipv4']['address']`.
Now that we've taken care of the `manager` code, let's work on the `worker` nodes. ### Worker {#worker} Just as easily as we created the _task_ to **join** a `manager` node to the cluster, we do the same for the `worker`. ```yaml --- - name: Add worker node to Swarm cluster community.docker.docker_swarm: state: join advertise_addr: "{{ hostvars[inventory_hostname]['ansible_default_ipv4']['address'] }}" join_token: "{{ hostvars[groups['leader'][0]]['clustering_swarm_cluster']['swarm_facts']['JoinTokens']['Worker'] }}" remote_addrs: [ "{{ hostvars[groups['leader'][0]]['ansible_default_ipv4']['address'] }}:2377" ] ```

Note

Déjà vu when it comes to the `join_token`, except that we use the `worker` token instead.
The _glue code_ you're looking for that does the magic is this. ```yaml --- - name: Bootstrap Swarm depedencies include_tasks: common.yml - name: Bootstrap leader node include_tasks: leader.yml when: inventory_hostname in groups['leader'] - name: Bootstrap manager node include_tasks: manager.yml when: inventory_hostname in groups['manager'] - name: Bootstrap worker node include_tasks: worker.yml when: inventory_hostname in groups['worker'] ``` Each of the tasks described above should be in its own file, as shown in the _glue code_, and they will **only** run on the group they are meant to run on. Following these tasks, I ended up with the cluster below. ```shell # docker node ls ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS ENGINE VERSION h4scu4nry2r9p129rsdt88ae2 * node001 Ready Active Leader 20.10.17 uyn43a9tsdn2n435untva9pae node002 Ready Active 20.10.17 ``` There, we see both nodes and they both seem to be in a `Ready` state. ## Conclusion {#conclusion} If you're _outside_ a professional setting and you find yourself needing to run a container orchestration platform, some platforms might be overkill. _Docker Swarm_ has great community support in _Ansible_ making the management of small clusters on low resource devices extremly easy. It comes with the added bonus of having built-in _service discovery_ and _networking_. Give it a try, you might be pleasently surprised like I was.