blog.lazkani.io-20200902-historical/posts/k3s/building-k3s-on-a-pi.org

#+BEGIN_COMMENT
.. title: Building k3s on a Pi
.. date: 2020-08-09
.. slug: building-k3s-on-a-pi
.. updated: 2020-08-09
.. status: published
.. tags: kubernetes, k3s, arm
.. category: k3s
.. authors: Elia el Lazkani
.. description: I have decided to make a better use of my pi, k3s came next.
.. type: text
#+END_COMMENT

I have had a *Pi* laying around used for a simple task for a while now.
A few days ago, I was browsing the web, learning more about privacy, when I stumbled upon [[https://adguard.com/en/welcome.html][AdGuard Home]].

I have been using it as my internal DNS on top of the security and privacy layers I add to my machine.
Its benefits can be argued but it is a DNS after all and I wanted to see what else it can do for me.
Anyway, I digress. I searched to see if I could find a container for *AdGuard Home* and I did.

At this point, I started thinking about what I could do to make the [[https://www.raspberrypi.org/][Pi]] more useful.

That's when [[https://k3s.io/][k3s]] came into the picture.

{{{TEASER_END}}}

* Pre-requisites
As this is not a *Pi* tutorial, I am going to be assuming that you have a /Raspberry Pi/ with *Raspberry Pi OS* /Buster/ installed on it.
The assumption does not mean you cannot install any other OS on the Pi and run this setup.
It only means that I have tested this on /Buster/ and that your milage will vary.

* Prepare the Pi
Now that you have /Buster/ already installed, let's go ahead and [[https://rancher.com/docs/k3s/latest/en/advanced/#enabling-legacy-iptables-on-raspbian-buster][fix]] a small default configuration issue with it.

*K3s* uses =iptables= to route things around correctly. /Buster/ uses =nftables= by default, let's switch it to =iptables=.

#+BEGIN_EXAMPLE
 $ sudo iptables -F
 $ sudo update-alternatives --set iptables /usr/sbin/iptables-legacy
 $ sudo update-alternatives --set ip6tables /usr/sbin/ip6tables-legacy
 $ sudo reboot
#+END_EXAMPLE

At this point, your /Pi/ should reboot. Your *OS* is configured for the next step.

* Pre-install Configuration
After testing *k3s* a few times, I found out that by /default/ it will deploy a few extra services like [[https://docs.traefik.io/][Traefik]].

Unfortunately, just like anything the /default/ configuration is just that. It's plain and not very useful from the start. You will need to tweak it.

This step could be done either /post/ or /pre/ deploy. Figuring out the /pre-deploy/ is a bit more involving but a bit more fun as well.

The first thing you need to know is that the normal behavior of *k3s* is to deploy anything found in =/var/lib/rancher/k3s/server/manifests/=.
So a good first step is, of course, to proceed with creating that.

#+BEGIN_EXAMPLE
 $ mkdir -p /var/lib/rancher/k3s/server/manifests/
#+END_EXAMPLE

The other thing to know is that *k3s* can deploy /Helm Charts/.
It will create the /manifests/ it will deploy by default, before beginning the setup, in the manifest path I mentioned.
If you would like to see what it deployed and how, visit that path after *k3s* runs.
I did, and I took their configuration of *Traefik* which I was unhappy with its /defaults/.

My next step was securing the /defaults/ as much as possible and I found out that *Traefik* can do [[https://docs.traefik.io/v2.0/middlewares/basicauth/][basic authentication]].
As a starting point, that's great. Let's create the credentials.

#+BEGIN_EXAMPLE
 $ htpasswd -c ./auth myUser
#+END_EXAMPLE

That was easy so far. Let's turn up the notch and create the manifest for *k3s*.

Create =traefik.yaml= in =/var/lib/rancher/k3s/server/manifests/= with the following content.

#+BEGIN_SRC yaml
---
apiVersion: helm.cattle.io/v1
kind: HelmChart
metadata:
  name: traefik
  namespace: kube-system
spec:
  chart: https://%{KUBERNETES_API}%/static/charts/traefik-1.81.0.tgz
  valuesContent: |-
    rbac:
      enabled: true
    ssl:
      enabled: true
    dashboard:
      enabled: true
      domain: traefik-ui.example.com
      auth:
        basic:
          myUser: $ars3$4A5tdstr$trSDDa4467Tsa54sTs.
    metrics:
      prometheus:
        enabled: false
    kubernetes:
      ingressEndpoint:
        useDefaultPublishedService: true
    image: "rancher/library-traefik"
    tolerations:
      - key: "CriticalAddonsOnly"
        operator: "Exists"
      - key: "node-role.kubernetes.io/master"
        operator: "Exists"
        effect: "NoSchedule"
#+END_SRC

It's a *Pi*, I don't need prometheus so I disabled it.
I also enabled the dashboard and added the credentials we created in the previous step.

Now, the /Helm Chart/ will deploy an ingress and expose the dashboard for you on the value of =domain=.

#+BEGIN_EXPORT html
<div class="admonition note">
<p class="admonition-title">Note</p>
#+END_EXPORT
I figured out the values to set in =valuesContent= by reading the /Helm Chart/
#+BEGIN_EXPORT html
</div>
#+END_EXPORT

* K3s
If everything is in place, you are ready to proceed.
You can install *k3s*, now, but before I get to that step, I will say a few things about *k3s*.

*K3s* has a smaller feature set than *k8s*, hence the smaller footprint.
Read the documentation to see if you need any of the missing features.
The second thing to mention is that *k3s* is a one binary deploy that uses *containerd*.
That's why we will use the script installation method as it adds the necessary *systemd* configuration for us.
It is a nice gesture.

Let's do that, shall we ?

#+BEGIN_EXAMPLE
 $ curl -sfL https://get.k3s.io | sh -s - --no-deploy traefik
#+END_EXAMPLE

#+BEGIN_EXPORT html
<div class="admonition note">
<p class="admonition-title">Note</p>
#+END_EXPORT
We need to make sure that *k3s* does not deploy its own *traefik* but ours.
Make sure to add =--no-deploy traefik= to our deployment command.
#+BEGIN_EXPORT html
</div>
#+END_EXPORT

Point =traefik.example.com= to your *Pi* =IP= in =/etc/hosts= on your machine.

#+BEGIN_EXAMPLE
 traefik.example.com  192.168.0.5
#+END_EXAMPLE

When the installation command is done, you should be able to visit [[http://traefik.example.com/][http://traefik.example.com/]]

You can get the /kubeconfig/ from the /Raspberry Pi/, you can find it in =/etc/rancher/k3s/k3s.yaml=. You will need to change the =server= *IP*.

* Conclusion
If you've made it so far, you should have a *k3s* cluster running on a single /Raspberry Pi/.
The next steps you might want to look into is disable the /metrics/ server and use the resources for other things.