blog.lazkani.io/content/posts/local-kubernetes-cluster-on-kvm.md

+++
title = "Local Kubernetes Cluster on KVM"
author = ["Elia el Lazkani"]
date = 2019-06-21T21:00:00+02:00
lastmod = 2021-06-28T00:00:56+02:00
tags = ["rancher", "rancheros", "kvm", "libvirt"]
categories = ["kubernetes"]
draft = false
+++

I wanted to explore _kubernetes_ even more for myself and for this blog. I've worked on pieces of this at work but not the totality of the work which I would like to understand for myself. I wanted, also to explore new tools and ways to leverage the power of _kubernetes_.

So far, I have been using _minikube_ to do the deployments but there is an inherit restriction that comes with using a single bundled node. Sure, it is easy to get it up and running but at some point I had to use `nodePort` to go around the IP restriction. This is a restriction that you will have in an actual _kubernetes_ cluster but I will show you later how to go around it. For now, let's just get a local cluster up and running.

<!--more-->


## Objective {#objective}

I needed a local _kubernetes_ cluster using all open source tools and easy to deploy. So I went with using _KVM_ as the hypervisor layer and installed `virt-manager` for shallow management. As an OS, I wanted something light and made for _kubernetes_. As I already know of Rancher (being an easy way to deploy _kubernetes_ and they have done a great job so far since the launch of their Rancer 2.0) I decided to try _RancherOS_. So let's see how all that works together.


## Requirements {#requirements}

Let's start by thinking about what we actually need. Rancher, the dashboard they offer is going to need a VM by itself and they [recommend](https://rancher.com/docs/rancher/v2.x/en/quick-start-guide/deployment/quickstart-vagrant/) _4GB of RAM_. I only have _16GB of RAM_ on my machine so I'll have to do the math to see how much I can afford to give this _dashboard_ and _manager_. By looking at the _RancherOS_ hardware [requirements](https://rancher.com/docs/os/v1.x/en/), I can tell that by giving a each node _2GB_ of RAM I should be able to host a _3 node cluster_ and with _2_ more for the _dashboard_ that puts me right on _8GB of RAM_. So we need to create _4 VMs_ with _2GB of RAM_ each.


## Installing RancherOS {#installing-rancheros}

Once all 4 nodes have been created, when you boot into the _RancherOS_ [ISO](https://rancher.com/docs/os/v1.x/en/installation/running-rancheros/workstation/boot-from-iso/) do the following.

<div class="admonition note">
<p class="admonition-title">Note</p>

Because I was using _libvirt_, I was able to do `virsh console <vm>` and run these commands.

</div>


## Virsh Console {#virsh-console}

If you are running these VMs on _libvirt_, then you can console into the box and run `vi`.

```text
 # virsh list
 Id   Name      State
 -------------------------
 21   kube01    running
 22   kube02    running
 23   kube03    running
 24   rancher   running

 # virsh console rancher
```


## Configuration {#configuration}

If you read the _RancherOS_ [documentation](https://rancher.com/docs/os/v1.x/en/), you'll find out that you can configure the _OS_ with a `YAML` configuration file so let's do that.

```text
 $ vi cloud-config.yml
```

And that file should hold.

```yaml
---
hostname: rancher.kube.loco
ssh_authorized_keys:
  - ssh-rsa AAA...
rancher:
  network:
    interfaces:
      eth0:
        address: 192.168.122.5/24
        dhcp: false
        gateway: 192.168.122.1
        mtu: 1500
```

Make sure that your **public** _ssh key_ is replaced in the example before and if you have a different network configuration for your VMs, change the network configuration here.

After you save that file, install the _OS_.

```text
 $ sudo ros install -c cloud-config.yml -d /dev/sda
```

Do the same for the rest of the servers and their names and IPs should be as follows (if you are following this tutorial):

```text
 192.168.122.5   rancher.kube.loco
 192.168.122.10  kube01.kube.loco
 192.168.122.11  kube02.kube.loco
 192.168.122.12  kube03.kube.loco
```


## Post Installation Configuration {#post-installation-configuration}

After _RancherOS_ has been installed, one will need to configure `/etc/hosts` and it should look like the following if one is working off of the _Rancher_ box.

```text
 $ sudo vi /etc/hosts
```

```text
 127.0.0.1       rancher.kube.loco
 192.168.122.5   rancher.kube.loco
 192.168.122.10  kube01.kube.loco
 192.168.122.11  kube02.kube.loco
 192.168.122.12  kube03.kube.loco
```

Do the same on the rest of the servers while changing the `127.0.0.1` hostname to the host of the server.


## Installing Rancher {#installing-rancher}

At this point, I have to stress a few facts:

-   This is not the Rancher recommended way to deploy _kubernetes_.
    -   The recommended way is of course [RKE](https://rancher.com/docs/rke/v0.1.x/en/).
-   This is for testing, so I did not take into consideration backup of anything.
    -   There are ways to backup Rancher configuration by mounting storage from the `rancher` docker container.

If those points are understood, let's go ahead and deploy Rancher.
First, `$ ssh rancher@192.168.122.5` then:

```text
 [rancher@rancher ~]$ docker run -d --restart=unless-stopped -p 80:80 -p 443:443 rancher/rancher
```

Give it a few minutes for the container to come up and the application as well. Meanwhile configure your `/etc/hosts` file on your machine.

```text
 192.168.122.5    rancher.kube.loco
```

Now that all that is out of the way, you can login to <https://rancher.kube.loco> and set your `admin` password and the `url` for Rancher.


## Deploying Kubernetes {#deploying-kubernetes}

Now that everything is ready, let's deploy _kubernetes_ the easy way.

At this point you should be greeted with a page that looks like the
following.

{{< figure src="/ox-hugo/01-add-cluster.png" caption="Figure 1: Add Cluster Page" target="_blank" link="/ox-hugo/01-add-cluster.png" >}}

Click on the **Add Cluser**

{{< figure src="/ox-hugo/02-custom-cluster.png" caption="Figure 2: Custom Cluster Page" target="_blank" link="/ox-hugo/02-custom-cluster.png" >}}

Make sure you choose **Custom** as a _provider_. Then fill in the **Cluser Name** in our case we'll call it **kube**.

{{< figure src="/ox-hugo/03-calico-networkProvider.png" caption="Figure 3: Network Provider: Calico (Optional)" target="_blank" link="/ox-hugo/03-calico-networkProvider.png" >}}

Optionally, you can choose your **Network Providor**, in my case I chose **Calico**. Then I clicked on **show advanced** at the bottom right corner then expanded the _newly shown tab_ **Advanced Cluster Options**.

{{< figure src="/ox-hugo/04-nginx-ingressDisabled.png" caption="Figure 4: Nginx Ingress Disabled" target="_blank" link="/ox-hugo/04-nginx-ingressDisabled.png" >}}

We will disable the **Nginx Ingress** and the **Pod Security Policy Support** for the time being. This will become more apparent why in the future, hopefully. Then hit **Next**.

{{< figure src="/ox-hugo/05-customize-nodes.png" caption="Figure 5: Customize Nodes" target="_blank" link="/ox-hugo/05-customize-nodes.png" >}}

Make sure that you select all **3 Node Roles**. Set the **Public Address** and the **Node Name** to the first node and then copy the command and paste it on the _first_ node.

Do the same for _all the rest_. Once the first docker image gets downloaded and ran you should see a message pop at the bottom.

{{< figure src="/ox-hugo/06-registered-nodes.png" caption="Figure 6: Registered Nodes" target="_blank" link="/ox-hugo/06-registered-nodes.png" >}}

<div class="admonition warning">
<p class="admonition-title">warning</p>

Do **NOT** click _done_ until you see all _3 nodes registered_.

</div>


## Finalizing {#finalizing}

Now that you have _3 registered nodes_, click **Done** and go grab yourself a cup of coffee. Maybe take a long walk, this will take time. Or if you are curious like me, you'd be looking at the logs, checking the containers in a quad pane `tmux` session.

After a long time has passed, our story ends with a refresh and a welcome with this page.

{{< figure src="/ox-hugo/07-kubernetes-cluster.png" caption="Figure 7: Kubernetes Cluster" target="_blank" link="/ox-hugo/07-kubernetes-cluster.png" >}}

Welcome to your Kubernetes Cluster.


## Conclusion {#conclusion}

At this point, you can check that all the nodes are healthy and you got yourself a kubernetes cluster. In future blog posts we will explore an avenue to deploy _multiple ingress controllers_ on the same cluster on the same `port: 80` by giving them each an IP external to the cluster.

But for now, you got yourself a kubernetes cluster to play with. Enjoy.