blog.lazkani.io/content/posts/local-kubernetes-cluster-on-kvm.md

199 lines
8.4 KiB
Markdown
Raw Normal View History

+++
title = "Local Kubernetes Cluster on KVM"
author = ["Elia el Lazkani"]
date = 2019-02-17
lastmod = 2019-06-21
tags = ["rancher", "rancheros", "kvm", "libvirt"]
categories = ["kubernetes"]
draft = false
+++
I wanted to explore _kubernetes_ even more for myself and for this blog. I've worked on pieces of this at work but not the totality of the work which I would like to understand for myself. I wanted, also to explore new tools and ways to leverage the power of _kubernetes_.
So far, I have been using _minikube_ to do the deployments but there is an inherit restriction that comes with using a single bundled node. Sure, it is easy to get it up and running but at some point I had to use `nodePort` to go around the IP restriction. This is a restriction that you will have in an actual _kubernetes_ cluster but I will show you later how to go around it. For now, let's just get a local cluster up and running.
<!--more-->
## Objective {#objective}
I needed a local _kubernetes_ cluster using all open source tools and easy to deploy. So I went with using _KVM_ as the hypervisor layer and installed `virt-manager` for shallow management. As an OS, I wanted something light and made for _kubernetes_. As I already know of Rancher (being an easy way to deploy _kubernetes_ and they have done a great job so far since the launch of their Rancer 2.0) I decided to try _RancherOS_. So let's see how all that works together.
## Requirements {#requirements}
Let's start by thinking about what we actually need. Rancher, the dashboard they offer is going to need a VM by itself and they [recommend](https://rancher.com/docs/rancher/v2.x/en/quick-start-guide/deployment/quickstart-vagrant/) _4GB of RAM_. I only have _16GB of RAM_ on my machine so I'll have to do the math to see how much I can afford to give this _dashboard_ and _manager_. By looking at the _RancherOS_ hardware [requirements](https://rancher.com/docs/os/v1.x/en/), I can tell that by giving a each node _2GB_ of RAM I should be able to host a _3 node cluster_ and with _2_ more for the _dashboard_ that puts me right on _8GB of RAM_. So we need to create _4 VMs_ with _2GB of RAM_ each.
## Installing RancherOS {#installing-rancheros}
Once all 4 nodes have been created, when you boot into the _RancherOS_ [ISO](https://rancher.com/docs/os/v1.x/en/installation/running-rancheros/workstation/boot-from-iso/) do the following.
<div class="admonition note">
<p class="admonition-title">Note</p>
Because I was using _libvirt_, I was able to do `virsh console <vm>` and run these commands.
</div>
## Virsh Console {#virsh-console}
If you are running these VMs on _libvirt_, then you can console into the box and run `vi`.
```text
# virsh list
Id Name State
-------------------------
21 kube01 running
22 kube02 running
23 kube03 running
24 rancher running
# virsh console rancher
```
## Configuration {#configuration}
If you read the _RancherOS_ [documentation](https://rancher.com/docs/os/v1.x/en/), you'll find out that you can configure the _OS_ with a `YAML` configuration file so let's do that.
```text
$ vi cloud-config.yml
```
And that file should hold.
```yaml
---
hostname: rancher.kube.loco
ssh_authorized_keys:
- ssh-rsa AAA...
rancher:
network:
interfaces:
eth0:
address: 192.168.122.5/24
dhcp: false
gateway: 192.168.122.1
mtu: 1500
```
Make sure that your **public** _ssh key_ is replaced in the example before and if you have a different network configuration for your VMs, change the network configuration here.
After you save that file, install the _OS_.
```text
$ sudo ros install -c cloud-config.yml -d /dev/sda
```
Do the same for the rest of the servers and their names and IPs should be as follows (if you are following this tutorial):
```text
192.168.122.5 rancher.kube.loco
192.168.122.10 kube01.kube.loco
192.168.122.11 kube02.kube.loco
192.168.122.12 kube03.kube.loco
```
## Post Installation Configuration {#post-installation-configuration}
After _RancherOS_ has been installed, one will need to configure `/etc/hosts` and it should look like the following if one is working off of the _Rancher_ box.
```text
$ sudo vi /etc/hosts
```
```text
127.0.0.1 rancher.kube.loco
192.168.122.5 rancher.kube.loco
192.168.122.10 kube01.kube.loco
192.168.122.11 kube02.kube.loco
192.168.122.12 kube03.kube.loco
```
Do the same on the rest of the servers while changing the `127.0.0.1` hostname to the host of the server.
## Installing Rancher {#installing-rancher}
At this point, I have to stress a few facts:
- This is not the Rancher recommended way to deploy _kubernetes_.
- The recommended way is of course [RKE](https://rancher.com/docs/rke/v0.1.x/en/).
- This is for testing, so I did not take into consideration backup of anything.
- There are ways to backup Rancher configuration by mounting storage from the `rancher` docker container.
If those points are understood, let's go ahead and deploy Rancher.
First, `$ ssh rancher@192.168.122.5` then:
```text
[rancher@rancher ~]$ docker run -d --restart=unless-stopped -p 80:80 -p 443:443 rancher/rancher
```
Give it a few minutes for the container to come up and the application as well. Meanwhile configure your `/etc/hosts` file on your machine.
```text
192.168.122.5 rancher.kube.loco
```
Now that all that is out of the way, you can login to <https://rancher.kube.loco> and set your `admin` password and the `url` for Rancher.
## Deploying Kubernetes {#deploying-kubernetes}
Now that everything is ready, let's deploy _kubernetes_ the easy way.
At this point you should be greeted with a page that looks like the
following.
{{< figure src="/ox-hugo/01-add-cluster.png" caption="Figure 1: Add Cluster Page" target="_blank" link="/ox-hugo/01-add-cluster.png" >}}
Click on the **Add Cluser**
{{< figure src="/ox-hugo/02-custom-cluster.png" caption="Figure 2: Custom Cluster Page" target="_blank" link="/ox-hugo/02-custom-cluster.png" >}}
Make sure you choose **Custom** as a _provider_. Then fill in the **Cluser Name** in our case we'll call it **kube**.
{{< figure src="/ox-hugo/03-calico-networkProvider.png" caption="Figure 3: Network Provider: Calico (Optional)" target="_blank" link="/ox-hugo/03-calico-networkProvider.png" >}}
Optionally, you can choose your **Network Providor**, in my case I chose **Calico**. Then I clicked on **show advanced** at the bottom right corner then expanded the _newly shown tab_ **Advanced Cluster Options**.
{{< figure src="/ox-hugo/04-nginx-ingressDisabled.png" caption="Figure 4: Nginx Ingress Disabled" target="_blank" link="/ox-hugo/04-nginx-ingressDisabled.png" >}}
We will disable the **Nginx Ingress** and the **Pod Security Policy Support** for the time being. This will become more apparent why in the future, hopefully. Then hit **Next**.
{{< figure src="/ox-hugo/05-customize-nodes.png" caption="Figure 5: Customize Nodes" target="_blank" link="/ox-hugo/05-customize-nodes.png" >}}
Make sure that you select all **3 Node Roles**. Set the **Public Address** and the **Node Name** to the first node and then copy the command and paste it on the _first_ node.
Do the same for _all the rest_. Once the first docker image gets downloaded and ran you should see a message pop at the bottom.
{{< figure src="/ox-hugo/06-registered-nodes.png" caption="Figure 6: Registered Nodes" target="_blank" link="/ox-hugo/06-registered-nodes.png" >}}
<div class="admonition warning">
<p class="admonition-title">warning</p>
Do **NOT** click _done_ until you see all _3 nodes registered_.
</div>
## Finalizing {#finalizing}
Now that you have _3 registered nodes_, click **Done** and go grab yourself a cup of coffee. Maybe take a long walk, this will take time. Or if you are curious like me, you'd be looking at the logs, checking the containers in a quad pane `tmux` session.
After a long time has passed, our story ends with a refresh and a welcome with this page.
{{< figure src="/ox-hugo/07-kubernetes-cluster.png" caption="Figure 7: Kubernetes Cluster" target="_blank" link="/ox-hugo/07-kubernetes-cluster.png" >}}
Welcome to your Kubernetes Cluster.
## Conclusion {#conclusion}
At this point, you can check that all the nodes are healthy and you got yourself a kubernetes cluster. In future blog posts we will explore an avenue to deploy _multiple ingress controllers_ on the same cluster on the same `port: 80` by giving them each an IP external to the cluster.
But for now, you got yourself a kubernetes cluster to play with. Enjoy.