I usually lurk around *Freenode* in a few projects that I use, can learn from and/or help with. This is a great opportunity to learn new things /all the time/.
This story is familiar in that manner, but that's where similarities diverge. Someone asked around =#Weechat= a question that caught my attention because it was, sort of, out of topic. The question was around how do you backup your stuff ?
#+hugo: more
I mean if I were asked that, I would've mentioned revision controlled off-site repositories for the code that I have.
For the personal stuff on the other hand, I would've admitted simple rudimentary solutions like =rsync=, =tar= and external drives.
So I was sort of happy with my backup solution, it has worked. Plain and simple.
I have to admit that, by modern standards it might not offer the ability to go back in time to a certain point.
But I use /file systems/ that offer /snapshot/ capabilities. I can recover from previous snapshots and send them somewhere safe.
Archiving and encrypting those is not a simple process, wish it was. That limits storage possibilities if you care to keep your data private.
But if you know me, you'd know that I'm always open to new ways of doing things.
I can't remember exactly the conversation but the name *BorgBackup* was mentioned (thank you however you are). That's when things changed.
**** BorgBackup
[[https://www.borgbackup.org/][Borg]] is defined as a
#+BEGIN_QUOTE
Deduplicating archiver with compression and encryption
#+END_QUOTE
Although this is a very accurate and encompassing definition, it doesn't really show you how /AWESOME/ this thing is.
I had to go to the docs first before I stumbled upon this video.
It can be a bit difficult to follow the video, I understand.
This is why I decided to write this post, to sort of explain to you how *Borg* can backup your stuff.
**** Encryption
Oh yeah, that's the *first* thing I look at when I consider any suggested backup solution. *Borg* offers built-in /encryption/ and /authentication/. You can read about it in details in the [[https://borgbackup.readthedocs.io/en/stable/usage/init.html#encryption-modes][docs]].
So that's a check.
**** Compression
This is another thing I look for in a suggested backup solution. And I'm happy to report that *Borg* has this under the belt as well.
*Borg* currently supports /LZ4/, /zlib/, /LZMA/ and /zstd/. You can also tune the level of compression. Pretty neat !
**** Full Backup
I've watched a few videos and read a bit of their documentation and they talk about *FULL BACKUP*.
Which means every time you run *Borg*, it will take a full backup of your stuff. A full backup at that point in time, don't forget.
The implication of this is that you have a versioned list of your backups, and you can go back in time to any of them.
Yes, you read that right. *Borg* does a full backup every time you run it. That's a pretty neat feature.
If you're a bit ahead of me, you were gonna say woooow there bud ! I have *Gigabytes* of data, what do you mean *FULL BACKUP*, you keep saying *FULL BACKUP*.
I mean *FULL BACKUP*, wait until you hear about the next feature.
**** Deduplication
Booyah ! It has deduplication. Ain't that awesome. I've watched a presentation by the project's original maintainer explain this.
I have one thing to say. It's pretty good. How good, you may ask ?
My answer would be, good enough to fool me into thinking that it was taking snapshots of my data.
It wasn't until I dug in deeper into the matter that I understood that it was a full backup and the deduping taking care of the rest.
**** Check
*Borg* offers a way to vefiry the consistency of the repository and the archives within. This way, you can make sure that your backups haven't been corrupted.
This is a very good feature, and a must in my opinion from a backup solution. *Borg* has /YOU/ covered.
**** Restore
A backup solution is nothing if you can't get your data backup.
*Borg* has a few ways for you to get your data.
You can either create an /archive/ file out of a backup. You can export a file, a directory or the whole directory tree from a backup.
You can also, if you like, mount a backup and get stuff out.
#+BEGIN_EXPORT html
<div class="admonition warning">
<p class="admonition-title">warning</p>
#+END_EXPORT
Mounting a *Borg* backup is done using /fuse/
#+BEGIN_EXPORT html
</div>
#+END_EXPORT
**** Conclusion
*Borg* is a great tool for backup. It comes in an easily installable self-contained binary so you can use it, pretty much, anywhere giving you no excuse /whatsoever/ not to use it.
Their documentation is very good, and *Borg* is easy to use.
It offers you all the features you need to do off-site and on-site backups of all your important data.
I'll be testing *Borg* moving forward for my data. I'll make sure to report back anything I find, in the future, related to the subject.
*** DONE Automating Borg :borgmatic:borgbackup:borg:
In the previous blog post entitle [[#borgbackup]], I talked about *borg*.
If you read that post, you would've noticed that *borg* has a lot of features.
With a lot of features come a lot of automation.
If you were thinking about using *borg*, you should either make a /simple cron/ or you're gonna have to write an elaborate script to take care of all the different steps.
What if I told you there's another way ? An easier way ! The *Borgmatic* way... What would you say ?
#+hugo: more
**** Borgmatic
*Borgmatic* is defined on their [[https://torsion.org/borgmatic/][website]] as follows.
#+BEGIN_QUOTE
borgmatic is simple, configuration-driven backup software for servers
and workstations. Protect your files with client-side encryption.
Backup your databases too. Monitor it all with integrated third-party
services.
#+END_QUOTE
If you go down to it, *borgmatic* uses *borg*'s /API/ to automate a list of configurable /tasks/.
This way, it saves you the trouble of writing your own scripts to automate these steps.
*Borgmatic* uses a /YAML/ configuration file. Let's configure a few tasks.
**** Location
First, let's start by configuring the locations that *borg* is going to be working with.
#+BEGIN_SRC yaml
location:
source_directories:
- /home/
repositories:
- user@backupserver:sourcehostname.borg
one_file_system: true
exclude_patterns:
- /home/*/.cache
- '*.pyc'
#+END_SRC
This tells *borg* that we need to backup our =/home= directories excluding a few patterns.
Let's not forget that we told *borg* where the repository is located at.
A few months ago, I went on a search for a solution for my /dotfiles/.
I tried projects likes [[https://www.gnu.org/software/stow/][GNU Stow]], [[https://github.com/anishathalye/dotbot][dotbot]] and a [[https://www.atlassian.com/git/tutorials/dotfiles][bare /git/ repository]].
Each one of these solutions has its advantages and its advantages, but I found mine in [[https://www.chezmoi.io/][/Chezmoi/]].
/Chezmoi/ ? That's *French* right ? How is learning *French* going to help me ?
#+hugo: more
**** Introduction
On a /*nix/ system, whether /Linux/, /BSD/ or even /Mac OS/ now, the applications one uses have their configuration saved in the user's home directory. These files are called /configuration/ files. Usually, these configuration files start with a =.= which on these systems designate hidden files (they do not show up with a simple =ls=). Due their names, these /configuration/ files are also referred to as /dotfiles/.
#+BEGIN_EXPORT html
<div class="admonition note">
<p class="admonition-title">Note</p>
#+END_EXPORT
I will be using /dotfiles/ and /configuration files/ interchangeably in this article, and they can be thought as such.
#+BEGIN_EXPORT html
</div>
#+END_EXPORT
One example of such files is the =.bashrc= file found in the user's /home directory/. It allows the user to configure /bash/ and change some behaviours.
Now that we understand what /dotfiles/ are, let's talk a little bit about the /previously mentioned/ solutions.
They deserve mentioning, especially if you're looking for such solution.
***** GNU Stow
/GNU Stow/ leverages the power of /symlinks/ to keep your /configuration/ in a *centralized* location.
Wherever your repository lives, /GNU Stow/ will mimic the internal structure of said repository in your *home directory* by /smartly symlinking/ everything.
I said /smartly/ because it tries to *minimize* the amount of /symlinks/ created by /symlinking/ to common root directories if possible.
By having all your configuration files under one directory structure, it is easier to push it to any public repository and share it with others.
The downsize is, you end-up with a lot of /symlinks/. It is also worth mentioning that not all applications behave well when their /configuration directories/ are /symlinked/. Otherwise, /GNU Stow/ is a great project.
***** Dotbot
/Dotbot/ is a /Python/ project that *aims* at automating your /dotfiles/. It gives you great control over what and how to manage your /dotfiles/.
Having it written in /Python/ means it is very easy to install; =pip=. It also means that it /should/ be easy to migrate it to different systems.
/Dotbot/ has a lot going for it. If the idea of having control over every aspect of your /dotfiles/, including the /possibility/ of the setup of the environment along with it, then /dotbot/ is for you.
Well, it's not for *me*.
***** Bare /Git/ Repository
This is arguably the /most elegant/ solution of them all.
The nice thing about this solution is its /simplicity/ and /cleanliness/. It is /essentially/ creating a /bare git/ repository /somewhere/ in your /home directory/ specifying the /home directory/ itself to be the /working directory/.
If you are wondering where one would use a /bare git/ repository in real life other than this use case.
Well, you have no other place to turn than any /git server/. On the server, /Gitea/ for example, your repository is only a /bare/ repository. One has to clone it to get the /working directory/ along with it.
Anyway, back to our topic. This is a great solution if you don't have to worry about things you would like to hide.
By hide, I mean things like /credentials/, /keys/ or /passwords/ which *never* belong in a /repository/.
You will need to find solutions for these types of files. I was looking for something /less involving/ and /more involved/.
**** /Chezmoi/ to the rescue ?
Isn't that what they *all* say ?
I like how the creator(s) defines [[https://www.chezmoi.io/][/Chezmoi/]]
#+BEGIN_QUOTE
Manage your dotfiles across multiple machines, securely.
#+END_QUOTE
Pretty basic, straight to the point. Unfortunately, it's a little bit harder to grasp the concept of how it works.
/Chezmoi/ basically /generates/ the /dotfiles/ from the /local repository/. These /dotfiles/ are saved in different forms in the /repository/ but they *always* generate the same output; the /dotfiles/. Think of /Chezmoi/ as a /dotfiles/ templating engine, at its basic form it saves your /dotfiles/ as is and /deploys/ them in *any* machine.
**** Working with /Chezmoi/
I think we should take a /quick/ look at /Chezmoi/ to see how it works.
/Chezmoi/ is written /Golang/ making it /fairly/ easy to [[https://www.chezmoi.io/docs/install/][install]] so I will forgo that boring part.
***** First run
To start using /Chezmoi/, one has to *initialize* a new /Chezmoi repository/.
#+BEGIN_SRC bash
chezmoi init
#+END_SRC
This will create a *new*/git repository/ in =~/.local/share/chezmoi=. This is now the *source state*, where /Chezmoi/ will get your /dotfiles/.
***** Plain /dotfiles/ management with /Chezmoi/
Now that we have a /Chezmoi/ repository. We can start to /populate/ it with /dotfiles/.
Let's assume that we would like to start managing one of our /dotfiles/ with /Chezmoi/.
I'm going with an /imaginary application/'s configuration directory.
This directory will hold different files with /versatile/ content types.
This is going to showcase some of /Chezmoi/'s capabilities.
#+BEGIN_EXPORT html
<div class="admonition note">
<p class="admonition-title">Note</p>
#+END_EXPORT
This is how I use /Chezmoi/. If you have a better way to do things, I'd like to hear about it!
#+BEGIN_EXPORT html
</div>
#+END_EXPORT
****** Adding a /dotfile/
This *DS9* application has its directory configuration in =~/.ds9/= where we find the =config=.
The configuration looks like any /generic//ini/ configuration.
#+BEGIN_SRC ini :tangle ~/.ds9/config
[character/sisko]
Name = Benjamin
Rank = Captain
Credentials = sisko-creds.cred
Mastodon = sisko-api.mastodon
#+END_SRC
/Nothing/ special about this file, let's add it to /Chezmoi/
#+BEGIN_SRC bash
chezmoi add ~/.ds9/config
#+END_SRC
****** Listing /dotfiles/
And /nothing/ happened... Hmm...
#+BEGIN_SRC bash
chezmoi managed
#+END_SRC
#+BEGIN_EXAMPLE
/home/user/.ds9
/home/user/.ds9/config
#+END_EXAMPLE
Okay, it seems that it is being managed.
****** Diffing /dotfiles/
We can /test/ it out by doing something like this.
#+BEGIN_SRC bash
mv ~/.ds9/config ~/.ds9/config.old
chezmoi diff
#+END_SRC
#+BEGIN_EXAMPLE
install -m 644 /dev/null /home/user/.ds9/config
--- a/home/user/.ds9/config
+++ b/home/user/.ds9/config
@@ -0,0 +1,5 @@
+[character/sisko]
+Name = Benjamin
+Rank = Captain
+Credentials = sisko-creds.cred
+Mastodon = sisko-api.mastodon
#+END_EXAMPLE
Alright, everything looks as it should be.
****** Apply /dotfiles/
But that's only a /diff/, how do I make /Chezmoi/ apply the changes because my /dotfile/ is still =config.old=.
Okay, we can actually get rid of the =config.old= file and make /Chezmoi/ regenerate the configuration.
#+BEGIN_SRC bash
rm ~/.ds9/config ~/.ds9/config.old
chezmoi -v apply
#+END_SRC
#+BEGIN_EXPORT html
<div class="admonition note">
<p class="admonition-title">Note</p>
#+END_EXPORT
I like to use the =-v= flag to check what is *actually* being applied.
#+BEGIN_EXPORT html
</div>
#+END_EXPORT
#+BEGIN_EXAMPLE
install -m 644 /dev/null /home/user/.ds9/config
--- a/home/user/.ds9/config
+++ b/home/user/.ds9/config
@@ -0,0 +1,5 @@
+[character/sisko]
+Name = Benjamin
+Rank = Captain
+Credentials = sisko-creds.cred
+Mastodon = sisko-api.mastodon
#+END_EXAMPLE
And we get the same output as the =diff=. Nice!
The configuration file was also recreated, that's awesome.
****** Editing /dotfiles/
If you've followed so far, you might have wondered... If I edit =~/.ds9/config=, then /Chezmoi/ is going to *override* it!
*YES*, *yes* it will.
#+BEGIN_EXPORT html
<div class="admonition warning">
<p class="admonition-title">warning</p>
#+END_EXPORT
Always use /Chezmoi/ to edit your managed /dotfiles/. Do *NOT* edit them directly.
*ALWAYS* use =chezmoi diff= before every /applying/.
#+BEGIN_EXPORT html
</div>
#+END_EXPORT
To /edit/ your managed /dotfile/, simply tell /Chezmoi/ about it.
#+BEGIN_SRC bash
chezmoi edit ~/.ds9/config
#+END_SRC
/Chezmoi/ will use your =$EDITOR= to open the file for you to edit. Once saved, it's saved in the /repository database/.
Be aware, at this point the changes are not reflected in your /home/ directory, *only* in the /Chezmoi source state/. Make sure you *diff* and then *apply* to make the changes in your /home/.
***** /Chezmoi/ repository management
As mentioned previously, the repository is found in =~/.local/share/chezmoi=.
I *always* forget where it is, luckily /Chezmoi/ has a solution for that.
#+BEGIN_SRC bash
chezmoi cd
#+END_SRC
Now, we are in the repository. We can work with it as a /regultar//git/ repository.
When you're done, don't forget to =exit=.
***** Other features
It is worth mentioning at this point that /Chezmoi/ offers a few more integrations.
****** Templating
Due to the fact that /Chezmoi/ is written in /Golang/, it can leverage the power of the /Golang [[https://www.chezmoi.io/docs/how-to/#use-templates-to-manage-files-that-vary-from-machine-to-machine][templating]]/ system.
One can replace /repeatable/ values like *email* or *name* with a template like ={{ .email }}= or ={{ .name }}=.
This will result in a replacement of these /templated variables/ with their real values in the resulting /dotfile/.
This is another reason why you should *always* edit your managed /dotfiles/ through /Chezmoi/.
Our /previous/ example would look a bit different.
#+BEGIN_SRC ini :tangle ~/.ds9/config
[character/sisko]
Name = {{ .sisko.name }}
Rank = {{ .sisko.rank }}
Credentials = sisko-creds.cred
Mastodon = sisko-api.mastodon
#+END_SRC
And we would add it a bit differently now.
#+BEGIN_SRC bash
chezmoi add --template ~/.ds9/config
#+END_SRC
#+BEGIN_EXPORT html
<div class="admonition warning">
<p class="admonition-title">warning</p>
#+END_EXPORT
Follow the [[https://www.chezmoi.io/docs/how-to/#use-templates-to-manage-files-that-vary-from-machine-to-machine][documentation]] to /configure/ the *values*.
#+BEGIN_EXPORT html
</div>
#+END_EXPORT
****** Password manager integration
Once you have the power of /templating/ on your side, you can always take it one step further.
/Chezmoi/ has integration with a big list of [[https://www.chezmoi.io/docs/how-to/#keep-data-private][password managers]]. These can be used directly into the /configuration files/.
In our /hypothetical/ example, we can think of the /credentials/ file (=~/.ds9/sisko-creds.cred=).
Do not /forget/ that this is also using the /templating/ engine. So you need to add as a /template/.
#+BEGIN_SRC bash
chezmoi add --template ~/.ds9/sisko-creds.cred
#+END_SRC
****** File encryption
Wait, what ! You almost slipped away right there old fellow.
We have our /Mastodon/*API* key in the =sisko-api.mastodon= file. The whole file cannot be pushed to a repository.
It turns out that /Chezmoi/ can use /gpg/ to [[https://www.chezmoi.io/docs/how-to/#use-gpg-to-keep-your-secrets][encrypt your files]] making it possible for you to push them.
To add a file encrypted to the /Chezmoi/ repository, use the following command.
#+BEGIN_SRC bash
chezmoi add --encrypt ~/.ds9/sisko-api.mastodon
#+END_SRC
****** Misc
There is a list of other features that /Chezmoi/ supports that I did not mention.
I did not use all the /features/ offered yet. You should check the [[https://www.chezmoi.io/][website]] for the full documentation.
**** Conclusion
I am fully migrated into /Chezmoi/ so far. I have used all the features above, and it has worked flawlessly so far.
I like the idea that it offers *all* the features I need while at the same time staying out of the way.
I find myself, often, editing the /dotfiles/ in my /home/ directory as a /dev/ version. Once I get to a configuration I like, I add it to /Chezmoi/. If I ever mess up badly, I ask /Chezmoi/ to override my changes.
I understand it adds a little bit of /overhead/ with the use of =chezmoi= commands, which I aliased to =cm=. But the end result is a /home/ directory which seems untouched by any tools (no symlinks, no copies, etc...) making it easier to migrate /out/ of /Chezmoi/ as a solution and into another one if I ever choose in the future.
When I first started using [[https://www.ansible.com/][ansible]], I did not know about [[https://molecule.readthedocs.io/en/latest/][molecule]]. It was a bit daunting to start a /role/ from scratch and trying to develop it without having the ability to test it. Then a co-worker of mine told me about molecule and everything changed.
#+hugo: more
I do not have any of the tools I need installed on this machine, so I will go through, step by step, how I set up ansible and molecule on any new machine I come across for writing ansible roles.
**** Requirements
What we are trying to achieve in this post, is a working ansible role that can be tested inside a docker container. To be able to achieve that, we need to install docker on the system. Follow the instructions on [[https://docs.docker.com/install/][installing docker]] found on the docker website.
**** Good Practices
First thing's first. Let's start by making sure that we have python installed properly on the system.
#+BEGIN_EXAMPLE
$ python --version
Python 3.7.1
#+END_EXAMPLE
Because in this case I have /python3/ installed, I can create a /virtualenv/ easier without the use of external tools.
Once all the steps above are complete, we can start by creating our first ansible role.
#+BEGIN_EXAMPLE
$ molecule init role -r example-role
--> Initializing new role example-role...
Initialized role in /home/elijah/sandbox/test-roles/example-role successfully.
$ tree example-role/
example-role/
├── defaults
│ └── main.yml
├── handlers
│ └── main.yml
├── meta
│ └── main.yml
├── molecule
│ └── default
│ ├── Dockerfile.j2
│ ├── INSTALL.rst
│ ├── molecule.yml
│ ├── playbook.yml
│ └── tests
│ ├── __pycache__
│ │ └── test_default.cpython-37.pyc
│ └── test_default.py
├── README.md
├── tasks
│ └── main.yml
└── vars
└── main.yml
9 directories, 12 files
#+END_EXAMPLE
You can find what each directory is for and how ansible works by visiting [[https://docs.ansible.com][docs.ansible.com]].
***** =meta/main.yml=
The meta file needs to modified and filled with information about the role. This is not a required file to modify if you are keeping this for yourself, for example. But it is a good idea to have as much information as possible if this is going to be released. In my case, I don't need any fanciness as this is just sample code.
#+BEGIN_SRC yaml
---
galaxy_info:
author: Elia el Lazkani
description: This is an example ansible role to showcase molecule at work
license: license (BDS-2)
min_ansible_version: 2.7
galaxy_tags: []
dependencies: []
#+END_SRC
***** =tasks/main.yml=
This is where the magic is set in motion. Tasks are the smallest entities in a role that do small and idempotent actions. Let's write a few simple tasks to create a user and install a service.
#+BEGIN_SRC yaml
---
# Create the user example
- name: Create 'example' user
user:
name: example
comment: Example user
shell: /bin/bash
state: present
create_home: yes
home: /home/example
# Install nginx
- name: Install nginx
apt:
name: nginx
state: present
update_cache: yes
notify: Restart nginx
#+END_SRC
***** =handlers/main.yml=
If you noticed, we are notifying a handler to be called after installing /nginx/. All handlers notified will run after all the tasks complete and each handler will only run once. This is a good way to make sure that you don't restart /nginx/ multiple times if you call the handler more than once.
#+BEGIN_SRC yaml
---
# Handler to restart nginx
- name: Restart nginx
service:
name: nginx
state: restarted
#+END_SRC
***** =molecule/default/molecule.yml=
It's time to configure molecule to do what we need. We need to start an ubuntu docker container, so we need to specify that in the molecule YAML file. All we need to do is change the image line to specify that we want an =ubuntu:bionic= image.
#+BEGIN_SRC yaml
---
dependency:
name: galaxy
driver:
name: docker
lint:
name: yamllint
platforms:
- name: instance
image: ubuntu:bionic
provisioner:
name: ansible
lint:
name: ansible-lint
scenario:
name: default
verifier:
name: testinfra
lint:
name: flake8
#+END_SRC
***** =molecule/default/playbook.yml=
This is the playbook that molecule will run. Make sure that you have all the steps that you need here. I will keep this as is.
#+BEGIN_SRC yaml
---
- name: Converge
hosts: all
roles:
- role: example-role
#+END_SRC
**** First Role Pass
This is time to test our role and see what's going on.
PLAY RECAP *********************************************************************
instance : ok=4 changed=3 unreachable=0 failed=0
#+END_EXAMPLE
It looks like the *converge* step succeeded.
**** Writing Tests
It is always a good practice to write unittests when you're writing code. Ansible roles should not be an exception. Molecule offers a way to run tests, which you can think of as unittest, to make sure that what the role gives you is what you were expecting. This helps future development of the role and keeps you from falling in previously solved traps.
***** =molecule/default/tests/test_default.py=
Molecule leverages the [[https://testinfra.readthedocs.io/en/latest/][testinfra]] project to run its tests. You can use other tools if you so wish, and there are many. In this example we will be using /testinfra/.
PLAY RECAP *********************************************************************
localhost : ok=2 changed=2 unreachable=0 failed=0
#+END_EXAMPLE
I have a few warning messages (that's likely because I am using /python 3.7/ and some of the libraries still don't fully support the new standards released with it) but all my tests passed.
**** Conclusion
Molecule is a great tool to test ansible roles quickly and while developing
them. It also comes bundled with a bunch of other features from different
projects that will test all aspects of your ansible code. I suggest you start
using it when writing new ansible roles.
** Container :@container:
*** DONE Linux Containers :linux:kernel:docker:podman:dockerfile:
Our story dates /all the way/ back to 2006, believe it or not. The first steps were taken towards what we know today as *containers*.
We'll discuss their history, how to build them and how to use them. Stick around! you might enjoy the ride.
#+hugo: more
**** History
***** 2006-2007 - The /[[https://lkml.org/lkml/2006/10/20/251][Generic Process Containers]]/ lands in Linux
This was renamed thereafter to /[[https://en.wikipedia.org/wiki/Cgroups][Control Groups]]/, popularily known as /cgroups/, and landed in /Linux/ version =2.6.24=.
/Cgroups/ are the first piece of the puzzle in /Linux Containers/. We will be talking about /cgroups/ in detail later.
***** 2008 - Namespaces
Even though /namespaces/ have been around since 2002, /Linux/ version =2.4.19=, they saw a [[https://www.redhat.com/en/blog/history-containers][rapid development]] beginning 2006 and into 2008.
/namespaces/ are the other piece of the puzzle in /Linux Containers/. We will talk about /namespaces/ in more details later.
***** 2008 - LXC
/LXC/ finally shows up!
/LXC/ is the first form of /containers/ on the /Linux/ kernel.
/LXC/ combined both /cgroups/ and /namespaces/ to provide isolated environments; containers.
#+BEGIN_EXPORT html
<div class="admonition note">
<p class="admonition-title">Note</p>
#+END_EXPORT
It is worth mentioning that /LXC/ runs a full /operating system/ containers from an image.
In other words, /LXC/ containers are meant to run more than one process.
#+BEGIN_EXPORT html
</div>
#+END_EXPORT
***** 2013 - Docker
/Docker/ offered a full set of tools for working with /containers/, making it easier than ever to work with them.
/Docker/ containers are designed to only run the application process.
Unlike /LXC/, the =PID==1= of a Docker container is excepted to be the application running in the contanier.
We will be discussing this topic in more detail later.
**** Concepts
***** /cgroups/
****** What are cgroups ?
Let's find out ! Better yet, let's use the tools at our disposal to find out together...
Open a *terminal* and run the following command.
#+BEGIN_SRC bash
man 7 cgroups
#+END_SRC
This should open the ~man~ pages for =cgroups=.
#+BEGIN_QUOTE
Control groups, usually referred to as cgroups, are a Linux kernel feature which allow processes to be organized into hierarchical groups whose usage of various types of resources can then be limited and monitored. The kernel's cgroup interface is provided through a pseudo-filesystem called cgroupfs. Grouping is implemented in the core cgroup kernel code, while resource tracking and limits are implemented in a set of per-resource-type subsystems (memory, CPU, and so on).
#+END_QUOTE
****** What does this all mean ?
This can all be simplified by explaining it in a different way.
Essentially, you can think of =cgroups= as a way for the /kernel/ to *limit* what you can *use*.
This gives us the ability to give a /container/ only *1* CPU out of the 4 available to the /kernel/.
Or maybe, limit the memory allowed to *512MB* to the container.
This way the container cannot overload the resources of the system in case they run a fork-bomb, for example.
But, =cgroups= do not limit what we can "/see/".
***** /namespaces/
****** /Namespaces/ to the rescue !
As we did before, let's check the ~man~ page for =namespaces=
#+BEGIN_SRC bash
man 7 namespaces
#+END_SRC
#+BEGIN_QUOTE
A namespace wraps a global system resource in an abstraction that makes it appear to the processes within the namespace that they have their own isolated instance of the global resource. Changes to the global resource are visible to other processes that are members of the namespace, but are invisible to other processes. One use of namespaces is to implement containers.
#+END_QUOTE
Wooow ! That's more mumbo jumbo ?!
****** Is it really simple ?
Let's simplify this one as well.
You can think of =namespaces= as a way for the /kernel/ to *limit* what we *see*.
There are multiple =namespaces=, like the =cgroup_namespaces= which /virtualizes/ the view of a process =cgroup=.
In other words, inside the =cgroup= the process with =PID= *1* is not =PID= on the *system*.
The =namespaces= manual page lists them, you check them out for more details. But I hope you get the gist of it !
***** Linux Containers
We are finally here! Let's talk /Linux Containers/.
The first topic we need to know about is *images*.
****** What are container images ?
We talked before that /Docker/ came in and offered tooling around /containers/.
One of those concepts which they used, in docker images, is *layers*.
First of all, an image is a /file-system/ representation of a container.
It is an on-disk, read-only, image. It sort of looks like your /Linux/*filesystem*.
Then, layers on top to add functionality. You might ask, what are these layers. We will see them in action.
Let's look at my system.
#+BEGIN_SRC bash
lsb_release -a
#+END_SRC
#+begin_example
LSB Version: n/a
Distributor ID: ManjaroLinux
Description: Manjaro Linux
Release: 20.2.1
Codename: Nibia
#+end_example
As you can see, I am running =Manjaro=. Keep that in mind.
Let's take a look at the kernel running on this machine.
#+BEGIN_SRC bash
uname -a
#+END_SRC
#+begin_example
Linux manjaro 5.10.15-1-MANJARO #1 SMP PREEMPT Wed Feb 10 10:42:47 UTC 2021 x86_64 GNU/Linux
#+end_example
So, it's /kernel version/=5.8.6=. Remember this one as well.
******* /neofetch/
I would like to /test/ a tool called =neofetch=. Why ?
- First reason, I am not that creative.
- Second, it's a nice tool, you'll see.
We can test =neofetch=
#+BEGIN_SRC bash
neofetch
#+END_SRC
#+begin_example
fish: Unknown command: neofetch
#+end_example
Look at that! We don't have it installed...
Not a big deal. We can download an image and test it inside.
****** Pulling an image
Let's download a docker image. I am using =podman=, an open source project that allows us to *use* containers.
#+BEGIN_EXPORT html
<div class="admonition note">
<p class="admonition-title">Note</p>
#+END_EXPORT
You might want to run these commands with =sudo= privileges.
As you can see, we just successfully built the container. We also got a =hash= as a name for it.
If you were careful, I used the =&&= command instead of using multiple =RUN=. You *can* use as many =RUN= commands ase you like.
But be careful, each one of those commands creates a *layer*. The /more/ layers you create, the /more/ time they require to *download*/*upload*.
It might not seem to be a lot of time to download a few extra layer on one system. But if we talk about /container orchestration/ platforms, it makes a big difference there.
Let's examine the build a bit more and see what we got.
#+BEGIN_EXAMPLE
STEP 1: FROM ubuntu:20.04
STEP 2: RUN apt-get update && apt-get install -y neofetch
#+END_EXAMPLE
The first step was to /download/ the base image so we could use it, then we added a *layer* which insatlled neofetch. If we list our *images*.
#+BEGIN_SRC bash :dir /sudo:: :results output
podman images
#+END_SRC
#+begin_example
REPOSITORY TAG IMAGE ID CREATED SIZE
localhost/neofetch-ubuntu 20.04 6486fa42efe5 5 minutes ago 241 MB
docker.io/library/ubuntu 20.04 f63181f19b2f 5 weeks ago 75.3 MB
#+end_example
We can see that we have =localhost/neofetch-ubuntu=. If we examine the =ID=, we can see that it is the same as the one given to us at the end of the build.
****** Running our container
Now that we created a /brand-spanking-new/ image, we can run it.
#+BEGIN_SRC bash :dir /sudo:: :results output
podman images
#+END_SRC
#+begin_example
REPOSITORY TAG IMAGE ID CREATED SIZE
localhost/neofetch-ubuntu 20.04 6486fa42efe5 6 minutes ago 241 MB
docker.io/library/ubuntu 20.04 f63181f19b2f 5 weeks ago 75.3 MB
#+end_example
First we list our *images*. Then we choose which one to run.
Finally thought /before/ I let you go. You may have noticed that I used =Podman= instead of =Docker=. In these examples, both commands should be interchangeable.
Remember kids, /containers/ are cool! They can be used for a wide variety of things. They are great at many things and with the help of /container orchestration/ platforms, they can scale better than ever. They are also very bad at certain things. Be careful where to use them, how to use and when to use them. Stay safe and mainly have fun!
*** DONE Playing with containers and Tor :docker:linux:@text_editors:ubuntu:fedora:proxy:privoxy:
As my followers well know, by now, I am a tinkerer at heart. Why do I do things ? No one knows ! I don't even know.
All I know, all I can tell you is that I like to see what can I do with the tools I have at hand. How can I bend them to my will.
Why, you may ask. The answer is a bit complicated; part of who I am, part of what I do as a DevOps. End line is, this time I was curious.
I went down a road that taught me so much more about /containers/, /docker/, /docker-compose/ and even /Linux/ itself.
The question I had was simple, *can I run a container only through Tor running in another container?*
#+hugo: more
**** Tor
I usually like to start topics that I haven't mentioned before with definitions. In this case, what is [[https://2019.www.torproject.org/index.html.en][Tor]], you may ask ?
#+begin_quote
Tor is free software and an open network that helps you defend against traffic analysis, a form of network surveillance that threatens personal freedom and privacy, confidential business activities and relationships, and state security.
#+end_quote
Although that /home/ page is obscure because it was replaced by the new /design/ of the website.
Although I love what *Tor* has done with all the services they offer, don't get me wrong.
But giving so much importance on the browser only and leaving the rest for dead when it comes to website, I have to say, I'm a bit sad.
Anyway, let's share the love for *Tor* and thank them for the beautiful project they offered humanity.
Now that we thanked them, let's abuse it.
***** Tor in a container
The task I set to discover relied on *Tor* being containerized.
The first thing I do is, simply, not re-invent the wheel.
Let's find out if someone already took that task.
With a litte bit of search, I found the [[https://hub.docker.com/r/dperson/torproxy][dperson/torproxy]] docker image.
It isn't ideal but I /believe/ it is written to be rebuilt.
Can we run it ?
#+begin_src bash
docker run -it -p 127.0.0.1:8118:8118 -d dperson/torproxy
#+end_src
#+begin_src bash
curl -Lx http://localhost:8118 http://jsonip.com/
#+end_src
And this is *definitely* not your IP. Don't take /my word/ for it!
Go to [[http://jsonip.com/][http://jsonip.com/]] in a browser and see for yourself.
Now that we *know* we can run *Tor* in a container effectively, let's kick it up a /notch/.
**** docker-compose
I will be /testing/ and making changes as I go along. For this reason, it's a good idea to use [[https://docs.docker.com/compose/][docker-compose]] to do this.
#+begin_quote
Compose is a tool for defining and running multi-container Docker applications. With Compose, you use a YAML file to configure your application’s services. Then, with a single command, you create and start all the services from your configuration.
#+end_quote
/Now/ that we saw what the *docker* team has to say about *docker-compose*, let's go ahead and use it.
First, let's implement what we just ran /ad-hoc/ in *docker-compose*.
#+begin_src yaml
---
version: '3.9'
services:
torproxy:
image: dperson/torproxy
container_name: torproxy
restart: unless-stopped
#+end_src
**** Air-gapped container
The next piece of the puzzle is to figure out *if* and *how* can we create an /air-gapped container/.
It turns out, we can create an =internal= network in /docker/ that has no access to the internet.
First, the /air-gapped container/.
#+begin_src yaml
air-gapped:
image: ubuntu
container_name: air-gapped
restart: unless-stopped
command:
- bash
- -c
- sleep infinity
networks:
- no-internet
#+end_src
Then comes the network.
#+begin_src yaml
networks:
no-internet:
driver: bridge
internal: true
#+end_src
Let's put it all together in a =docker-compose.yaml= file and run it.
#+begin_src bash
docker-compose up -d
#+end_src
Keep that terminal open, and let's put the /hypothesis/ to the test and see if rises up to be a /theory/.
I've been playing around with containers for a few years now. I find them very useful.
If you host your own, like I do, you probably write a lot of /nginx/ configurations, maybe /apache/.
If that's the case, then you have your own solution to get certificates.
I'm also assuming that you are using /let's encrypt/ with /certbot/ or something.
Well, I didn't want to anymore. It was time to consolidate. Here comes Traefik.
#+hugo: more
**** Traefik
So [[https://doc.traefik.io/traefik/][Traefik]] is
#+begin_quote
an open-source Edge Router that makes publishing your services a fun and easy experience. It receives requests on behalf of your system and finds out which components are responsible for handling them.
#+end_quote
Which made me realize, I still need /nginx/ somewhere. We'll see when we get to it. Let's focus on /Traefik/.
***** Configuration
If you run a lot of containers and manage them, then you probably use /docker-compose/.
I'm still using =version 2.3=, I know I am due to an upgrade but I'm working on it slowly.
It's a bigger project... One step at a time.
Let's start from the top, literally.
#+NAME: docker-compose-header
#+begin_src yaml
---
version: '2.3'
services:
#+end_src
#+BEGIN_EXPORT html
<div class="admonition note">
<p class="admonition-title">Note</p>
#+END_EXPORT
Upgrading to =version 3.x= of /docker-compose/ requires the creation of /network/ to /link/ containers together. It's worth investing into, this is not a /docker-compose/ tutorial.
#+BEGIN_EXPORT html
</div>
#+END_EXPORT
Then comes the service.
#+NAME: docker-compose-service-traefik
#+begin_src yaml
traefik:
container_name: traefik
image: "traefik:latest"
restart: unless-stopped
mem_limit: 40m
mem_reservation: 25m
#+end_src
and of course, who can forget the volume mounting.
#+NAME: docker-compose-traefik-volumes
#+begin_src yaml
volumes:
- "/var/run/docker.sock:/var/run/docker.sock:ro"
#+end_src
***** Design
Now let's talk design to see how we're going to configuse this bad boy.
I want to /Traefik/ to listen on ports =80= and =443= at a minimum to serve traffic.
Let's do that.
#+NAME: docker-compose-traefik-config-listeners
#+begin_src yaml
command:
- --entrypoints.web.address=:80
- --entrypoints.websecure.address=:443
#+end_src
and let's not forget to map them.
#+NAME: docker-compose-traefik-port-mapping
#+begin_src yaml
ports:
- "80:80"
- "443:443"
#+end_src
Next, we would like to redirect =http= to =https= always.
/Let's Encrypt/ have set limits on *how many* certificates you can request per certain amount of time. To test your certificate request and renewal processes, use their staging infrastructure. It is made for such purpose.
#+BEGIN_EXPORT html
</div>
#+END_EXPORT
Then we mount it, for persistence.
#+NAME: docker-compose-traefik-volumes-acme
#+begin_src yaml
- "./traefik/acme.json:/acme.json"
#+end_src
Let's not forget to add our /Cloudflare/*API* credentials as environment variables for /Traefik/ to use.
#+NAME: docker-compose-traefik-environment
#+begin_src yaml
environment:
- CLOUDFLARE_EMAIL=<your-cloudflare@email.here>
- CLOUDFLARE_API_KEY=<your-api-key-goes-here>
#+end_src
***** Dashboard
Now let's configure /Traefik/ a bit more with a bit of labeling.
First, we specify the /host//Traefik/ should listen for to service the /dashboard/.
With a little bit of /Traefik/ documentation searching and a lot of help from =htpasswd=, we can create a =basicauth= login to protect the dashboard from public use.
I'm not going to go into details about the /middleware/ flags configured here but you're welcome to check the /Traefik/ middleware [[https://doc.traefik.io/traefik/middlewares/overview/][docs]].
[engine x] is an HTTP and reverse proxy server, a mail proxy server, and a generic TCP/UDP proxy server, originally written by Igor Sysoev.
#+end_quote
In this example, we're going to assume you have a /static blog/ generated by a /static blog generator/ of your choice and you would like to serve it for people to read it.
So let's do this quickly as there isn't much to tell except when it comes to labels.
We are mounting the blog directory from our /host/ to =/usr/share/nginx/html/blog= as *read-only* into the /nginx/ container. We are also providing /nginx/ with a template configuration and passing the variables as /environment/ variables as you noticed. It is also mounted as *read-only*. The configuration template looks like the following, if you're wondering.
#+begin_src nginx
server {
listen ${NGINX_BLOG_PORT};
server_name localhost;
root /usr/share/nginx/html/${NGINX_BLOG_HOST};
location / {
index index.html;
try_files $uri $uri/ =404;
}
}
#+end_src
***** Traefik configuration
So, /Traefik/ configuration at this point is a little bit tricky for the first time.
First, we configure the /host/ like we did before.
I have had a *Pi* laying around used for a simple task for a while now.
A few days ago, I was browsing the web, learning more about privacy, when I stumbled upon [[https://adguard.com/en/welcome.html][AdGuard Home]].
I have been using it as my internal DNS on top of the security and privacy layers I add to my machine.
Its benefits can be argued but it is a DNS after all and I wanted to see what else it can do for me.
Anyway, I digress. I searched to see if I could find a container for *AdGuard Home* and I did.
At this point, I started thinking about what I could do to make the [[https://www.raspberrypi.org/][Pi]] more useful.
That's when [[https://k3s.io/][k3s]] came into the picture.
#+hugo: more
**** Pre-requisites
As this is not a *Pi* tutorial, I am going to be assuming that you have a /Raspberry Pi/ with *Raspberry Pi OS*/Buster/ installed on it.
The assumption does not mean you cannot install any other OS on the Pi and run this setup.
It only means that I have tested this on /Buster/ and that your milage will vary.
**** Prepare the Pi
Now that you have /Buster/ already installed, let's go ahead and [[https://rancher.com/docs/k3s/latest/en/advanced/#enabling-legacy-iptables-on-raspbian-buster][fix]] a small default configuration issue with it.
*K3s* uses =iptables= to route things around correctly. /Buster/ uses =nftables= by default, let's switch it to =iptables=.
At this point, your /Pi/ should reboot. Your *OS* is configured for the next step.
**** Pre-install Configuration
After testing *k3s* a few times, I found out that by /default/ it will deploy a few extra services like [[https://docs.traefik.io/][Traefik]].
Unfortunately, just like anything the /default/ configuration is just that. It's plain and not very useful from the start. You will need to tweak it.
This step could be done either /post/ or /pre/ deploy. Figuring out the /pre-deploy/ is a bit more involving but a bit more fun as well.
The first thing you need to know is that the normal behavior of *k3s* is to deploy anything found in =/var/lib/rancher/k3s/server/manifests/=.
So a good first step is, of course, to proceed with creating that.
#+BEGIN_EXAMPLE
$ mkdir -p /var/lib/rancher/k3s/server/manifests/
#+END_EXAMPLE
The other thing to know is that *k3s* can deploy /Helm Charts/.
It will create the /manifests/ it will deploy by default, before beginning the setup, in the manifest path I mentioned.
If you would like to see what it deployed and how, visit that path after *k3s* runs.
I did, and I took their configuration of *Traefik* which I was unhappy with its /defaults/.
My next step was securing the /defaults/ as much as possible and I found out that *Traefik* can do [[https://docs.traefik.io/v2.0/middlewares/basicauth/][basic authentication]].
As a starting point, that's great. Let's create the credentials.
#+BEGIN_EXAMPLE
$ htpasswd -c ./auth myUser
#+END_EXAMPLE
That was easy so far. Let's turn up the notch and create the manifest for *k3s*.
Create =traefik.yaml= in =/var/lib/rancher/k3s/server/manifests/= with the following content.
If you have ever worked with /kubernetes/, you'd know that minikube out of the box does not give you what you need for a quick setup. I'm sure you can go =minikube start=, everything's up... Great... =kubectl get pods -n kube-system=... It works, let's move on...
But what if it's not let's move on to something else. We need to look at this as a local test environment in capabilities. We can learn so much from it before applying to the lab. But, as always, there are a few tweaks we need to perform to give it the magic it needs to be a real environment.
#+hugo: more
**** Prerequisites
If you are looking into /kubernetes/, I would suppose that you know your linux's ABCs and you can install and configure /minikube/ and its prerequisites prior to the beginning of this tutorial.
You can find the guide to install /minikube/ and configure it on the /minikube/ [[https://kubernetes.io/docs/setup/minikube/][webpage]].
Anyway, make sure you have /minikube/ installed, /kubectl/ and whatever driver dependencies you need to run it under that driver. In my case, I am using /kvm2/ which will be reflected in the commands given to start /minikube/.
**** Starting /minikube/
Let's start minikube.
#+BEGIN_EXAMPLE
$ minikube start --vm-driver=kvm2
Starting local Kubernetes v1.13.2 cluster...
Starting VM...
Getting VM IP address...
Moving files into cluster...
Setting up certs...
Connecting to cluster...
Setting up kubeconfig...
Stopping extra container runtimes...
Starting cluster components...
Verifying apiserver health ...
Kubectl is now configured to use the cluster.
Loading cached images from config file.
Everything looks great. Please enjoy minikube!
#+END_EXAMPLE
Great... At this point we have a cluster that's running, let's verify.
#+BEGIN_EXAMPLE
# Id Name State
--------------------------
3 minikube running
#+END_EXAMPLE
For me, I can check =virsh=. If you used /VirtualBox/ you can check that.
Now what ? Well, now we deploy a few addons that we need to deploy in production as well for a functioning /kubernetes/ cluster.
Let's check the list of add-ons available out of the box.
#+BEGIN_EXAMPLE
$ minikube addons list
- addon-manager: enabled
- dashboard: enabled
- default-storageclass: enabled
- efk: disabled
- freshpod: disabled
- gvisor: disabled
- heapster: enabled
- ingress: enabled
- kube-dns: disabled
- metrics-server: enabled
- nvidia-driver-installer: disabled
- nvidia-gpu-device-plugin: disabled
- registry: disabled
- registry-creds: disabled
- storage-provisioner: enabled
- storage-provisioner-gluster: disabled
#+END_EXAMPLE
Make sure you have /dashboard/, /heapster/, /ingress/ and /metrics-server/*enabled*. You can enable add-ons with =kubectl addons enable=.
**** What's the problem then ?
Here's the problem that comes next. How do you access the dashboard or anything running in the cluster ? Everyone online suggests you proxy a port and you access the dashboard. Is that really how it should work ? Is that how production system do it ?
The answer is of course not. They use different types of /ingresses/ at their disposal. In this case, /minikube/ was kind enough to provide one for us, the default /kubernetes ingress controller/, It's a great option for an ingress controller that's solid enough for production use. Fine, a lot of babble. Yes sure but this babble is important. So how do we access stuff on a cluster ?
To answer that question we need to understand a few things. Yes, you can use a =NodePort= on your service and access it that way. But do you really want to manage these ports ? What's in use and what's not ? Besides, wouldn't it be better if you can use one port for all of the services ? How you may ask ?
We've been doing it for years, and by we I mean /ops/ and /devops/ people. You have to understand that the kubernetes ingress controller is simply an /nginx/ under the covers. We've always been able to configure /nginx/ to listen for a specific /hostname/ and redirect it where we want to. It shouldn't be that hard to do right ?
Well this is what an ingress controller does. It uses the default ports to route traffic from the outside according to hostname called. Let's look at our cluster and see what we need.
#+BEGIN_EXAMPLE
$ kubectl get services --all-namespaces
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
In my case, you can see that I have a few things that are in =NodePort= configuration and you can access them on those ports. But the /kubernetes-dashboard/ is a =ClusterIP= and we can't get to it. So let's change that by adding an ingress to the service.
**** Ingress
An ingress is an object of kind =ingress= that configures the ingress controller of your choice.
#+BEGIN_SRC yaml
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: kubernetes-dashboard
namespace: kube-system
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /
spec:
rules:
- host: dashboard.kube.local
http:
paths:
- path: /
backend:
serviceName: kubernetes-dashboard
servicePort: 80
#+END_SRC
Save that to a file =kube-dashboard-ingress.yaml= or something then run.
Now all we need to know is the IP of our kubernetes cluster of /one/.
Don't worry /minikube/ makes it easy for us.
#+BEGIN_EXAMPLE
$ minikube ip
192.168.39.79
#+END_EXAMPLE
Now let's add that host to our =/etc/hosts= file.
#+BEGIN_EXAMPLE
192.168.39.79 dashboard.kube.local
#+END_EXAMPLE
Now if you go to [[http://dashboard.kube.local]] in your browser, you will be welcomed with the dashboard. How is that so ? Well as I explained, point it to the nodes of the cluster with the proper hostname and it works.
You can deploy multiple services that can be accessed this way, you can also integrate this with a service mesh or a service discovery which could find the up and running nodes that can redirect you to point to at all times. But this is the clean way to expose services outside the cluster.
*** DONE Your First Minikube Helm Deployment :minikube:ingress:helm:prometheus:grafana:
In the last post, we have configured a basic /minikube/ cluster. In this post we will deploy a few items we will need in a cluster and maybe in the future, experiment with it a bit.
#+hugo: more
**** Prerequisite
During this post and probably during future posts, we will be using /helm/ to deploy to our /minikube/ cluster. Some offered by the helm team, others by the community and maybe our own. We need to install =helm= on our machine. It should be as easy as downloading the binary but if you can find it in your package manager go that route.
**** Deploying Tiller
Before we can start with the deployments using =helm=, we need to deploy /tiller/. It's a service that manages communications with the client and deployments.
#+BEGIN_EXAMPLE
$ helm init --history-max=10
Creating ~/.helm
Creating ~/.helm/repository
Creating ~/.helm/repository/cache
Creating ~/.helm/repository/local
Creating ~/.helm/plugins
Creating ~/.helm/starters
Creating ~/.helm/cache/archive
Creating ~/.helm/repository/repositories.yaml
Adding stable repo with URL: https://kubernetes-charts.storage.googleapis.com
Adding local repo with URL: http://127.0.0.1:8879/charts
$HELM_HOME has been configured at ~/.helm.
Tiller (the Helm server-side component) has been installed into your Kubernetes Cluster.
Please note: by default, Tiller is deployed with an insecure 'allow unauthenticated users' policy.
To prevent this, run ``helm init`` with the --tiller-tls-verify flag.
For more information on securing your installation see: https://docs.helm.sh/using_helm/#securing-your-helm-installation
#+END_EXAMPLE
/Tiller/ is deployed, give it a few minutes for the pods to come up.
**** Deploy Prometheus
We often need to monitor multiple aspects of the cluster easily. Sometimes maybe even write our applications to (let's say) publish metrics to prometheus. And I said 'let's say' because technically we offer an endpoint that a prometheus exporter will consume regularly and publish to the prometheus server. Anyway, let's deploy prometheus.
NOTES: The Prometheus Operator has been installed. Check its status by
running: kubectl --namespace kube-prometheus get pods -l
"release=prometheus-operator"
Visit [[https://github.com/coreos/prometheus-operator]] for
instructions on how to create & configure Alertmanager and Prometheus
instances using the Operator.
#+END_EXAMPLE
At this point, prometheus has been deployed to the cluster. Give it a few minutes for all the pods to come up. Let's keep on working to get access to the rest of the consoles offered by the prometheus deployment.
**** Prometheus Console
Let's write an ingress configuration to expose the prometheus console.
First off we need to list all the service deployed for prometheus.
#+BEGIN_EXAMPLE
$ kubectl get service prometheus-operator-prometheus -o yaml -n kube-prometheus
I wanted to explore /kubernetes/ even more for myself and for this blog. I've worked on pieces of this at work but not the totality of the work which I would like to understand for myself. I wanted, also to explore new tools and ways to leverage the power of /kubernetes/.
So far, I have been using /minikube/ to do the deployments but there is an inherit restriction that comes with using a single bundled node. Sure, it is easy to get it up and running but at some point I had to use =nodePort= to go around the IP restriction. This is a restriction that you will have in an actual /kubernetes/ cluster but I will show you later how to go around it. For now, let's just get a local cluster up and running.
#+hugo: more
**** Objective
I needed a local /kubernetes/ cluster using all open source tools and easy to deploy. So I went with using /KVM/ as the hypervisor layer and installed =virt-manager= for shallow management. As an OS, I wanted something light and made for /kubernetes/. As I already know of Rancher (being an easy way to deploy /kubernetes/ and they have done a great job so far since the launch of their Rancer 2.0) I decided to try /RancherOS/. So let's see how all that works together.
**** Requirements
Let's start by thinking about what we actually need. Rancher, the dashboard they offer is going to need a VM by itself and they [[https://rancher.com/docs/rancher/v2.x/en/quick-start-guide/deployment/quickstart-vagrant/][recommend]] /4GB of RAM/. I only have /16GB of RAM/ on my machine so I'll have to do the math to see how much I can afford to give this /dashboard/ and /manager/. By looking at the /RancherOS/ hardware [[https://rancher.com/docs/os/v1.x/en/][requirements]], I can tell that by giving a each node /2GB/ of RAM I should be able to host a /3 node cluster/ and with /2/ more for the /dashboard/ that puts me right on /8GB of RAM/. So we need to create /4 VMs/ with /2GB of RAM/ each.
**** Installing RancherOS
Once all 4 nodes have been created, when you boot into the /RancherOS/ [[https://rancher.com/docs/os/v1.x/en/installation/running-rancheros/workstation/boot-from-iso/][ISO]] do the following.
#+BEGIN_EXPORT html
<div class="admonition note">
<p class="admonition-title">Note</p>
#+END_EXPORT
Because I was using /libvirt/, I was able to do =virsh console <vm>= and run these commands.
#+BEGIN_EXPORT html
</div>
#+END_EXPORT
**** Virsh Console
If you are running these VMs on /libvirt/, then you can console into the box and run =vi=.
#+BEGIN_EXAMPLE
# virsh list
Id Name State
-------------------------
21 kube01 running
22 kube02 running
23 kube03 running
24 rancher running
# virsh console rancher
#+END_EXAMPLE
**** Configuration
If you read the /RancherOS/ [[https://rancher.com/docs/os/v1.x/en/][documentation]], you'll find out that you can configure the /OS/ with a =YAML= configuration file so let's do that.
#+BEGIN_EXAMPLE
$ vi cloud-config.yml
#+END_EXAMPLE
And that file should hold.
#+BEGIN_SRC yaml
---
hostname: rancher.kube.loco
ssh_authorized_keys:
- ssh-rsa AAA...
rancher:
network:
interfaces:
eth0:
address: 192.168.122.5/24
dhcp: false
gateway: 192.168.122.1
mtu: 1500
#+END_SRC
Make sure that your *public*/ssh key/ is replaced in the example before and if you have a different network configuration for your VMs, change the network configuration here.
Do the same for the rest of the servers and their names and IPs should be as follows (if you are following this tutorial):
#+BEGIN_EXAMPLE
192.168.122.5 rancher.kube.loco
192.168.122.10 kube01.kube.loco
192.168.122.11 kube02.kube.loco
192.168.122.12 kube03.kube.loco
#+END_EXAMPLE
**** Post Installation Configuration
After /RancherOS/ has been installed, one will need to configure =/etc/hosts= and it should look like the following if one is working off of the /Rancher/ box.
#+BEGIN_EXAMPLE
$ sudo vi /etc/hosts
#+END_EXAMPLE
#+BEGIN_EXAMPLE
127.0.0.1 rancher.kube.loco
192.168.122.5 rancher.kube.loco
192.168.122.10 kube01.kube.loco
192.168.122.11 kube02.kube.loco
192.168.122.12 kube03.kube.loco
#+END_EXAMPLE
Do the same on the rest of the servers while changing the =127.0.0.1= hostname to the host of the server.
**** Installing Rancher
At this point, I have to stress a few facts:
- This is not the Rancher recommended way to deploy /kubernetes/.
- The recommended way is of course [[https://rancher.com/docs/rke/v0.1.x/en/][RKE]].
- This is for testing, so I did not take into consideration backup of anything.
- There are ways to backup Rancher configuration by mounting storage from the =rancher= docker container.
If those points are understood, let's go ahead and deploy Rancher.
Optionally, you can choose your *Network Providor*, in my case I chose *Calico*. Then I clicked on *show advanced* at the bottom right corner then expanded the /newly shown tab/*Advanced Cluster Options*.
We will disable the *Nginx Ingress* and the *Pod Security Policy Support* for the time being. This will become more apparent why in the future, hopefully. Then hit *Next*.
Make sure that you select all *3 Node Roles*. Set the *Public Address* and the *Node Name* to the first node and then copy the command and paste it on the /first/ node.
Do the same for /all the rest/. Once the first docker image gets downloaded and ran you should see a message pop at the bottom.
Do *NOT* click /done/ until you see all /3 nodes registered/.
#+BEGIN_EXPORT html
</div>
#+END_EXPORT
**** Finalizing
Now that you have /3 registered nodes/, click *Done* and go grab yourself a cup of coffee. Maybe take a long walk, this will take time. Or if you are curious like me, you'd be looking at the logs, checking the containers in a quad pane =tmux= session.
After a long time has passed, our story ends with a refresh and a welcome with this page.
At this point, you can check that all the nodes are healthy and you got yourself a kubernetes cluster. In future blog posts we will explore an avenue to deploy /multiple ingress controllers/ on the same cluster on the same =port: 80= by giving them each an IP external to the cluster.
But for now, you got yourself a kubernetes cluster to play with. Enjoy.
*** DONE Deploying Helm in your Kubernetes Cluster :helm:tiller:
In the previous post in the /kubernetes/ series, we deployed a small /kubernetes/ cluster locally on /KVM/. In future posts we will be deploying more things into the cluster. This will enable us to test different projects, ingresses, service meshes, and more from the open source community, build specifically for /kubernetes/. To help with this future quest, we will be leveraging a kubernetes package manager. You've read it right, helm is a kubernetes package manager. Let's get started shall we ?
#+hugo: more
**** Helm
As mentioned above, helm is a kubernetes package manager. You can read more about the helm project on their [[https://helm.sh/][homepage]]. It offers a way to Go template the deployments of service and package them into a portable package that can be installed using the helm command line.
Generally, you would install the helm binary on your machine and install it into the cluster. In our case, the /RBACs/ deployed in the kubernetes cluster by rancher prevent the default installation from working. Not a problem, we can go around the problem and we will in this post. This is a win for us because this will give us the opportunity to learn more about helm and kubernetes.
#+BEGIN_EXPORT html
<div class="admonition note">
<p class="admonition-title">Note</p>
#+END_EXPORT
This is not a production recommended way to deploy helm. I would *NOT* deploy helm this way on a production cluster. I would restrict the permissions of any =ServiceAccount= deployed in the cluster to its bare minimum requirements.
#+BEGIN_EXPORT html
</div>
#+END_EXPORT
**** What are we going to do ?
We need to understand a bit of what's going on and what we are trying to do. To be able to do that, we need to understand how /helm/ works. From a high level, the =helm= command line tool will deploy a service called /Tiller/ as a =Deployment=.
The /Tiller/ service talks to the /kubernetes//API/ and manages the deployment process while the =helm= command line tool talks to /Tiller/ from its end. So a proper deployment of /Tiller/ in a /kubernetes/ sense is to create a =ServiceAccount=, give the =ServiceAccount= the proper permissions to be able to do what it needs to do and you got yourself a working /Tiller/.
**** Service Account
This is where we start by creating a =ServiceAccount=. The =ServiceAccount= looks like this.
#+BEGIN_SRC yaml
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: tiller
namespace: kube-system
#+END_SRC
We de deploy the =ServiceAccount= to the cluster. Save it to =ServiceAccount.yaml=.
#+BEGIN_EXAMPLE
$ kubectl apply -f ServiceAccount.yaml
serviceaccount/tiller created
#+END_EXAMPLE
#+BEGIN_EXPORT html
<div class="admonition note">
<p class="admonition-title">Note</p>
#+END_EXPORT
To read more about =ServiceAccount= and their uses please visit the /kubernetes/ documentation page on the [[https://kubernetes.io/docs/reference/access-authn-authz/service-accounts-admin/][topic]].
#+BEGIN_EXPORT html
</div>
#+END_EXPORT
**** Cluster Role Binding
We have /Tiller/ (=ServiceAccount=) deployed in =kube-system= (=namespace=). We need to give it access.
***** Option 1
We have the option of either creating a =Role= which would restrict /Tiller/ to the current =namespace=, then tie them together with a =RoleBinding=.
This option will restrict /Tiller/ to that =namespace= and that =namespace= only.
***** Option 2
Another option is to create a =ClusterRole= and tie the =ServiceAccount= to that =ClusterRole= with a =ClusterRoleBinding= and this will give /Tiller/ access across /namespaces/.
***** Option 3
In our case, we already know that =ClustRole==cluster-admin= already exists in the cluster so we are going to give /Tiller/=cluster-admin= access.
#+BEGIN_SRC yaml
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: tiller
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
subjects:
- kind: ServiceAccount
name: tiller
namespace: kube-system
#+END_SRC
Save the following in =ClusterRoleBinding.yaml= and then
#+BEGIN_EXAMPLE
$ kubectl apply -f ClusterRoleBinding.yaml
clusterrolebinding.rbac.authorization.k8s.io/tiller created
#+END_EXAMPLE
**** Deploying Tiller
Now that we have all the basics deployed, we can finally deploy /Tiller/ in the cluster.
Adding stable repo with URL: https://kubernetes-charts.storage.googleapis.com
Adding local repo with URL: http://127.0.0.1:8879/charts
$HELM_HOME has been configured at ~/.helm.
Tiller (the Helm server-side component) has been installed into your Kubernetes Cluster.
Please note: by default, Tiller is deployed with an insecure 'allow unauthenticated users' policy.
To prevent this, run `helm init` with the --tiller-tls-verify flag.
For more information on securing your installation see: https://docs.helm.sh/using_helm/#securing-your-helm-installation
Happy Helming!
#+END_EXAMPLE
#+BEGIN_EXPORT html
<div class="admonition note">
<p class="admonition-title">Note</p>
#+END_EXPORT
Please make sure you read the helm installation documentation if you are deploying this in a production environment. You can find how you can make it more secure [[https://helm.sh/docs/using_helm/#securing-your-helm-installation][there]].
#+BEGIN_EXPORT html
</div>
#+END_EXPORT
After a few minutes, your /Tiller/ deployment or as it's commonly known as a =helm install= or a =helm init=. If you want to check that everything has been deployed properly you can run.
Everything seems to be working properly. In future posts, we will be leveraging the power and convenience of helm to expand our cluster's capabilities and learn more about what we can do with kubernetes.
** MISC :@misc:
*** DONE A Quick ZFS Overview on Linux :zfs:file_system:
I have, for years, been interested in /file systems/. Specifically a /file system/ to run my personal systems on. For most people *Ext4* is good enough and that is totally fine. But, as a power user, I like to have more control, more features and more options out of my file system.
I have played with most of file sytsems on Linux, and have been using *Btrfs* for a few years now. I have worked with NAS systems running on *ZFS* and have been very impressed by it. The only problem is that *ZFS* wasn't been well suppored on Linux at the time. *Btrfs* promissed to be the *ZFS* replacement for Linux nativetly, especially that it was backed up by a bunch of the giants like Oracle and RedHat. My decision at that point was made, and yes that was before RedHat's support for *XFS* which is impressive on its own. Recently though, a new project gave everyone hope. [[http://www.open-zfs.org/wiki/Main_Page][OpenZFS]] came to life and so did [[https://zfsonlinux.org/][ZFS on Linux]].
#+hugo: more
Linux has had *ZFS* support for a while now but mostly to manage a *ZFS*/file system/, so I kept watching until I saw a blog post by *Ubuntu* entitled [[https://ubuntu.com/blog/enhancing-our-zfs-support-on-ubuntu-19-10-an-introduction][Enhancing our ZFS support on Ubuntu 19.10 -- an introduction]].
In the blog post above, I read the following:
#+BEGIN_QUOTE
We want to support ZFS on root as an experimental installer option, initially for desktop, but keeping the layout extensible for server later on. The desktop will be the first beneficiary in Ubuntu 19.10. Note the use of the term ‘experimental' though!
#+END_QUOTE
My eyes widened at this point. I know that *Ubuntu* has had native *ZFS* support since 2016 but now I could install it with one click. At that point I was all in, and I went back to *Ubuntu*.
**** Ubuntu on root ZFS
You heard me right, the *Ubuntu* installer offers an 'experimental' install on *ZFS*. I made the decision based on the well tested stability of *ZFS* in production environments and its ability to offer me the flexibility and the ability to backup and recover my data easily.
In other words, if *Ubuntu* doesn't work, *ZFS* is there and I can install whatever I like on top and if you are familiar with *ZFS* you know exactly what I mean and I have barely scratched the ice on its capabilities.
So here I was with *Ubuntu* installed on my laptop on root *ZFS*. So I had to do it.
#+BEGIN_EXAMPLE
# zpool status -v
pool: bpool
state: ONLINE
status: The pool is formatted using a legacy on-disk format. The pool can
still be used, but some features are unavailable.
action: Upgrade the pool using 'zpool upgrade'. Once this is done, the
pool will no longer be accessible on software that does not support
feature flags.
scan: none requested
config:
NAME STATE READ WRITE CKSUM
bpool ONLINE 0 0 0
nvme0n1p4 ONLINE 0 0 0
errors: No known data errors
pool: rpool
state: ONLINE
scan: none requested
config:
NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
nvme0n1p5 ONLINE 0 0 0
errors: No known data errors
#+END_EXAMPLE
#+BEGIN_EXPORT html
<div class="admonition note">
<p class="admonition-title">Note</p>
#+END_EXPORT
I have read somewhere in a blog about *Ubuntu* that I should not run an upgrade on the boot pool.
#+BEGIN_EXPORT html
</div>
#+END_EXPORT
and it's running on...
#+BEGIN_EXAMPLE
# uname -s -v -i -o
Linux #28-Ubuntu SMP Wed Dec 18 05:37:46 UTC 2019 x86_64 GNU/Linux
#+END_EXAMPLE
Well that was pretty easy.
**** ZFS Pools
Let's take a look at how the installer has configured the /pools/.
#+BEGIN_EXAMPLE
# zpool list
NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
bpool 1,88G 158M 1,72G - - - 8% 1.00x ONLINE -
rpool 472G 7,91G 464G - - 0% 1% 1.00x ONLINE -
#+END_EXAMPLE
So it creates a /boot/ pool and a /root/ pool. Maybe looking at the
*datasets* would give us a better idea.
**** ZFS Datasets
Let's look at the sanitized version of the datasets.
I was asked recently about how I have my email client setup. As I naturally do, I replied with something along the lines of the following.
#+begin_quote
I use isync, notmuch, afew and msmtp with emacs as an interface, let me get you a link on how I did my setup from my blog.
#+end_quote
To my surprise, I never wrote about the topic. I guess this is as better time as any to do so.
Let's dig in.
#+hugo: more
**** Bird's-eye View
Looking at the big list of tools mentioned in the title, I /could/ understand how one could get intimidated but I *assure* you these are very basic, yet very powerful, tools.
First task is to divide and conquer, as usual. We start by the first piece of the puzzle, understand email.
In a very simplified way of thinking of email is that each email is simply a file. This file has all the information needed as to who sent it to whom, from which server, etc...
The bottom line is that it's simply a file in a folder somewhere on a server. Even though this might not be the case on the server, in this setup it will most certainly be the case locally on your filesystem. Thinking about it in terms of files in directories also makes sense because it will most likely be synchronized back with the server that way as well.
Now you might ask, what tool would offer us such a way to synchronize emails and my answer would be... Very many, of course... come on this is /Linux/ and /Open Source/ ! Don't ask silly questions... But to what's relevant to my setup it's /isync/.
Now that I have the emails locally on my filesystem, I need a way to interact with them. Some prefer to work with directories, I prefer to work with tags instead. That's where /notmuch/ comes in. You can think of it as an email tagging and querying system. To make my life simpler, I utilize /afew/ to handle a few basic email tasks to save me from writing a lot of /notmuch/ rules.
I already make use of /emacs/ extensively in my day to day life and having a /notmuch/ interface in /emacs/ is great. I can use /emacs/ to view, tag, search and send email.
Oh wait, right... I wouldn't be able to send email without /msmtp/.
**** isync
[[https://isync.sourceforge.io/][isync]] is defined as
#+begin_quote
a command line application which synchronizes mailboxes.
#+end_quote
While isync currently supports *Maildir* and *IMAP4* mailboxes, it has the very logical command of =mbsync=. Of course !
Now, /isync/ is very well documented in the =man= pages.
#+begin_src bash
man mbsync
#+end_src
Everything you need is there, have fun reading.
While you read the =man= pages to figure out what you want, I already did that and here's what I want in my =~/.mbsyncrc=.
#+begin_src conf
##########################
# Personal Configuration #
##########################
# Name Account
IMAPAccount Personal
Host email.hostname.com
User personal@email.hostname.com
Pass "yourPassword"
# One can use a command which returns the password
The following will synchronize both ways the following folders:
- Remote "Inbox" with local "Inbox"
- Remote "Archive" with local "Archive"
- Remote "Sent" with local "Sent"
- Remote "Junk" with local "Trash"
Those are the only directories I care about.
With the configuration in place, we can try to sync the emails.
#+begin_src bash
mbsync -C -a -V
#+end_src
**** notmuch
You can read more about [[https://notmuchmail.org/][notmuch]] on their webpage. Their explanation is interesting to say the least.
What /notmuch/ does, is create a database where it saves all the tags and relevant information for all the emails. This makes it extremely fast to query and do different operations on large numbers of emails.
I use /notmuch/ mostly indirectly through /emacs/, so my configuration is very simple. All I want from /notmuch/ is to tag all *new* emails with the =new= tag.
#+begin_src conf
# .notmuch-config - Configuration file for the notmuch mail system
#
# For more information about notmuch, see https://notmuchmail.org
# Database configuration
#
# The only value supported here is 'path' which should be the top-level
# directory where your mail currently exists and to where mail will be
# delivered in the future. Files should be individual email messages.
# Notmuch will store its database within a sub-directory of the path
# configured here named ".notmuch".
#
[database]
path=/home/user/.mail/
# User configuration
#
# Here is where you can let notmuch know how you would like to be
# addressed. Valid settings are
#
# name Your full name.
# primary_email Your primary email address.
# other_email A list (separated by ';') of other email addresses
# at which you receive email.
#
# Notmuch will use the various email addresses configured here when
# formatting replies. It will avoid including your own addresses in the
# recipient list of replies, and will set the From address based on the
# address to which the original email was addressed.
# tags A list (separated by ';') of the tags that will be
# added to all messages incorporated by "notmuch new".
#
# ignore A list (separated by ';') of file and directory names
# that will not be searched for messages by "notmuch new".
#
# NOTE: *Every* file/directory that goes by one of those
# names will be ignored, independent of its depth/location
# in the mail store.
#
[new]
tags=new;
#tags=unread;inbox;
ignore=
# Search configuration
#
# The following option is supported here:
#
# exclude_tags
# A ;-separated list of tags that will be excluded from
# search results by default. Using an excluded tag in a
# query will override that exclusion.
#
[search]
exclude_tags=deleted;spam;
# Maildir compatibility configuration
#
# The following option is supported here:
#
# synchronize_flags Valid values are true and false.
#
# If true, then the following maildir flags (in message filenames)
# will be synchronized with the corresponding notmuch tags:
#
# Flag Tag
# ---- -------
# D draft
# F flagged
# P passed
# R replied
# S unread (added when 'S' flag is not present)
#
# The "notmuch new" command will notice flag changes in filenames
# and update tags, while the "notmuch tag" and "notmuch restore"
# commands will notice tag changes and update flags in filenames
#
[maildir]
synchronize_flags=true
#+end_src
Now that /notmuch/ is configured the way I want it to, I use it as follows.
#+begin_src bash
notmuch new
#+end_src
Yup, that simple.
This will tag all new emails with the =new= tag.
**** afew
Once all the new emails have been properly tagged with the =new= tag by /notmuch/, /afew/ comes in.
[[https://github.com/afewmail/afew][/afew/]] is defined as an initial tagging script for /notmuch/. The reason of using it will become evident very soon but let me quote some of what their Github page says.
#+begin_quote
It can do basic thing such as adding tags based on email headers or maildir folders, handling killed threads and spam.
In move mode, afew will move mails between maildir folders according to configurable rules that can contain arbitrary notmuch queries to match against any searchable attributes.
#+end_quote
This is where the bulk of the configuration is, in all honesty. At this stage, I had to make a decision of how would I like to manage my emails ?
I think it should be simple if I save them as folders on the server as it doesn't support tags. I can derive the basic tags from the folders and keep a backup of my database for all the rest of the tags.
Basically, I make sure that all the emails, in their folders, are tagged properly. I make sure the emails which need to be moved are moved to their designated folders. The rest is simply the inbox.
#+BEGIN_EXPORT html
<div class="admonition note">
<p class="admonition-title">Note</p>
#+END_EXPORT
The *read* / *unread* tag is automatically handled between /notmuch/ and /isync/. It's seemlessly synchronized between the tools.
#+BEGIN_EXPORT html
</div>
#+END_EXPORT
With the configuration in place, I run /afew/.
#+begin_src bash
afew -v -t --new
#+end_src
For moving the emails, I use /afew/ as well but I apply it on all emails and not just the ones tagged with =new=.
#+begin_src bash
afew -v -m --all
#+end_src
**** msmtp
[[https://marlam.de/msmtp/][/msmtp/]] is an SMTP client. It sends email.
The configuration is very simple.
#+begin_src conf
# Set default values for all following accounts.
defaults
auth on
tls on
tls_trust_file /etc/ssl/certs/ca-certificates.crt
logfile ~/.msmtp.log
# Mail
account personal
host email.hostname.com
port 587
from personal@email.hostname.com
user personal@email.hostname.com
password yourPassword
# One can use a command which returns the password
# Such as a password manager or a bash script
# passwordeval sh script/path
# Set a default account
account default : personal
#+end_src
**** Emacs
I use [[https://github.com/hlissner/doom-emacs][/Doom/]] as a configuration framework for /Emacs/. /notmuch/ comes as a modules which I enabled, but you might want to check the /notmuch/'s /Emacs/ [[https://notmuchmail.org/notmuch-emacs/][Documentation]] page for help with installation and configuration.
I wanted to configure the /notmuch/ interface a bit to show me what I'm usually interested in.
#+begin_src elisp
(setq +notmuch-sync-backend 'mbsync)
(setq notmuch-saved-searches '((:name "Unread"
:query "tag:inbox and tag:unread"
:count-query "tag:inbox and tag:unread"
:sort-order newest-first)
(:name "Inbox"
:query "tag:inbox"
:count-query "tag:inbox"
:sort-order newest-first)
(:name "Archive"
:query "tag:archive"
:count-query "tag:archive"
:sort-order newest-first)
(:name "Sent"
:query "tag:sent or tag:replied"
:count-query "tag:sent or tag:replied"
:sort-order newest-first)
(:name "Trash"
:query "tag:deleted"
:count-query "tag:deleted"
:sort-order newest-first))
)
#+end_src
Now, all I have to do is simply open the =notmuch= interface in /Emacs/.
**** Conclusion
To put everything together, I wrote a /bash script/ with the commands provided above in series. This script can be called by a *cron* or /even/*manually* to synchronize emails.
From the /Emacs/ interface I can do pretty much everything I need to do.
Future improvements I have to think about is the best way to do email notifications. There are a lot of different ways I can approach this. I can use notmuch to query for what I want. I could maybe even try querying the information out of the [[https://xapian.org/][Xapian]] database. But that's food for thought.
I want email to be simple and this makes it simple for me. How are you making email simple for you ?
*** DONE Email IMAP Setup with isync :email:isync:imap:
The blog post "[[#email-setup-with-isync-notmuch-afew-msmtp-and-emacs]]" prompted a few questions. The questions were around synchronizing email in general.
I did promise to write up more blog posts to explain the pieces I brushed over quickly for brevity and ease of understanding. Or so I thought !
#+hugo: more
**** Maildir
Let's talk *Maildir*. [[https://en.wikipedia.org/wiki/Maildir][Wikipedia]] defines it as the following.
#+begin_quote
The Maildir e-mail format is a common way of storing email messages in which each message is stored in a separate file with a unique name, and each mail folder is a file system directory. The local file system handles file locking as messages are added, moved and deleted. A major design goal of Maildir is to eliminate the need for program code to handle file locking and unlocking.
#+end_quote
It is basically what I mentioned before. Think of your emails as folders and files. The image will get clearer, so let's dig even deeper.
If you go into a *Maildir* directory, let's say *Inbox* and list all the directories in there, you'll find tree of them.
#+begin_src bash
$ ls
cur/ new/ tmp/
#+end_src
These directories have a purpose.
- =tmp/=: This directory stores all temporary files and files in the process of being delivered.
- =new/=: This directory stores all new files that have not yet been /seen/ by any email client.
- =cur/=: This directory stores all the files that have been previously seen.
This is basically how emails are going to be represented on your disk. You will need to find an /email client/ which can parse these files and work with them.
**** IMAP
The *Internet Mail Access Protocol*, shortened to *[[https://en.wikipedia.org/wiki/Internet_Message_Access_Protocol][IMAP]]*, is an
#+begin_quote
Internet standard protocol used by email clients to retrieve email messages from a mail server over a TCP/IP connection.
#+end_quote
In simple terms, it is a way of communication that allows synchronization between a /client/ and an /email server/.
**** What can you do with that information ?
Now, you have all the pieces of the puzzle to figure out how to think about your email on disk and how to synchronize it.
It might be a good idea to dive a little bit into my configuration and why I chose these settings to begin with. Shall we ?
**** isync
Most /email servers/ nowadays offer you an *IMAP* (*POP3* was another protocol used widely back in the day) endpoint to connect to. You might be using /Outlook/ or /Thunderbird/ or maybe even /Claws-mail/ as an /email client/. They usually show you the emails in a neat *GUI* (Graphical User Interface) with all the /read/ and /unread/ mail and the /folders/. If you've had the chance to configure one of these clients a few years ago, you would've needed to find the *IMAP*/host/ and /port/ of the server. These clients /talk/*IMAP* too.
[[https://isync.sourceforge.io/][isync]] is an application to synchronize mailboxes. I use it to connect to my /email server/ using *IMAP* and synchronize my emails to my hard drive as a *Maildir*.
***** IMAP
The very first section of the configuration is the *IMAP* section.
#+begin_src conf
IMAPAccount Personal
Host email.hostname.com
User personal@email.hostname.com
Pass "yourPassword"
# One can use a command which returns the password
In here, we configure the *IMAP* settings. Most notably here is of course =Host=, =User= and =Pass/PassCmd=. These settings refer to your server and you should populate them with that information.
The =IMAPStore= is used further in the configuration, this gives a name for the *IMAP*/Store/. In simple terms, if you want to refer to your /server/ you use =personal-remote=.
***** Maildir
The next section of the configuration is the *Maildir* part. You can think of this as where do you want /your emails/ to be saved /on disk/.
#+begin_src conf
MaildirStore personal-local
Subfolders Verbatim
Path ~/.mail/
Inbox ~/.mail/Inbox
#+end_src
This should be self explanatory but I'd like to point out the =MaildirStore= key. This refers to /email/ on /disk/. So, if you want to refer to your /emails on disk/ you use =personal-local=.
At this point, you are thinking to yourself what the hell does that mean ? What is this dude talking about ! Don't worry, I got you.
***** Synchronize to your taste
This is where all what you've learned comes together. The fun part ! The part where you get to choose how you want to do things.
Here's what I want. I want to /synchronize/ my /server/*Inbox* with my /on disk/*Inbox* both ways. If the *Inbox* folder does not exist /on disk/, create it. The name of the *Inbox* on the server is =Inbox=.
This can be translated to the following.
#+begin_src conf
Channel sync-personal-inbox
Master :personal-remote:"Inbox"
Slave :personal-local:Inbox
Create Slave
SyncState *
CopyArrivalDate yes
#+end_src
I want to do the same with =Archive= and =Sent=.
#+begin_src conf
Channel sync-personal-archive
Master :personal-remote:"Archive"
Slave :personal-local:Archive
Create Slave
SyncState *
CopyArrivalDate yes
Channel sync-personal-sent
Master :personal-remote:"Sent"
Slave :personal-local:Sent
Create Slave
SyncState *
CopyArrivalDate yes
#+end_src
At this point, I still have my /trash/. The /trash/ on the server is called =Junk= but I want it to be =Trash= on disk. I can do that easily as follows.
#+begin_src conf
Channel sync-personal-trash
Master :personal-remote:"Junk"
Slave :personal-local:Trash
Create Slave
SyncState *
CopyArrivalDate yes
#+end_src
I choose to /synchronize/ my /emails/ both ways. If you prefer, for example, not to download the /sent/ emails and only /synchronize/ them up to the server, you can do that with =SyncState=. Check the =mbsync= manual pages.
***** Tie the knot
At the end, add all the channel names configured above under the save /Group/ with the same account name.
#+begin_src conf
Group Personal
Channel sync-personal-inbox
Channel sync-personal-archive
Channel sync-personal-sent
Channel sync-personal-trash
#+end_src
**** Conclusion
This is pretty much it. It is that simple. This is how I synchronize my email. How do you ?
*** DONE A Python Environment Setup :python:pipx:pyenv:virtual_environment:virtualfish:
I've been told that =python= package management is bad. I have seen some really bad practices online, asking you to run commands here and there without an understanding of the bigger picture, what they do and sometimes with escalated privileges.
Along the years, I have compiled a list of practices I follow, and a list of tools I use. I hope to be able to share some of the knowledge I've acquired and show you a different way of doing things. You might learn about a new tool, or a new use for a tool. Come along for the ride !
#+hugo: more
**** Python
As most know, [[https://www.python.org/][Python]] is an interpreted programming language. I am not going to go into the details of the language in this post, I will only talk about management.
If you want to develop in Python, you need to install libraries. You can find /some/ in your package manager but let's face it =pip= is your way.
The majority of /Linux/ distributions will have Python installed as a lot of system packages now rely on it, even some package managers.
Okay, this is the last time I actually use the system's Python. What ? Why ? You ask !
**** pyenv
I introduce you to [[https://github.com/pyenv/pyenv][pyenv]]. Pyenv is a Python version management tool, it allows you to install and manage different versions of Python as a /user/.
Beautiful, music to my ears.
Let's get it from the package manager, this is a great use of the package manager if it offers an up to date version of the package.
#+begin_src bash
sudo pacman -S pyenv
#+end_src
If you're not using an /Archlinux/ based distribution follow the instructions on their [[https://github.com/pyenv/pyenv#installation][webpage]].
Alright ! Now that we've got ourselves pyenv, let's configure it real quickly.
Following the docs, I created =~/.config/fish/config.d/pyenv.fish= and in it I put the following.
#+begin_src fish
# Add pyenv executable to PATH by running
# the following interactively:
set -Ux PYENV_ROOT $HOME/.pyenv
set -U fish_user_paths $PYENV_ROOT/bin $fish_user_paths
# Load pyenv automatically by appending
# the following to ~/.config/fish/config.fish:
status is-login; and pyenv init --path | source
#+end_src
Open a new shell and you're all ready to continue along, you're all locked, loaded and ready to go!
***** Setup the environment
This is the first building block of my environment. We first start by querying for Python versions available for us.
#+begin_src bash
pyenv install --list
#+end_src
Then, we install the latest Python version. Yes, even if it's an upgrade, I'll handle the upgrade, as well, as we go along.
Set everything up to use the new installed version.
First, we set the global Python version for our /user/.
#+begin_src bash
pyenv global 3.9.5
#+end_src
Then, we switch our current shell's Python version, instead of opening a new shell.
#+begin_src bash
pyenv shell 3.9.5
#+end_src
That was easy. We test that everything works as expected by checking the version.
#+begin_src bash
pyenv version
#+end_src
Now, if you do a =which= on the =python= executable, you will find that it is in the =pyenv= shims' directory.
***** Upgrade
In the *future*, the upgrade path is exactly the same as the setup path shown above. You query for the list of Python versions available, choose the latest and move on from there.
Very easy, very simple.
**** pip
[[https://pypi.org/project/pip/][pip]] is the package installer for Python.
At this stage, you have to understand that you are using a Python version installed by /pyenv/ as your /user/. The pip provided, if you do a =which=, is also in the same shims directory.
Using =pip= at this stage as a /user/ is better than running it as /root/ but it is also not touching your system; just your user. We can do *one* better. I'm going to use =pip= as a /user/ once !
I know, you will have a lot of questions at this point as to why. You will see, patience is a virtue.
**** pipx
Meet [[https://github.com/pypa/pipx][pipx]], this tool is the *amazing* companion for a /DevOps/, and /developer/ alike. Why ? You would ask.
It, basically, creates Python /virtual environments/ for packages you want to have access to /globally/. For example, I'd like to have access to a Python *LSP* server on the go.
This way my text editor has access to it too and, of course, can make use of it freely. Anyway, let's cut this short and show you. You will understand better.
Let's use the only =pip= command as a /user/ to install =pipx=.
#+begin_src bash
pip install --user pipx
#+end_src
#+BEGIN_EXPORT html
<div class="admonition warning">
<p class="admonition-title">warning</p>
#+END_EXPORT
You are setting yourself up for a *world of hurt* if you use =sudo= with =pip= or run it as =root=. *ONLY* run commands as =root= or with escalated privileges when you know what you're doing.
#+BEGIN_EXPORT html
</div>
#+END_EXPORT
***** LSP Server
As I gave the *LSP* server as an example, let's go ahead and install it with some other Python packages needed for global things like /emacs/.
#+begin_src bash
pipx install black
pipx install ipython
pipx install isort
pipx install nose
pipx install pytest
pipx install python-lsp-server
#+end_src
Now each one is in it's own happy little /virtual environment/ separated from any other dependency but its own. Isn't that lovely ?
If you try to run =ipython=, you will see that it will actually work. If you look deeper at it, you will see that it is pointing to =~/.local/bin/ipython= which is a symlink to the actual package in a /pipx//virtual environment/.
***** Upgrade
After you *set* a new Python version with /pyenv/, you simply reinstall everything.
#+begin_src bash
pipx reinstall-all
#+end_src
And like magic, everything get recreated using the new version of Python /newly/ set.
**** virtualfish
Now that /pipx/ is installed, let's go head and install something to manage our Python /virtual environments/ on-demand, for use whenever we need to, for targeted projects.
Some popular choices people use are [[https://pipenv.pypa.io/en/latest/][Pipenv]], [[https://python-poetry.org/][Poetry]], [[https://virtualenv.pypa.io/en/latest/][virtualenv]] and plain and simple python with the =venv= module.
You're welcome to play with all of them. Considering I use /fish/ as my default /shell/, I like to use [[https://virtualfish.readthedocs.io/en/latest/][virtualfish]].
Let's install it.
#+begin_src bash
pipx install virtualfish
#+end_src
This offers me a new command; =vf=. With =vf=, I can create Python /virtual environments/ and they will all be saved in a directory of my choosing.
***** Setup
Let's create one for [[https://docs.ansible.com/ansible/latest/index.html][Ansible]].
#+begin_src bash
vf new ansible
#+end_src
This should *activate* it. Then, we install /Ansible/.
#+begin_src bash
pip install ansible molecule docker
#+end_src
At this stage, you will notice that you have =ansible= installed. You will also notice that all the /pipx/ packages are also still available.
If you want to tie /virtualfish/ to a specific directory, use =vf connect=.
***** Upgrade
To /upgrade/ the Python version of all of our /virtual environments/, /virtualfish/ makes it as easy as
#+begin_src bash
vf upgrade
#+end_src
And we're done !
**** Workflow
At this stage, you have an idea about the tools I use and where their scope falls. I like them because they are /limited/ to their own scope, each has its own little domain where it reigns.
- I use *pyenv* to install and manage different versions of Python for testing purposes while I stay on the latest.
- I use *pipx* for the commands that I need access to /globally/ as a user.
- I use *virtualfish* to create one or more /virtual environment/ per project I work on.
With this setup, I can test with different versions of Python by creating different /virtual environments/ with different version each, or two versions of the tool you're testing as you keep the Python version static.
It could also be different versions of a library, testing forward compatibility for example.
At each step, I have an upgrade path to keep all my environments running the latest versions. I also have a lot of flexibility by using =requirements.txt= files and others for /development/ sometimes or even /testing/.
**** Conclusion
As you can see, with a little bit of knowledge and by standing on the shoulders of giants, you can easily manage a Python environment entirely as a /user/.
You have full access to a wide array of Python distributions to play with. Endless different versions of packages, /globally/ and /locally/ installed.
If you create /virtual environments/ for each of your projects, you won't fall in the common pitfalls of versioning hell.
Keep your /virtual environments/ numerous and dedicated to projects, small sets, and you won't face any major problems with keeping your system clean yet up to date.
*** DONE My Path Down The Road of Cloudflare's Redirect Loop :cloudflare:cdn:
In a previous post entitled "[[#automating-borg]]", I showed you how you can automate your *borg* backups with *borgmatic*.
After I started using *borgmatic* for my backups and hooked it to a /cron/ running every 2 hours, I got interested into knowing what's happening to my backups at all times.
My experience comes handy in here, I know I need a monitoring system. I also know that traditional monitoring systems are too complex for my use case.
I need something simple. I need something I can deploy myself.
#+hugo: more
**** Choosing a monitoring system
I already know I don't want a traditional monitoring system like /nagios/ or /sensu/ or /prometheus/. It is not needed, it's an overkill.
I went through the list of hooks that *borgmatic* offers out of the box and checked each project.
I came across [[https://healthchecks.io/][HealthChecks]].
**** HealthChecks
The [[https://healthchecks.io/][HealthChecks]] project works in a simple manner.
It simply offers syou an endpoint which you need to ping within a certain period, otherwise you get paged.
It has a lot of integrations from simple emails to other third party services that will call or message you or even trigger push notifications to your phone.
In my case, a simple email is enough. After all, they are simply backups and if they failed now, they will work when cron runs again in 2 hours.
**** Deploy
Let's create a docker-compose service configuration that looks like the
We start by defining a few variables for the website hostname to monitor, the check ID provided by /healthchecks/ and finally the /healthchecks/ base link for the monitors.
Once those are set, we simply use =curl= with a couple of special flags to make sure that it fails properly if something goes wrong.
We start the /healthchecks/ timer, run the website check and either call the passing or the failing /healthchecks/ endpoint depending on the outcomes.
I have recently blogged about moving to /emacs/ and the reasons behind it.
Since then, I have used /Orgmode/ a lot more. And I have begun to like it even more. I had a plan to move the blog to /[[https://gohugo.io/][Hugo]]/. After giving it a try, I had inconsistent results. I must've been doing something wrong. I've spend a lot more time than I anticipated on it. At some point, it becomes an endeavor with diminishing returns. So I ditched that idea.
But why did I want to move to /Hugo/ in the first place ?
#+hugo: more
**** Why /Hugo/ you may ask
Well, the answer to that question is very simple; /Orgmode/.
The long answer is that the default /Nikola/ markup language and the most worked on is /reStructuredText/. It can support other formats. /Orgmode/ also seems widely supported and can be easily manipulated. So I want to move to /Orgmode/ instead of /rst/.
But what are the odds ?
Damn... It has plugins and you can find an [[https://plugins.getnikola.com/v8/orgmode/][orgmode]] page where you find
#+BEGIN_EXAMPLE
$ nikola plugin -i orgmode
#+END_EXAMPLE
Where the heck did that come from ? Okay that was easy.
Turns out /Nikola/ supports /Orgmode/.
**** Nikola /Orgmode/ plugin installation
The page suggests running.
#+BEGIN_EXAMPLE
$ nikola plugin -i orgmode
#+END_EXAMPLE
Followed by
#+BEGIN_SRC python
# NOTE: Needs additional configuration in init.el file.
# Add the orgmode compiler to your COMPILERS dict.
[2020-08-31 23:16:29] INFO: auto: Serving on http://127.0.0.1:8000/ ...
[2020-08-31 23:16:36] INFO: auto: Server is shutting down.
#+END_EXAMPLE
I knew there was a catch !
You might be looking for the error message and it might take you a while. It took me a bit to find out what was wrong. The error is actually the following.
#+BEGIN_EXAMPLE
Please install htmlize from https://github.com/hniksic/emacs-htmlize
#+END_EXAMPLE
It turns out that the plugin is a /python/ script that calls /emacs/ with a configuration =init.el=. I know I have /htmlize/ installed on my /doom/ system but /Nikola/ does not see it.
After looking around the internet, I found the =init.el= file I'm looking for. It's in =plugins/orgmode/init.el= and it has the following few lines at the top.
#+BEGIN_SRC emacs-lisp
(require 'package)
(setq package-load-list '((htmlize t)))
(package-initialize)
#+END_SRC
Okay, that's what's trying to load /htmlize/. Let's try to add it to the =load-path= as follows.
In my case, the path to =htmlize= is =~/.emacs.d/.local/straight/build/htmlize=.
If you don't have it installed, simply =git clone= the repository in a directory and =load-path= that path.
#+BEGIN_EXPORT html
</div>
#+END_EXPORT
Now, let's try /Nikola/.
#+BEGIN_EXAMPLE
$ nikola auto
[2020-08-31 23:30:32] INFO: auto: Rebuilding the site...
Scanning posts..........done!
[2020-08-31 23:30:36] INFO: auto: Serving on http://127.0.0.1:8000/ ...
#+END_EXAMPLE
Woohoo ! It works. Now let's move to the next steps. Writing our first blog post.
**** First /Org/ post
Let's create this blog post.
#+BEGIN_EXPORT html
<div class="admonition warning">
<p class="admonition-title">warning</p>
#+END_EXPORT
It is very important to use the =nikola= command line interface to create the post. I spent too much time trying to figure out the /header/ settings.
#+BEGIN_EXPORT html
</div>
#+END_EXPORT
#+BEGIN_EXAMPLE
$ nikola new_post -1 -f orgmode -t orgmode posts/misc/welcome-back-to-the-old-world.org
#+END_EXAMPLE
Now edit the /org/ file and save it. /Nikola/ should pick it up and render it.
**** Yes, I have made more changes
***** Theme
I have moved the blog to the /[[https://themes.getnikola.com/v8/willy-theme/][willy-theme]]/ which offers /light/ and *dark* modes and good code highlighting.
***** Blog post format
You might have also noticed that there were big changes to the repository. All the blog posts have been converted to /Orgmode/ now, both /pages/ and /posts/.
I used [[https://pandoc.org/][pandoc]] to do the initial conversion from /rst/ to /Orgmode/ as follows.
I know, I know. It does a pretty good initial job but you will need to touch up the posts. Fortunately, I did not have a lot of blog posts yet. Unfortunately, I had enough for the task to take a few days. For me, it was worth it.
**** Conclusion
This was a long overdue project, I am happy to finally put it behind me and move foward with something simple that works with my current flow.
After publishing my /blog/ in new form yesterday night, I have received some suggestions for changes to the theme.
First off, I noticed that the footer is not showing after the blog was deployed. That reminded me that I have made changes to the original theme on disk. The pipeline, though, install the theme fresh before deploying the website.
I needed to fix that. Here's how I did it.
#+hugo: more
**** Create a new theme
This might be counter intuitive but /themes/ in /Nikola/ can actually have parents. So what we need to do is clone the theme we want to modify while keeping it as parent to our theme. I'll show you.
First, create your new theme.
#+BEGIN_EXAMPLE
$ nikola theme --new custom-willy-theme --parent willy-theme --engine=jinja
#+END_EXAMPLE
#+BEGIN_EXPORT html
<div class="admonition note">
<p class="admonition-title">Note</p>
#+END_EXPORT
I had to use =--engine=jinja= because /willy-theme/ uses jinja templating. If you are using the /mako/ engine, you don't need to add thihs as the *default* is /mako/.
#+BEGIN_EXPORT html
</div>
#+END_EXPORT
#+BEGIN_EXPORT html
<div class="admonition warning">
<p class="admonition-title">warning</p>
#+END_EXPORT
You will /probably/ need both themes in your =themes/= directory. The /willy-theme/ needs to be installed before creating your /custom/ theme from it.
#+BEGIN_EXPORT html
</div>
#+END_EXPORT
This should create =themes/custom-willy-theme/=. If we look inside, we'll see one file that describes this /theme/ with its *parent*.
Go to your =conf.py= and change the /theme/ to =custom-willy-theme=.
**** Let's talk hierarchy
Now that we have our own /custom theme/ out of the /willy-theme/, if we rebuild the blog we can see that nothing changes. Of course, we have not made any modifications. But did you ever ask yourself the question, why did the site not change ?
If your theme points to a *parent*, whatever /Nikola/ expects will have to be *your theme first* with a *failover to the parent* theme. Ok, if you've followed so far, you will need to know what /Nikola/ is expecting right ?
You can dig into the /documentation/ here to find out what you can do, but I wanted to change a few things to the theme. I wanted to add a footer, for example.
It turns out for /willy-theme/ that is located in the =templates/base.tmpl=. All I did was the following
I made my modification to the =base.tmpl= and rendered the blog. It was that simple. My changes were made.
**** Conclusion
You can always clone the /theme repository/ and make your modifications to it. But maintenance becomes an issue. This seems to be a cleaner way for me to make modifications on the original /theme/ I'm using. This is how you can too.
The topic of /git/ came up recently a lot at work. Questions were asked about why I like to do what I do and the reasoning beind.
Today, I joined =#dgplug= on [[https://freenode.net/][freenode]] and it turns out it was class time and the topic is /git/ and writing a post on it.
Which got me thinking... Why not do that ?
#+hugo: more
**** Requirements
I'd like to start my post with a requirement, /git/. It has to be installed on your machine, obviously, for you to be able to follow along.
**** A Few Concepts
I'm going to try to explain a few concepts in a very simple way. That means I am sacrificing accuracy for ease of understanding.
***** What is revision control?
[[https://en.wikipedia.org/wiki/Version_control][Wikipedia]] describes it as:
#+BEGIN_QUOTE
"A component of software configuration management, version control,
also known as revision control or source control, is the management
of changes to documents, computer programs, large web sites, and
other collections of information."
#+END_QUOTE
In simple terms, it keeps track of what you did and when as long as you log that on every change that deserve to be saved.
This is a very good way to keep backups of previous changes, also a way to have a history documenting who changed what and for what reason (NO! Not to blame, to understand why and how to fix it).
***** What is a git commit?
You can read all about what a commit is on the manual page of [[https://git-scm.com/docs/git-commit][git-commit]].
But the simple way to understand this is, it takes a snapshot of your work and names it a /SHA/ number (very long string of letters and numbers). A /SHA/ is a unique name that is derived from information from the current commit and every commit that came before since the beginning of the tree.
In other words, there is an extremely low chance that 2 commits would ever have the same /SHA/. Let's not also forget the security implication from this. If you have a clone of a repository and someone changed a commit somewhere in the tree history, every commit including the one changed and newer will have to change names. At that point, your fork will have a mismatch and you can know that the history was changed.
***** What is the =git add= thingy for?
Well the [[https://git-scm.com/docs/git-add][git-add]] manual page is very descriptive about the subject but, once again, I'll try to explain it in metaphors.
Think of it this way, =git-commit= saves the changes, but what changes ? That's exactly the question to answer. What changes ?
What if I want to commit some changes but not others ? What if I want to commit all the code in one commit and all the comments in another ?
That's where the "staging area" comes in play. You use =git-add= to stage files to be committed. And whenever you run the =git-commit= command, it will commit whatever is staged to be committed, right ?
**** Practice
Now that we've already explained a few concepts, let's see how this all fits together.
***** Step 1: Basic git configuration
The [[https://git-scm.com/book/en/v2/Getting-Started-First-Time-Git-Setup][Getting Started - First-Time Git Setup]] has more detailed setup but I took out what's quick and easy for now.
This is easy. If you want to be able to commit, you need to create a project to work on. A "project" can be translated to a repository and everything in that directory will be tracked.
So let's create a repository
#+BEGIN_EXAMPLE
$ # Navigate to where you'd like to create the repository
$ cd ~/Documents/Projects/
$ # Create repository directory
$ mkdir example
$ # Navigate into the newly created directory
$ cd example
$ # Create the repository
$ git init
#+END_EXAMPLE
Yeah, it was only one command =git init=. Told you it was easy, didn't I?
***** Step 3: Make a change
Let's create a file called =README.md= in the current directory (=~/Documents/Projects/example=) and put the following in it.
#+BEGIN_SRC markdown
# Example
This is an example repository.
#+END_SRC
And save it of course.
***** Step 4: Staging changes
If you go back to the command line and check the following command, you'll see a similar result.
#+BEGIN_EXAMPLE
$ git status
On branch master
No commits yet
Untracked files:
(use "git add <file>..." to include in what will be committed)
README.md
nothing added to commit but untracked files present (use "git add" to track)
#+END_EXAMPLE
and =README.md= is in red (if you have colors enabled). This means that there is file that is not tracked in your repository. We would like to track that one, let's stage it.
#+BEGIN_EXAMPLE
$ git add README.md
$ git status
On branch master
No commits yet
Changes to be committed:
(use "git rm --cached <file>..." to unstage)
new file: README.md
#+END_EXAMPLE
And =README.md= would now become green (if you have colors enabled). This means that if you commit now, this new file will be added and tracked in the future for changes. Technically though, it is being tracked for changes right now.
Let's prove it.
#+BEGIN_EXAMPLE
$ echo "This repository is trying to give you a hands on experience with git to complement the post." >> README.md
$ git status
On branch master
No commits yet
Changes to be committed:
(use "git rm --cached <file>..." to unstage)
new file: README.md
Changes not staged for commit:
(use "git add <file>..." to update what will be committed)
(use "git checkout -- <file>..." to discard changes in working directory)
modified: README.md
#+END_EXAMPLE
As you can see, git figured out that the file has been changed. Now let's add these changes too and move forward.
#+BEGIN_EXAMPLE
$ git add README.md
$ git status
On branch master
No commits yet
Changes to be committed:
(use "git rm --cached <file>..." to unstage)
new file: README.md
#+END_EXAMPLE
***** Step 5: Committing
This will be as easy as the rest. Let's commit these changes with a good commit message to describe the changes.
You can definitely see who committed it, when and what the message was. You also have access to the changes made in this commit.
**** Conclusion
I'm going to end this post here, and will continue to build up the knowledge in new posts to come. For now, I think it's a good idea to simply work with commits.
Next concepts to cover would be branching and merging.
*** DONE Git! Branching and Merging :git:branch:merge:
In the previous post about /git/, we had a look at what /git/ is and got our feet wet with a bit of it.
In this post, I will be moving forward with the topic, I will be talking about branches, how to work with them and finally what merging is and how it works.
#+hugo: more
**** Requirements
The same requirement we had from the last post, obviously /git/.
**** Branching and Merging
***** What is a branch?
/git/ [[https://git-scm.com/book/en/v1/Git-Branching-What-a-Branch-Is][documentation]] describes it as:
#+BEGIN_QUOTE
"A branch in Git is simply a lightweight movable pointer to one of the[se] commits."
#+END_QUOTE
Usually, people coming from /svn/ think of *branches* differently. In /git/, a branch is simply a pointer to a commit.
So let's verify that claim to see if it's true.
Remember our example repository from the last post ? We'll be using it here.
The commit is, of course, different because this is a different computer with a different repository from scratch. Anyway, it seems from the log message that both /mybranch/ and /master/ are pointing to same commit /SHA/. Technically they are pointing to *HEAD*.
From reading the output of log, we can see that the /master/ branch points to a different commit than /mybranch/.
To visualize this, let's look at it in a different way.
#+BEGIN_EXAMPLE
$ git log --graph --oneline --all
* b30f4e0 (HEAD -> mybranch) Adding an empty line
* 643a353 (master) Second commit
#+END_EXAMPLE
What the above suggests is that our two branches have different contents at this stage. In other words, if I switch back to the /master/ branch what do you think we will find in =README.md= ?
#+BEGIN_EXAMPLE
$ git checkout master
Switched to branch 'master'
$ cat README.md
# Example
This is an example repository.
This repository is trying to give you a hands on experience with git to complement the post.
$
#+END_EXAMPLE
And if we switch back to /mybranch/.
#+BEGIN_EXAMPLE
$ git checkout mybranch
Switched to branch 'mybranch'
$ cat README.md
# Example
This is an example repository.
This repository is trying to give you a hands on experience with git to complement the post.
$
#+END_EXAMPLE
Let's add another commit to make easier to see the changes than an empty line.
#+BEGIN_EXAMPLE
$ echo "Let's add a line to mybranch." >> README.md
$ git add README.md
$ git commit -m "Adding more commits to mybranch"
[mybranch f25dd5d] Adding more commits to mybranch
1 file changed, 1 insertion(+)
#+END_EXAMPLE
Now let's check the tree again.
#+BEGIN_EXAMPLE
$ git log --graph --oneline --all
* f25dd5d (HEAD -> mybranch) Adding more commits to mybranch
* b30f4e0 Adding an empty line
* 643a353 (master) Second commit
#+END_EXAMPLE
Let's also check the difference between our /master/ branch and /mybranch/.
#+BEGIN_EXAMPLE
$ git diff master mybranch
diff --git a/README.md b/README.md
index b4734ad..f07e71e 100644
--- a/README.md
+++ b/README.md
@@ -2,3 +2,5 @@
This is an example repository.
This repository is trying to give you a hands on experience with git to complement the post.
+
+Let's add a line to mybranch.
#+END_EXAMPLE
The =+= suggests an addition and =-= suggests a deletion of a line. As we can see from the =+= shown before the two lines added to the =README.md= file, /mybranch/ has these additions.
You can read more about /git/ branches in the /git/ [[https://git-scm.com/book/en/v1/Git-Branching-What-a-Branch-Is][documentation]] page.
***** What is merging ?
That's all fine so far, but how do I get these changes from /mybranch/ to the /master/ branch ?
The answer to that is also as easy as all the steps taken so far. /git/ merges *from* a branch you specify *to* the branch you are currently on.
#+BEGIN_EXAMPLE
$ # Checking which branch we are on
$ git branch
master
* mybranch
$ # We are on mybranch and we need to put these changes into master
$ # First we need to move to our master branch
$ git checkout master
Switched to branch 'master'
$ # Now we can merge from mybranch
$ git merge mybranch
Updating 643a353..f25dd5d
Fast-forward
README.md | 2 ++
1 file changed, 2 insertions(+)
#+END_EXAMPLE
As we can see. The changes in /mybranch/ have been merged into the /master/ branch.
I'll explain to you how I like to work and my personal merging strategy. I will keep out some details as they use concepts that are more advanced than what has been discussed so far.
***** /master/ branch
To me, the /master/ branch stays always up to date with the *remote*/master/ branch. In other words, I do not make commits against the /master/ branch in the project I'm working on.
***** branch
If I want to work on the project, I start by updating the /master/ branch and then branching it as we've seen before. The name of the branch is always indicative on what it holds, or what kind of work I am doing on it.
As long as I am working on my dev branch, I keep updating the /master/ branch and then porting the changes into my dev branch. This way, at the end the code is compatible and I am testing with the latest version of the code. This is very helpful and makes merging later a breeze.
***** merging
After my work is done, I push my branch to the remote server and ask for the maintainer of the project to merge my changes into the /master/ branch after reviewing it, of course. To explain this in a very simple manner, all that mumbo jumpo talk previously simply means someone else did the merge into master.
**** Conclusion
In this post, I talked about what are branches. We went ahead and worked a little bit with branches and then mentioned merging. At the end of the post I talked a bit about my merging strategy.
In the next post, I will be talking about remotes.
In the previous post, we talked about branching and merging. We will say a few last words on branches in this post and dive into remotes.
What are remotes ? What are they for ? How are they used ?
Coming right up.
**** Requirements
In this post, we will need another requirement.
- First, you obviously need /git/.
- Second, you will need a git repository on a git server. Easier way is to create an account on [[https://gitlab.com][Gitlab]], [[https://github.com][GitHub]] or other similar services.
**** Branches
I have a few more things I need to say about branches...
If you came to the same conclusion that branches in /git/ are /cheap/, you are correct.
This is very important because this encourages you to create more branches.
A lot of short living branches is a great way to work. Small features added here and there.
Small projects to test new features, etc...
Second conclusion you can come up with from the previous post is that the /master/ branch is not a /special/ branch.
People use it as a /special/ branch, or the branch of *truth* by convention /only/.
I should also note that some services like *Gitlab* offer master branch protection on their own which would not allow master history overwriting.
The best next topic that comes after /branches/ is a topic extremely similar to it, *remotes*.
**** Remotes
The description of =git-remote= from the [[https://git-scm.com/docs/git-remote][manual page]] is simply
#+BEGIN_QUOTE
Manage the set of repositories ("remotes") whose branches you track.
#+END_QUOTE
That's exactly what it is.
A way to manage /remote/ repositories.
Now we will be talking about managing them in a bit but let's talk about how to use them.
I found the best way to think to work with them is that you can think of them as /branches/.
That's exactly why I thought this would be best fit after that blog post.
***** Listing
Let's list them on our project and see what's what.
#+BEGIN_EXAMPLE
$ git remote -v
#+END_EXAMPLE
Okay! Nothing...
Alright, let's change that.
We don't have a /remote/ repository we can manage.
We need to create one.
***** Adding a remote
So I went to *Gitlab* and I created a new repository.
After creating the repository, you will get a box with commands that look similar to the following.
In the previous topic, I talked about git remotes because it felt
natural after branching and merging.
Now, the time has come to talk a little bit about =rebase= and some good
cases to use it for.
#+hugo: more
**** Requirements
This has not changed people, it is still /git/.
**** Rebase
In /git/ there are 2 ways of integrating your changes from one branch
into another.
We already talked about one; =git-merge=. For more information about =git-merge= consult the [[https://git-scm.com/book/en/v2/Git-Branching-Basic-Branching-and-Merging#_basic_merging][git basic branching and merging]] manual.
The other is =git-rebase=.
While =git-rebase= has a lot of different uses, the basic use of it is described in the [[https://git-scm.com/book/en/v2/Git-Branching-Rebasing][git branching rebasing]] manual as:
#+BEGIN_QUOTE
"With the =rebase= command, you can take all the changes that were committed on one branch and replay them on a different branch."
#+END_QUOTE
In other words, all the commits you have made into the branch you are on will be set aside.
Then, all the changes in the branch you are rebasing from will be applied to your branch.
Finally, all your changes, that were set aside previously, will be applied back to your branch.
The beauty about this process is that you can keep your branch updated with upstream, while coding your changes.
By the end of the process of adding your feature, your changes are ready to be merged upstream straight away.
This is due to the fact that all the conflicts would've been resolved in each rebase.
#+BEGIN_EXPORT html
<div class="admonition note">
<p class="admonition-title">Note</p>
#+END_EXPORT
Branch and branch often!
if you merge, merge and merge often!
or rebase, and rebase often!
#+BEGIN_EXPORT html
</div>
#+END_EXPORT
***** Usage
Rebase is used just like merge in our case.
First, let's create a branch and make a change in that branch.
#+BEGIN_EXAMPLE
$ git checkout -b rebasing-example
Switched to a new branch 'rebasing-example'
$ printf "\n# Rebase\n\nThis is a rebase branch.\n" >> README.md
$ git add README.md
$ git commit -m "Adding rebase section"
[rebasing-example 4cd0ffe] Adding rebase section
1 file changed, 4 insertions(+)
$
#+END_EXAMPLE
Now let's assume someone (or yourself) made a change to the =master= branch.
#+BEGIN_EXAMPLE
$ git checkout master
Switched to branch 'master'
Your branch is up to date with 'origin/master'.
$ printf "# Master\n\nThis is a master branch" >> master.md
$ git add master.md
$ git commit -m "Adding master file"
[master 7fbdab9] Adding master file
1 file changed, 3 insertions(+)
create mode 100644 master.md
$
#+END_EXAMPLE
I want to take a look at how the tree looks like before I attempt any changes.
#+BEGIN_EXAMPLE
$ git log --graph --oneline --all
* 7fbdab9 (HEAD -> master) Adding master file
| * 4cd0ffe (rebasing-example) Adding rebase section
|/
* 4f6bb31 (origin/master) Adding the git remote section
* 0bd01aa Second commit
#+END_EXAMPLE
After both of our commits, the tree diverged.
We are pointing to the *master* branch, I know that because =HEAD= points to /master/.
That commit is different than the commit that =rebase-example= branch points to.
These changes were introduced by someone else while I was adding the rebase section in the =README.md= file and they might be crucial for my application.
In short, I was those changes in the code I am working on right now.
Let's do that.
#+BEGIN_EXAMPLE
$ git checkout rebasing-example
Switched to branch 'rebasing-example'
$ git rebase master
First, rewinding head to replay your work on top of it...
Applying: Adding rebase section
#+END_EXAMPLE
And, let's look at the tree of course.
#+BEGIN_EXAMPLE
$ git log --graph --oneline --all
* 1b2aa4a (HEAD -> rebasing-example) Adding rebase section
* 7fbdab9 (master) Adding master file
* 4f6bb31 (origin/master) Adding the git remote section
* 0bd01aa Second commit
#+END_EXAMPLE
The tree lookr linear now. =HEAD= is pointing to our branch.
That commit points to the =7fbdab9= commit which the /master/ branch also points to.
So rebase set aside =1b2aa4a= to apply =7fbdab9= and then re-applied it back. Pretty neat huh ?!
**** My Strategy
I'm going to be honest with you. I do not know the different kinds of merge strategies.
I've glazed at names of a few but I've never looked at them closely enough to see which one is what.
What I use, I've used for a while. I learned it from somewhere and changed a few things in it to make it work for me.
First of all, I always fork a repository.
I tend to stay away from creating a branch on the upstream repository unless it's my own personal project.
On my fork, I freely roam. I am the king of my own fork and I create as many branches as I please.
I start with an assumption. The assumption is that my /master/ branch is, for all intents and purposes, upstream.
This means I keep it up to date with upstream's main branch.
When I make a branch, I make a branch from /master/, this way I know it's up to date with upstream.
I do my work on my branch. Every few hours, I update my /master/ branch. After I update my /master/
branch, I /rebase/ the /master/ branch into my branch and voilà I'm up to date.
By the time my changes are ready to be merged back into upstream for any
reason, they are ready to go.
That *MR* is gonna be ready to be merged in a jiffy.
**** Conclusion
From what I've read, I use one of those strategies described on some
website. I don't know which one. But to me, it doesn't matter because it
works for me. And if I need to adapt that for one reason or another, I
can.
*** DONE Git binary clean up :git:git_filter_repo:git_lfs:
When I first started this blog, I simply started with experiments. The first iteration was a /wordpress/ which was followed, very fast, by /joomla/. Neither of them lasted long. They are simply not for me.
I am lucky to be a part of a small group started in =#dgplug= on /Freenode/. In mentioned group, I have access to a lot of cool and awesome people who can put me to shame in development. On the flip side, I live by a /motto/ that says:
#+BEGIN_QUOTE
Always surround yourself with people smarter than yourself.
#+END_QUOTE
It's the best way to learn. Anyway, back to the topic at hand, they introduced me to /static blog generators/. There my journey started but it started with a trial. I didn't give too much thought to the repository. It moved from /GitHub/ to /Gitlab/ and finally /here/.
But, of course, you know how projects go, right ?
Once you start with one, closely follows other ones that crop up along the way. I put them on my *TODO*, literally. One of those items was that I committed all the images to the repository. It wasn't until a few days ago until I added a =.gitattributes= file. Shameful, I know.
No more ! Today it all changed.
#+hugo: more
**** First step first
Let's talk about what we need to do a little bit before we start. Plan it out in our head before doing the actual work.
I will itemize them here to make it easy to follow:
- Clone a fresh repository to do the work in
- Remove all the images from the /git/ repository
- Add the images again to /git lfs/
Sounds simple enough, doesn't it ?
#+BEGIN_EXPORT html
<div class="admonition warning">
<p class="admonition-title">warning</p>
#+END_EXPORT
If you follow along this blog post, here's what you can expect.
- You *WILL* lose /all the files you delete from disk/, as well, so make a copy
- You *WILL* re-write history. This means that the /SHA/ of every commit since the first image was committed *WILL* mostly likely change.
- You *WILL* end up essentially with a new repository that shares very little similarities with the original, so *BACKUP*!.
#+BEGIN_EXPORT html
</div>
#+END_EXPORT
Now that we got the /warning/ out of the way, let's begin the serious work.
**** Clone the repository
I bet you can do this with your eyes closed by now.
Yes, I use =-v= when I commit from the shell, try it.
The interesting part from the previous step is that /git-filter-repo/ left us without a /remote/. As I said, this repository resembles very little the original one so the decision made by /git-filter-repo/ is correct.
Let's add a *new empty repository*/remote/ to our new repository and push.
Total 338 (delta 128), reused 316 (delta 127), pack-reused 0
remote: Resolving deltas: 100% (128/128), done.
remote: . Processing 1 references
remote: Processed 1 references in total
To git.project42.io:Elia/blog.lazkani.io.git
* [new branch] master -> master
Branch 'master' set up to track remote branch 'master' from 'origin'.
#+END_EXAMPLE
And the deed is done.
#+BEGIN_EXPORT html
<div class="admonition note">
<p class="admonition-title">Note</p>
#+END_EXPORT
If you were extremely observant so war, you might've noticed that I used the same link again while I said a *new repository*.
Indeed, I did. The old repository was renamed and archived [[https://gitea.project42.io/Elia/blog.lazkani.io-20200902-historical][here]]. A new one with the name of the previous one was created instead.
#+BEGIN_EXPORT html
</div>
#+END_EXPORT
**** Conclusion
After I pushed the repository you can notice the change in size. It's not insignificant.
I think it's clearner now. The *1.2MB* size on the /repository/ is no longer
bothering me.
** RSS :@rss:
*** DONE Yet Another RSS Reader Move ? :emacs:org_mode:configuration:
The more I get comfortable with /emacs/ and /doom/, the more I tend to move things to it. This means that I am getting things done faster, without the need to get bogged down in the weeds of things.
This also means that, sometimes, I get to decommission a service that I host for my own personal use. If I can do it with a /text file/ in /git/, why would I host a full-on service to do it for me ?
You might say, well, then you can access it from anywhere ! Security much ?!
if I don't have my machine, I will not access my passwords. In practice, the reality is that I am tied to my own machine. On one hand, I cannot access my services online without my machine and if I am on the move it is highly unlikely for me to access my /rss/.
Oh yeah ! /rss/ ! That's what we are here for right ? Let's dive in...
#+hugo: more
**** Introduction
I hosted an instance of /[[https://miniflux.app/][miniflux]]/ on a /vps/ for my /rss/. /Miniflux/ is a great project, I highly recommend it. I have used it for a number of years without any issues at all; hassle free. I love it !
But with time, we have to move on. I have had my eye on the /rss/ configuration in the /doom/~init.el~ since I installed it. Now comes the time for me to try it out.
I will go with my process with you so you can see what I did. There might be better ways of doing things than this, if you know how ping me !
**** Doom documentation
The nice thing about /doom/ is that it is documented. The ~rss~ is a /doom/~module~ so we will look in the /doom/~modules~ manual.
We can achieve this by hitting ~SPC h d m~ and then searching for ~rss~. The documentation will give us a bit of informaton to get started, like for example that it uses ~elfeed~ as a package.
**** Elfeed
The creators of [[https://github.com/skeeto/elfeed][elfeed]] describe it as.
#+BEGIN_QUOTE
... an extensible web feed reader for Emacs, supporting both Atom and RSS.
#+END_QUOTE
The project looks well documented, that's very good. It has extensions, /org/ one... wait /org/ one ? What does it do ?
**** Elfeed Org
What is this thing [[https://github.com/remyhonig/elfeed-org][elfeed-org]] ?
#+BEGIN_QUOTE
Configure the Elfeed RSS reader with an Orgmode file
#+END_QUOTE
Sweet ! That's what I'm talking about. A neatly written /org/ file as configuration.
It is always a good idea to go through documentation, at least quickly. Skim it over, you don't know what you would miss in there. I've been doing this for a long time, there is no way I can miss any... oh wait... I missed this...
***** Import/Export OPML?
Whaaaat ?
#+BEGIN_QUOTE
Use ~elfeed-org-import-opml~ to import an OPML file to an elfeed-org structured tree.
#+END_QUOTE
Alright, that sounds easy. Let's export from /miniflux/ and import in /elfeed/.
**** Configuration
Before we import and whatnot, let's figure out what we are importing and where.
After reading the documentation of both ~elfeed~ and ~elfeed-org~, it says we need to set ~rmh-elfeed-org-files~ which is a /list/.
In my /doom/ configuration, I think I need to do the following.
This way we can guarantee where the file is, or we can go digging where the default is and copy from there.
This is just another file in my /org/ collection. Nothing special about it, it gets tagged and searched like everything else.
#+BEGIN_EXPORT html
<div class="admonition note">
<p class="admonition-title">Note</p>
#+END_EXPORT
I added the ~(elfeed-org)~ in the block to load ~elfeed-org~ after I had to load it manually a few times. This made it work on my system, I might be doing it wrong so your milage may vary.
The ~after!~ section is /doom/ specific.
#+BEGIN_EXPORT html
</div>
#+END_EXPORT
I also added the following line above the ~rmh-elfeed-org-files~ line.
#+BEGIN_SRC emacs-lisp
(setq elfeed-search-filter "@1-month-ago")
#+END_SRC
I simply wanted to see a span of /a month/ instead of the default /2 weeks/.
/Elfeed-org/ by default *inherits tagging* and *ignores text*. In this way, I can cascade /tags/ and when it's time to sort I can search for ~+xkcd~ and I get only /xkcd/ posts. I can also do something similar to filter on ~+general +europe~ for specifically getting /Europe/'s /Reddit news/.
The other reason for the /org/ integration is the documentation aspect for the future. I have only recently migrated to /elfeed/ so the documentation is still somewhat lacking, even for me. Not to worry though, as is the custom with the other migrations so far I ended up documenting a lot of it in better ways.
**** The big finish ?
Okay, okay ! That's a lot of babbling let's get to it, shall we ?
Now that everything is configured the way we like. Let's /reload/ everything and try ~M-x~~elfeed~.
Yeah, I know not very impressive huh ? We didn't add any /hooks/ to update and fetch things. I like to do that manually. The documentation, though, describes how to do that, if you like. For now, let's do it ourselves ~M-x~~elfeed-update~. You should be greeted with something like this.
There was nothing hard about the setup, whatsoever. It took me a bit to go through the relevant bits of the documentation for /my use cases/ which are, I admit, simple. I can now decommission my /miniflux/ instance as I have already found my future /rss/ reader.
** IRC :@irc:
*** DONE Weechat, SSH and Notification :weechat:notification:ssh:
I have been on IRC for as long as I have been using /Linux/ and that is a long time. Throughout the years, I have moved between /terminal IRC/ clients. In this current iteration, I am using [[https://weechat.org/][Weechat]].
#+hugo: more
There are many ways one can use /weechat/ and the one I chose is to run it in /tmux/ on a /cloud server/. In other words, I have a /Linux/ server running on one of the many cloud providers on which I have /tmux/ and /weechat/ installed and configured the way I like them. If you run a setup like mine, then you might face the same issue I have with IRC notifications.
**** Why?
/Weechat/ can cause a terminal bell which will show on some /terminals/ and /window managers/ as a notification. But you only know that /weechat/ pinged. Furthermore, if this is happening on a server that you are /ssh/'ing to, and with various shell configurations, this system might not even work. I wanted something more useful than that so I went on the hunt for the plugins available to see if any one of them could offer me a solution. I found many official plugins that did things in a similar fashion and each in a different and interesting way but none the way I want them to work.
**** Solution
After trying multiple solutions offered online which included various plugins, I decided to write my own. That's when /weenotify/ was born. If you know my background then you know, already, that I am big on open source so /weenotify/ was first released on [[https://gitlab.com/elazkani/weenotify][Gitlab]]. After a few changes, requested by a weechat developer (*FlashCode* in *#weechat* on [[https://freenode.net/][Freenode]]), /weenotify/ became as an [[https://weechat.org/scripts/source/weenotify.py.html/][official weechat plugin]].
**** Weenotify
Without getting into too many details, /weenotify/ acts as both a weechat plugin and a server. The main function is to intercept weechat notifications and patch them through the system's notification system. In simple terms, if someone mentions your name, you will get a pop-up notification on your system with information about that. The script can be configured to work locally, if you run weechat on your own machine, or to open a socket and send the notification to /weenotify/ running as a server. In the latter configuration, /weenotify/ will display the notification on the system the server is running on.
**** Configuration
Let's look at the configuration to accomplish this... As mentioned in the beginning of the post, I run weechat in /tmux/ on a server. So I /ssh/ to the server before attaching /tmux/. The safest way to do this is to *port forward over ssh* and this can be done easily by /ssh/'ing using the following example.
#+BEGIN_EXAMPLE
$ ssh -R 5431:localhost:5431 server.example.com
#+END_EXAMPLE
At this point, you should have port *5431* forwarded between the server and your machine.
Once the previous step is done, you can test if it works by trying to run the /weenotify/ script in server mode on your machine using the following command.
#+BEGIN_EXAMPLE
$ python weenotify.py -s
Starting server...
Server listening locally on port 5431...
#+END_EXAMPLE
The server is now running, you can test port forwarding from the server to make sure everything is working as expected.
#+BEGIN_EXAMPLE
$ telnet localhost 5431
Trying ::1...
Connected to localhost.
Escape character is '^]'.
#+END_EXAMPLE
If the connection is successful then you know that port forwarding is working as expected. You can close the connection by hitting =Ctrl= + =]=.
Now we are ready to install the plugin in weechat and configure it. In weechat, run the following command.
#+BEGIN_EXAMPLE
/script search weenotify
#+END_EXAMPLE
At which point, you should be greeted with the buffer shown in the screenshot below.
You can install the plugin with =Alt=+ =i= and make sure it autoloads with =Alt= +=A=.
You can get more information about working with weechat scripts by reading the help menu.
You can get the scripts help menu by running the following in weechat.
#+BEGIN_EXAMPLE
/help script
#+END_EXAMPLE
The /weenotify/ plugin is installed at this stage and only needs to be configured. The plugin has a list of values that can be configured. My configuration looks like the following.
Each one of those configuration options can be set as shown in the example below in weechat.
#+BEGIN_EXAMPLE
/set plugins.var.python.weenotify.enable on
#+END_EXAMPLE
Make sure that the plugin *enable* value is *on* and that the *mode* is *remote*, if you're following this post and using ssh with port forwarding. Otherwise, If you want the plugin to work locally, make sure you set the *mode* to *local*.
If you followed this post so far, then whenever someone highlights you on weechat you should get a pop-up on your system notifying you about it.
*** DONE Weechat and Emacs :weechat:emacs:weechat_el:
In the last few blog posts, I mentioned a few migrations caused by my /VSCode/
discovery a few weeks ago [[#emacs-and-org-mode]].
As I was configuring /Doom/, I noticed that there was a configuration for /weechat/ in there. I checked it out very briefly and found that it was a /[[https://github.com/the-kenny/weechat.el][weechat.el]]/ package for /Emacs/.
#+hugo: more
At the time, I didn't have too much time to spend on this so I quickly passed it over with plans to come back to it, /eventually/.
The time has come for me to configure and try this at least !
I already have my /weechat/ installation running remotely behind an /nginx/*reverse proxy*. I tried to connecting using that endpoint, unfortunately no dice.
**** The Problem
As I was asking in /#weechat.el/ on *freenode* for help, the very quick to help /[[https://github.com/flashcode][FlashCode]]/ sprung into action. He wasn't able to help me but he pointed me in the right direction.
I asked why would /Glowing Bear/ work but not /weechat.el/ ?
The answer was along the line that /Glowing Bear/ uses a /websocket/. Alright that made sense. Maybe /weechat.el/ does not do /websocket/.
**** The Solution
So, we are behind an /nginx/*reverse proxy* instance. What we need to do is expose our service as a /TCP reverse proxy/ instead of our usual /HTTP/ one. We are moving down networking layers to the *TCP/IP* instead of *HTTP*.
What we need to do is add a /stream/ section to our /nginx/ to accomplish this.
#+BEGIN_EXAMPLE
stream {
server {
listen 9000 ssl;
ssl_certificate /path/to/chain.pem;
ssl_certificate_key /path/to/cert.pem;
proxy_pass 127.0.0.1:9000;
}
}
#+END_EXAMPLE
#+BEGIN_EXPORT html
<div class="admonition warning">
<p class="admonition-title">warning</p>
#+END_EXPORT
The =stream= section has to be outside the =http= section.
If you add this configuration next to your other =server= sections, it will fail.
#+BEGIN_EXPORT html
</div>
#+END_EXPORT
In the previous block we make a few assumptions.
- We are behind SSL: I use the /nginx/ reverse proxy for /SSL termination/ as it handles reloading certificates automatically. If I leave it to /weechat/, I have to reload the /certificates/ manually and often.
- Weechat is listening on port 9000 locally: The /weechat/ relay needs to be configured to listen on *localhost* and on port *9000* for this configuration to work. Make sure to change it to fit your needs.
Now that the configuration is out of the way, let's test it.
Open emacs and run =M-x= followed by =weechat-connect=. This should get you going.
**** Conclusion
It was a nice path down the road of packets. It's always a good day when you learn new things. I have never used /TCP/ forwarding with /nginx/ before but I'm glad it is supported.
Now that you know how to do the same as well, I hope you give both projects a try. I think they are worth it.
I'm also thankful to have so many different awesome projects created by the open source community.
** Text Editors :@text_editors:
*** DONE Emacs and Org-mode :emacs:org_mode:configuration:
I have recently found out, late I know, that the /VSCode/ distribution of the so called /Code - OSS/ is exactly that; a distribution.
Let me make it clear, the /VSCode/ binaries you download from *Microsoft* has an upstream the *GitHub repository* named [[https://github.com/Microsoft/vscode][VSCode]] but in fact is not exactly the same code.
*Microsoft* has already added a few gifts for you, including *telemetry*, not cool huh ?!
Well, they tell you this in the documentation, urrrmmm [[https://github.com/microsoft/vscode/wiki/Differences-between-the-repository-and-Visual-Studio-Code][somewhere]].
#+hugo: more
At the same time, I was giving /Jupyter Notebook/ a try. I worked on my previous post in it before writing down the final result as a blog post.
But at the back of my mind, there was always [[https://orgmode.org/][Org-mode]].
Putting one and one together, you've guessed it. I have moved to *Emacs*... again... for the umm I can't remember time.
But this time, it is different ! I hope...
**** Back story
I was using /Jupyter Notebooks/ as a way to write down notes. Organize things.
I had a work around the /output/ and was able to clean it.
But let's face it, it might work but it is designed more towards other goals.
I want to write notes and the best way to work with notes is to keep in the text, literally.
I found a /VSCode/ extension that can handle /Org-mode/ in some capacity (I haven't tested it) so I decided to switch to /Emacs/ and keep the extention as a backup.
**** Emacs Distribution of Doom
Haha ! Very funny, I know. I went with [[https://github.com/hlissner/emacs-doom-themes][Doom]].
Why? You may ask. I don't really have a good answer for you except the following.
* I didn't want to start from scratch, I wanted something with batteries included.
* At the same time, I've tried /Doom/ before and I like how it does things.
It is logical to me while at the same time very configurable.
* I was able to get up and running very quickly. Granted, my needs are few.
* I got /Python/ and /Golang/ auto-completion and /evil/ mode. I'm good to go !
Now let's dig down to my main focus here. Sure I switched editors but it was for a reason; *Org-mode*.
**** Org-mode Configuration
I will be talking about two different configuartion options here.
I am new to emacs so I will try to explain everything.
The two options are related to the difference between a /vanilla/ configuration and /Doom/'s version of the configuration.
The differences are minor but they are worth talking about.
***** New Org File
If you've used /Org-mode/ before and created /org files/, you already know that you need to set a few values at the top of the file. These include the /title/, /author/, /description/ and a different other values to change setting and/or behavior.
It is a bit of a manual labor to write these few lines at the beginning of every file. I wanted to automate that. So I got inspiration from [[https://gitlab.com/shakthimaan/operation-blue-moon][shakthimaan]].
I used his method to create a small =define-skeleton= for a header.
You can use this later with =M-x= + =genrate-new-header-org=.
#+BEGIN_EXPORT html
<div class="admonition note">
<p class="admonition-title">Note</p>
#+END_EXPORT
=M-x= is the *Meta* key and *x* combination.
Your *Meta* key can differ between the *Alt* on /Linux/ and *Command* on /Mac OS X/.
=M-x= will open a prompt for you to write in. Write the name you gave the skeleton, in this case it is =generate-new-header-org= and then hit the /Return/.
#+BEGIN_EXPORT html
</div>
#+END_EXPORT
***** New Task
[[https://gitlab.com/shakthimaan/operation-blue-moon][shakthimaan]] already created something for this. It looks like the following.
#+BEGIN_SRC emacs-lisp
;; Create a new skeleton to generate a new =Task=
(define-skeleton insert-org-entry
"Prompt for task, estimate and category"
nil
'(setq task (skeleton-read "Task: "))
'(setq estimate (skeleton-read "Estimate: "))
'(setq owner (skeleton-read "Owner: "))
'(setq category (skeleton-read "Category: "))
'(setq timestamp (format-time-string "%s"))
"** " task \n
":PROPERTIES:" \n
":ESTIMATED: " estimate \n
":ACTUAL:" \n
":OWNER: " owner \n
":ID: " category "." timestamp \n
":TASKID: " category "." timestamp \n
":END:")
#+END_SRC
This can also be used like the one above with =M-x= + =insert-org-entry=.
***** Doom specific configuration
Whatever defined so far should work if you just add it to your configuration but if you use /Doom/ it would a nice touch to integrate it with the workflow.
In =~/.doom.d/config.el=, wrap the previous definitions with =(after! org)=.
It's a nice touch to add these skeletons after /Org-mode/ has loaded.
:n :desc "Generate New Header Org" "G" 'generate-new-header-org
:n :desc "New Task Entry" "N" 'insert-org-entry
))
))
)
#+END_SRC
**** What do I do now ?
You might be asking yourself at this point, what does this all mean ?
What do I do with this ? Where do I go ?
Well here's the thing. You find yourself wanting to create a new /org file/.
You do so in emacs and follow it with =M-x= + =generate-new-header-org= (or =SPC m G= in *Doom*). /Emacs/ will ask you a few questions in the bottom left corner and once you answer then, your header should be all set.
You can follow that with =M-x= + =insert-org-entry= (or =SPC m N=) to generate a task. This will also ask you for input in the bottom left corner.
**** Conclusion
This should help me pick up the usage of /Org-mode/ faster. It is also a good
idea if you've already configured your /Emacs/ to read all your /org file/ for a
I was working on a /project/ that required a lot of manual steps. I /generally/ lean towards *automating everything* but in /some cases/ that is, unfortunately, not possible.
Documenting such project is not an easy task to accomplish, especially with so many moving parts and different outputs.
Since I have been using /org-mode/ more frequently for /documentation/ and /organization/ in general, I gravitated towards it as a first instinct.
I wasn't sure of the capabilities of /org-mode/ in such unfamiliar settings but I was really surprised by the outcome.
#+hugo: more
**** Introduction
If you haven't checked [[https://orgmode.org/][org-mode]] already, you should.
As its main capability it is to keep it simple for writing things down and organizing them, /org-mode/ is great to keep track of the steps taking along the way.
The ability to quickly move between /plain text/ and into /code blocks/ is excellent. Coupling /org-mode/ with /[[https://orgmode.org/worg/org-contrib/babel/intro.html][org-babel]]/ gives you the ability to run the /source code/ blocks and get the output back into the /org/ file itself. That is extremely neat.
With those two abilities alone, I could document things as I go along. This included both the commands I am running and the output I got back. *Fantastic*.
After some search online, I found out that this method is called /literal coding/. It consists of having the /plain text/ documentation and the /code/ in the same file and with the help of both previously mentioned /emacs/ packages one can get things working.
That sounds like fun!
**** Emacs Configuration
After digesting all the information I mentioned so far, that got me thinking. What about /emacs/?
A quick look online got me the answer. It is possible to do with /emacs/ as well. Alright, let's get right into it shall we ?
First step, I added the following line to my /main/ configuration. In my case, my /main/ configuration file is the /doom/ distribution's configuration file.
Make sure /org-mode/ and /org-babel/ are both *installed* and *configured* on your system before trying to run ~org-babel-load-file~
#+BEGIN_EXPORT html
</div>
#+END_EXPORT
**** Org-mode Conversion
After I pointed my /main emacs configuration/ to the /org/ configuration file I desire to use, I copied all the content of my /main emacs configuration/ in an ~emacs-lisp~ source code block.
#+BEGIN_EXAMPLE
#+BEGIN_SRC emacs-lisp
... some code ...
#+END_SRC
#+END_EXAMPLE
I, then, reloaded my /emacs/ to double check that everything works as expected and /it did/.
***** Document the code
Now that we have everything in one /org/ file, we can go ahead and start documenting it. Let's see an example of /before/ and /after/.
I started small, bits and pieces. I took a /snippet/ of my configuration that looked like the following.
#+BEGIN_SRC org
,#+BEGIN_SRC emacs-lisp
(setq display-line-numbers-type t)
(setq display-line-numbers-type 'relative)
(after! evil
(map! :map evil-window-map
(:leader
(:prefix ("w" . "Select Window")
:n :desc "Left" "<left>" 'evil-window-left
:n :desc "Up" "<up>" 'evil-window-up
:n :desc "Down" "<down>" 'evil-window-down
:n :desc "Right" "<right>" 'evil-window-right))))
,#+END_SRC
#+END_SRC
I converted it to something that looks very familiar to /org/ users out there.
#+BEGIN_SRC org
,* Line Numbering
,** Enable line numbering
Enabling line numbering by turning the flag on.
,#+BEGIN_SRC emacs-lisp
(setq display-line-numbers-type t)
,#+END_SRC
,** Configure /relative/ line numbering
Let's also make sure it's the /relative/ line numbering.
This helps jumping short distances very fast.
,#+BEGIN_SRC emacs-lisp
(setq display-line-numbers-type 'relative)
,#+END_SRC
,* Evil
,** Navigation
I'd like to use the /arrows/ to move around. ~hjkl~ is not very helpful or pleasant on /Colemak/.
,#+BEGIN_SRC emacs-lisp
(after! evil
(map! :map evil-window-map
(:leader
(:prefix ("w" . "Select Window")
:n :desc "Left" "<left>" 'evil-window-left
:n :desc "Up" "<up>" 'evil-window-up
:n :desc "Down" "<down>" 'evil-window-down
:n :desc "Right" "<right>" 'evil-window-right))))
,#+END_SRC
#+END_SRC
It might not be much a looker in such a block, but trust me, if you have an /org-mode/ parser it will make total sense. It will export to /html/ very well too.
Most importantly, the /emacs/ configuration still works.
**** Conclusion
I went through my /emacs configuration/ and transformed it into a /documented org/ file. My configuration looks a little bit neater now and that's great.
The capabilities of /literal programming/ goes way beyond this post, which goes without saying, and this is not the only use case for it.
*** DONE Bookmark with Org-capture :org_mode:emacs:org_capture:org_web_tools:org_cliplink:
I was reading, and watching, [[https://cestlaz.github.io/about/][Mike Zamansky]]'s blog post [[https://cestlaz.github.io/stories/emacs/][series]] about /org-capture/ and how he manages his bookmarks. His blog and video series are a big recommendation from me, he is teaching me tons every time I watch his videos. His inspirational videos were what made me dig down on how I could do what he's doing but... my way...
I stumbled across [[https://dewaka.com/blog/2020/04/08/bookmarking-with-org-mode/][this]] blog post that describes the process of using =org-cliplink= to insert the /title/ of the post into an /org-mode/ link. Basically, what I wanted to do is provide a link and get an /org-mode/ link. Sounds simple enough. Let's dig in.
#+hugo: more
**** Org Capture Templates
I will assume that you went through Mike's [[https://cestlaz.github.io/posts/using-emacs-23-capture-1/][part 1]] and [[https://cestlaz.github.io/posts/using-emacs-24-capture-2/][part 2]] posts to understand what =org-capture-templates= are and how they work. I essentially learned it from him and I do not think I can do a better job than a teacher.
Now that we understand where we need to start from, let's explain the situation. We need to find a way to call =org-capture= and provide it with a /template/. This /template/ will need to take a /url/ and add an /org-mode//url/ in our bookmarks. It will look something like the following.
I formatted it a bit so it would have some properties. I simply used the =%t= to put the /timestamp/ of when I took the bookmark. I used the =%?= to drop me at the end for editing. Then =some-function-here-to-call= a function to call to generate our /bookmark section/ with a title.
The blog post I eluded to earlier solved it by using [[https://github.com/rexim/org-cliplink][org-cliplink]]. While =org-cliplink= is great for getting /titles/ and manipulating them, I don't really need that functionality. I can do it manually. Sometimes, though, I would like to copy a page... Maybe if there is a project that /could/ attempt to do someth... Got it... [[https://github.com/alphapapa/org-web-tools][org-web-tools]].
***** Configuring /org-capture/ with /org-web-tools/
You would assume that you would be able to just pop =(org-web-tools-insert-link-for-url)= in the previous block and you're all done. But uhhh....
#+BEGIN_EXAMPLE
Wrong number of arguments: (1 . 1), 0
#+END_EXAMPLE
No dice. What would seem to be the problem ?
We look at the definition and we find this.
#+BEGIN_SRC emacs-lisp
(defun org-web-tools-insert-link-for-url (url)
"Insert Org link to URL using title of HTML page at URL.
If URL is not given, look for first URL in `kill-ring'."
I don't know why, exactly, it doesn't work by calling it straight away because I do not know /emacs-lisp/ at all. If you do, let me know. I suspect it has something to do with =(interactive)= and the list provided to it as arguments.
Anyway, I can see it is using =org-web-tools--org-link-for-url=, which the documentation suggests does the same thing as =org-web-tools-insert-link-for-url=, but is not exposed with =(interactive)=. Okay, we have bits and pieces of the puzzle. Let's put it together.
Now if we copy a link into the /clipboard/ and then call =org-capture= with the option =b=, we get prompted to edit the following before adding it to our /bookmarks/.
#+BEGIN_SRC org
** [[https://cestlaz.github.io/stories/emacs/][Using Emacs Series - C'est la Z]]
:PROPERTIES:
:TIMESTAMP: <2020-09-17 do>
:END:
#+END_SRC
Works like a charm.
***** Custom URL
What if we need to modify the url in some way before providing it. I have that use case. All I needed to do is create a function that takes /input/ from the user and provide it to =org-web-tools--org-link-for-url=. How hard can that be ?! uhoh! I said the curse phrase didn't I ?
#+BEGIN_SRC emacs-lisp
(defun org-web-tools-insert-link-for-given-url ()
"Extend =org-web-tools-inster-link-for-url= to take a user given URL"
(interactive)
(let ((url (read-string "Link: ")))
(org-web-tools--org-link-for-url url)))
#+END_SRC
We can, then, hook the whole thing up to our =org-capture-templates= and we get.
Now, you should have a working setup... =org-cliplink= willing !
**** Conclusion
I thought this was going to be harder to pull off but, alas, it was simple, even for someone who doesn't know /emacs-lisp/, to figure out. I hope I'd get more familiar with /emacs-lisp/ with time and be able to do more. Until next time, I recommend you hook =org-capture= into your workflow. Make sure it fits your work style, otherwise you will not use it, and make your path a more productive one.
*** DONE Calendar Organization with Org :emacs:org_mode:calendar:organization:
I have been having /some/ issues with my calendar. Recurring stuff have been going out of wack for some reason. In general, the setup I've had for the past few years have now become a problem I need to fix.
I decided to turn to my trusted /emacs/, like I usually do. /Doom/ comes bundled with something. Let's figure out what it is and how to configure it together.
#+hugo: more
**** Calendar in Emacs
I dug deeper into /Doom/'s /Calendar/ module and I found out that it is using [[https://github.com/kiwanami/emacs-calfw][calfw]].
I went to /GitHub/ and checked the project out. It's another emacs package, I'm going to assume you know how to install it.
