blog.lazkani.io-20200902-historical/posts/configuration-management/ansible-testing-with-molecule.org

#+BEGIN_COMMENT
.. title: Ansible testing with Molecule
.. date: 2019-01-11
.. slug: ansible-testing-with-molecule
.. updated: 2019-06-21
.. status: published
.. tags: configuration management, ansible, molecule,
.. category: configuration management
.. authors: Elia el Lazkani
.. description: A fast way to create a testable ansible role using molecule.
.. type: text
#+END_COMMENT

When I first started using [[https://www.ansible.com/][ansible]], I did not know about [[https://molecule.readthedocs.io/en/latest/][molecule]]. It was a bit daunting to start a /role/ from scratch and trying to develop it without having the ability to test it. Then a co-worker of mine told me about molecule and everything changed.

{{{TEASER_END}}}

I do not have any of the tools I need installed on this machine, so I will go through, step by step, how I set up ansible and molecule on any new machine I come across for writing ansible roles.

* Requirements
What we are trying to achieve in this post, is a working ansible role that can be tested inside a docker container. To be able to achieve that, we need to install docker on the system. Follow the instructions on [[https://docs.docker.com/install/][installing docker]] found on the docker website.

* Good Practices
First thing's first. Let's start by making sure that we have python installed properly on the system.

#+BEGIN_EXAMPLE
 $ python --version
 Python 3.7.1
#+END_EXAMPLE

Because in this case I have /python3/ installed, I can create a /virtualenv/ easier without the use of external tools.

#+BEGIN_EXAMPLE
 # Create the directory to work with
 $ mkdir -p sandbox/test-roles
 # Navigate to the directory
 $ cd sandbox/test-roles/
 # Create the virtualenv
 ~/sandbox/test-roles $ python -m venv .ansible-venv
 # Activate the virtualenv
 ~/sandbox/test-roles $ source .ansible-venv/bin/activate
 # Check that your virtualenv activated properly
 (.ansible-venv) ~/sandbox/test-roles $ which python
 /home/elijah/sandbox/test-roles/.ansible-venv/bin/python
#+END_EXAMPLE

At this point, we can install the required dependencies.

#+BEGIN_EXAMPLE
 $ pip install ansible molecule docker
 Collecting ansible
   Downloading https://files.pythonhosted.org/packages/56/fb/b661ae256c5e4a5c42859860f59f9a1a0b82fbc481306b30e3c5159d519d/ansible-2.7.5.tar.gz (11.8MB)
     100% |████████████████████████████████| 11.8MB 3.8MB/s
 Collecting molecule
   Downloading https://files.pythonhosted.org/packages/84/97/e5764079cb7942d0fa68b832cb9948274abb42b72d9b7fe4a214e7943786/molecule-2.19.0-py3-none-any.whl (180kB)
     100% |████████████████████████████████| 184kB 2.2MB/s

 ...

 Successfully built ansible ansible-lint anyconfig cerberus psutil click-completion tabulate tree-format pathspec future pycparser arrow
 Installing collected packages: MarkupSafe, jinja2, PyYAML, six, pycparser, cffi, pynacl, idna, asn1crypto, cryptography, bcrypt, paramiko, ansible, pbr, git-url-parse, monotonic, fasteners, click, colorama, sh, python-gilt, ansible-lint, pathspec, yamllint, anyconfig, cerberus, psutil, more-itertools, py, attrs, pluggy, atomicwrites, pytest, testinfra, ptyprocess, pexpect, click-completion, tabulate, future, chardet, binaryornot, poyo, urllib3, certifi, requests, python-dateutil, arrow, jinja2-time, whichcraft, cookiecutter, tree-format, molecule, docker-pycreds, websocket-client, docker
 Successfully installed MarkupSafe-1.1.0 PyYAML-3.13 ansible-2.7.5 ansible-lint-3.4.23 anyconfig-0.9.7 arrow-0.13.0 asn1crypto-0.24.0 atomicwrites-1.2.1 attrs-18.2.0 bcrypt-3.1.5 binaryornot-0.4.4 cerberus-1.2 certifi-2018.11.29 cffi-1.11.5 chardet-3.0.4 click-6.7 click-completion-0.3.1 colorama-0.3.9 cookiecutter-1.6.0 cryptography-2.4.2 docker-3.7.0 docker-pycreds-0.4.0 fasteners-0.14.1 future-0.17.1 git-url-parse-1.1.0 idna-2.8 jinja2-2.10 jinja2-time-0.2.0 molecule-2.19.0 monotonic-1.5 more-itertools-5.0.0 paramiko-2.4.2 pathspec-0.5.9 pbr-4.1.0 pexpect-4.6.0 pluggy-0.8.1 poyo-0.4.2 psutil-5.4.6 ptyprocess-0.6.0 py-1.7.0 pycparser-2.19 pynacl-1.3.0 pytest-4.1.0 python-dateutil-2.7.5 python-gilt-1.2.1 requests-2.21.0 sh-1.12.14 six-1.11.0 tabulate-0.8.2 testinfra-1.16.0 tree-format-0.1.2 urllib3-1.24.1 websocket-client-0.54.0 whichcraft-0.5.2 yamllint-1.11.1
#+END_EXAMPLE

* Creating your first ansible role
Once all the steps above are complete, we can start by creating our first ansible role.

#+BEGIN_EXAMPLE
 $ molecule init role -r example-role
 --> Initializing new role example-role...
 Initialized role in /home/elijah/sandbox/test-roles/example-role successfully.

 $ tree example-role/
 example-role/
 ├── defaults
 │   └── main.yml
 ├── handlers
 │   └── main.yml
 ├── meta
 │   └── main.yml
 ├── molecule
 │   └── default
 │       ├── Dockerfile.j2
 │       ├── INSTALL.rst
 │       ├── molecule.yml
 │       ├── playbook.yml
 │       └── tests
 │           ├── __pycache__
 │           │   └── test_default.cpython-37.pyc
 │           └── test_default.py
 ├── README.md
 ├── tasks
 │   └── main.yml
 └── vars
     └── main.yml

 9 directories, 12 files
#+END_EXAMPLE

You can find what each directory is for and how ansible works by visiting [[https://docs.ansible.com][docs.ansible.com]].

** =meta/main.yml=
The meta file needs to modified and filled with information about the role. This is not a required file to modify if you are keeping this for yourself, for example. But it is a good idea to have as much information as possible if this is going to be released. In my case, I don't need any fanciness as this is just sample code.

#+BEGIN_SRC yaml
---
galaxy_info:
  author: Elia el Lazkani
  description: This is an example ansible role to showcase molecule at work
  license: license (BDS-2)
  min_ansible_version: 2.7
  galaxy_tags: []
dependencies: []
#+END_SRC

** =tasks/main.yml=
This is where the magic is set in motion. Tasks are the smallest entities in a role that do small and idempotent actions. Let's write a few simple tasks to create a user and install a service.

#+BEGIN_SRC yaml
---
# Create the user example
- name: Create 'example' user
  user:
    name: example
    comment: Example user
    shell: /bin/bash
    state: present
    create_home: yes
    home: /home/example

# Install nginx
- name: Install nginx
  apt:
    name: nginx
    state: present
    update_cache: yes
  notify: Restart nginx
#+END_SRC

** =handlers/main.yml=
If you noticed, we are notifying a handler to be called after installing /nginx/. All handlers notified will run after all the tasks complete and each handler will only run once. This is a good way to make sure that you don't restart /nginx/ multiple times if you call the handler more than once.

#+BEGIN_SRC yaml
---
# Handler to restart nginx
- name: Restart nginx
  service:
    name: nginx
    state: restarted
#+END_SRC

** =molecule/default/molecule.yml=
It's time to configure molecule to do what we need. We need to start an ubuntu docker container, so we need to specify that in the molecule YAML file. All we need to do is change the image line to specify that we want an =ubuntu:bionic= image.

#+BEGIN_SRC yaml
---
dependency:
  name: galaxy
driver:
  name: docker
lint:
  name: yamllint
platforms:
  - name: instance
    image: ubuntu:bionic
provisioner:
  name: ansible
  lint:
    name: ansible-lint
scenario:
  name: default
verifier:
  name: testinfra
  lint:
    name: flake8
#+END_SRC

** =molecule/default/playbook.yml=
This is the playbook that molecule will run. Make sure that you have all the steps that you need here. I will keep this as is.

#+BEGIN_SRC yaml
---
- name: Converge
  hosts: all
  roles:
    - role: example-role
#+END_SRC

* First Role Pass
This is time to test our role and see what's going on.

#+BEGIN_EXAMPLE
 (.ansible-role) ~/sandbox/test-roles/example-role/ $ molecule converge
 --> Validating schema /home/elijah/sandbox/test-roles/example-role/molecule/default/molecule.yml.
 Validation completed successfully.
 --> Test matrix

 └── default
     ├── dependency
     ├── create
     ├── prepare
     └── converge

 --> Scenario: 'default'
 --> Action: 'dependency'
 Skipping, missing the requirements file.
 --> Scenario: 'default'
 --> Action: 'create'

     PLAY [Create] ******************************************************************

     TASK [Log into a Docker registry] **********************************************
     skipping: [localhost] => (item=None)

     TASK [Create Dockerfiles from image names] *************************************
     changed: [localhost] => (item=None)
     changed: [localhost]

     TASK [Discover local Docker images] ********************************************
     ok: [localhost] => (item=None)
     ok: [localhost]

     TASK [Build an Ansible compatible image] ***************************************
     changed: [localhost] => (item=None)
     changed: [localhost]

     TASK [Create docker network(s)] ************************************************

     TASK [Create molecule instance(s)] *********************************************
     changed: [localhost] => (item=None)
     changed: [localhost]

     TASK [Wait for instance(s) creation to complete] *******************************
     changed: [localhost] => (item=None)
     changed: [localhost]

     PLAY RECAP *********************************************************************
     localhost                  : ok=5    changed=4    unreachable=0    failed=0


 --> Scenario: 'default'
 --> Action: 'prepare'
 Skipping, prepare playbook not configured.
 --> Scenario: 'default'
 --> Action: 'converge'

     PLAY [Converge] ****************************************************************

     TASK [Gathering Facts] *********************************************************
     ok: [instance]

     TASK [example-role : Create 'example' user] ************************************
     changed: [instance]

     TASK [example-role : Install nginx] ********************************************
     changed: [instance]

     RUNNING HANDLER [example-role : Restart nginx] *********************************
     changed: [instance]

     PLAY RECAP *********************************************************************
     instance                   : ok=4    changed=3    unreachable=0    failed=0
#+END_EXAMPLE

It looks like the *converge* step succeeded.

* Writing Tests
It is always a good practice to write unittests when you're writing code. Ansible roles should not be an exception. Molecule offers a way to run tests, which you can think of as unittest, to make sure that what the role gives you is what you were expecting. This helps future development of the role and keeps you from falling in previously solved traps.

** =molecule/default/tests/test_default.py=
Molecule leverages the [[https://testinfra.readthedocs.io/en/latest/][testinfra]] project to run its tests. You can use other tools if you so wish, and there are many. In this example we will be using /testinfra/.

#+BEGIN_SRC python
import os

import testinfra.utils.ansible_runner

testinfra_hosts = testinfra.utils.ansible_runner.AnsibleRunner(
    os.environ['MOLECULE_INVENTORY_FILE']).get_hosts('all')


def test_hosts_file(host):
    f = host.file('/etc/hosts')

    assert f.exists
    assert f.user == 'root'
    assert f.group == 'root'


def test_user_created(host):
    user = host.user("example")
    assert user.name == "example"
    assert user.home == "/home/example"


def test_user_home_exists(host):
    user_home = host.file("/home/example")
    assert user_home.exists
    assert user_home.is_directory


def test_nginx_is_installed(host):
    nginx = host.package("nginx")
    assert nginx.is_installed


def test_nginx_running_and_enabled(host):
    nginx = host.service("nginx")
    assert nginx.is_running
#+END_SRC

#+BEGIN_EXPORT html
<div class="admonition warning">
<p class="admonition-title">warning</p>
#+END_EXPORT
Uncomment =truthy: disable= in =.yamllint= found at the base of the role.
#+BEGIN_EXPORT html
</div>
#+END_EXPORT

#+BEGIN_EXAMPLE
 (.ansible_venv) ~/sandbox/test-roles/example-role $ molecule test
 --> Validating schema /home/elijah/sandbox/test-roles/example-role/molecule/default/molecule.yml.
 Validation completed successfully.
 --> Test matrix

 └── default
     ├── lint
     ├── destroy
     ├── dependency
     ├── syntax
     ├── create
     ├── prepare
     ├── converge
     ├── idempotence
     ├── side_effect
     ├── verify
     └── destroy

 --> Scenario: 'default'
 --> Action: 'lint'
 --> Executing Yamllint on files found in /home/elijah/sandbox/test-roles/example-role/...
 Lint completed successfully.
 --> Executing Flake8 on files found in /home/elijah/sandbox/test-roles/example-role/molecule/default/tests/...
 /home/elijah/.virtualenvs/world/lib/python3.7/site-packages/pycodestyle.py:113: FutureWarning: Possible nested set at position 1
   EXTRANEOUS_WHITESPACE_REGEX = re.compile(r'[[({] | []}),;:]')
 Lint completed successfully.
 --> Executing Ansible Lint on /home/elijah/sandbox/test-roles/example-role/molecule/default/playbook.yml...
 Lint completed successfully.
 --> Scenario: 'default'
 --> Action: 'destroy'

     PLAY [Destroy] *****************************************************************

     TASK [Destroy molecule instance(s)] ********************************************
     changed: [localhost] => (item=None)
     changed: [localhost]

     TASK [Wait for instance(s) deletion to complete] *******************************
     ok: [localhost] => (item=None)
     ok: [localhost]

     TASK [Delete docker network(s)] ************************************************

     PLAY RECAP *********************************************************************
     localhost                  : ok=2    changed=1    unreachable=0    failed=0


 --> Scenario: 'default'
 --> Action: 'dependency'
 Skipping, missing the requirements file.
 --> Scenario: 'default'
 --> Action: 'syntax'

     playbook: /home/elijah/sandbox/test-roles/example-role/molecule/default/playbook.yml

 --> Scenario: 'default'
 --> Action: 'create'

     PLAY [Create] ******************************************************************

     TASK [Log into a Docker registry] **********************************************
     skipping: [localhost] => (item=None)

     TASK [Create Dockerfiles from image names] *************************************
     changed: [localhost] => (item=None)
     changed: [localhost]

     TASK [Discover local Docker images] ********************************************
     ok: [localhost] => (item=None)
     ok: [localhost]

     TASK [Build an Ansible compatible image] ***************************************
     changed: [localhost] => (item=None)
     changed: [localhost]

     TASK [Create docker network(s)] ************************************************

     TASK [Create molecule instance(s)] *********************************************
     changed: [localhost] => (item=None)
     changed: [localhost]

     TASK [Wait for instance(s) creation to complete] *******************************
     changed: [localhost] => (item=None)
     changed: [localhost]

     PLAY RECAP *********************************************************************
     localhost                  : ok=5    changed=4    unreachable=0    failed=0


 --> Scenario: 'default'
 --> Action: 'prepare'
 Skipping, prepare playbook not configured.
 --> Scenario: 'default'
 --> Action: 'converge'

     PLAY [Converge] ****************************************************************

     TASK [Gathering Facts] *********************************************************
     ok: [instance]

     TASK [example-role : Create 'example' user] ************************************
     changed: [instance]

     TASK [example-role : Install nginx] ********************************************
     changed: [instance]

     RUNNING HANDLER [example-role : Restart nginx] *********************************
     changed: [instance]

     PLAY RECAP *********************************************************************
     instance                   : ok=4    changed=3    unreachable=0    failed=0


 --> Scenario: 'default'
 --> Action: 'idempotence'
 Idempotence completed successfully.
 --> Scenario: 'default'
 --> Action: 'side_effect'
 Skipping, side effect playbook not configured.
 --> Scenario: 'default'
 --> Action: 'verify'
 --> Executing Testinfra tests found in /home/elijah/sandbox/test-roles/example-role/molecule/default/tests/...
     ============================= test session starts ==============================
     platform linux -- Python 3.7.1, pytest-4.1.0, py-1.7.0, pluggy-0.8.1
     rootdir: /home/elijah/sandbox/test-roles/example-role/molecule/default, inifile:
     plugins: testinfra-1.16.0
 collected 5 items

     tests/test_default.py .....                                              [100%]

     =============================== warnings summary ===============================

 ...

     ==================== 5 passed, 7 warnings in 27.37 seconds =====================
 Verifier completed successfully.
 --> Scenario: 'default'
 --> Action: 'destroy'

     PLAY [Destroy] *****************************************************************

     TASK [Destroy molecule instance(s)] ********************************************
     changed: [localhost] => (item=None)
     changed: [localhost]

     TASK [Wait for instance(s) deletion to complete] *******************************
     changed: [localhost] => (item=None)
     changed: [localhost]

     TASK [Delete docker network(s)] ************************************************

     PLAY RECAP *********************************************************************
     localhost                  : ok=2    changed=2    unreachable=0    failed=0
#+END_EXAMPLE

I have a few warning messages (that's likely because I am using /python 3.7/ and some of the libraries still don't fully support the new standards released with it) but all my tests passed.

* Conclusion
Molecule is a great tool to test ansible roles quickly and while developing them. It also comes bundled with a bunch of other features from different projects that will test all aspects of your ansible code. I suggest you start using it when writing new ansible roles.