Testing Ansible roles with Molecule, libvirt (vagrant-libvirt) and QEMU/KVM
I maintain a small Ansible role for WireGuard. It was mainly made for Ubuntu. After a while people started contributing code esp. for other OSes. As I only use Ubuntu and Archlinux personally I needed to trust contributors that their code works as I couldn’t test the parts that were special for a specific OS. That wasn’t great. I needed something to test the role for all OSes. That’s were Molecule comes in.
To quote the description of Molecule: “Molecule project is designed to aid in the development and testing of Ansible roles. Molecule provides support for testing with multiple instances, operating systems and distributions, virtualization providers, test frameworks and testing scenarios.”
Most people will use Docker as a driver which is also Molecule’s default. That means if you want to test your role with Ubuntu 20.04 e.g. Molecule will instruct the Docker driver to launch a Docker container with Ubuntu 20.04. But for my WireGuard role Docker isn’t good enough. I needed a “real” OS as some OSes still don’t have a kernel that supports WireGuard out of the box like Ubuntu 18.04 e.g. In such a case we need a DKMS module package and for this you need the correct kernel header package installed. If I would do this with Docker Ansible would report the kernel version of the host where the Docker container runs on. But that could (and probably will) be different from the Docker container. Also I needed
systemd which also requires some changes to make it work with Docker.
The next natural fit would be to use the Vagrant driver with the VirtualBox provider to create virtual machines (VM) with a fully installed Linux operating system. But why a second virtualization solution if I already have a few virtual machines running locally with libvirt and KVM (Kernel-based Virtual Machine)? That two components provide fast virtualization out of the box that basically comes with every Linux OS.
This instructions are mainly for Archlinux here but that only matters for installing the required packages. Everything else is the same no matter which Linux distribution you use. I’ll mention the required packages for Ubuntu 20.04 too but that’s mostly an educated guess (but I tried to install the packages in a virtual machine at least) ;-)
Before you start make sure that you add yourself to the
libvirt group (for Ubuntu 20.04 you may also need to be part of group
Next we need a few packages:
For Archlinux that’s
sudo pacman -S \ ansible \ qemu \ vagrant \ ebtables \ dnsmasq \ libvirt \ python-pip
and for Ubuntu 20.04 it’s most probably:
sudo apt-get install \ ansible \ qemu \ vagrant \ ebtables \ dnsmasq \ libvirt-daemon-system \ libvirt-clients \ libvirt-dev \ pkg-config \ python3-pip
There are also a few Python packages needed. While they are partly also part of the OS package repository I’d recommend to install them via Python’s package manger PIP to get the latest versions. Archlinux packages are normally quite up2date but in case for molecule-vagrant that’s currently (20200930) not the case but we definitely need the latest version here. Using the
--user flag makes sure that the Python packages are installed to the Python user install directory for your platform (typically
~/.local/). You can also create your own virtual env. with
python3 -m venv ... (at least since version 3.3 and up) but that’s out of scope of this blog post. Either way it helps to don’t mess around with the system Python packages. So:
pip3 install --user python-vagrant pip3 install --user testinfra pip3 install --user libvirt-python pip3 install --user molecule pip3 install --user molecule-vagrant pip3 install --user rich
molecule-vagrant needs to be at least version
0.3 or higher. With a lower version you’ll be out of luck!
Now I installed the
libvirt plugin for
vagrant plugin install vagrant-libvirt
Also make sure that
$HOME/.local/bin is part of your
To demonstrate how I’ve implemented Molecule in my WireGuard role lets have a look how the role directory looked before I started:
CHANGELOG.md defaults .gitignore handlers meta README.md tasks templates
Since the role implementation was already there the initialization of Molecule looks like this:
molecule init scenario kvm --role-name githubixx.ansible_role_wireguard --driver-name vagrant
This command adds a few files and directories:
├── molecule │ └── kvm │ ├── converge.yml │ ├── INSTALL.rst │ ├── molecule.yml │ └── verify.yml └── .yamllint
As you can see above inside the
molecule directory there is currently a folder called
kvm. That’s the name of the scenario I defined above with
molecule init scenario kvm .... If you don’t specify a scenario name here the directory would have been called
Scenarios are the starting point for a lot of powerful functionality that Molecule offers. For now, we can think of a scenario as a test suite for the role. You can create as many scenarios as you like and Molecule will run one after the other. One example could be that maybe later I’ll decide that I don’t only want to test with KVM virtual machines but also within a Google Cloud project. So I’ll just create an additional scenario
google-cloud e.g. When you create more scenarios it might make sense to share some files between them by creating a
shared folder and point from the scenarios to the files in that folder. But that’s a different story… ;-)
The content of the files in
molecule/kvm/ directory is pretty rudimentary. So I replaced the content. Lets first have a look at
molecule.yml. That’s basically the configuration file for the resources that are needed to run the test suite.
The first thing that is defined in this file is the dependency manager:
dependency: name: galaxy
Since we’re using Ansible it of course makes sense that
ansible-galaxy is used as dependency manager. As the WireGuard role don’t need any other Ansible roles nothing more needs to be defined here.
Next I defined the driver. Molecule uses the driver to delegate the task of creating instances:
driver: name: vagrant provider: name: libvirt type: libvirt options: memory: 192 cpus: 2
By default Molecule would use Docker but as already stated above I used the
vagrant driver with the
libvirt provider. So instead of Docker container this configuration will start virtual machines via
platforms: - name: test-wg-ubuntu2004 box: generic/ubuntu2004 interfaces: - auto_config: true network_name: private_network type: static ip: 192.168.10.10 groups: - vpn - name: test-wg-ubuntu1804 box: generic/ubuntu1804 interfaces: - auto_config: true network_name: private_network type: static ip: 192.168.10.20 groups: - vpn ... - name: test-wg-arch box: archlinux/archlinux interfaces: - auto_config: true network_name: private_network type: static ip: 192.168.10.80 groups: - vpn
platforms the virtual machines are defined that should be started. In the example above you only see three VMs defined but the WireGuard role defines a few more but that doesn’t matter. After the
box is defined. That’s basically preconfigured virtual machine images. You can find quite a few of them at https://app.vagrantup.com/boxes/search . Since in my case I’m only interested in VM images that can be used with the
libvirt provider you can use this URL: https://app.vagrantup.com/boxes/search?provider=libvirt
I mostly use the generic/* boxes. Besides the boxes for Ubuntu 18 and 20 and Archlinux I also defined VMs for CentOS, Fedora and Debian in different versions. I also tried to use generic/arch but this box has a problem that was not that easy to solve. It doesn’t contain
Python by default and that’s rather bad if you want to configure a VM with Ansible ;-) Luckily there is also an official Archlinux box archlinux/archlinux which contained Python out of the box. But that one had another problem. But I’ll come back to that later.
For most roles you normally don’t need to define static IPs for the interfaces. But in case of WireGuard an endpoint needs to be defined so that WireGuard can contact its peers. That’s why I configured an interface with static IPs for every VM.
And finally every VM is part of the
vpn group. That’s basically the host groups which you normally define in Ansible’s
While not strongly needed it may make sense to already download all the Vagrant VM images/boxes. This speeds up the process later. The VM images are quite large so be prepared to wait a little bit in case your internet connection isn’t that fast.
In case of the WireGuard role the following VM images/boxes are needed:
vagrant box add generic/ubuntu2004 --provider libvirt vagrant box add generic/ubuntu1804 --provider libvirt vagrant box add generic/debian10 --provider libvirt vagrant box add generic/fedora31 --provider libvirt vagrant box add generic/fedora32 --provider libvirt vagrant box add generic/centos8 --provider libvirt vagrant box add generic/centos7 --provider libvirt vagrant box add archlinux/archlinux --provider libvirt
But lets continue with the next part of
provisioner: name: ansible connection_options: ansible_ssh_user: vagrant ansible_become: true log: true lint: name: ansible-lint inventory: host_vars: test-wg-ubuntu2004: wireguard_address: "10.10.10.10/24" wireguard_port: 51820 wireguard_persistent_keepalive: "30" wireguard_endpoint: "192.168.10.10" test-wg-ubuntu1804: wireguard_address: "10.10.10.20/24" wireguard_port: 51820 wireguard_persistent_keepalive: "30" wireguard_endpoint: "192.168.10.20" ...
The sample above don’t shows all
host_vars. As already mentioned above basically the same host variables were also defined for CentOS, Fedora, Debian and Archlinux.
All the Vagrant images contain a user
vagrant that has
sudo permissions. That’s why I defined
ansible_become to tell Ansible to use that user to login and to use
sudo (which is the default method for
For all hosts defined in
platforms above I defined a few host variables. As you can see
wireguard_endpoint host variable matches
interfaces.ip defined in
platforms. And every host becomes a private WireGuard interface IP defined in
wireguard_address. So if everything works well later we should be able to run
ping 10.10.10.20 from
test-wg-ubuntu2004 and should get some response if the VPN works as intended.
scenario: name: kvm test_sequence: - prepare - converge
As mentioned above I created a scenario
kvm. Molecule treats scenarios as a first-class citizens, with a top-level configuration syntax. A scenario allows Molecule test a role in a particular way. A scenario is a self-contained directory containing everything necessary for testing the role in a particular way.
If you’ve a look at the documentation various sequences can be defined like
test sequence. For now the
test_sequence defined above is good enough to prepare/start the VMs and install the role on all virtual machines by using the host variables defined in the inventory.
That’s basically what we need for
molecule.yml for now. As already mentioned in
platforms above I had one problem with the
Archlinux box. During my first run I figured out that Ansible was not able to install the required packages as
Archlinux package manager
pacman was not initialized. But I don’t wanted to add this task to the WireGuard role because this is not the responsibility of that role to init a package manager.
So I finally came up with the idea to modify
converge.yml. So the result looked like this:
--- - hosts: all remote_user: vagrant become: true gather_facts: true tasks: - name: Init pacman raw: | pacman-key --init pacman-key --populate archlinux changed_when: false ignore_errors: true when: ansible_distribution|lower == 'archlinux' - name: Include WireGuard role include_role: name: githubixx.ansible_role_wireguard
As you can see there is task that gets only executed if Ansible sets the internal
ansible_distribution variable to
archlinux. And it does exactly what I needed. So afterwards only the WireGuard role needs to be included.
Now that everything is setup
molecule converge -s kvm
should now start the test VMs and install the role on all hosts accordingly. If you don’t already downloaded the VM images/boxes as mentioned above it could take quite a while the first time to execute this command as all VM images needs to be downloaded first.
Now currently Molecule only creates the virtual machines and installs the role with the provided variables. This only makes sure that the role can be installed but that doesn’t mean that everything also works as intended. This is where Testinfra comes in. With Testinfra you can write unit tests in Python to test actual state of your servers configured by management tools like Ansible. But since I only wanted to demonstrate how to use Molecule with
libvirt this blog post ends here ;-)