How to test your Ansible playbook with Vagrant

How it all started

Infrastructure as Code is getting more and more attention and for good reasons.
Previously infrastructures where created and maintained by hand, which works well until...

So when I needed to create a new database cluster I looked into a setup that would be reproducible and it should be testable before it was deployed on the actual servers.

For stateless things we can used docker but that wasn't a great fit for the database servers as we need to have persistent storage and scaling and/or changes where not expected that often.

I went for a golden oldie: Ansible
While I had worked with it before, it was mainly putting some roles together run the script and validate, manually, that it worked.

This time around I didn't want to 'test' the provisioning manually. As the setup was a bit larger and the overhead (commit, deploy) was bigger.

So I thought there should be a smarter and better way.

Start of the journey

As I am mainly a dev engineer, I didn't had thorough expertise on Ansible. (still don't) So Google it was to get the journey started. Looking through all the things, I mostly saw tests for Ansible Roles. Or ways to test the playbook on the target systems, but I wanted to test the playbook before deployment during the build and/or locally.

Turned out it was not that hard to create, but getting all the information was.
Because of the combination of Ansible and Vagrant and probably my lack of knowledge on those topics 😃~

The project setup

First off, what are we actually going to provision?
We will be provisioning a 3 server cluster of a database server, RethinkDB
You probably didn't heard from it before, and you can forget about it after reading this.

There will be some specific commands for rethinkDB, but most of it will be about the testing setup and ways you should test.

Project layout

A pretty normal layout for Ansible projects.
The part we will focus on is the test folder.

Test

There are a few levels of testing that can be done for the Ansible playbook.

You can have Ansible roles unit tests, which will validate if the roles 'works'.
And you can have tests for the playbook.

In my case the roles I created where specific for my situation so I didn't, wanted to, create unit tests as that would have been a bit overkill.

Instead I focussed on a few other things:

The role itself should validate if the work it did was actually executed and working as intended.

Making sure roles are doing what they should

So for instance the install-rethinkdb role should validate that it did install the package provided.

To do so, we get the version that is installed by executing the rethinkdb -v command and save the output. In another step we validate that the output is what we expect.
This way we are verifying that the installation of the (new) version went successful as the version is being used.


    - name: Install rethinkdb
      yum:
        name: "/tmp/rethinkdb-{{ rethinkdb_version }}.x86_64.rpm"
        state: present
        allow_downgrade: yes
        disable_gpg_check: yes        

    - name: Get installed rethinkdb version
      command: rethinkdb -v
      register: rethinkdb_version_installed

    - name: Validate rethinkdb version used is the same as wanted version
      assert:
        that:
          - "'{{ rethinkdb_version }}' in rethinkdb_version_installed.stdout"

In another role I updated the configuration and needed to restart the database in order to get the configuration working.

For this we restart the service via systemd and wait for couple of seconds.
We then get the service_facts and validate that the service is in a running state.

- name: Make sure rethinkdb service is restarted
    systemd:
      state: restarted
      name: rethinkdb.service
      enabled: yes
      daemon_reload: yes

  - name: Pause for 10 seconds to start the service
    pause:
      seconds: 10

  - name: Get service statuses
    service_facts:
    register: service_state_rethinkdb

  - name: Validate status of rethinkdb service
    assert:
      that:
        - "'running' in service_state_rethinkdb.ansible_facts.services['rethinkdb.service'].state"

The take way here is that you should look at your roles as parts that will have a single purpose and that you should also validate that they do what you expect. Role tests and such are great but verifying it during runtime is a must, as that is what matters in the end

Testing the playbook in a controlled (local/build) environment

For testing the playbook we need servers to executed the playbook on.
In comes, if you looked at the layout you could have guessed it, Vagrant

So here is the Vagrantfile that I used, I added some comments in there to help you understand what is going on.

In short, it will provision 3 servers that are in an own private network and execute the (test) playbook on those servers. Effectively creating a local database cluster.

# -*- mode: ruby -*-
# vi: set ft=ruby :

# Copy certificate files to destination they are expected.
FileUtils.cp %w(tests/files/ca.pem tests/files/cert.pem tests/files/key.pem), 'roles/ansible-role-rethinkdb-configure/files/'
FileUtils.cp %w(tests/files/vagrant1.local.net.crt tests/files/vagrant1.local.net.key), 'files/'
FileUtils.cp %w(tests/files/vagrant2.local.net.crt tests/files/vagrant2.local.net.key), 'files/'
FileUtils.cp %w(tests/files/vagrant3.local.net.crt tests/files/vagrant3.local.net.key), 'files/'

Vagrant.configure("2") do |config|

  # If you have issues with SSL certificates add this
  config.vm.box_download_insecure = true
  # We want to use redhat7 as the target servers are also redhat 7
  config.vm.box = "generic/rhel7"

  # To save overhead and time we use linked_clones (https://www.vagrantup.com/docs/providers/virtualbox/configuration#linked-clones)
  config.vm.provider "virtualbox" do |v|
    v.linked_clone = true
  end

  # We want to provision 3 servers for our cluster
  N = 3
  (1..N).each do |machine_id|
    # Give the server a unique name
    config.vm.define "vagrant#{machine_id}" do |machine|
      # same for the hostname of the VM
      machine.vm.hostname = "vagrant#{machine_id}"
      # Here we will be setting up the private network for the cluster.
      # In this network the cluster can communicate to each other.
      # We also want to expose (forward) some ports that are used inside the VM to the host (so that we can connect to the dashboard)
      # In this example I only added it for server1.
      if machine_id === 1
        # Private network instruction for vagrant
        machine.vm.network "private_network", ip: "192.168.77.#{20+machine_id}", virtualbox__intnet: "network", name: "network"
        # Forward all the ports below to the host. auto_correct is set to true to chose an available port if the one specified is taken
        machine.vm.network "forwarded_port", guest: 10082, host: 10082, auto_correct: true
        machine.vm.network "forwarded_port", guest: 10081, host: 10081, auto_correct: true
        machine.vm.network "forwarded_port", guest: 8443, host: 8443, auto_correct: true
        machine.vm.network "forwarded_port", guest: 9088, host: 9088, auto_correct: true
        machine.vm.network "forwarded_port", guest: 9089, host: 9089, auto_correct: true
      else
        # The other servers (2,3) should be in the private network
        machine.vm.network "private_network", ip: "192.168.77.#{20+machine_id}", virtualbox__intnet: "network"
      end

      # Only execute when all the machines are up and ready.
      if machine_id == N
        machine.vm.provision :ansible do |ansible|
          # Disable default limit to connect to all the machines
          ansible.limit = "all"
          # Playbook that should be executed, in this case the test playbook
          ansible.playbook = "tests/test.yml"
          # Please log it all
          ansible.verbose = true
          # We need to provied a few extra vars to the playbook
          ansible.extra_vars = {
            RETHINKDB_DATABASE_ADMIN_PASSWORD: '',
            NGINX_DASHBOARD_PASSWORD: 'pasword'
          }
          # These groups are the same as you normally specify in the inventorie file
          ansible.groups = {
            leader: ["vagrant1"],
            followers: ["vagrant2", "vagrant3"],
          }
        end
      end
    end
  end

end

The 'test' playbook looks like this:
I am using a specific playbook for tests as I needed to add a few things that are normally there if you get a server from your provider. At least that was the case in my project.

---

- name: Install rethinkdb 
  hosts: all # Executed this on all 3 servers
  become: yes # become root

  # pre_tasks are there to create the rethinkdb user. 
  # These will be executed before the rest.
  pre_tasks:
    - name: Add group "rethinkdb01"
      group:
        name: rethinkdb01
        state: present

    - name: Add user "ansi_rethinkdb01"
      user:
        name: ansi_rethinkdb01
        groups: rethinkdb01
        shell: /sbin/nologin
        append: yes
        comment: "vagrant nologin User"
        state: present

    - name: Add user "rethinkdb01"
      user:
        name: ansi_rethinkdb01
        groups: rethinkdb01
        shell: /sbin/nologin
        append: yes
        comment: "vagrant nologin User"
        state: present

    - name: Allow 'rethinkdb01' group to have passwordless sudo
      lineinfile:
        dest: /etc/sudoers
        state: present
        regexp: '^%rethinkdb01'
        line: '%rethinkdb01 ALL=(ALL) NOPASSWD: ALL'
        validate: 'visudo -cf %s'

    - name: Add sudoers users to rethinkdb01 group
      user: name=ansi_rethinkdb01 groups=rethinkdb01 append=yes state=present createhome=yes

    - name: Add sudoers users to rethinkdb01 group
      user: name=rethinkdb01 groups=rethinkdb01 append=yes state=present createhome=yes

# As the firewall was interrupting with the private network I disabled it
    - name: Stop and disable firewalld.
      service:
        name: firewalld
        state: stopped
        enabled: False

 # Rethinkdb can be memory happy so we needed to increase the virtual memory on the real servers. Here to test if it works.
    - name: Increase virtual memory
      shell: echo "vm.max_map_count=262144" >> /etc/sysctl.d/rethinkdb.conf && sudo sysctl --system

# We need to have  machine certificates for cluster security.
    - name: Copy certificate files
      copy:
        src: "{{ item.src }}"
        dest: "{{ item.dest }}"
        group: "{{ target_group }}"
        owner: "{{ target_user }}"
        force: yes
      loop:
          - {src: '{{ ansible_hostname }}.local.net.crt', dest: '/etc/ssl/certs/{{ ansible_hostname }}.local.net.crt'}
          - {src: '{{ ansible_hostname }}.local.net.key', dest: '/etc/ssl/certs/{{ ansible_hostname }}.local.net.key'}

# The roles that will be executed.
  roles:
    - name: ../roles/ansible-role-install-rethinkdb-offline
    - name: ../roles/ansible-role-rethinkdb-configure
    - name: ../roles/ansible-role-install-node-exporter-offline

# This is the real 'testing' part of the playbook
# As described before each role should make sure it does what it should do.
# But in order to make sure the provisioning works as a whole we will be testing some things here again

  post_tasks:
    - name: Get virtual memory config
      shell: sysctl vm.max_map_count
      register: rethinkdb_virtual_memory

    - name: Validate virtual memory config is correct
      assert:
        that:
          - "'vm.max_map_count = 262144' in rethinkdb_virtual_memory.stdout"

    - name: Get service status
      service_facts:
      register: service_state_rethinkdb

    - name: Validate status of rethinkdb service
      assert:
        that:
          - "'running' in service_state_rethinkdb.ansible_facts.services['rethinkdb.service'].state"

# Call the node_exporter locally and save the output
    - name: Check if node exporter is working
      uri:
        url: https://localhost:10081/metrics
        return_content: yes
        validate_certs: no
      register: node_exporter_response

# Validate that there is some output.  
    - name: Validate node exporter running
      assert:
        that:
          - "'go_goroutines' in node_exporter_response.content"

- name: install horizon and proxy
  hosts: leader # Only execute this on the servers in the `leader` group
  become: yes

  roles:
    - name: ../roles/ansible-role-rethinkdb-horizon
    - name: ../roles/ansible-role-rethinkdb-exporter
    - name: ../roles/ansible-role-rethinkdb-nginx

  post_tasks:
    - name: Get service status
      service_facts:
      register: service_state_rethinkdb

    - name: Validate status of rethinkdb service
      assert:
        that:
          - "'running' in service_state_rethinkdb.ansible_facts.services['rethinkdb.service'].state"

    - name: Validate status of rethinkdb-exporter service
      assert:
        that:
          - "'running' in service_state_rethinkdb.ansible_facts.services['rethinkdb-exporter.service'].state"
      run_once: true    

# The rethinkdb rethinkdb_exporter should provided metrics, and we want to check the results so save it to a variable.
    - name: Check if all nodes are connected
      uri:
        url: https://localhost:10082/metrics
        return_content: yes
        validate_certs: no
      register: exporter_response
      run_once: true

# Validate that there are three servers connected to the cluster according to the rethinkdb_exporter
    - name: Validate number of nodes running
      assert:
        that:
          - "'rethinkdb_cluster_servers_total {{ rethinkdb_number_of_nodes }}' in exporter_response.content"
      run_once: true

...

In the post_taks you can add as many assertions as you want/need. Make sure that you at least test the basics, so that the thing you installed is up and working.

Execute it

Well we only need to run one command to test the whole thing:

 vagrant up

Or if you want to execute it on Azure DevOps:

stages:
  - stage: Verifying
    pool:
      vmImage: 'macOS-latest'
    jobs:
      - job: Test_the_setup
        condition: or(eq('${{parameters.testRethinkDBSetup}}', true), eq(variables.isMain, false))
        steps: 
          - script: brew install ansible
            displayName: 'Install ansible'
          - script: |
              cd rethinkdb
              vagrant up    
            displayName: 'Run rethinkdb ansible scripts'

This will start the Vagrantfile and in effect create the VM's and start the playbook execution.

vagrant1: Calculating and comparing box checksum...
==> vagrant1: Successfully added box 'generic/rhel7' (v3.2.10) for 'virtualbox'!
==> vagrant1: Preparing master VM for linked clones...
    vagrant1: This is a one time operation. Once the master VM is prepared,
    vagrant1: it will be used as a base for linked clones, making the creation
    vagrant1: of new VMs take milliseconds on a modern system.
==> vagrant1: Importing base box 'generic/rhel7'...

---
PLAY [Install rethinkdb] *******************************************************

TASK [Gathering Facts] *********************************************************
ok: [vagrant2]
ok: [vagrant3]
ok: [vagrant1]

TASK [Add group "rethinkdb01"] *************************************************
changed: [vagrant2] => {"changed": true, "gid": 1001, "name": "rethinkdb01", "state": "present", "system": false}
changed: [vagrant1] => {"changed": true, "gid": 1001, "name": "rethinkdb01", "state": "present", "system": false}
changed: [vagrant3] => {"changed": true, "gid": 1001, "name": "rethinkdb01", "state": "present", "system": false}
---

PLAY RECAP *********************************************************************
vagrant1                   : ok=79   changed=48   unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
vagrant2                   : ok=44   changed=27   unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
vagrant3                   : ok=44   changed=27   unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

With this setup you are sure that the provisioned servers are working as intended and that if you make a change it is validated. Even after the creator left 😄

We now have a local running database cluster that is tested and, as a bonus, you can use it for development.

If you have any question regarding testing Ansible playbooks and or setting up a local Vagrant (cluster) environment, please let me know! (Not sure if I can answer them but will do my best 😰)

Also any other remarks/improvements are always welcome.

Blog