Configuration management is an essential piece of any computer system consisting of multiple components. If implemented correctly, it can make maintenance a lot easier and predictable for anyone involved.

This becomes especially powerful when the system is managed with machine-readable configuration files instead of using various interactive tools. Spinning up and configuring each component should be automatic and consistent.

SaltStack (also known by the less searchable name Salt) is one such configuration management system. It doesn't only support configuration management, but also remote execution and event driven automation but let's focus on getting started with Salt configuration management.

Simple architectural overview

Before we dive into a small example, let's take a look at a very brief and simplistic overview of how Salt works so you'll know what it's all about.

At its core Salt operates by setting up a small agent, called a minion, on all the system components you want to configure or operate (e.g. server). This minion talks to a Salt master which knows how the minion should be configured based on some configuration files written in YAML.

In the configuration management corner of Salt, these YAML configuration files define the state the component should find itself in. The minion itself then makes sure it is in the desired state by doing some Python-fu.

So basically to get Salt working, all you need to do is to set up a Salt master and define the configuration in a YAML file. Then you just install the minion on the system components you want the Salt master to control.

Example

To see it all in action let's set up a small Docker-based example. I'm fully aware that it's kind of weird to use Salt to configure Docker images because the Dockerfile is essentially a configuration file providing consistency, but Docker just makes it simpler to showcase the simplicity and for you to try out Salt.

We're going to create one Debian image with Salt master and create configuration files to control one Ubuntu image and one Alpine image.

First, let's create our two minions.

Alpine minion

The only thing you have to do really, is to install the Salt minion agent, it will automatically be configured with sane defaults although we'll change them slightly.

Installing Salt minion on Alpine is as easy as just running:

apk add salt-minion

(in Docker we'll add the --no-cache flag because we want to clean up after ourselves)

We have to make one small change to make the configuration files for the states easier. We're going to change something called the minion id.

Normally when the salt minion agent starts up it will by default take the hostname and set it as a minion id. The id is how the Salt master will recognize the different minions.

In the configuration management setup we need to target our minions which will allow us to define different states for different machines. There are many ways to target minions but we're going to target ours using the minion ids.

In Docker the hostnames are going to be a random string (e.g. 71665beda883) which makes predefined targeting difficult. What we're going to do is prefix the minion id with a string that makes it easier for us to target the machine.

For the Alpine example, we're going to use the prefix locomotive. To do that we'll create a docker entrypoint file which echoes "locomotive-$HOSTNAME" into the file where the salt minion stores its id, /etc/salt/minion_id. Afterwards we'll start up the salt minion.

So this is our docker-entrypoint.sh for alpine:

echo "locomotive-$HOSTNAME" > /etc/salt/minion_id
salt-minion

Now all we have to do to create the Alpine Salt minion, is install the Salt minion, copy our docker entrypoint file into the image, and set the command to run the docker entrypoint file.

FROM alpine

RUN apk add --no-cache salt-minion

COPY docker-entrypoint.sh /

CMD ["sh", "/docker-entrypoint.sh"]

Let's build the Docker image as alpine-minion (assuming we're in a folder where we have the alpine Dockerfile):

docker build -t alpine-minion .

Alright, onwards to the next minion.

Ubuntu minion

The Ubuntu minion is going to be almost exactly the same. The difference is going to be that we use apt to install the salt minion instead of apk, and we're going to use the prefix cow instead of locomotive (because we want to target the minions differently).

So the docker-entrypoint.sh is going to be like this:

echo "cow-$HOSTNAME" > /etc/salt/minion_id
salt-minion

The Dockerfile is going to look like this:

FROM ubuntu

RUN apt-get update && \
    apt-get install -y salt-minion

COPY docker-entrypoint.sh /

CMD ["sh", "/docker-entrypoint.sh"]

Let's build this one as ubuntu-minion (again assuming we're now in the ubuntu minion's directory):

docker build -t ubuntu-minion .

That's enough of the minions, let's move on to the states for our minions.

Salt states

You may be wondering why I chose the locomotive and cow prefix for the alpine and ubuntu images, respectively. That's because we're going to install two different software packages based on the prefix.

As I said before, we're going to target the minions (based on their minion id) and install the different packages based on the prefix. When compiling the state for a specific minion, Salt always starts looking in a file called top.sls. The sls file extension stands for SaLt State file, but don't let that fool you, it's just a YAML file. All state definition files should have the sls file extension.

We're going to create three state files. One to install two software packages: cowsay and fortune. One to install software packages sl and fortune (we install fortune in both this way for the example). Lastly we create the top.sls file where Salt starts its journey in.

Make sure a package is installed

Let's start by making sure the packages cowsay and fortune are installed. A Salt state file can contain multiple states, all defined by state names as the root elements of the state file. We're just going to have one in a file called cow.sls to make sure we install cowsay and fortune.

We can call the state what we want, but to be explicit let's call it install fun cow tools. The only thing we have to make sure when giving these names is that the for any given minion, no two states can have the same name.

The state element in the YAML file is a dictionary of the states we want. Salt comes with a lot of built in state modules we can use (that's the beauty of Salt) but the Salt state module we want to use is called pkg.

We use the pkg state module to define that the minions where this state applies need to have packages installed (a function in the pkg module called installed). The pkg.installed function can take in a keyword argument pkgs which for its value is a list of packages to install, in our case: fortune and cowsay.

The cow.sls file will look like this:

install cow tools:
  pkg.installed:
    - pkgs:
      - fortune
      - cowsay

The locomotive file is going to be similar, but instead of installing cowsay, we're going to install sl in addition to fortune. We'll also use the name install locomotive tools for the state. So the file locomotive.sls will look like this:

install locomotive tools:
  pkg.installed:
    - pkgs:
      - fortune
      - sl

As you can see, the files use the same salt modules and functions even if we know that behind the scenes one will be executed on an alpine machine and the other on an ubuntu machine. Salt will use the appropriate package manager.

top.sls

Let's tie this all together in the top.sls and target the machines based on their prefix. Salt allows use of an asterisk/glob when targeting so it's fairly easy to target the minion ids with these prefixes.

First we must define an environment we use. We'll use the default Salt environment called base. That's going to be the root element and the value is a dictionary. The key elements of that dictionary are the targeting patterns. Their values are what state files it should apply. So something like this:

<environment>:
  <target pattern>:
    - <state file>
    - <state file>

The state files are given as the name without the file extension. The state files can be in subdirectories but then we have to use the Python convention where directories and files are separated by a period. So a file cow.sls in the subdirectory farm is included as farm.cow.

Now this may seem complicated but it's really not. It's just hard to describe. If we assume the files are in the same directory as the top.sls file, our top.sls will look like this:

base:
  locomotive-*:
    - locomotive
  cow-*:
    - cow

This means any minion id that starts with locomotive- is going to apply the states found in the locomotive.sls file in the same folder as the top.sls file.

This is the gist of how we build up and define the state of a system with Salt, using YAML files. Now we all we need is the master that can tell the minions what to do.

Salt master

Let's use Debian for the Salt mater. Just like with the minions, The only thing we have to do to get Salt up and running with sane defaults, is to install the salt-master package. So it's going to be a simple Docker image we need (we'll do the magic that we need to make our Salt system operational, when we mount configuration files):

FROM debian:10-slim

RUN apt-get update && \
    apt-get install -y salt-master

CMD salt-master

Let's build this image as salt-master (assuming we're in the same directory as the Salt master Dockerfile):

docker build -t salt-master .

Now with all images we need let's make it operational!

Salt master configuration

First off, we're going to need a basic configuration file that will tell the Salt master where our state files are.

Thanks to Salt's sane defaults, the configuration file is super-simple. We only have to define a file root where our state environment can be found. It is possible to manage a lot of different Salt environments with different states using the same Salt master and configuration file but we're going to stick to the default base environment.

The configuration file just needs to point to the directory with the root of our state files for a base Salt environment. We'll create a file called master.config with only three lines:

file_roots:
  base:
    - /srv/salt/

Our state files (those we created earlier) need to be made available at /srv/salt and we need to mount our master.config in our Salt master as the file /etc/salt/master.

Docker compose environment

To make all of this easier we'll use Docker compose to set up all of our services. Before we create the Docker compose file let's first have a look at how our local file system should look when we've created our Docker compose file (assuming the Dockerfiles we used earlier are stored elsewhere):

salt-example/
  |
  |- docker-compose.yml
  |- master.config
  `- states/
       |
       |- cow.sls
       |- locomotive.sls
       `- top.sls

Now we can create our Docker compose file.

There are a few things we have to make sure we do. Two I have already mentioned: one is to mount the master.config as /etc/salt/master in the Salt master service, the other to mount our states/ directory as /srv/salt/ in the Salt master service.

The third thing I haven't mentioned, but it's very important. By default minions will look for the Salt master at the domain name salt. This is configurable but we are using defaults, so we have to make sure the Salt master service in the Docker compose file is called salt (which makes it available as salt to our minions who share the docker compose network).

So the docker-compose.yml file will define three services, salt (with files and directories mounted properly), our alpine minion and the ubuntu minion:

version: "3"
services:
  salt:
    image: salt-master
    volumes:
      - ./master.config:/etc/salt/master
      - ./states:/srv/salt
  alpine-minion:
    image: alpine-minion
  ubuntu-minion:
    image: ubuntu-minion

That is all. We're ready to start everything up. Just type:

docker-compose up -d

Congratulations! If there haven't been any typos, you now have your first Salt configuration management environment up and running. But it actually doesn't do anything.

Accepting minions

If Salt would just start handing out states to any machine that contacts the Salt master and fits into a specific minion id pattern, we'd have a problem on our hands. To avoid this we need to accept what minions we want the Salt master to manage.

When the minion contacts the Salt master it sends its public key. If we haven't accepted the minion before, we need to accept it (that is to say, if we trust it) using a tool installed on the Salt master called salt-key.

Before we continue, let's open up a shell on the Salt master docker container:

docker-compose exec salt bash

Now in the Salt master bash shell you can list all minion keys that are accepted, denied, rejected and unaccepted, by running salt-key with the -l all option:

salt-key -l all

You should see an output similar to this one:

Accepted Keys:
Denied Keys:
Unaccepted Keys:
cow-7d7f4faf9802
locomotive-1fbb4f5c409d
Rejected Keys:

As you can see there should be two unaccepted keys (one with a cow prefix and another with a locomotive prefix).

If we do not accept them we will not be able to manage their configurations so let's go ahead and accept all the keys we've gotten using the -A flag:

salt-key -A

You'll be asked to confirm if you want to accept the two listed keys. Just enter Y if all looks good (or simply press Enter because accepting is the default). Now you should see something like:

Key for minion cow-7d7f4faf9802 accepted.
Key for minion locomotive-1fbb4f5c409d accepted.

Now we can control our minions. They may have shutdown because it took some time to accept their keys (they don't want to wait forever). Let's just be sure they're up by exiting the Salt master:

exit

Then we run docker compose up again:

docker-compose up -d

Then we can attach to the Salt master again:

docker-compose exec salt bash

If you want to be sure we've accepted our keys and they're still there just type:

salt-key -l all

You should see something like:

Accepted Keys:
cow-7d7f4faf9802
locomotive-1fbb4f5c409d
Denied Keys:
Unaccepted Keys:
Rejected Keys:

Applying state

The states still haven't been applied to our minions. They don't know what packages they should install so they haven't installed anything. To trigger the configuration we will have to execute a state.apply on the minions we want to apply the state.

To do this we target our minions and we can safely just target all of our minions using the glob (asterisk) and tell it to apply the states. To do that we type the following command:

salt '*' state.apply

Notice the quotes around the glob. This is needed because we're passing the string with the targeting pattern.

You should see a lot of stuff on your screen but near the end of it all, you'll see a summary like:

Summary for locomotive-1fbb4f5c409d
------------
Succeeded: 1 (changed=1)
Failed:    0
------------
Total states run:     1
Total run time:   5.633 s

This tells you all states were run successfully and one change was made (it installed the packages) on the locomotive machine (Alpine). You'll see a similar report for the cow machine (Ubuntu)

You can run the state.apply function as often as you like. Because it is a state, it defines how your minion should look so what it actually does before installing packages in our case, is to check if all the packages are installed. If they aren't or some are missing, it goes ahead and installs the missing ones.

The state apply does the magic of making sure that the state is fulfilled. Running state.apply again should result in a report for (one for the Alpine machine and another for the Ubuntu machine) like this one:

locomotive-1fbb4f5c409d:
----------
          ID: install locomotive tools
    Function: pkg.installed
      Result: True
     Comment: All specified packages are already installed
     Started: 23:53:42.277146
    Duration: 723.556 ms
     Changes:   

Summary for locomotive-03f0f2574592
------------
Succeeded: 1
Failed:    0
------------
Total states run:     1
Total run time: 723.556 ms

This is awesome. Salt tells us everything is great, but is it? Let's check. First exit the Salt master

exit

Alright now we're ready to check if Salt did what we wanted it to.

Does it work?

The cowsay package in Ubuntu is installed in /usr/games/cowsay so to see it in action we'll run this:

docker-compose exec ubuntu-minion /usr/games/cowsay I think I need some Salt

You'll see a bovine ascii art telling you it needs some Salt. Let's make see if we have fortune installed (also in /usr/games/):

docker-compose exec ubuntu-minion /usr/games/fortune

Enjoy the wisdom!

Alpine should also have some packages (in this case we don't have to access it at /usr/games/ because it is already in the path):

docker-compose exec alpine-minion fortune

Woah! More wisdom! Wait, maybe we installed the cow on Alpine as well:

docker-compose exec alpine-minion cowsay Salt told me not to

Nope. You should get an error telling you that cowsay isn't found in the path. But sl is:

docker-compose exec alpine-minion sl

Here's a joke because you got this far: What sound does a sick train make? -- Ah-Ah-Ah Choo choo!

So these were the basics of Salt. In future posts I want to dive deeper into Salt but I needed to get the basics out of the way.

This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.

Cover image by Jorge Royan: Four salt shakers on a table with salt spilled from one of them