The first article in this series went over an example of how to set up the Salt configuration management tool to automate a computer infrastructure.

Salt is very powerful and in this next section of the series, we'll take a look at two important components, grains and the pillar. At a first glance grains and the pillar seem to serve the same purpose but there is an important distinction between them.

What are grains and the pillar?

Grains and the pillar are used by Salt to store data about the minion it is controlling. They are both in the end a key value store. In fact they are what is called in Python a dictionary (if you're into Perl or Ruby, they're known as a hash).

For example, the following is an example of some grains on the alpine minion we created in the first article in the series:

ipv4:
    - 127.0.0.1
    - 172.25.0.3
ipv6:
kernel:
    Linux
kernelrelease:
    4.15.0-65-generic

If you want to check out all of the grains, you can execute the following command in the salt master docker container (to get a shell, type this in the same directory as you keep your docker compose file: docker-compose exec salt bash):

salt 'locomotive-*' grains.items

The pillar data looks the same except the keys and values are different. However if you run the command to list the pillar items on your salt master, you won't get a similar result now:

salt 'locomotive-*' pillar.items

You'll get nothing as a result:

locomotive-03f0f2574592:
    ----------

That's because while many grains are automatically generated when the salt minion starts up, the pillar data isn't. In fact the salt minion doesn't ever generate the pillar data. That's the big distinction between grains and the pillar.

Grains are generated and/or stored on the minions themselves. Pillar data are generated and/or stored on the salt master.

This distinction allows the pillar data to contain sensitive data you don't want lying around on the minion. It could be minion configurations or variables you want to manage in a central place, well or just any other data.

Grains on the other hand are useful for information you won't necessarily know when you configure the minion like IP addresses, operating system, CPU information etc. It can also be arbitrary data, like a role you want to configure and store on the minion itself.

For example you could on the machine create a grain called role with value webserver. Then the salt master could pick up that grain and from there decide how to configure it, without changing anything in the salt master configuration files. The pillar data for that machine may then include information such as software licenses you need for a particular application, users allowed to log into web servers etc.

One good thing to keep in mind is that grains are fairly static, they change infrequently because they're just information about the system (either automatically generated or custom data). The pillar on the other hand is better suited for data that will change more frequently.

Configuring grains

Most of the grains you normally need are automatically generated. However, if you ever find yourself in a position where you want to configure them you can store them in the minion configuration file on the machine. Location depends on operating system, for example on Unix systems you find it at /etc/salt/minion except in FreeBSD where it's stored at /usr/local/etc/salt/minion. On Windows machines the configuration is found at C:\salt\conf\minion.

The configuration file is a YAML file and you can add grains as a root element in the configuration file with the grains you want to store. For example you could add this to the minion config:

grains:
  roles:
    - webserver

This would make the grain roles available with a list as its value, containing the webserver role (it doesn't have to be a list of strings but I try to choose a data structure applicable to the context and a server could have multiple roles so that's why I'd use a list for the roles).

Another place where you can store grains is in /etc/salt/grains. I like explicit so I like this location more than the minion config. It's a YAML file but if you want to use /etc/salt/grains you don't need the root grains element. So the equivalent /etc/salt/grains file to the one above is:

roles:
  - webserver

Let's adapt the example from the first post in this series and configure grains for the two salt minions.

Let's create two grains files, one for our alpine container and another for the Ubuntu container, defining roles for the machines. Create a file called alpine.grains in the same location as your docker-compose.yml file (we're going to mount it later on). Here are the contents of the alpine.grains file:

roles:
  - locomotive
  - fortuneteller

Create a similar file for the Ubuntu container called ubuntu.grains with these contents:

roles:
  - cow
  - fortuneteller

Now let's update our docker-compose.yml We just have to mount the two files we created at /etc/salt/grains on both machines so the docker-compose.yml file should look like this:

version: "3"
services:
  salt:
    image: salt-master
    volumes:
      - ./master.config:/etc/salt/master
      - ./states:/srv/salt
  alpine-minion:
    image: alpine-minion
      - ./alpine.grains:/etc/salt/grains
  ubuntu-minion:
    image: ubuntu-minion
    volumes:
      - ./ubuntu.grains:/etc/salt/grains

Let' get all of them up and running using these new grains and therefore their new roles. Run this command from the same directory:

docker-compose up -d

Still running first post containers?

If you shut down and removed the containers you created in the first post, you'll just have to accept keys like you did in the first post.

If you are still running the same containers, you should see the minions being updated (recreated), but not the salt master (it's up to date).

If you try to check things out with salt master now, you'll notice that it doesn't manage your salt minions any longer. The reason is that because we recreated the docker containers they now have a new minion id (it's a new machine basically). So if you're not following this guide adamantly and want to play around, you'll have to accept the keys as shown in the first blog post in the series.

Now we're ready to use those newly defined roles somehow.

Usefulness of grains

Targeting minions

Grains are very useful for targeting minions. Remember in the first article how we targeted the machines using the minion id? We can also target them using grains.

Let's update the configuration file for the salt master to target based on grains instead of minion ids. First let's create a special state file for the fortune program. Remove the line in both cow.sls and locomotive.sls where we tell salt to install the fortune package. Then create a similar state file called fortune.sls:

install fortuneteller tools:
  pkg.installed:
    - pkgs:
      - fortune

Alright. Now we can change the top.sls file to target based on grains instead of minion ids. The pattern for it is to say key:value. Then you can, before you list what state files to include, begin by telling salt what you want to match with your pattern (in our case the grain).

So to apply the apache.sls file to machines with the role webserver you would write something like this:

'roles:webserver':
  - match: grain
  - apache

Notice the ' around roles:webserver. This is so that the YAML file gets processed correctly. There is a shorthand version to achieve the same. You could instead write this:

'G@roles:webserver':
  - apache

Let's modify the top file so we target each role specifically and apply the appropriate state files. The top file could look like this then (I choose the more explicit targeting pattern):

base:
  'roles:cow':
    - match: grain
    - cow
  'roles:fortuneteller':
    - match: grain
    - fortune
  'roles:locomotive':
    - match: grain
    - locomotive

Logic in state files

Grains can also be useful in the state files themselves. For example, we can check the operating system and do different things based on the operating system. You'll see more on this when we go through templating in the Salt files but to give you a taste, let's solve the problem we had in the first article of the series, where the Ubuntu machine installed the packages in /usr/games/.

To solve that problem we'll put an if statement that checks the os grain to see if it is an Ubuntu operating system. If it is, then we make a symbolic link to /usr/bin. We'll need to change all state files because Ubuntu does this with all packages.

Before we change the files, let's first see how we look up the grain value in the state file. The grain we're after to check the operating system is the os grain. It becomes available as grains['os'] (if you know Python, you'll understand that this is just a dictionary lookup).

So the trick is to add a Python if statement inside {% %} then include the state configurations you want before you end with a {% endif %}. I'll cover this better in a later post but to start cow.sls should have this content now:

install cow tools:
  pkg.installed:
    - pkgs:
      - cowsay

{% if grains['os'] == 'Ubuntu' %}
make cowsay available on Ubuntu:
  file.symlink:
    - name: /usr/bin/cowsay
    - target: /usr/games/cowsay
{% endif %}

To dissect this a little bit, first you see the state we defined in the first post of the series and modified a few minutes ago (to remove fortune). Then we create our if statement which checks if the os grain is equal to Ubuntu. If it is, then we create a state called make cowsay available on Ubuntu. This states uses the file state module to create a symlink to /usr/games/cowsay (the target) at /usr/bin/cowsay (defined in name of the symlink). Then we close the if statement.

fortune.sls is going to be similar

install fortuneteller tools:
  pkg.installed:
    - pkgs:
      - fortune

{% if grains['os'] == 'Ubuntu' %}
make fortune available on Ubuntu:
  file.symlink:
    - name: /usr/bin/fortune
    - target: /usr/games/fortune
{% endif %}

So will locomotive.sls

install locomotive tools:
  pkg.installed:
    - pkgs:
      - sl

{% if grains['os'] == 'Ubuntu' %}
make sl available on Ubuntu:
  file.symlink:
    - name: /usr/bin/sl
    - target: /usr/games/sl
{% endif %}

Trying out the grain based configuration

Let's test our new grain based configuration. First let's stop and remove all of our docker containers because we want a fresh slate (this way we'll also get around the new minion ids discussed earlier and easily remove the older containers.

Type in these docker-compose commands to get stop containers, remove them, start up new ones and finally enter bash shell on the salt master (answering yes when prompted):

docker-compose stop
docker-compose rm
docker-compose up -d
docker-compose exec salt bash

Once you've entered the bash shell on the new salt master, accept the minion keys with this command (pressing Enter for yes when prompted):

salt-key -A

Now we're ready to test our grain based targeting in the top.sls. Let's apply the states to all machines (targeting them with the glob *):

salt '*' state.apply

This will install all the programs, based on the roles defined in the grains and create the symlinks on the Ubuntu machines thanks to the if statement in the states. If you look at the output report, you'll see 2 states succeeded in the Alpine container while four changed in the Ubuntu container.

Targeting in the command line

Another place you can use to target based on grains is on the command line. Instead of using the glob or minion id when you run the salt command on the salt master, you can target based on grain by using the -G option.

As an example, let's run the fortune command on all fortuneteller minions (both Ubuntu and Alpine containers). To run a command via the salt command on the salt master you type cmd.run instead of the state.apply we've used earlier and then you tell salt which command to run (in our case fortune). Go ahead and type this in your salt master's bash shell:

salt -G roles:fortuneteller cmd.run fortune

As you can see, we give salt the -G option followed by the grain key and value we want to target (much like we targeted them in the top.sls file. Then we tell salt to run the fortune command with the cmd.run fortune. The output should be some nice words of wisdom:

cow-e54f99c12817:
    Don't feed the bats tonight.
locomotive-68af4abb8a98:
    Put no trust in cryptic comments.

We can also only ask Alpine minions to deliver fortunes:

salt -G os:Alpine cmd.run fortune

This returns only a single wisdom because the grain only matched a single minion:

locomotive-68af4abb8a98:
    The meek shall inherit the earth -- they are too weak to refuse.

OK. That's enough about grains for now, let's quickly turn to the pillar.

Configuring the pillar

As I said previously. The pillar offers the same data structure as grains but instead of being generated and hosted on the salt minions, the pillar data are generated on the salt master and handed to the minion.

That means that one big difference between the pillar and grains is that the pillar needs targeting, so that the pillar data can be handed to the right minion. Besides that, they're just the same.

Let's move the exact programs we need to install for each role into the pillar. That makes it more flexible because we don't have to hard code the programs to install on each minion based on the role. It's going to simplify maintenance and our state code.

Configuring the pillar is a mix between states and grains. We have a top.sls file for the pillar where we do the targeting. There we include files, which should return a data structure similar to grains.

Pillar configuration files

We begin by creating a folder called pillar. This folder we're going to mount on our salt master. This is the folder that will hold our pillar data. Let's first create files for each role. The root element in each file will be packages (could be anything we want but we want it to be explicit right?). Followed by the name of the program as a key with a value which is just a description of the package we install. Something like this:

packages:
  package-name: package description

We won't really use the package description, we're only interested in the package name. However, salt will automatically merge dictionaries in a smart way (which is useful and cool) but unfortunately not append to lists. That's fine though because this gives us a nice place to put a description or comment with an Easter egg.

So first create a file called cow.sls with this content:

packages:
  cowsay: A program with a cow

The second file called fortuneteller.sls with this content:

packages:
  fortune: A program for a fortune teller

Lastly create the third file called locomotive.sls with this content:

packages:
  sl: Tech Model Railroad Club

Finally we need our top.sls file. It's going to be exactly the same as our state file (you can copy it if you want). First we define the environment we want it to be used in, then we target the roles and include the sls files we just created. So go ahead and create the top.sls with this content:

base:
  'roles:cow':
    - match: grain
    - cow
  'roles:fortuneteller':
    - match: grain
    - fortune
  'roles:locomotive':
    - match: grain
    - locomotive

As you can see we can match grains when targeting our pillar. That's pretty cool. We use the roles grain to match the roles.

Be careful, because matching grains in the pillar comes with a caveat. When you use grains to match because they can be configured on the minion itself which means a minion can send a grain to get data it shouldn't get. In our case we're just declaring packages to install, so that's not so harmful.

Configuring the master

Next we'll reconfigure our salt master to read from our newly created pillar directory. For that we have to mount the pillar directory with docker compose and configure the master to read the pillar files from that directory. We begin by changing the master configuration.

We only need to add a new configuration variable called pillar_roots which is similar to the preexisting file_roots. It tells in what folder we keep the pillar files for a particular environment (in our case base).

We will mount the directory with the pillar data at /srv/pillar. Update the master.config file and add the pillar_root and point the base environment to /srv/pillar, so your master.config file should look like this:

file_roots:
  base:
    - /srv/salt/
pillar_roots:
  base:
    - /srv/pillar

Then we need to change our docker-compose.yml file to mount our new pillar directory at /srv/pillar.

version: "3"
services:
  salt:
    image: salt-master
    volumes:
      - ./master.config:/etc/salt/master
      - ./states:/srv/salt
      - ./pillar:/srv/pillar
  alpine-minion:
    image: alpine-minion
    volumes:
      - ./alpine.grains:/etc/salt/grains
  ubuntu-minion:
    image: ubuntu-minion
    volumes:
      - ./ubuntu.grains:/etc/salt/grains

That's it. That's all you need.

Do we have pillar data?

Let's have a look at the pillar data we've created. Again, we want to start with a clean slate so we stop our docker containers and remove them (press y when prompted if we're sure we want to remove them). Then we spin them back up again and attach to the bash shell of salt master:

docker-compose stop
docker-compose rm
docker-compose up -d
docker-compose exec salt bash

Once we're attached to the salt master, we accept the salt keys for the new minions (press enter to accept them when prompted):

salt-key -A

To have a look at the pillar data we just created type the following command

salt '*' pillar.items

You should now see the packages listed based on the roles of the machines. Something like this:

cow-e9e01577bbd8:
    ----------
    packages:
        ----------
        cowsay:
            A program with a cow
        fortune:
            A program for a fortune teller
locomotive-023378ead2d9:
    ----------
    packages:
        ----------
        fortune:
            A program for a fortune teller
        sl:
            Tech Model Railroad Club

We still aren't using this pillar data. To do that properly, we'll have to dive into the next part of this series: templating (I've teased it a little bit in this article though).

Little by little we'll get a better understanding of many of Salt's features.

This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.

Cover image by Yair Aronshtam: Salt pillar at the Dead Sea, Israel