Open Credo

September 14, 2015 | DevOps, Hashicorp, Open Source

SaltStack – Using Consul as an External Pillar Source

Recently I was working on a project that was using SaltStack for configuration management and Consul for service discovery. It occurred to me that using Consul’s key/value store would be great place to store data needed for my Salt runs, but unfortunately Consul was not supported in SaltStack as an official data store at that point in time. Being an open source project however, this provided an excellent opportunity to contribute back and this blog post looks to provide some details on how this works, as well as a practical demo on how you can take advantage of Consul as an external data store.


Brett Mack

SaltStack – Using Consul as an External Pillar Source

The functionality is available via the v2015.8.0 release although it should be noted that the latest stable release (at time of writing) is 2015.5.5. Before getting into the low level details, let’s set some context and get an overview of these technologies, how they fit together, and the benefits they can provide! If you’d rather jump straight to the code, click here.


SaltStack is one of the 4 major configuration management tools available that also includes Puppet, Chef, and Ansible. Salt can be used in 2 ways: using salt-ssh to provide remote execution on one or many hosts much like Ansible, or in a master – minion set up much like Puppet where minions will connect to a master to get their instructions. Structurally, a Salt project is made up of one or more “Formulas” and a “Top File” which controls the formulas that are run on each host. Formulas are written in YAML making them remarkable easy to read, write and maintain. In addition to formulas, Salt also supports “Pillars”. Pillars are data structures which are also usually written in YAML and provide a data source during a Salt run. These can be used to modify the formulas which run on a host, make formulas more dynamic, or even edit values within config files on a per-host basis. When I said pillar data was usually written in YAML, this is because it’s the default, it doesn’t have to be. Salt supports external pillars, much like hiera as used by puppet, allowing you to get your pillar data from many different sources. Using Consul as an external pillar will form the basis of our demo.

Consul is one of many great open source tools by HashiCorp. Made up of various components, it provides several key features including: service discovery, health checking, and fundamentally, being able to operate as a consistent, highly available distributed key/kalue store. For this blog, it’s the last of these features we will concentrate on. Consul’s key/value store took inspiration from Etcd, to the point they used the etcd Jepson template to test they could provide a consistent, highly available key/value store even during network partitions. However unlike Etcd, Consul can run with one, or many, servers and multiple agent nodes each communicating using Serf’s gossip protocol. For the purpose of this demo we will be using a single server but as your infrastructure grows you can add more servers and run a very lightweight Consul agent on each of your nodes, eliminating single point of failures.

The Benefits

Using Consul as an external data store with Salt allows us to cleanly separate our data (which may change often), from our formulas which would normally be stored as a git repo. Another benefit of using Consul is that it allows in depth permission management on a per-folder basis. This means you could allow each member of a development team to be able to add gems or packages to a staging environment but still require one of DevOps/SysAdmin team to then add this to a production machine.

Basic Example

Let’s start to get something up and running. For this demo you will require both Vagrant and Git. First clone the repository located at, once that is done cd into the saltstack-consul-demo directory. Notice the Vagrantfile, this is going to instruct Vagrant to spin up 3 boxes- 1 Salt master and 2 Salt minions. During the provisioning of these boxes SaltStack will be installed using the salt-bootstrap script. We use this method to make sure we install the version we need that includes the Consul module as described in the overview. To start the provisioning type in:

vagrant up

This may take a little while as it downloads the base box and then provisions each of our boxes.

When the Salt minions start up, the first thing they do is try to connect to the master. By default they look for the master using the hostname ‘salt’ although this is configurable. In our Vagrantfile we’ve used the vagrant-hosts and vagrant-hostmanager plugins on each of the minions to modify their /etc/hosts file so they look at the salt-master when using the ‘salt’ address. For security reasons when the minions first connect, they offer up a key which the master must accept before it will send over any data. So you should connect to the master and accept these keys first. To connect to the master run the command:

vagrant ssh master

Now you’re connected and can see which minions have tried to connect by running:

sudo salt-key -L

You should see 3 keys in the unaccepted list. One for each of the minions, and one for the master. We will also use Salt to further configure the master box. To accept all the unaccepted keys run:

sudo salt-key -A

When prompted enter ‘y’ to accept. You’re now going to use Salt to actually set up Consul on our salt-master. You may have noticed that vagrant has been configured to mount the Salt folder in the git repo to /srv/salt on the master box, this is the directory that Salt checks for it’s scripts by default, and you will find the top file in here which instructs Salt to apply the Consul formula on the ‘salt-master’ box. The yaml for this formula can be found in /srv/salt/consul/init.sls. It’s all pretty self explanatory so check through it and when you’re happy run the following Salt command:

sudo salt ‘salt-master’ state.highstate

You should now have a single Consul instance installed with a web UI that you can connect to on

You should now go ahead and create 2 folders- salt-private and salt-shared. You can create folders using the Create Key form. Just ensure you follow the name of the key with ‘/’ and Consul will know you want to create a folder.

The salt-shared folder is expected to contain common pillar data that we want to send to every host. The salt-private folder will contain pillar data on a per-host basis. Let’s use the salt-shared folder to provide a list of packages that should be installed on every host. Click salt-shared on the left to enter the salt-shared folder. In here create a key called ‘pkgs’.

Make sure you separate each package with a new line. You will want to do this because although by default Consul simply gives us one value for each key, YAML supports multiple data types and the Consul pillar script is able to perform certain data type manipulations. Multiple lines will be returned as an array unless the first and last character of the data is ‘“‘, in this case the script will remove each ‘“‘ and return the text as a value. Any sub-folders will be returned as a hash (Technically this is a Dict as this is python, but despite writing the Consul pillar script I’m still primarily a Ruby developer so I’m going to say hash). Now that you have some actual pillar data, you need to tell Salt to use Consul as an external pillar source. Use your favourite editor to modify the Salt master config file (I use vim)

sudo vim /etc/salt/master

Add the following lines anywhere in this file

  consul.port: 8500

  - consul: consul_config root=salt-shared
  - consul: consul_config root=salt-private/%(minion_id)s

Restart the the salt master to pick up the new settings:

sudo service salt-master restart

Check that you can see the pillar data for each of the hosts by issuing the following command:

sudo salt ‘*’ pillar.items

The output should look something like this:

        - curl
        - vim
        - unzip
        - pv
        - httpie
        - curl
        - vim
        - unzip
        - pv
        - httpie
        - curl
        - vim
        - unzip
        - pv
        - httpie

This pillar data is now ready to be put to use with a new Salt formula. Create the file /srv/salt/pkgs.sls with the following content

{% for pkg in pillar.get('pkgs', []) %}
{{ pkg }}:
{% endfor %}

Salt uses the Jinja templating language and here we’re looping through the pkgs pillar array and saying each package should be installed. Edit /srv/salt/top.sls to tell Salt to run this formula on each host. The top.sls file should now look like this:

    - consul
    - pkgs

Apply this by running:

sudo salt ‘*’ state.highstate

Check the output from this command, you should see a comment for each package confirming it was either already installed on that node or has now been installed on that node along with other information such as the version that was installed. Now add some other packages that should only be installed on minion1. Go back to the Consul web UI and create a folder called ‘minion1’ within the ‘salt-private’ folder. In here create a ‘pkgs’ key with the value:


The Consul pillar script will merge the shared pkgs array with the new pkgs array for minion1. If you now run the previous command again you should see that the extra packages have now been installed on minion1, and only minion1.

Hierarchical Example

The previous section demonstrated how to make use of a  pretty simple structure, let’s try something more complex. Consul’s key value store is hierarchical, you can use this to create more complex formulas with multiple attributes. Let’s consider some data which may be used to represent a user. A user can require a username, a list of groups to belong to, and could even have a public key that should be added to the authorized_keys file to allow them to connect via ssh. You can model this using a hierarchy. Create a folder ‘users’ in the ‘salt-shared’ folder. Each sub-folder under here will represent 1 user, so go ahead and create a folder in here named ‘bob’. In this folder we will create 3 keys with the data below:


Robert Smith




ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCbLCq3Xk7+Qk69lOcMrsRC/FMOPr+zuRCiNhfQgktv/JR5bv5t2QUv2PrZ9hkGRIt7HMErWQXdP8o5Cehqp2rz1IU6zf8exw0HAshuyJCR0UqVMWJB2IRDj6zmKcc8jCjav8kyS0skI7YKG34FmCBq5cNef2v7UzrOT25OEpOQpe2Tm6PWt16Sh+CnefDU721AsjZl4zd0tQUpcDpACiRM8yLbPR/e88Qe2ZKltiHLCR6XYqnIv2rpO+UO/RKcAaOCDfV2w95brWlKIDEiOSAXflzuHVt40JwVElukgghwYhVhpgN1feW4i9sl60WThC0rIJHXav1XDeVRCwsZHb8v bob@hostname

To put this data to use create /srv/salt/users.sls with the following content:

{% for user, userinfo in pillar.get('users', {}).items() %}
{{ user }}:
    - fullname: {{ userinfo.get('fullname', '') }}
    - gid_from_name: True
    - optional_groups:
    {% for group in userinfo.get('groups', []) %}
      - {{ group }}
    {% endfor %}
  {% if userinfo.get('public_key', '') != '' %}
    - user: {{ user }}
    - name: {{ userinfo.get('public_key', '') }}
    - require:
      - user: {{ user }}
  {% endif %}
{% endfor %}

Add this to the top file for each host just as you did for the pkgs and run:

sudo salt ‘*’ state.highstate

You should now have the user ‘bob’ on every host. You can test this by typing exit to leave the salt-master box and using the provided private key to connect to minion1 via the command below

ssh -i ./bob_rsa


I hope this has demonstrated the value that Consul can add as an external pillar source. Using the web UI is an easy way to visualise and store your data, and with Consul’s hierarchical data structure along with Jinja templating you can create complex formulas in few lines that can change per-host depending on the data you provide.


This blog is written exclusively by the OpenCredo team. We do not accept external contributions.



Twitter LinkedIn Facebook Email