DRYing out Terraform

HCL is very declaritive and, at heart I’m an imperative kind of guy (yes, yes I know its the 21st century), so it was always going to be a bit of a struggle to wrap my brain around its declarative style. But, I am also quite fond of not repeating (aka copy & pasting) great chunks of code and this is where HCL shows its limitations. In this blog post I’m going to describe one of the limitations in HCL and how I got round it.

The challange

I’m working on some Terraform templates that are intended to be sufficiently generic such that, with the right variable values, they can be used to provision different infrastructure in Azure or AWS. Consequently, when I wanted to define the template for creating different clusters of VMs I was intrigued to find out how DRY I could make my template. I know about the count feature, and the resource definitions for provisioning the VMs are held in a module template, but I wanted to pass in several properties so that I could have multiple VMs: one of, say, a medium size with a hostname prefix of ‘master’ and the other of, say, a large size with a prefix of ‘node’. In other words, a list of maps where each map contains the key/value pairs of the dynamic properties.

Trying with a list of maps

I tried code similar to this:

variable "host1" {
  type = "map"

  default = {
    size    = "Standard_A1"
    prefix = "master"
  }
}

variable "host2" {
  type = "map"

  default = {
    size    = "Standard_A2"
    prefixe = "node"
  }
}

resource "null_resource" "hosts" {
  triggers {
    list_of_hosts = ["${var.cluster1}, ${var.cluster2}"]
  }
}

But when I executed terraform plan Terraform said no, with the following error message:

There are warnings and/or errors related to your configuration. Please
fix these before continuing.

Errors:

  * At column 1, line 1: output of an HIL expression must be a string, or a single list (argument 1 is TypeMap) in:

${var.host1}, ${var.host2}

Although since Terraform v0.7, Maps and Lists are considered first class variable types this only applies one level deep. If you try anything fancy, like a list of maps or a map of lists you will quickly hit brick walls, especially if you want to use the various interpolation functions to manipulate them. There are various Open issues in Terraform’s github repo which describe some of these brick walls and possible work-arounds: here, here and here.

Using a string and some careful parsing

After some Googling I came across this blog post: https://serialseb.com/blog/2016/05/11/terraform-working-around-no-count-on-module/. Instead of using Maps and Lists the author defines all the values in a single string and uses a cunning combination of the split and element interpolation functions to extract the required values. I’m not going to reproduce the entire post here, but following the author’s example my single string would look like this:

variable clusters = ${default = "Standard_A1:master,Standard_A2,node"}

His solution certainly works but it is important to get the ordering of the values correct in the string definition and it isn’t going to be obvious what property values need to be put in the string and in what order.

Using a Map instead

Inspired by what I had learnt I wondered about using a Map to hold the properties instead of a string: with each key representing a dynamic property (such as VM size) and the value containing the list of property values in the form of a comma delimited string. The following code provides an example of this approach:

variable "instances" {
  type = "map"

  default = {
    # Ensure that each property contains the same number of items
    roles    = "master,node"
    sizes    = "Standard_A1,Standard_A2"
    prefixes = "master,node"
  }
}

resource "null_resource" "vms" {
  # count could use any of the properties to get its value
  count = "${length(split(",",var.instances["roles"]))}"

  triggers {
    role        = "${element(split(",", var.instances["roles"]), count.index)}"
    size        = "${element(split(",", var.instances["sizes"]), count.index)}"
    host_prefix = "${element(split(",", var.instances["prefixes"]), count.index)}"
  }
}

The instances variable holds the map of dynamic properties to be used in creating VM resources and the vms null_resource is responsible for creating a list where each element contains a role, size and host_prefix.
You can execute terraform apply against this template and you should see this result:

null_resource.vms.0: Creating...
  triggers.%:           "" => "3"
  triggers.host_prefix: "" => "master"
  triggers.role:        "" => "master"
  triggers.size:        "" => "Standard_A1"
null_resource.vms.1: Creating...
  triggers.%:           "" => "3"
  triggers.host_prefix: "" => "node"
  triggers.role:        "" => "node"
  triggers.size:        "" => "Standard_A2"
null_resource.vms.0: Creation complete
null_resource.vms.1: Creation complete

Apply complete! Resources: 2 added, 0 changed, 0 destroyed.

You can see that the null_resource.vms resource contains 2 elements with containing the 3 dynamic properties. These can then be referenced by other Terraform resources; here is an example:

resource "azurerm_virtual_machine" "host-vm" {
  name      = "host${count.index}"
  vm_size   = "${element(null_resource.vms.*.triggers.size, count.index)}"
  tags {
      role = ${element(null_resource.vms.*.triggers.role, count.index)}
  }
  count     = "${length(split(",",var.instances["roles"]))}"

  ....

}

One final limitation to be aware of, is how Terraform determines the value of count; it seems to do this before processing any of the null_resources. Thus, it is ok to use the value from a variable or the length of a list type variable but it won’t work if you try to use the length of a null_resource. For example, this will fail:
count = "${length(null_resource.vms)}"

Final thoughts

The point of this blog post is to demonstrate how some of the newer features in HCL and Terraform can be used to reduce the amount of boilerplate in .tf files and make them DRYer. The next challenge is to include a ‘VM count’ for each type of instance so that I can specify that I want 3 ‘master’ VMs and 5 ‘node’ VMs, but I’ll save that for a future article.

This blog is written exclusively by the OpenCredo team. We do not accept external contributions.