Mar 22, 2021

Iterations in Terraform Part 2: list and map expressions

List and Map expressions are powerful terraform constructs that can be used to create and transform data structures for better use in your resources and modules. In this post I will show you why, when, and how to use them in your infrastructure code. I will refrain from printing Terraform outputs. Each of the short snippets are simple enough and have sufficient explanation, but you can easily run most of them on your laptop as they are self-sufficient(with the exception of a few snippets that do have resources relying on variables and provider configuration).

List expressions

List expression is an expression that produces a new list based on the value of an existing list(or map). This concept should be familiar to those of you who knows Python because it is akin to list comprehensions. There is nothing special about expression itself, though their capacity for transforming collections allows to write very concise and flexible infrastructure code. We’ll get to this later, and now let’s look at couple of examples.

Very simple and straightforward, creates list of numbers 1 to 10:

output "the_list" {
  value = [ for num in range(10): num ]
}

Same as before but with a condition that keeps just the odd numbers:

output "the_odd_list" {
  value = [ for num in range(10): num if num % 2  == 1]
}

You can use any kind of expressions to form a list item:

locals {
  the_list = [1, 2, 3, 4, 5]
}

output "squares_list" {
  value = [ for num in local.the_list: num * num ]
}

output "another_squares_list" {
  value = [ for num in local.the_list: pow(num, 2) ]
}

You can create nested lists and then flatten them. Example below gets all possible combinations of elements from two lists. Please note that both number and letter are available in the innermost list expression:

locals {
  numbers = [1, 2, 3]
  letters = ["a", "b", "c"]

  all_combinations = flatten(
    [ for number in local.numbers:
      [ for letter in local.letters:
        {
          "number" = number
          "letter" = letter
        }
      ]
    ]
  )
}

output "result" {
  value = local.all_combinations
}

You are not limited to using variables as a source. Using a list of resources created with count is very common. Resources are referenced in locals block before it was even declared but in Terraform order of code blocks doesn’t matter:

locals {
  service_account_emails = [
    for sa in google_service_account.service_accounts: sa.email
  ]
)

resource "google_service_account" "service_accounts" {
  count        = 5
  account_id   = var.service_account_id
  display_name = var.service_account_display_name
}

output "service_account_emails" {
  value = local.service_account_emails
}

Last but not least you can declare two variables rather than one in the list expression. In that case first one will get an index of the source list element and the second will get the value:

locals {
  the_list          = [5, 4, 3, 2, 1]
  list_with_indexes = [ for index, value in local.the_list: "${index}-${value}" ]
}

output "list_with_indexes" {
  value = local.list_with_indexes
}

Using this syntax you can use maps as a source collection for the list expression. Your variables will obtain element key and value(instead of index and value as in previous example):

locals {
  the_map = {
    a = 1
    b = 2
  }
  list_from_map = [ for key, value in local.the_map: "${key} = ${value}" ]
}

output "the_new_list" {
  value = local.list_from_map
}

Now when we had a good look at the syntax and all sorts of list expression variations, it’s time to see some practical example.

Lets assume we have a team of individuals that are working on the same project but require individual sandboxes in the cloud to perform their daily activities. In GCP terms that would be projects. Each developer may request multiple Service Accounts in each of their projects. At least one of those Service Accounts will need a JSON key(for Terraform) created to allow developers to deploy resources within their sandboxes.

As you can see, the request is generic so we should be able to leverage same set of resources to deploy all sandboxes at once. Although it’s not a software development, DRY principle applies here just as much. At the same time the number of projects and service accounts in each of them is unknown so the code needs to be flexible, no hard-coded stuff. The cherry on the cake will be the optional flag in a Service Account config to signal that it needs a JSON authentication key generated.

Here’s the snippet to complete the task:

locals {
  # we start with a `projects` map, where key is project id and value is it's config
  projects = {
    project_1 = {
      # `service_accounts` is the list of service accounts to create
      # in a given project
      service_accounts = [
        {
          account_id = "iac-deployer"
          # `create_key` defines if we should create a key for the service account
          create_key = true
        },
        {
          account_id = "cloud-function"
        }
      ]
    }
    project_2 = {
      # `service_accounts` is the list of service accounts to create in a project
      service_accounts = [
        {
          account_id = "iac-deployer"
          create_key = true
        },
        {
          account_id = "testing-vm"
        },
        {
          account_id = "testing-cloudsql"
        }
      ]
    }
  }

  # we'll start with transforming the source map into more "consumable" form
  # one list for all service accounts that needs keys
  # the default `false` value provided in the `lookup` makes
  # the `create_key` flag optional
  sa_with_keys = flatten(
    [ for project_id, config in local.projects:
      [ for service_account in lookup(config, "service_accounts", []):
        {
          "project_id" = project_id
          "account_id" = lookup(service_account, "account_id")
        } if lookup(service_account, "create_key", false) == true
      ]
    ]
  )

  # and one list for all service accounts without keys
  sa_without_keys = flatten(
    [ for project_id, config in local.projects:
      [ for service_account in lookup(config, "service_accounts", []):
        {
          "project_id" = project_id
          "account_id" = lookup(service_account, "account_id")
        } if lookup(service_account, "create_key", false) == false
      ]
    ]
  )
}

# now we can proceed to service account creation
# not the best example of resource re-usability but we'll get another shot
# with map expressions
resource "google_service_account" "with_keys" {
  count        = length(local.sa_with_keys)
  project      = local.sa_with_keys[count.index].project_id
  account_id   = local.sa_with_keys[count.index].account_id
}

resource "google_service_account" "without_keys" {
  count        = length(local.sa_without_keys)
  project      = local.sa_without_keys[count.index].project_id
  account_id   = local.sa_without_keys[count.index].account_id
}

# and the keys
# because both keys and service accounts are based on the same list we can
# pull this trick and be sure that right service accounts will get keys
resource "google_service_account_key" "with_keys" {
  count              = length(local.sa_with_keys)
  service_account_id = google_service_account.with_keys[count.index].name
}

This was rather small snippet but it is sufficient to illustrate how the inputs can be composed to reflect the user requirements, and then transformed to fit the resource block constraints.

Map expressions

Map expressions are almost identical to list expressions with the only difference that they create a maps and have slightly different syntax. As I mentioned in Part 1, using maps is preferable way because resources created with from maps(for_each) are less fragile than lists(count). Maps expressions are similar to Python’s dictionary comprehensions. Here are couple of examples:

Create a map from the list:

output "the_list" {
  value = { for index, num in range(10): index => num }
}

Previous example was not particularly useful due to the fact it use indexes as keys. Though sometimes we do want to convert list to map to leverage for_each versatility. In such cases finding a way to create a unique key without relying on list item index is paramount:

locals {
  the_list = [
    {
      project_id = "project_1"
      account_id = "account_1"
    },
    {
      project_id = "project_2"
      account_id = "account_2"
    },
  ]
}
output "the_map" {
  value = {
    for elem in local.the_list: "${elem.project_id}-${elem.account_id}" => elem
  }
}

Your map expressions can have conditional expressions to filter unwanted items. You can check either key or value(including some of it’s sub-elements):

locals {
  the_map = {
    el1 = {
      name = "John"
      team = "Dev"
    }
    el2 = {
      name = "Jane"
      team = "Ops"
    }
    el3 = {
      name = "Jim"
      team = "Ops"
    }
  }
}

output "filtered_map" {
  value = { for key, value in local.the_map: key => value if value.team == "Ops" }
}

All previous examples transformed lists to maps. Of course maps can be created from other maps. Following example is a bit superficial and brittle, though very clear. It inverts keys and values:

locals {
  the_map = {
    a = "z"
    b = "y"
    c = "x"
  }
}

output "the_new_map" {
  value = { for key, value in local.the_map: value => key }
}

Now for a more practical example let’s look again at the Service Account creation use case and try to improve it. We want our code to be DRY but unfortunately first attempt had some duplicated resources which doesn’t look nice. In addition to that Service Accounts and their keys are one of those resource which suffers a lot from the count flaw(deletion of the item in the middle of the list). We definitely need a map here.

The snippet below use a combination of list and map expressions to get a list of objects with all information about each individual service account to create and then merges it into a single map. Each map item must have a unique key. In this case it is "${project_id}/${service_account.account_id}" and its sole purpose is to provide this uniqueness. Although it is not consumed in the service_account resource it will be present in the Terraform state Resource ID thus will come handy when referencing service_account_id in the service_account_key resource. Please also note that the merge function of terraform accepts multiple maps so we have to use ellipsis operator(...) to unpack a list as a set of individual values:

locals {
  projects = {
    project_1 = {
      service_accounts = [
        {
          account_id = "terraform"
          create_key = true
        },
        {
          account_id = "cloud-function"
        }
      ]
    }
  }

  service_accounts = merge(
    [ for project_id, config in local.projects:
      { for service_account in lookup(config, "service_accounts", []) : "${project_id}/${service_account.account_id}" => {
          "project_id" = project_id
          "account_id" = lookup(service_account, "account_id")
          "create_key" = lookup(service_account, "create_key", false)
        }
      }
    ]...
  )
}

# so now we can create all service accounts at once
resource "google_service_account" "default" {
  for_each   = local.service_accounts
  project    = each.value.project_id
  account_id = each.value.account_id
}


# an alternative of creating multiple local variables would be a simple filter
# expression within the resource block
resource "google_service_account_key" "default" {
  for_each           = { for key, value in local.service_accounts: key => value if value.create_key == true }
  service_account_id = google_service_account.default[each.key].name
}

Conclusion

List and map expressions are indispensable tools in the toolbox of an infrastructure engineer writing IaC with Terraform. Imagining how to compose the data in a coherent way and then transforming it to fit the resource or module interface is essential for writing concise, flexible and repeatable infrastructure code. I hope this material will help you to feel comfortable working with list and map expressions and creating resources in bulk. Good luck!