Terraform variable definitions in YAML

In my previous posts dedicated to Terraform I have described various ways of creating resources in bulk. Designing objects composed of basic data types such as string, number, bool, list, and map allows you to decouple “the logic” from the configuration. As a consequence of extensively applying this approach you can end up having a huge configuration files that are hard to read and navigate through. Naturally one starts wondering if there’s a way to break it into manageable chunks. For some time I’ve been doing this by splitting big configs into smaller files and then using a bash “glue code” to concat them before plan/apply. It did the trick very well but looked hacky. Every time I had to on-board another person I had to go though a lengthy explanation of what’s going on there and why. Recently I came across one Terraform Module from the Google Cloud Fabric repo. It’s author was able to implement the same idea in a much cleaner way. In this post I’ll share with you what is it about and how you can use it.

Overview

As you might have guessed from the title, it has something to do with YAML. Although you could use JSON or even HCL configs just as well, YAML is considered to be better suited for the human reader. Besides due to it’s wide use in many CI/CD systems and Kubernetes more people are familiar with it. Therefore the ultimate goal of writing IaC that can be easily read/updated by people who don’t have extensive experience using Terraform becomes easily achievable.

The approach is based on usage of 3 built-in terraform functions:

• fileset will return a set of filenames in the given path based on a given pattern.
• file will return a content of a file on a given path as a string.
• yamldecode will parse a string and return a HCL compliant value(typically a map/object).

Combining them will convert a list of YAML files into a list of objects. The cherry on top of this cake is the ellipsis symbol ... which will expand a list of objects into separate arguments if a merge function call te produce the ultimate config, ready to be ingested by the for_each meta-argument of any terraform resource/module.

Example

Consider a scenario when you want to group your firewall rules based on the team name they are created for. Such grouping would keep your IaC repo nice and tidy and would simplify any change review process as it would be easier to see at a glance what file you are changing and what teams would be affected by it:

$ tree ./firewall_configs

./firewall_configs
├── team_a
│   ├── app.yaml
│   └── db.yaml
└── team_b
    ├── backend.yaml
    └── frontend.yaml

Each of those files represents a map of firewall_name => firewall configuration, something like this:

fwl_allow_https_ingres:
  allow:
    - protocol: "tcp"
      ports:
        - "443"
  target_tags:
    - "https_server"
  source_ranges:
    - "0.0.0.0/0"

The transformation chain could look like this:

locals {
  # you start with pointing to where all of your firewall configs are stored
  root_path        = "./firewall_configs"
  # then you obtain all individual config paths in all subfolders
  config_list      = fileset(local.root_path, "**/*.yaml")
  # next you read those files and get their string content
  yaml_string_list = [ for name in local.config_list: file("${local.root_path}/${name}") ]
  # then you perform the yaml decoding sieving empty values(files with no content or commented content)
  firewall_configs = [ for yaml_string in local.yaml_string_list: yamldecode(yaml_string) if length(yaml_string) > 0 ]
  # and finally merging all configs into a single map
  final_config     = merge(local.firewall_configs...)
}

I will leave the consumption of such config map out of the scope of this article as it was covered in slightest details in my previous “Iterations in Terraform” series(1, 2). The only additional important aspect to keep in mind is that your firewall rule names must follow some sort of naming convention that ensure their uniqueness. The way merge works will consume duplicates without warnings with later key/value pair overwriting earlier ones. If you want to check out the concrete example with Firewall Rules please check out the Google Cloud VPC Firewall - Yaml Module in Google’s Cloud Foundation Fabric.