Problem

Extracting values from a Kubernetes secret is quite easy but requires quite a bit of typing. Here’s one of the ways you can do it:

kubectl get secret someapp-credentials -o jsonpath='{.data.username}' | base64 -d
kubectl get secret someapp-credentials -o jsonpath='{.data.password}' | base64 -d

Not so bad. Though for a person like me, who prefers to have k as an alias to kubectl because it’s just too much to type, doing this task regularly can become annoying pretty fast.

This is actually something I used to do a lot. But one day the level of frustration when typing all these commands reached the threshold so I sit to come up with a simple script that could become my “Swiss Army Knife” when it comes to working with Kubernetes secrets.

Solution

The core idea here is to support two use cases:

1. Print the decoded value of a single key in a secret

2. Print the entire data section of a secret with all values being decoded

Ideally it should support both getting secrets from the namespace of the current context and any other namespace in a very concise manner(typing -n namespace every time is just too much trouble). Below is the script itself. It utilizes a bit of bash, basic linux commands, and jq.

#!/bin/bash

if [ -z "$1" ]; then
  echo "Usage: $0 [<namespace>/]<secret> [<key>]"
  exit 1
fi

namespace_or_secret=$(echo $1 | awk -F '/' '{print $1}')
secret=$(echo $1 | awk -F '/' '{print $2}')

result=""
if [ -z "$secret" ]; then
  secret=$namespace_or_secret
  result=$(kubectl get secret $secret -o jsonpath='{.data}' | jq 'map_values(@base64d)')
else
  namespace=$namespace_or_secret
  result=$(kubectl -n $namespace get secret $secret -o jsonpath='{.data}' | jq 'map_values(@base64d)')
fi

if [ -z "$2" ]; then
  echo $result | jq '.'
else
  echo $result | jq -r --arg k "$2" '.[$k]'
fi

Now I can get a decrypted content of any secret I need in any namespace with one very simple command(assuming you put it in /usr/local/bin/ksecret or $HOME/.local/bin/ksecret and grant chmod +x):

ksecret kube-system/default-token-bf4vd

the output looks like this:

{
  "ca.crt": "-----BEGIN CERTIFICATE-----\nMIIC/jCCAeagAwIBAgIBADANBgkqhkiG9...",
  "namespace": "kube-system",
  "token": "eyJhbGciOiJSUzx1NiIsImtpZCI6ImRvTHJHVHZqWjNhU05pYlpDYl84cUlscTN..."
}

or if I need just the token:

ksecret kube-system/default-token-bf4vd token

the output will be raw string ready for copy/paste:

eyJhbGciOiJSUzI1NiI...

It also supports getting keys with characters reserved by jq like .:

ksecret kube-system/default-token-bf4vd ca.crt

the output will be a properly formatted certificate:

-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----

Overview

As you might have guessed from the title, it has something to do with YAML. Although you could use JSON or even HCL configs just as well, YAML is considered to be better suited for the human reader. Besides due to it’s wide use in many CI/CD systems and Kubernetes more people are familiar with it. Therefore the ultimate goal of writing IaC that can be easily read/updated by people who don’t have extensive experience using Terraform becomes easily achievable.

The approach is based on usage of 3 built-in terraform functions:

• fileset will return a set of filenames in the given path based on a given pattern.
• file will return a content of a file on a given path as a string.
• yamldecode will parse a string and return a HCL compliant value(typically a map/object).

Combining them will convert a list of YAML files into a list of objects. The cherry on top of this cake is the ellipsis symbol ... which will expand a list of objects into separate arguments if a merge function call te produce the ultimate config, ready to be ingested by the for_each meta-argument of any terraform resource/module.

Example

Consider a scenario when you want to group your firewall rules based on the team name they are created for. Such grouping would keep your IaC repo nice and tidy and would simplify any change review process as it would be easier to see at a glance what file you are changing and what teams would be affected by it:

$ tree ./firewall_configs

./firewall_configs
├── team_a
│   ├── app.yaml
│   └── db.yaml
└── team_b
    ├── backend.yaml
    └── frontend.yaml

Each of those files represents a map of firewall_name => firewall configuration, something like this:

fwl_allow_https_ingres:
  allow:
    - protocol: "tcp"
      ports:
        - "443"
  target_tags:
    - "https_server"
  source_ranges:
    - "0.0.0.0/0"

The transformation chain could look like this:

locals {
  # you start with pointing to where all of your firewall configs are stored
  root_path        = "./firewall_configs"
  # then you obtain all individual config paths in all subfolders
  config_list      = fileset(local.root_path, "**/*.yaml")
  # next you read those files and get their string content
  yaml_string_list = [ for name in local.config_list: file("${local.root_path}/${name}") ]
  # then you perform the yaml decoding sieving empty values(files with no content or commented content)
  firewall_configs = [ for yaml_string in local.yaml_string_list: yamldecode(yaml_string) if length(yaml_string) > 0 ]
  # and finally merging all configs into a single map
  final_config     = merge(local.firewall_configs...)
}

I will leave the consumption of such config map out of the scope of this article as it was covered in slightest details in my previous “Iterations in Terraform” series(1, 2). The only additional important aspect to keep in mind is that your firewall rule names must follow some sort of naming convention that ensure their uniqueness. The way merge works will consume duplicates without warnings with later key/value pair overwriting earlier ones. If you want to check out the concrete example with Firewall Rules please check out the Google Cloud VPC Firewall - Yaml Module in Google’s Cloud Foundation Fabric.

List expressions

List expression is an expression that produces a new list based on the value of an existing list(or map). This concept should be familiar to those of you who knows Python because it is akin to list comprehensions. There is nothing special about expression itself, though their capacity for transforming collections allows to write very concise and flexible infrastructure code. We’ll get to this later, and now let’s look at couple of examples.

Very simple and straightforward, creates list of numbers 1 to 10:

output "the_list" {
  value = [ for num in range(10): num ]
}

Same as before but with a condition that keeps just the odd numbers:

output "the_odd_list" {
  value = [ for num in range(10): num if num % 2  == 1]
}

You can use any kind of expressions to form a list item:

locals {
  the_list = [1, 2, 3, 4, 5]
}

output "squares_list" {
  value = [ for num in local.the_list: num * num ]
}

output "another_squares_list" {
  value = [ for num in local.the_list: pow(num, 2) ]
}

You can create nested lists and then flatten them. Example below gets all possible combinations of elements from two lists. Please note that both number and letter are available in the innermost list expression:

locals {
  numbers = [1, 2, 3]
  letters = ["a", "b", "c"]

  all_combinations = flatten(
    [ for number in local.numbers:
      [ for letter in local.letters:
        {
          "number" = number
          "letter" = letter
        }
      ]
    ]
  )
}

output "result" {
  value = local.all_combinations
}

You are not limited to using variables as a source. Using a list of resources created with count is very common. Resources are referenced in locals block before it was even declared but in Terraform order of code blocks doesn’t matter:

locals {
  service_account_emails = [
    for sa in google_service_account.service_accounts: sa.email
  ]
)

resource "google_service_account" "service_accounts" {
  count        = 5
  account_id   = var.service_account_id
  display_name = var.service_account_display_name
}

output "service_account_emails" {
  value = local.service_account_emails
}

Last but not least you can declare two variables rather than one in the list expression. In that case first one will get an index of the source list element and the second will get the value:

locals {
  the_list          = [5, 4, 3, 2, 1]
  list_with_indexes = [ for index, value in local.the_list: "${index}-${value}" ]
}

output "list_with_indexes" {
  value = local.list_with_indexes
}

Using this syntax you can use maps as a source collection for the list expression. Your variables will obtain element key and value(instead of index and value as in previous example):

locals {
  the_map = {
    a = 1
    b = 2
  }
  list_from_map = [ for key, value in local.the_map: "${key} = ${value}" ]
}

output "the_new_list" {
  value = local.list_from_map
}

Now when we had a good look at the syntax and all sorts of list expression variations, it’s time to see some practical example.

Lets assume we have a team of individuals that are working on the same project but require individual sandboxes in the cloud to perform their daily activities. In GCP terms that would be projects. Each developer may request multiple Service Accounts in each of their projects. At least one of those Service Accounts will need a JSON key(for Terraform) created to allow developers to deploy resources within their sandboxes.

As you can see, the request is generic so we should be able to leverage same set of resources to deploy all sandboxes at once. Although it’s not a software development, DRY principle applies here just as much. At the same time the number of projects and service accounts in each of them is unknown so the code needs to be flexible, no hard-coded stuff. The cherry on the cake will be the optional flag in a Service Account config to signal that it needs a JSON authentication key generated.

Here’s the snippet to complete the task:

locals {
  # we start with a `projects` map, where key is project id and value is it's config
  projects = {
    project_1 = {
      # `service_accounts` is the list of service accounts to create
      # in a given project
      service_accounts = [
        {
          account_id = "iac-deployer"
          # `create_key` defines if we should create a key for the service account
          create_key = true
        },
        {
          account_id = "cloud-function"
        }
      ]
    }
    project_2 = {
      # `service_accounts` is the list of service accounts to create in a project
      service_accounts = [
        {
          account_id = "iac-deployer"
          create_key = true
        },
        {
          account_id = "testing-vm"
        },
        {
          account_id = "testing-cloudsql"
        }
      ]
    }
  }

  # we'll start with transforming the source map into more "consumable" form
  # one list for all service accounts that needs keys
  # the default `false` value provided in the `lookup` makes
  # the `create_key` flag optional
  sa_with_keys = flatten(
    [ for project_id, config in local.projects:
      [ for service_account in lookup(config, "service_accounts", []):
        {
          "project_id" = project_id
          "account_id" = lookup(service_account, "account_id")
        } if lookup(service_account, "create_key", false) == true
      ]
    ]
  )

  # and one list for all service accounts without keys
  sa_without_keys = flatten(
    [ for project_id, config in local.projects:
      [ for service_account in lookup(config, "service_accounts", []):
        {
          "project_id" = project_id
          "account_id" = lookup(service_account, "account_id")
        } if lookup(service_account, "create_key", false) == false
      ]
    ]
  )
}

# now we can proceed to service account creation
# not the best example of resource re-usability but we'll get another shot
# with map expressions
resource "google_service_account" "with_keys" {
  count        = length(local.sa_with_keys)
  project      = local.sa_with_keys[count.index].project_id
  account_id   = local.sa_with_keys[count.index].account_id
}

resource "google_service_account" "without_keys" {
  count        = length(local.sa_without_keys)
  project      = local.sa_without_keys[count.index].project_id
  account_id   = local.sa_without_keys[count.index].account_id
}

# and the keys
# because both keys and service accounts are based on the same list we can
# pull this trick and be sure that right service accounts will get keys
resource "google_service_account_key" "with_keys" {
  count              = length(local.sa_with_keys)
  service_account_id = google_service_account.with_keys[count.index].name
}

This was rather small snippet but it is sufficient to illustrate how the inputs can be composed to reflect the user requirements, and then transformed to fit the resource block constraints.

Map expressions

Map expressions are almost identical to list expressions with the only difference that they create a maps and have slightly different syntax. As I mentioned in Part 1, using maps is preferable way because resources created with from maps(for_each) are less fragile than lists(count). Maps expressions are similar to Python’s dictionary comprehensions. Here are couple of examples:

Create a map from the list:

output "the_list" {
  value = { for index, num in range(10): index => num }
}

Previous example was not particularly useful due to the fact it use indexes as keys. Though sometimes we do want to convert list to map to leverage for_each versatility. In such cases finding a way to create a unique key without relying on list item index is paramount:

locals {
  the_list = [
    {
      project_id = "project_1"
      account_id = "account_1"
    },
    {
      project_id = "project_2"
      account_id = "account_2"
    },
  ]
}
output "the_map" {
  value = {
    for elem in local.the_list: "${elem.project_id}-${elem.account_id}" => elem
  }
}

Your map expressions can have conditional expressions to filter unwanted items. You can check either key or value(including some of it’s sub-elements):

locals {
  the_map = {
    el1 = {
      name = "John"
      team = "Dev"
    }
    el2 = {
      name = "Jane"
      team = "Ops"
    }
    el3 = {
      name = "Jim"
      team = "Ops"
    }
  }
}

output "filtered_map" {
  value = { for key, value in local.the_map: key => value if value.team == "Ops" }
}

All previous examples transformed lists to maps. Of course maps can be created from other maps. Following example is a bit superficial and brittle, though very clear. It inverts keys and values:

locals {
  the_map = {
    a = "z"
    b = "y"
    c = "x"
  }
}

output "the_new_map" {
  value = { for key, value in local.the_map: value => key }
}

Now for a more practical example let’s look again at the Service Account creation use case and try to improve it. We want our code to be DRY but unfortunately first attempt had some duplicated resources which doesn’t look nice. In addition to that Service Accounts and their keys are one of those resource which suffers a lot from the count flaw(deletion of the item in the middle of the list). We definitely need a map here.

The snippet below use a combination of list and map expressions to get a list of objects with all information about each individual service account to create and then merges it into a single map. Each map item must have a unique key. In this case it is "${project_id}/${service_account.account_id}" and its sole purpose is to provide this uniqueness. Although it is not consumed in the service_account resource it will be present in the Terraform state Resource ID thus will come handy when referencing service_account_id in the service_account_key resource. Please also note that the merge function of terraform accepts multiple maps so we have to use ellipsis operator(...) to unpack a list as a set of individual values:

locals {
  projects = {
    project_1 = {
      service_accounts = [
        {
          account_id = "terraform"
          create_key = true
        },
        {
          account_id = "cloud-function"
        }
      ]
    }
  }

  service_accounts = merge(
    [ for project_id, config in local.projects:
      { for service_account in lookup(config, "service_accounts", []) : "${project_id}/${service_account.account_id}" => {
          "project_id" = project_id
          "account_id" = lookup(service_account, "account_id")
          "create_key" = lookup(service_account, "create_key", false)
        }
      }
    ]...
  )
}

# so now we can create all service accounts at once
resource "google_service_account" "default" {
  for_each   = local.service_accounts
  project    = each.value.project_id
  account_id = each.value.account_id
}


# an alternative of creating multiple local variables would be a simple filter
# expression within the resource block
resource "google_service_account_key" "default" {
  for_each           = { for key, value in local.service_accounts: key => value if value.create_key == true }
  service_account_id = google_service_account.default[each.key].name
}

Conclusion

List and map expressions are indispensable tools in the toolbox of an infrastructure engineer writing IaC with Terraform. Imagining how to compose the data in a coherent way and then transforming it to fit the resource or module interface is essential for writing concise, flexible and repeatable infrastructure code. I hope this material will help you to feel comfortable working with list and map expressions and creating resources in bulk. Good luck!

Theory

Let’s start with a reviewing some theoretic review. Meta-arguments are the kind of arguments that can be used in all resources and modules regardless of what they do. There are five meta-arguments at the moment with count and for_each being the most often used.

count

The oldest way to create resources iteratively is to use meta-argument count. It takes a number and creates that exact number of copies of the resources. On each iteration it exposes an object count with a single argument index inside of the resource block. By accessing count.index you can get an index number of current iteration. You would use this index to somehow ensure that resource arguments which have to be unique - are unique. Some examples of that would be using count.index interpolation in your strings or as a way to pick an item from some list.

There are multiple ways you can use count in your code:

1. The most primitive one is to create identical resources with slightly different names:

   resource "google_service_account" "service_accounts" {
     count        = length(3)
     account_id   = "${var.account_id}-${count.index}"
     display_name = var.display_name
   }

2. More useful scenario is to reference an item in the list of config objects. This way each of the resources you create can have fully customizable config. Please also note that using length function acts as a toggle in a way. By providing an empty list you disable resource creation.

   variable "service_accounts" {
     type    = list(map(string))
     default = [
       {account_id = "one-sa", display_name = "Custom SA"},
       {account_id = "two-sa", display_name = "Yet another SA"},
     ]
   }
   
   resource "google_service_account" "service_accounts" {
     count        = length(var.service_accounts)
     account_id   = var.service_accounts[count.index].account_id
     display_name = var.service_accounts[count.index].display_name
   }

3. Sometimes you are in a situations when you need to create resource only if certain conditions are met(eg certain parameter is set to true). So it can act as a toggle if combined with conditional expression:

   resource "google_service_account" "service_account" {
     count        = var.create_service_account == true ? 1 : 0
     account_id   = var.service_account_id
     display_name = var.service_account_display_name
   }

4. You can also combine toggle feature with creation from the list of configs:

   resource "google_service_account" "service_accounts" {
     count        = var.create_service_accounts == true ? length(var.service_accounts) : 0
     account_id   = var.service_accounts[count.index].account_id
     display_name = var.service_accounts[count.index].display_name
   }

As you can see it is very easy to use and quite powerful too. Unfortunately there is one particularly nasty side effect. Resources created using count are referenced in the state by their list index. If you delete a single item in the middle of the underlying list, all the items to the right of it will have their index recalculated, causing recreation of all those resources. Nonetheless it still can be used in scenarios when underlying list is not supposed to change frequently or items will be deleted only from the end of the list(scaling up/down).

If you want to reference an argument of a resource created with count in other resources/outputs, you will have to use a list index or splat expression.

One item:

resource "google_service_account" "service_accounts" {
  count        = length(3)
  account_id   = "${var.account_id}-${count.index}"
  display_name = var.display_name
}

output "service_account_email" {
  value = google_service_account.service_accounts[0].email
}

All items:

# ...
output "service_account_emails" {
  value = google_service_account.service_accounts[*].email
}

for_each

This meta-argument is relatively new. Originally for_each was introduced in Terraform 0.12.0 as a way of creating “dynamic” resource blocks (blocks you can use within the same resource multiple times) but since Terraform 0.12.6 it’s usage was extended as a count alternative for resource creation. On each iteration it exposes each object with two arguments - key and value, which will have map element key and value correspondingly. Although map is the main resource type to use with for_each you can also use it with sets (or lists converted to sets with toset function).

When used with maps, for_each is not a subject to the side effect that count has. Due to the fact that every item in the map has a unique key and resources created with for_each are referenced in the state by the key name of a corresponding map element, deleting an item from the map will not affect other map elements resulting in more predictable “plan”.

The main use case scenario is simple creation of multiple resources, but there is also a way you can implement a toggle feature.

1. Creating multiple resources with fully customizable configs. Please also note that using empty map works as toggle by itself just like the empty list did in count = length(....

   variable "service_accounts" {
     type    = map(map(string))
     default = {
       "one-sa" = {
         display_name = "Custom SA"
         description  = "A custom SA"
         },
       "two-sa" = {
         display_name = "Yet another SA"
         description  = "A custom SA"
       },
     }
   }
   
   resource "google_service_account" "service_account" {
     for_each     = var.service_accounts
     account_id   = each.key
     display_name = each.value.display_name
     description  = each.value.description
   }

2. Create single resources with a toggle. Looks ugly and does exactly the same thing as count toggle. I would not recommend using it but for the sake of completeness:

   resource "google_service_account" "service_accounts" {
     for_each     = var.create_service_accounts == true ? map(var.account_id, var.display_name) : {}
     account_id   = each.key
     display_name = each.value
   }

3. Create multiple resources with a toggle:

   resource "google_service_account" "service_accounts" {
     for_each     = var.create_service_accounts == true ? var.service_accounts : {}
     account_id   = each.key
     display_name = each.value.display_name
     description  = each.value.description
   }

When it comes to referencing resource values in the outputs, If you want to reference an argument of a resource created with for_each, you would need to address each resource by the key it was created with:

output "service_account_email" {
  value = google_service_account.service_accounts["one-sa"].email
}

Splat expressions will not work with maps but you can use list and map expressions to achieve the same result:

output "service_account_email" {
  value = [
    for k, v in google_service_account.service_accounts: v.email
  ]
}

which would produce following output:

service_account_email = [
  "one-sa@.....",
  "two-sa@....."
]

or

output "service_account_email" {
  value = {
    for k, v in google_service_account.service_accounts:
      k => v.email
  }
}

which would produce following output:

service_account_email = {
  one-sa = "one-sa@.....",
  two-sa = "two-sa@....."
}

There is one thing you should be aware of when using for_each. It will not be able to calculate the plan if the keys of the underlying map are composed from the outputs of any other resource. Consider the following snippet:

locals {
  numbers = ["one", "two", "three"]
  the_map = {
    for number in local.numbers:
      "${number}-${random_string.random.result}" => "some text"
  }
}

resource "random_string" "random" {
  length           = 2
  special          = false
}

resource "local_file" "foo" {
  for_each = local.the_map
  content  = each.value
  filename = each.key
}

Even though it is clear to the human reader that the number of elements in the map does not depend on the result of random_string resource, terraform will not be able to create the plan. So try to avoid such scenarios.

Error: Invalid for_each argument

  on main.tf line 13, in resource "local_file" "foo":
  13:     for_each = local.the_map

The "for_each" value depends on resource attributes that cannot be determined
until apply, so Terraform cannot predict how many instances will be created.
To work around this, use the -target argument to first apply only the
resources that the for_each depends on.

Practice

Now as a practical example I will assemble some primitive certificate creation tool. It is not production-grade but might come handy for experiments in the home lab. The code will allow to either generate the CA or import the external one. I will be using TLS and Local providers with most of the aforementioned ways of using count and for_each.

Let’s start with the variable declaration. I’m going to use two composite variables, one for CA configuration and one for all leaf certificates.

cat variables.tf

variable "root_ca" {
  description = "Root CA Specification"
  type = object({
    algorithm     = string
    rsa_bits      = number
    ecdsa_curve   = string
    external_cert = string
    external_key  = string
    subject       = map(string)
  })
  default = null
}

variable "certificates" {
  description = "Leaf Certs Specification"
  default     = {}
}

You have probably noticed one significant difference between the root_ca variable and the certificates. First one has an object type declared while second has none. While certificates is a map of objects, keeping its type unset allows me to have “optional parameters” in the config. Omitting such parameters will not be interpreted by terraform as an error and will make the tfvars file significantly shorter in some cases. Terraform is smart enough to deduce variable type automatically during the plan/apply. If I were using strictly defined type I would have to set all key/value pairs declared in the object type. Even empty ones would have to be explicitly set to null.

Next I’ll put some values to have a better perspective on the possible way of consuming them:

cat terraform.tfvars

root_ca = {
  algorithm     = "RSA"
  rsa_bits      = 4096
  ecdsa_curve   = null
  external_cert = null
  external_key  = null
  subject = {
    CN = "Root CA"
    O  = "Example Org"
  }
}
certificates = {
  "example.com" = {
    algorithm = "RSA"
    rsa_bits  = 2048
    subject = {
      CN = "Example Leaf Cert"
      O  = "Example Org"
    }
  }
  "website.com" = {
    algorithm = "ECDSA"
    subject = {
      CN = "Some Website"
      O  = "Example Org"
    }
  }
  "anotherone.com" = {
    subject = {
      CN = "Leaf Cert"
      O  = "Example Org"
    }
  }
}

As you can see, the keys of certificates variable are domain names which make them unique to a certain degree. The “website.com” value has only algorithm set and “anotherone.com” omits even that. In both cases missing values will be calculated automatically according to the sane defaults I will put in my main.tf. At the same time omitting root_ca.ecdsa_curve would be an error.

Finally the heart of this tool, the resources.

cat main.tf

# the locals block is prefect for reusable expressions, it has a toggle which
# defines if CA cert and key must be generated or imported from file
locals {
  use_external_cert = lookup(var.root_ca, "external_key", null) != null && lookup(var.root_ca, "external_cert", null) != null
}
# next two data sources import the external cert and key file only it toggle
# is true
data "local_file" "ca-key" {
  count    = local.use_external_cert == true ? 1 : 0
  filename = var.root_ca.external_key
}

data "local_file" "ca-cert" {
  count    = local.use_external_cert == true ? 1 : 0
  filename = var.root_ca.external_cert
}

# Root CA Config #
# same logic goes into the self-signed CA config, but the toggle is reversed
resource "tls_private_key" "root" {
  count       = local.use_external_cert == true ? 0 : 1
  algorithm   = var.root_ca.algorithm
  rsa_bits    = var.root_ca.algorithm == "RSA" ? var.root_ca.rsa_bits : null
  ecdsa_curve = var.root_ca.algorithm == "ECDSA" ? var.root_ca.ecdsa_curve : null
}

# notice that in subject block we cant reference CN and O as we did with with
# algorithm so usilg `lookup` function is desirable
resource "tls_self_signed_cert" "root" {
  count           = local.use_external_cert == true ? 0 : 1
  key_algorithm   = tls_private_key.root[0].algorithm
  private_key_pem = tls_private_key.root[0].private_key_pem

  subject {
    common_name  = lookup(var.root_ca.subject, "CN")
    organization = lookup(var.root_ca.subject, "O")
  }

  validity_period_hours = 87600 # 10 years

  allowed_uses = [
    "crl_signing",
    "cert_signing"
  ]
  is_ca_certificate = true
}

# Leaf Certs Config #
# leaf config is not as strict as root ca, because there was no explicit type
# declaration. Again the `lookup` function saves the day providing defaults for
# all omitted values
resource "tls_private_key" "leaf" {
  for_each    = var.certificates
  algorithm   = lookup(each.value, "algorithm", "RSA")
  rsa_bits    = lookup(each.value, "algorithm", "RSA") == "RSA" ? lookup(each.value, "rsa_bits", 2048) : null
  ecdsa_curve = lookup(each.value, "algorithm", "RSA") == "ECDSA" ? lookup(each.value, "ecdsa_curve", "P224") : null
}

resource "tls_cert_request" "leaf" {
  for_each        = var.certificates
  key_algorithm   = tls_private_key.leaf[each.key].algorithm
  private_key_pem = tls_private_key.leaf[each.key].private_key_pem

  subject {
    common_name  = lookup(each.value.subject, "CN")
    organization = lookup(each.value.subject, "O")
  }

  dns_names = [each.key]
}

# another resource with local toggle use, this time to either get
# the generated CA or the external one
resource "tls_locally_signed_cert" "leaf" {
  for_each           = var.certificates
  cert_request_pem   = tls_cert_request.leaf[each.key].cert_request_pem
  ca_key_algorithm   = var.root_ca.algorithm
  ca_private_key_pem = local.use_external_cert == true ? data.local_file.ca-key[0].content : tls_private_key.root[0].private_key_pem
  ca_cert_pem        = local.use_external_cert == true ? data.local_file.ca-cert[0].content : tls_self_signed_cert.root[0].cert_pem

  validity_period_hours = 8760 # 1 year

  allowed_uses = [
    "key_encipherment",
    "digital_signature",
    "server_auth",
  ]
}

# File Outputs #
# and finally file outputs
# nothing new here, same tricks
resource "local_file" "ca-cert" {
  count    = local.use_external_cert == true ? 0 : 1
  content  = tls_self_signed_cert.root[0].cert_pem
  filename = "ca.pem"
}

resource "local_file" "ca-key" {
  count    = local.use_external_cert == true ? 0 : 1
  content  = tls_private_key.root[0].private_key_pem
  filename = "ca-key.pem"
}

resource "local_file" "leaf-certs" {
  for_each = var.certificates
  content  = tls_locally_signed_cert.leaf[each.key].cert_pem
  filename = "${each.key}-cert.pem"
}

resource "local_file" "leaf-keys" {
  for_each = var.certificates
  content  = tls_private_key.leaf[each.key].private_key_pem
  filename = "${each.key}-key.pem"
}

Now run it and you will get the CA certificate, all leaf certificates and corresponding private keys:

terraform init
terraform plan -out main.tfplan
terraform apply main.tfplan
ls -l | grep pem

Output:

-rwxr-xr-x 1 alex alex  1554 Mar  9 23:52 anotherone.com-cert.pem
-rwxr-xr-x 1 alex alex  1679 Mar  9 23:52 anotherone.com-key.pem
-rwxr-xr-x 1 alex alex  3243 Mar  9 23:52 ca-key.pem
-rwxr-xr-x 1 alex alex  1834 Mar  9 23:52 ca.pem
-rwxr-xr-x 1 alex alex  1558 Mar  9 23:52 example.com-cert.pem
-rwxr-xr-x 1 alex alex  1675 Mar  9 23:52 example.com-key.pem
-rwxr-xr-x 1 alex alex  1265 Mar  9 23:52 website.com-cert.pem
-rwxr-xr-x 1 alex alex   207 Mar  9 23:52 website.com-key.pem

To test the script with imported CA you can delete the state and all leaf certs and then replace the root_ca if your terraform.tfvars with the following content:

root_ca = {
  algorithm     = "RSA"
  rsa_bits      = 4096
  ecdsa_curve   = null
  external_cert = "ca.pem"
  external_key  = "ca-key.pem"
  subject = {
    CN = "Root CA"
    O  = "Example Org"
  }
}

and then repeat the Terraform routine

rm -f terraform.tfstate*
ls | grep .com | xargs rm -f
terraform plan -out main.tfplan
terraform apply main.tfplan
ls -l | grep pem

Output:

-rwxr-xr-x 1 alex alex  1554 Mar 10 21:27 anotherone.com-cert.pem
-rwxr-xr-x 1 alex alex  1679 Mar 10 21:27 anotherone.com-key.pem
-rwxr-xr-x 1 alex alex  3243 Mar 10 21:25 ca-key.pem
-rwxr-xr-x 1 alex alex  1834 Mar 10 21:25 ca.pem
-rwxr-xr-x 1 alex alex  1562 Mar 10 21:27 example.com-cert.pem
-rwxr-xr-x 1 alex alex  1675 Mar 10 21:27 example.com-key.pem
-rwxr-xr-x 1 alex alex  1265 Mar 10 21:27 website.com-cert.pem
-rwxr-xr-x 1 alex alex   207 Mar 10 21:27 website.com-key.pem

Conclusion

I hope this article shed some light on the potential and the ways of using the iteration mechanisms in Terraform. In part two I am going to cover the list and map expressions for building robust data structures that can be consumed by the count and for_each meta-arguments. Combining those you will be able to write the code to support “self-serve” infrastructure deployment processes and create complex customizable resource hierarchies.

Stay tuned.

Dynamic Secret Engines: What do they do?

In a nutshell dynamic secrets engines provide a way to off-load credential management from the 3rd party apps/services(e.g. databases, message brokers, cloud providers, etc) giving you in exchange a generic mechanism of creating, rotating and revoking credentials with configurable TTL, RBAC, etc. All done via contacting a single API over HTTPS.

Dynamic Secret Engines: How they do it?

You would configure the secret engine with an “admin”-like credentials(I will call them “root” credentials) of a target app/service giving it permissions to create, update, or delete users, passwords, tokens, etc. Then the end user would access Vault to obtain short-lived credentials with a given access scope. Vault would track the status of such credentials via the “lease” mechanism and revoke them(by sending appropriate instructions to the target app/service) when the lease expires.

The “root” credentials are decently privileged and long-lived at the same time. Although once they are written to the Vault, they can’t be extracted in any way making a leak an unlikely event and giving you some peace of mind.

Dynamic Secret Engines: Why and how would you use them?

Dynamic secret engines give you a powerful mechanism to take control over the secrets sprawl and put an end to the long-lived credentials. At the same time, Vault’s audit trail will have an ultimate record of all access attempts. After Secret Engine is configured you would create one or more Role(some secret engines call it RoleSets) which would be associated with a specific access scope on the target. In other words speaking a Role/RoleSet is a Vault’s internal representation of an identity in a target app/service(such as user, service account, etc). The capabilities of a RoleSet greatly depends on the Secret Engine implementation and capabilities of the target app/service. By reading(or writing sometimes) from the Role/RoleSet path you are obtaining a credentials of such identity.

The end-user access is controlled by the Vault’s policies. You can grant access to the user/group on a specific path where secrets engine Roles/RoleSets are mounted. The granularity of the access granted is limited by the target service capabilities and your imagination.

Practice

Now let’s look closely at the Google Cloud secret engine in action. We’ll start as always with setting up Vault server. To keep it simple, I’ll use the ‘filesystem’ storage backend and no SSL/TLS. Config file first:

cat <<EOF > config.hcl
listener "tcp" {
  address     = "127.0.0.1:8200"
  tls_disable = "true"
}

storage "file" {
  path = "$(pwd)/data"
}

disable_mlock = "true"
api_addr      = "http://127.0.0.1:8200"
EOF

And the start the server:

vault server -config ./config.hcl

In another terminal window initialize and unseal it (just one key share would do for this test) and then set up environment variables for further configuration:

export VAULT_ADDR="http://127.0.0.1:8200"
vault operator init -key-shares=1 -key-threshold=1 -format=json | tee init_info.json
vault operator unseal $(jq -r ".unseal_keys_hex[0]" init_info.json)
export VAULT_TOKEN="$(jq -r ".root_token" init_info.json)"

Now enable the secret engine:

vault secrets enable gcp

At this point we’ll need a “root” GCP Service Account key. The Service Account must have following permissions:

iam.serviceAccounts.create
iam.serviceAccounts.delete
iam.serviceAccounts.get
iam.serviceAccounts.list
iam.serviceAccounts.update
iam.serviceAccountKeys.create
iam.serviceAccountKeys.delete
iam.serviceAccountKeys.get
iam.serviceAccountKeys.list

Plus additional set of permission of a following pattern:

<service>.<resource>.getIamPolicy
<service>.<resource>.setIamPolicy

First block will give the “root” service account permissions to create child Service Accounts and their keys. Second block will allow to grant them roles according to the RoleSet specification.

<service>.<resource> will define what kind of GCP services and resources Vault will be able to grant access to. I will be granting pre-defined roles, though in real use cases you would probably go with a Custom Role.

gcloud iam service-accounts create vault-dyn-engine
SA_EMAIL="$(gcloud iam service-accounts list --project vault-dynamic-secrets| grep vault-dyn-engine | awk '{print $1}')"

gcloud iam service-accounts keys create credentials.json --iam-account=$SA_EMAIL --project vault-dynamic-secrets
PROJECT_ID="$(jq -r '.project_id' credentials.json)"

gcloud projects add-iam-policy-binding $PROJECT_ID \
    --member=serviceAccount:$SA_EMAIL \
    --role=roles/resourcemanager.projectIamAdmin

gcloud projects add-iam-policy-binding $PROJECT_ID \
    --member=serviceAccount:$SA_EMAIL \
    --role=roles/iam.serviceAccountAdmin

gcloud projects add-iam-policy-binding $PROJECT_ID \
    --member=serviceAccount:$SA_EMAIL \
    --role=roles/iam.serviceAccountKeyAdmin

Back to the Vault configuration. Time to create a RoleSet with a Project Viewer role:

vault write gcp/config credentials=@credentials.json

vault write gcp/roleset/project_viewer \
    project="vault-dynamic-secrets" \
    secret_type="service_account_key"  \
    bindings=-<<EOF
      resource "//cloudresourcemanager.googleapis.com/projects/vault-dynamic-secrets" {
        roles = ["roles/viewer"]
      }
EOF

After this step you should be able to see a new service account in GCP:

IAM roles granted to this Service Account will match one you put it the RoleSet:

Now the secret engine is ready for use. At this point you would create policies and grant your Vault users write permission ot the path gcp/key/project_viewer. I’ll omit this part and use the same root token I used for configuration:

vault write gcp/key/project_viewer ttl=5s -format=json > result.json

jq -r ".data.private_key_data" result.json | base64 -d > generated_key.json

echo "Lease duration: $(jq -r '.lease_duration' result.json)"

I use a silly TTL value of 5 sec, but it shows that keys are created and then shortly after destroyed:

And to finish the demo let’s delete the RoleSet:

vault delete gcp/roleset/project_viewer

The moment RoleSet is deleted, corresponding Service Account in GCP is deleted too.

Pitfalls and workarounds

Thinking about the lease mechanism and decoupled nature of a target app/service and a secrets manager, two corner cases inevitably come to mind:

1. What will happen if Vault crashes, reboots, or simply gets sealed? Typically you would address this issue by configuring HA-Vault. The OSS Vault allows only Active/Standby mode, when only one instance of Vault is serving user requests at all time. Enterprise Vault gives you “Performance Standby Nodes” feature which allows standby nodes to serve Read request, increasing the request throughput your Vault can handle. Though even if all of Vault nodes will go down at the same time somehow you will be ok. Due to the fact that secret lifetime gets controlled by the Vault and it needs to send the request to the target app/service to revoke the secret, revocation stops working until Vault is down/sealed. The good news is that the moment you unseal it, Vault will reassess all it’s leases and will revoke all secrets associated with expired leases.

2. What will happen if the target app/service is down, or the connectivity is disrupted in any way? At the first glance this scenario looks similar to the first one, Vault is aware of all expired leases and can repeat the revocation once connection is restored. In practice there is one significant limitation. Vault makes a limited number of attempts to revoke the secret and then just stops trying. In the test I have performed with Google Cloud secrets engine(and quick research confirmed that it is valid for other dynamic secret engines too) Vault made 6 retries starting with 40sec back-off timer, and then gradually increasing it(+10sec, +20sec, +40sec, etc). What that means is if your target app/service connectivity were not restored in ~13 mins, the secret will be hanging there indefinitely.

This is a sad news indeed, but there are things you can do to address it:

1. You can manually revoke such secrets(or build the automation around the Vault API and your logging service that captures Vault logs and can detect the event of failed revocation)
2. If your Vault gets sealed/unsealed all expired leases will be reassessed and revoked accordingly. If you are using manual unsealing it’s not making things much better. If you do use auto-unsealing, you can potentially build a workflow with a daily seal/unseal routine. Not a bulletproof solution but it is something at least.

Conclusion

Vault dynamic secret’s engines is an extremely powerful and handy tool that can make life of you SecOps teams much easier. Though as it typically is the case, it’s not perfect. There are pitfalls and edge cases. The “maximum revocation attempt” issue in the Vault’s Git repository is still open. Let’s hope it’ll be addressed and resolved in the near future. Meanwhile if you are planning to use Dynamic Secret Engines, consider if a app/service downtime is a frequent event for you and plan your remediation procedures accordingly.

Theory

Lets assume we have an org with multiple developer teams. All of them use central CI system to build and deploy their software to the cloud. Each team might have more than one project to develop/maintain and obviously security is the priority for everyone so all secrets must be in the central well protected location. Most of CI’s have some sort of secrets storage, but each pipeline has to be configured individually (the only one that did it right is Concourse CI, those folks created beautiful way of integrating 3rd party secrets managers). Now imagine what effort would it take to maintain a dozen of secrets across 3+ environments… Pain and suffering, isn’t it?

What we want is to make this process transparent. Knowing where secrets are stored and how to get them should be the only CI concern. Managing those secrets(adding, updating, deleting, rotating, etc) should be “externalized” to the lack of better word.

Hashicorp Vault is an example of a software that can make it happen. It is API-driven and has a powerful RBAC system. Key/Value secrets engine is a simple and straightforward but at the same time robust and flexible way of storing arbitrary string data. I will demonstrate how to use it to fetch secrets for your CI/CD pipelines.

Lets think for a moment about typical deployment procedure. We try to keep all environments alike, that’s the only way we can guarante(more or less) that if it works flawlessly in staging it will work just as well in production. That means that infrastructure code should be the same for all the environments. Obviously there will be variables like names, credentials, certificates, etc. We need a way to fetch them in a generic way but with ability to detect the target environment type and get the correct secret for environment.

Vault use paths as a mount/access point for secret engines. We could leverage that to make single script to fetch environment secrets using following path structure:

ci-kv/TEAM/PROJECT/ENVIRONMENT/SECRET

Practice

We’ll start with Vault configuration. Setting up production grade Vault server is out of the scope of this post and I will be using dev-server for this proof of concept.

vault server -dev

The rest will be done in a separate shell. First enable the secrets engine:

export VAULT_ADDR='http://127.0.0.1:8200'
vault secrets enable -path=ci-kv -version=1 kv

Then write some secrets for staging environment:

vault kv put ci-kv/dev_team_a/project_x/staging/username \
    value=staging_user

vault kv put ci-kv/dev_team_a/project_x/staging/password \
    value=staging_password

And some for production:

vault kv put ci-kv/dev_team_a/project_x/production/username \
    value=production_user

vault kv put ci-kv/dev_team_a/project_x/production/password \
    value=production_password

Next goes the policy to access all secrets. You will need read and list permissions:

cat <<EOF > policy.hcl
path "ci-kv/dev_team_a/*" {
  capabilities = ["list", "read"]
}
EOF
vault policy write dev_team_a_ci policy.hcl

As you can see I’m granting the CI permissions to read all secrets of dev_team_a regardless of the project but you could go as granular as you like.

The next step would be to create an AppRole credentials and bind them with the policy you just created:

vault auth enable approle
vault write auth/approle/role/dev_team_a_ci \
    token_max_ttl=5m \
    policies=dev_team_a_ci
vault read auth/approle/role/dev_team_a_ci/role-id
vault write -f auth/approle/role/dev_team_a_ci/secret-id

That will produce Role ID and Secret ID necessary to obtain a token. You will get an output similar to following:

Key        Value
---        -----
role_id    f73079cb-619d-6764-c8dd-c5f7c42d58c5

Key                   Value
---                   -----
secret_id             362c3f32-6e94-c942-1669-bff27bda8763
secret_id_accessor    b2a6e6b6-1d54-8a5c-fff4-23257f4d7a62

Now when all configurations are done lets write a small shell script to retrieve all secrets. To do the trick you will need curl and jq.

cat <<'EOF' > run_with_secrets.sh
#!/bin/sh

set -o nounset
set -o errexit

# get the token
TOKEN="$(curl \
    --silent \
    --request POST \
    --data "{\"role_id\":\"$ROLE_ID\",\"secret_id\":\"$SECRET_ID\"}" \
    $VAULT_URL/v1/auth/approle/login | jq -r .auth.client_token)"

# fetch secrets list
SECRETS="$(curl \
    --silent \
    --header "X-Vault-Token: $TOKEN" \
    --request LIST \
    $VAULT_URL/v1/$KV_PATH_PREFIX/$ENVIRONMENT | jq -r ".data.keys[]")"

# fetch secrets and put them in corresponding environment variables
for SECRET in $SECRETS; do
  VALUE=$(curl \
    --silent \
    --header "X-Vault-Token: $TOKEN" \
    "$VAULT_URL/v1/$KV_PATH_PREFIX/$ENVIRONMENT/$SECRET" | jq -r ".data.value")
  export $SECRET="$VALUE"
done

# finally run the deployment command that needs all those secrets
exec "$@"
EOF

chmod +x ./run_with_secrets.sh

Now lets test the result. Generally speaking that should be your deployment tool that would reference the environment variables. But to keep it simple a short script that just greps secrets out of environment variables should be enough:

cat <<EOF > tester.sh
env | grep username
env | grep password
EOF

chmod +x ./tester.sh

Now run it:

# following would be configured in your CI
export VAULT_URL="http://127.0.0.1:8200"
export KV_PATH_PREFIX="ci-kv/dev_team_a/project_x"
export ROLE_ID="f73079cb-619d-6764-c8dd-c5f7c42d58c5"
export SECRET_ID="362c3f32-6e94-c942-1669-bff27bda8763"

# environment can be obtained either from envvar set by CI or from a branch name
export ENVIRONMENT="staging"
./run_with_secrets.sh ./tester.sh

export ENVIRONMENT="production"
./run_with_secrets.sh ./tester.sh

Outputs should perfectly match values you configured in Vault. Obviously same script would work with any number of secrets equally well. Not to mention that you will be able to dynamically upload or rotate secrets without a need to update your CI pipeline config. The only inputs we’ll need are Vault URL, KV path prefix, AppRole credentials, and environment name. You can also easily modify the script to dump secrets into a file rather than environment variables(for example if you need a tfvars file for your Terraform code).

To put those values in the Vault you could use either the built-in UI or a custom frontend. It supports integration with authentication backends like LDAP or OIDC so letting users in should be very easy. By properly configuring policies, you can give your users write/update access which will allow them to create/update secrets but not view them.

Room for improvement

If you wonder what’s gonna happen if Vault was configured incorrectly, or there’s a typo in the path, or even AppRole has expired, the answer is nothing good. The script has no error handling or even decent output to help with troubleshooting when something goes wong. Also there is not that much flexibility in the way Vault operations are performed, login method is hardcoded, etc. In other words, there is plenty of space for improvement.

You can have a look at what was my approach to converting it into a more robust version at Trollabs/vault-secrets-fetcher. And I would be happy to hear what would you do to further improve it.

In fact GitOps works great in any scenario where declarative configuration is used. I have successfully used it to provision Cloud Infrastructure with Terraform and I am going to share some insights, tips and tricks I’ve learned over the last year.

So what is GitOps?

In a nutshell, GitOps is an approach for handling Ops tasks using Git as a centerpiece. We all know that git is amazing collaboration tool. It helps us to track changes in code, and reverse them in the matter of minutes when it is necessary. All modern Continuous Integration(CI) systems work in combination with Git. They observe the state of a git repository and trigger jobs when the state changes.

Git repo becomes in a way the single source of truth, because every time the repo state changes new CI job is triggered to reflect that change on the target environment. That means your provisioning procedure might run multiple times in a row on the same or almost the same infrastructure code, so it must be idempotent. This is the only constraint that GitOps puts on the tools you can use. Terraform is a good example of such tool due to the fact that it keeps track of the infrastructure state and (re)deploys only the delta.

Bells and whistles

Git itself is a small CLI tool and is rarely used on its own, typically one will use a Git based code hosting service. The most popular are GitHub, GitLab, and BitBucket. They all offer the distributed version control and source code management functionality of Git but also bring some additional features (bug tracking, pull requests, integrated ci, etc).

All of them provide capability(either integrated or with help of WebHooks) to launch scripts on events such as push, pull request creation or update, creation of the comment in the PR discussion, etc. Moreover they give you access to their API allowing you to build complex scenarios.

Another common and incredibly useful features are merge checks and branch protection. They empower you to enforce how and when your code “moves” between branches of the repo. Here are some examples of what you can do with it:

1. Deny direct push to the branch unless it was done by merging a Pull Request
2. Prevent merging a Pull Request unless it was approved by one ore more people of a certain group (eg code owners)
3. Prevent merge if last CI build on the branch has failed
4. Prevent merge if discussions on the PR were not resolved

Use them to create “gates” between different environments.

The GitOps Workflow

I’ve diverted a bit into the hosted Git solutions and their feature set to give a little bit perspective of what they have to offer. Now lets see how you can use them to build more or less useful GitOps workflow.

As you can see the diagram is very generic and can be applied with any hosted Git offering, CI system or Cloud Provider. Pull Request event is used only once to validate the code in the testing environment before pushing it to staging. After successful validation neither the code nor deployment process will change so you can use the same job/script. To make this possible you will have to store environment-specific variables, secrets and credentials in some sort of Secret Management system and then fetch them at runtime. Variables that are the same for all environments and are not sensitive can be stored in git together with the infrastructure code.

I prefer Hashicorp Vault’s KV engine for storing secrets and credentials. The KV path can be composed as something like:

kv/TEAM_NAME/BRANCH_NAME/*

That way you can write a generic script that will fetch all secrets from the prefix and write them to the .tfvars file or set as environment variables. You can use it with any number of different teams and environments using BRANCH_NAME as a pointer to the environment.

To get an idea of how you can achieve that you can check out my post Injecting secrets into CI/CD from Hashicorp Vault.

What else you can do?

Of course you can also have CI jobs for other PRs if you have a suitable action to launch. For example you could generate and send a notification to the Slack channel with a URL of the PR and a results of test/deployment to previous environment. Generally speaking appropriate notifications is a very important aspect of a GitOps workflow. Having a message sent to your Slack channel or a Hipchat room with a good summary helps a lot. Just try to keep the noise down.

Conclusion

I believe that GitOps is extremely useful approach and definitely will be a nice addition to the DevOps engineer’s toolbox. At the same time it’s not a silver bullet and in some use case scenarios might not make sense. So before going wild transforming everything into GitOps workflows try it in a “proof of concept” and see if it adds value. I hope this article will bring you some inspiration and will help you to build GitOps workflows in your organization.