Injecting secrets into CI/CD from Hashicorp Vault

Ever since I started my DevOps engineer journey my experience of interacting with secrets from CI system could be summarized in 3 words - “pain and suffering”. I’ve been wondering “Wouldn’t it be dreamy to have a tool that can take all the secrets from a central location and inject them into my build at runtime?”. I’ve been planning to explore this idea for a long time and now, finally, the time has come. We’ll make this world a better place and lift the weight of CI/CD secrets management off of DevOps engineers shoulders by putting in place some automation.

Theory

Lets assume we have an org with multiple developer teams. All of them use central CI system to build and deploy their software to the cloud. Each team might have more than one project to develop/maintain and obviously security is the priority for everyone so all secrets must be in the central well protected location. Most of CI’s have some sort of secrets storage, but each pipeline has to be configured individually (the only one that did it right is Concourse CI, those folks created beautiful way of integrating 3rd party secrets managers). Now imagine what effort would it take to maintain a dozen of secrets across 3+ environments… Pain and suffering, isn’t it?

What we want is to make this process transparent. Knowing where secrets are stored and how to get them should be the only CI concern. Managing those secrets(adding, updating, deleting, rotating, etc) should be “externalized” to the lack of better word.

Hashicorp Vault is an example of a software that can make it happen. It is API-driven and has a powerful RBAC system. Key/Value secrets engine is a simple and straightforward but at the same time robust and flexible way of storing arbitrary string data. I will demonstrate how to use it to fetch secrets for your CI/CD pipelines.

Lets think for a moment about typical deployment procedure. We try to keep all environments alike, that’s the only way we can guarante(more or less) that if it works flawlessly in staging it will work just as well in production. That means that infrastructure code should be the same for all the environments. Obviously there will be variables like names, credentials, certificates, etc. We need a way to fetch them in a generic way but with ability to detect the target environment type and get the correct secret for environment.

Vault use paths as a mount/access point for secret engines. We could leverage that to make single script to fetch environment secrets using following path structure:

ci-kv/TEAM/PROJECT/ENVIRONMENT/SECRET

Practice

We’ll start with Vault configuration. Setting up production grade Vault server is out of the scope of this post and I will be using dev-server for this proof of concept.

vault server -dev

The rest will be done in a separate shell. First enable the secrets engine:

export VAULT_ADDR='http://127.0.0.1:8200'
vault secrets enable -path=ci-kv -version=1 kv

Then write some secrets for staging environment:

vault kv put ci-kv/dev_team_a/project_x/staging/username \
    value=staging_user

vault kv put ci-kv/dev_team_a/project_x/staging/password \
    value=staging_password

And some for production:

vault kv put ci-kv/dev_team_a/project_x/production/username \
    value=production_user

vault kv put ci-kv/dev_team_a/project_x/production/password \
    value=production_password

Next goes the policy to access all secrets. You will need read and list permissions:

cat <<EOF > policy.hcl
path "ci-kv/dev_team_a/*" {
  capabilities = ["list", "read"]
}
EOF
vault policy write dev_team_a_ci policy.hcl

As you can see I’m granting the CI permissions to read all secrets of dev_team_a regardless of the project but you could go as granular as you like.

The next step would be to create an AppRole credentials and bind them with the policy you just created:

vault auth enable approle
vault write auth/approle/role/dev_team_a_ci \
    token_max_ttl=5m \
    policies=dev_team_a_ci
vault read auth/approle/role/dev_team_a_ci/role-id
vault write -f auth/approle/role/dev_team_a_ci/secret-id

That will produce Role ID and Secret ID necessary to obtain a token. You will get an output similar to following:

Key        Value
---        -----
role_id    f73079cb-619d-6764-c8dd-c5f7c42d58c5

Key                   Value
---                   -----
secret_id             362c3f32-6e94-c942-1669-bff27bda8763
secret_id_accessor    b2a6e6b6-1d54-8a5c-fff4-23257f4d7a62

Now when all configurations are done lets write a small shell script to retrieve all secrets. To do the trick you will need curl and jq.

cat <<'EOF' > run_with_secrets.sh
#!/bin/sh

set -o nounset
set -o errexit

# get the token
TOKEN="$(curl \
    --silent \
    --request POST \
    --data "{\"role_id\":\"$ROLE_ID\",\"secret_id\":\"$SECRET_ID\"}" \
    $VAULT_URL/v1/auth/approle/login | jq -r .auth.client_token)"

# fetch secrets list
SECRETS="$(curl \
    --silent \
    --header "X-Vault-Token: $TOKEN" \
    --request LIST \
    $VAULT_URL/v1/$KV_PATH_PREFIX/$ENVIRONMENT | jq -r ".data.keys[]")"

# fetch secrets and put them in corresponding environment variables
for SECRET in $SECRETS; do
  VALUE=$(curl \
    --silent \
    --header "X-Vault-Token: $TOKEN" \
    "$VAULT_URL/v1/$KV_PATH_PREFIX/$ENVIRONMENT/$SECRET" | jq -r ".data.value")
  export $SECRET="$VALUE"
done

# finally run the deployment command that needs all those secrets
exec "$@"
EOF

chmod +x ./run_with_secrets.sh

Now lets test the result. Generally speaking that should be your deployment tool that would reference the environment variables. But to keep it simple a short script that just greps secrets out of environment variables should be enough:

cat <<EOF > tester.sh
env | grep username
env | grep password
EOF

chmod +x ./tester.sh

Now run it:

# following would be configured in your CI
export VAULT_URL="http://127.0.0.1:8200"
export KV_PATH_PREFIX="ci-kv/dev_team_a/project_x"
export ROLE_ID="f73079cb-619d-6764-c8dd-c5f7c42d58c5"
export SECRET_ID="362c3f32-6e94-c942-1669-bff27bda8763"

# environment can be obtained either from envvar set by CI or from a branch name
export ENVIRONMENT="staging"
./run_with_secrets.sh ./tester.sh

export ENVIRONMENT="production"
./run_with_secrets.sh ./tester.sh

Outputs should perfectly match values you configured in Vault. Obviously same script would work with any number of secrets equally well. Not to mention that you will be able to dynamically upload or rotate secrets without a need to update your CI pipeline config. The only inputs we’ll need are Vault URL, KV path prefix, AppRole credentials, and environment name. You can also easily modify the script to dump secrets into a file rather than environment variables(for example if you need a tfvars file for your Terraform code).

To put those values in the Vault you could use either the built-in UI or a custom frontend. It supports integration with authentication backends like LDAP or OIDC so letting users in should be very easy. By properly configuring policies, you can give your users write/update access which will allow them to create/update secrets but not view them.

Room for improvement

If you wonder what’s gonna happen if Vault was configured incorrectly, or there’s a typo in the path, or even AppRole has expired, the answer is nothing good. The script has no error handling or even decent output to help with troubleshooting when something goes wong. Also there is not that much flexibility in the way Vault operations are performed, login method is hardcoded, etc. In other words, there is plenty of space for improvement.

You can have a look at what was my approach to converting it into a more robust version at Trollabs/vault-secrets-fetcher. And I would be happy to hear what would you do to further improve it.