Using Hashicorp Vault

Posted on: 2023-03-19

I've been using Ansible to manage configuration on the servers that I manage, but I recently began looking for a proper secrets management solution. Hashicorp Vault is a popular free option in the enterprise so I decided to set it up. While the install is very easy, it does require a bit of configuration to make it work, so in this post I document the way I use it.

The first step is to set the Hasicorp repo and install vault:

apt install lsb-release jq
wget -O - https://apt.releases.hashicorp.com/gpg | gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | tee /etc/apt/sources.list.d/hashicorp.list
apt update && apt install vault
vault version

If it installed correctly, you should see the version of Vault you have. Next, we need to configure the /etc/vault.d/vault.hcl configuration file:

ui = true

disable_mlock = true

storage "file" {
  path = "/opt/vault/data"
}

api_addr = "https://vault.example.com:8200"

listener "tcp" {
  address       = "0.0.0.0:8200"
  tls_cert_file = "/etc/letsencrypt/live/vault.example.com/fullchain.pem"
  tls_key_file  = "/etc/letsencrypt/live/vault.example.com/privkey.pem"
}

Here, I use the default file storage backend, but I use a LetsEncrypt SSL certificate. You can make your own or use a self signed certificate if you want. From here, you can start Vault with systemctl start vault and you should be able to access the web UI on port 8200 from which you can setup the initial root token and unseal the vault. The root token is used to login as a superuser, while the unseal token is needed to unlock all the secrets. These should be saved securely, since you need to unseal your vault every time the service restarts.

The next step is to enable the kv secrets type and the approle authentication so you can access secrets through the API:

export VAULT_ADDR=https://vault.example.com:8200/
vault login
vault auth enable approle
vault secrets enable kv

The first command is to tell the Vault CLI where the endpoint is, then you need to login with your root token. After that, we create the auth and secrets endpoints. Next, let's create a read-only policy we can use with an API user, so we don't need to use the root user with our scripts and applications in the /etc/vault.d/readonly-kv-policy.hcl file:

path "kv/data/*" {
  capabilities = ["read", "list"]
}

Ideally you should restrict the path to only one application per token, but in this case we give full access to all the secrets stored in kv. Then, we'll run a few commands to add this policy and create our user:

vault policy write readonly-kv readonly-kv-policy.hcl
vault write auth/approle/role/readonly-kv-role token_policies="readonly-kv" token_ttl="1h" token_max_ttl="4h"
vault write -f auth/approle/role/readonly-kv-role/secret-id
vault read auth/approle/role/readonly-kv-role/role-id

This should give you a role_id and a secret_id that are attached to this role. We'll use those to read secrets, but first let's store a few secrets using the CLI:

vault kv put kv/myapp username="test" password="blah"

Here we stored 2 key/value pairs in the myapp path. Now, let's write a small shell script to read from it, using the API user we created:

#!/bin/bash

VAULT_ADDR="https://vault.example.com:8200"
ROLE_ID="aaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee"
SECRET_ID="aaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee"
TOKEN=$(curl -s --request POST --data "{\"role_id\":\"$ROLE_ID\", \"secret_id\":\"$SECRET_ID\"}" $VAULT_ADDR/v1/auth/approle/login | jq -r .auth.client_token)

if [ $# -eq 1 ]; then
        SECRET=$(curl -s --header "X-Vault-Token: $TOKEN" $VAULT_ADDR/v1/kv/data/$1 | jq -r .data.data )
        echo $SECRET
elif [ $# -eq 2 ]; then
        SECRET=$(curl -s --header "X-Vault-Token: $TOKEN" $VAULT_ADDR/v1/kv/data/$1 | jq -r .data.data.$2 )
        echo $SECRET
else
        echo "Usage: $0  [secret name]"
        exit 1
fi

The way this script works is that you can get a list of secrets in JSON format using ./vault.sh myapp or the value of a single secret with ./vault.sh myapp username. From here, you can use this in any shell script, or through other applications like Ansible. For example, I get my AWS keys form Vault using this script and assign it to Ansible variables:

AWS_SECRET: "{{ lookup('pipe', '~/vault.sh myapp aws_secret') }}"

That should give you a good head start to using Vault. There are a lot more to explore of course, like you can use the web UI to do some (but not all) of what we did using the CLI, and you should make sure you secure your vault properly if you intend to store production secrets in it.