Secrets Management
For practical reasons, some secrets will not be stored inside the Vault. Those manifests that might be committed into this repository must have its content replaced with <FILL-IN>. You also have a simple script available (check_secrets.sh) that outputs every manifest file with at least one <FILL-IN> entry. You can use this script to check if you have correctly filled in all secrets for your purpose.
By default, as stated in the official documentation, kubernetes secrets are, by default, stored in plain text, which makes them vulnerable in case an attacker is able to read data from the server. Although the probabilities of this occurring are small, it is a risk we deemed unworthy to take.
Vault is a solution that we found in order to securely store encrypted secrets, among other features such as PKI infraestructure.
However, Vault itself is not enough, since it works as a repository of secrets, which means that, by itself, does not sync the secrets to the kubernetes pods, so a solution is needed for the pods to have access to the secrets that are held on the vault repository.
Due to this reason, we decided to use the Vault Operator which uses semantics native to kubernetes, since it is just one more kubernetes operator, which makes it easy to sync secrets between Vault and the pods that use them.
The other alternative would be to use a sidecar injector, but that was deemed as a more complicated process, since it is not something native to kubernetes like the operator pattern is as well as it required to make annotations to the .yaml files of the deployed services, increasing the complexity of deployments.
There is a script called setup-vault-dev.sh contained in the services/vault folder that you can run in order to deploy vault and vault operator in deploy mode.
./services/vault/deploy-vault-dev.shThe steps to make a pod have access to the secrets can either be done with the Vault UI or via the issuing of commands in the terminal. The examples in this document will mainly show examples of the CLI commands as the using of the user interface is fairly intuitive due to the great job made by the Vault.
In order to connect to vault inside the terminal you must issue the command:
kubectl exec --stdin=true --tty=true vault-0 -n vault -- /bin/shThis will open a shell inside of the vault pod.
In production environments, we will not be able to issue commands in the terminal without an authentication token unless we explicitly authenticate.
Some ways include:
-
Using the root token
$\rightarrow$ export VAULT_TOKEN=<root-token>inside thevault-cli. -
Using username and password
$\rightarrow$ vault login -method=userpass username=<username> password=<password>
When running in development mode (with services/vault/vault-dev-values.yaml), in order to be able to access the UI, you will be prompted by a login page where you have to input a token.
In this case, because the value specified on the yaml is the root token, you just have to write 'root` as the token.
In order for other pods to be able to communicate and be authenticated with the vault service, we need to setup an authentication method for the vault method, which we do in this case using the kubernetes auth method.
vault auth enable -path demo-auth-mount kubernetesvault write auth/demo-auth-mount/config \
kubernetes_host="https://$KUBERNETES_PORT_443_TCP_ADDR:443"Below is a picture of all supported vault authentication methods to date (5th June 2024):
We enable a secrets engine at a specified path and then we create a policy for that folder indicating which namespace and service accounts will have access to that path, making it easy to isolate secrets from pods that do not need to have access to it.
There is more than one type of secret engine and the way to configure those will change depending on the type. Below follows a picture of all the possible secret engines available taken at 5th June 2024.
For example, we can use a different secret engine for a database connection than to just a normal secret like a jwt token. With database secrets we can create users and then use them on our database pods.
Generic example
vault secrets enable -path=<path_name> <type>Specific example
vault secrets enable -path=kvv2 kv-v2In this specific example we are enabling a secrets engine of type kv-v2.
It is important to point out that vault already comes with a default kv-v2
secrets engine called secret.
This creates a filesystem point inside vault where secrets can be stored.
An important characteristic is that in vault, when you create a secret, it will not be a single key pair value. Instead, in vault when you create a secret, the secret is actually a dictionary.
So, for example, you could have a secret named postgres-db, which would like this inside vault:
{
_raw: # this is metadata from the secret
'username': 'cool-username',
'password': 'strong-password',
}vault kv put kvv2/webapp/config username="static-user" password="static-password"The webapp/config part is just the folder the secret will be put on inside the secrets engine.
Generic example
vault policy write <policy_name> - <<EOF
path "<path_name>/*" {
capabilities = ["read"]
}
EOFSpecific example:
vault policy write dev - <<EOF
path "kv2/* {
capabilities = ["read"]
}
EOFThis will be responsbile to allow pods with this role to be able to access the path of the secrets.
vault write <auth-endpoint>/role/<name_of_role> \
bound_service_account_names=default \
bound_service_account_namespaces=app \
policies=dev \
audience=vault \
ttl=24hIn this case, the auth-endpoint would be demo-auth-mount as defined on step 1.
We need to setup an authentication method in the secret in order to specify its role so that the vault pod can check if that role has access to a specific path where secrets could be stored.
As specified here:
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultAuth
metadata:
name: secret-auth-wathever-name-you-like
namespace: app
spec:
method: kubernetes
mount: demo-auth-mount
kubernetes:
role: role1
serviceAccount: default
audiences:
- vaultThe mount is the name of authentication endpoint we created on step 1.
The role is the role we created for the policy.
As for the secret, as stated here would look like this:
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultStaticSecret
metadata:
name: demo-secret
namespace: app
spec:
type: kv-v2
# mount path
mount: kvv2
# path of the secret
path: webapp/config
# dest k8s secret
destination:
name: secretkv
create: true
# static secret refresh interval
refreshAfter: 30s
# Name of the CRD to authenticate to Vault
vaultAuthRef: secret-auth-wathever-name-you-likeThe vaultAuthRef is the name specified on the metadata of the yaml of the authentication of the service.
It is worth it to point out that the secret needs to be inside the same namespace that was specified on the secret authentication yaml.
You can find a full example of a YAML deployment with vault, you can find it here.
The more relevant part is here:
spec:
volumes:
- name: secrets
secret:
secretName: "demo-secret"
containers:
- name: example
image: nginx:latest
env:
- name: DB_PASSWORD
valueFrom:
secretKeyRef:
name: "vso-db-demo"
key: password
- name: DB_USERNAME
valueFrom:
secretKeyRef:
name: "vso-db-demo"
key: username
volumeMounts:
- name: secrets
mountPath: /etc/secrets
readOnly: trueIn this specific example, this allows to pass environment variables to the deployed application that this yaml deploys and the values are fetched from the vault secrets engine.
As you can see in this part
secretKeyRef:
name: "demo-secret"
key: passwordIt reiterates with was said before that vault secrets are dictionaries, and because of that, we tell the name of the secret and then the key that we want to fetch from the dictionary.
As expected, the data inside Vault is encrypted and as all encryption methods, it requires encryption and decryption keys.
In the default config, when Vault is initialized, it creates a key which is stored with the data. However, to prevent storing the plaintext version of the key, the key is encrypted with a root key. The root key is the same as the root token and is the token that all users of Vault need to know in order to be able to decrypt the data and interact with.
In the default config Vault uses an algorithm for secret sharing among a group called Shamir's secret sharing.
In order to protect the value of the root token, this token is built from possibly more than one key, in order to make it harder to find the value of the root token. These keys from which the root token is built from are called key shares.
For example, if we have 3 key shares and a threshold of 2, it means that in order to obtain the value of the root token we need two of those key shares. It may be easier to think of the threshold as the quorum needed to reach the actual value of the root token.
Exactly, that is what we need to discover, which is why we need to unseal the vault, in order to obtain the plaintext value of the root token, which means:
-
Specifying the number of key shares
-
The threshold (quorum) needed to obtain the plaintext value of the root token
After specifying those parameters and clicking on Initialize, we are able to begin unsealing the vault and getting the value of the root token.
1.3. Isn't it insecure for a vault instance to allow us just to get the plaintext version of the root key?
It is not insecure because vault only allows this type of requests on an instance which has no data and was just initialized..
For obvious security reasons, the operation of getting a plaintext version of the root token is not normally allowed.
For further details as why vault starts sealed, you can them here