As more and more organizations making the shift to cloud native technologies, Kubernetes has become the de facto choice to orchestrate container based applications. As applications grow in size, the number of microservices increases and so does the data they process. Hence, handling data, especially sensitive data becomes critical. Out of the box, Kubernetes supports 'Secrets' objects to store sensitive information like passwords, tokens, ssh keys, etc. securely.

Kubernetes secret eliminates the need of hard-coding sensitive data in the application code. Secrets provide this sensitive information as data mount or expose them as environment variables.

Let’s take a quick example of how this secret object is getting used within the Kubernetes cluster.

Create a secret object:

apiVersion: v1
kind: Secret
metadata:
 name: rabbitmq
 namespace: default
type: Opaque
data:
 RABBITMQ_PASSWORD: cmFiYml0bXE=

Now, let's understand how to Inject this secret object inside a pod using an environment variable.

apiVersion: apps/v1
kind: Pod
metadata:
 name: nginx
spec:
  containers:
  - name: nginx
    image: nginx:latest
    env:
      - name: RABBITMQ_PASSWORD
        valueFrom:
          secretKeyRef:
            name: rabbitmq
            key: RABBITMQ_PASSWORD

That’s how we can easily inject a secret inside a pod and secure sensitive information. However, there is a challenge with this approach. Our sensitive information is still not fully secure because the yaml definition of secrets are base64 encoded, and anyone can easily decode base64 encoded secrets.

To overcome this challenge with managing secrets within Kubernetes cluster, we need a better secret management system that provides a single source of credentials, secrets, security policies, etc. There are many solutions already available in the open source world like Bank-Vaults, AWS Secrets Manager, and Cloud KMS, but in this article, we will be focusing on HashiCorp Vault as it is currently widely adopted within the cloud native ecosystem. In this blog post, we will discuss how you can create a production grade secret management system using Hashicorp Vault.

What is HashiCorp Vault?

HashiCorp Vault is a secret management tool that is used to store sensitive values and access it securely. A secret can be anything, such as API encryption keys, passwords, or certificates. Vault provides encryption services and supports authentication and authorization.

We can run Vault in high-availability (HA) mode and standalone mode. The standalone mode runs a single Vault server which is less secure and less resilient that is not recommended for production grade setup.

Benefit of using HashiCorp Vault in HA using Raft

Vault can run in the multi-server mode for high availability to protect against outages. To persist the encrypted data, Vault supports many storage backends like Consul, MySQL, DynamoDB, etc.

But there are some challenges using these standard backend storages like:

• Increased complexity with addition of storage backends.
• Additional configuration needed to allow Vault to operate as a client.
• Vault configuration to connect to the provider as a client.

So we need a solution which minimizes these challenges and at the same time, allows us to implement and manage security efficiently.

In this article we are focusing on Integrated storage to persist the data which has below benefits:

• Integrated into Vault (reducing the total administration).
• All configuration stays within Vault.
• Supports failover and multi-cluster replication.
• Get rid of additional network requests to backe.

In a Kubernetes cluster, to deploy Vault in high availability mode, it would be deployed as a cluster of pods. When deploying Vault in HA mode, all the nodes in a Vault cluster will have a replicated copy of Vault's data. Under the hood, Raft Consensus Algorithm is used to replicate the data across all the nodes.

Deploy Vault in HA and Vault Auto Unseal

Before we jump into deploying Vault, it is important to understand what happens when we deploy a Vault pod. After deployment, the Vault pods start in a sealed state. That means until we unseal the Vault server, almost no operations are allowed. Here, unsealing is the process of decrypting the data inside Vault and allowing other services to access it.

The data that is generally stored in Vault is encrypted and it needs an encryption key to decrypt the data. As we see in the below diagram, to decrypt the data, Vault must decrypt the encryption key first, which requires the master key. Unsealing is the process of getting access to this master key. The master key is also stored alongside all other Vault data, but it is encrypted by another mechanism: the unseal key. By default the Vault config uses Shamir seals.

Vault Auto Unseal

To make the Vault server secure, there is also an API that seals it. This process puts away the master key in memory and requires another unseal process to restore it. So in case a pod gets restarted, every time someone has to manually unseal the Vault using vault operator unseal command.

To avoid the manual step of unsealing, Vault supports automatic unsealing via cloud based key management services like Azure Key Vault, Amazon KMS, AliCloud KMS, and Google Cloud KMS which enables us to let trusted cloud providers take care of the unsealing process, as seen in the below diagram.

We are going to set up an auto-unsealing Vault on Kubernetes with Azure Key Vault. Let’s start by setting up the required Azure Services:\
Prerequisite: You need to have an Azure cloud account. If not, you can create trial account for free.

• Create an Azure Key Vault by following official documentation and note down the Subscription ID and Directory ID, which will be required in later steps.

• Next step is to register an App by following official documentation which establishes a trust relationship between our app and the Microsoft identity platform. This trust is unidirectional: your app trusts the Microsoft identity platform, and not the other way around. Here we will create a Service Principal which will work as our access layer to the Azure Key Vault.

It is very important to copy the value of Application (client ID) and client secret from register an App section mentioned above.

• Once this is done, let's switch back to Azure Key Vault, and under IAM add a new Role Assignment as Key Vault Contributor for our newly created Service Principal:

• Once the Service Principal is assigned to the Azure Key Vault, we need to provide specific access permissions to it. Vault needs Get permissions at key level and Unwrap Key and Wrap Key at Cryptographic level, so under access policy, create a new one with those selected:

• Select the Service Principal in the list, Click on Add and Save the configuration

• The last step on Azure portal is to create a key on Key Vault, which will be the unsealing key for Vault:

Once the above steps are done, it's time to install the HashiCorp Vault. The recommended way to deploy a Vault in the Kubernetes cluster is using the Vault’s official Helm chart. To deploy Vault in HA with auto unsealing use the below-mentioned values.yml file.

injector:
  enabled: false
server:
  image:
    repository: "hashicorp/vault"
    tag: "1.9.0"
    # Overrides the default Image Pull Policy
    pullPolicy: IfNotPresent

  # Configure the Update Strategy Type for the StatefulSet
  updateStrategyType: "OnDelete"
  resources:
    requests:
      memory: 256Mi
      cpu: 250m
    limits:
      memory: 256Mi
      cpu: 250m
  ha:
    enabled: true
    replicas: 3
    raft:
      enabled: true
      config: |
        ui = true
        listener "tcp" {
          tls_disable = 1
          address = "[::]:8200"
          cluster_address = "[::]:8201"
        }
        seal "azurekeyvault" {
          tenant_id       = "0fd16XXX-xxxx-xxxx-xxxx-xxxx"
          client_id       = "ab509eca-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
          client_secret   = "UTb7Q~xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
          vault_name      = "vault-k8s-data"
          key_name        = "vault-k8s-unsealer-key"
          subscription_id = "c83a96b1-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
        }
        storage "raft" {
          path = "/vault/data"
        }
        service_registration "kubernetes" {}
  dataStorage:
    enabled: true
    size: 500Mi

The values mentioned under seal “azurekeyvault” in the above configuration are derived from Azure portal while setting up azure services.

Note:

Proceed with the installation following the steps mentioned below:

$ helm repo add hashicorp https://helm.releases.hashicorp.com
"hashicorp" has been added to your repositories

$ helm install vault hashicorp/vault -f values.yaml
NAME: vault
LAST DEPLOYED: Sat Mar  5 22:14:51 2022
NAMESPACE: default
STATUS: deployed
REVISION: 1
NOTES:
Thank you for installing HashiCorp Vault!

Now that you have deployed Vault, you should look over the docs on using
Vault with Kubernetes available here:

https://www.vaultproject.io/docs/


Your release is named vault. To learn more about the release, try:

$ helm status vault
$ helm get manifest vault

Let’s verify the installation by running below-mentioned command:

$ kubectl get pods -n vault
NAME                                           READY   STATUS    RESTARTS 
vault-0                                        0/1     Running   0      
vault-1                                        0/1     Running   0     
vault-2                                        0/1     Running   0

As discussed earlier, Vault starts in a sealed state, and that’s why all the Vault pods are in running status but none of the Vault pods are "ready". To unseal Vault, we need to initialize it. Just exec into pod vault-0 and initialize your Vault instance (and keep your generated keys safe) as shown below:

$ kubectl exec -it vault-0 vault operator init

Recovery Key 1: FKjt5wkzN5bUBIuR52KrPP1c2Il/f7RZdn5E+ipfNF8s
Recovery Key 2: FCzUyduESPyavh6QtqWZpdnUDKa3bEEpBHbX3NgTrCiU
Recovery Key 3: Tf7FVEpj5tdJLqQqNw/Jt0OytRI5FAZYig/yafSVz3Xg
Recovery Key 4: duLpa/6IozTOR0mkO7sp0CwmnI+1DsC6d2+oZG/A1CIZ
Recovery Key 5: pyVFs/rRFEk9rSn57Ru+KeuAQzW6eurl3j0/pS/JRpXD
Initial Root Token: s.d0LAlSnAerb4a7d6ibkfxrZy
Success! Vault is initialized
Recovery key initialized with 5 key shares and a key threshold of 3. Please
securely distribute the key shares printed above.

Now, we see the Vault unsealer in action:

$ kubectl get pods -n vault
NAME                                           READY   STATUS    RESTARTS
vault-0                                        1/1     Running   0      
vault-1                                        0/1     Running   0     
vault-2                                        0/1     Running   0         

However, our replica vault-1 and vault-2 are still not ready. They are follower pods of the leader vault-0. In order to make vault-0 visible, we need to login using our Initial Root Token.

$ kubectl exec -it vault-0 -- vault login s.d0LAlSnAerb4a7d6ibkfxrZy

Success! You are now authenticated. The token information displayed below
is already stored in the token helper. You do NOT need to run "vault login"
again. Future Vault requests will automatically use this token.
Key                  Value
---                  -----
token                s.d0LAlSnAerb4a7d6ibkfxrZy
token_accessor       hJEFibLTbUP4sgA8X4tMBqZ8
token_duration       ∞
token_renewable      false
token_policies       ["root"]
identity_policies    []
policies             ["root"]

Next, let’s join vault-1 and vault-2 to Vault-0 to make the Vault setup Highly available by running commands as shown below:

$ kubectl exec -it vault-1 -- vault operator raft join http://vault-0.vault-internal:8200

Key       Value
---       -----
Joined    true

$ kubectl exec -it vault-2 -- vault operator raft join http://vault-0.vault-internal:8200

Key       Value
---       -----
Joined    true

Let’s check the pod status and this time you would be able to see them in running as well as Ready mode:

$ kubectl get pods -n vault
NAME                                           READY   STATUS    RESTARTS   
vault-0                                        1/1     Running   0     
vault-1                                        1/1     Running   0     
vault-2                                        1/1     Running   0     

The last but important step is to verify whether the HA setup is correct or not, is by running the below command against each Vault pod and making sure that the value of HA Enabled parameters is true.

$ kubectl exec -it -n vault vault-0 -- vault status
Key                      Value
---                      -----
Recovery Seal Type       shamir
Initialized              true
Sealed                   false
Total Recovery Shares    5
Threshold                3
Version                  1.9.0
Storage Type             raft
Cluster Name             vault-cluster-30882e80
Cluster ID               1afbe13a-e951-482d-266b-e31693d17e20
HA Enabled               true
HA Cluster               https://vault-0.vault-internal:8201
HA Mode                  active
Active Since             2022-01-19T04:39:37.586622342Z
Raft Committed Index     61
Raft Applied Index       61

$ kubectl exec -it -n vault vault-1 -- vault status
Key                      Value
---                      -----
…
HA Enabled               true
HA Cluster               https://vault-0.vault-internal:8201
HA Mode                  standby
Active Node Address      http://10.244.0.17:8200
Raft Committed Index     61
Raft Applied Index       61

$ kubectl exec -it -n vault vault-2 -- vault status
Key                      Value
---                      -----
HA Enabled               true
HA Cluster               https://vault-0.vault-internal:8201
HA Mode                  standby
Active Node Address      http://10.244.0.17:8200
Raft Committed Index     61
Raft Applied Index       61

As you can see vault-0 is active and vault-1 and vault-2 is standby. Now, Let’s try deleting vault-0 and run the same command, you will notice that vault-1 has become active and vault-0 becomes standby.

kubectl -n vault delete pod vault-0

$ kubectl exec -it -n vault vault-1 -- vault status
Key                      Value
---                      -----
…
HA Enabled               true
HA Cluster               https://vault-1.vault-internal:8201
HA Mode                  active
Active Node Address      http://10.244.0.18:8200
Raft Committed Index     61
Raft Applied Index       61

$ kubectl exec -it -n vault vault-0 -- vault status
Key                      Value
---                      -----
…
HA Enabled               true
HA Cluster               https://vault-1.vault-internal:8201
HA Mode                  standby
Active Since             2022-01-19T04:39:37.586622342Z
Raft Committed Index     61
Raft Applied Index       61

Vault Web UI

Finally to be able to access the Vault web interface; let’s port-forward the service.

$ kubectl port-forward -n vault svc/vault 8200:8200`

Vault UI will be accessible on http://localhost:8200

To login in Vault, enter the token that we generated when we initialized Vault for the first time (Initial Root Token).

Secrets Engine is a Vault component that stores or generates a secret. Vault supports different types of Secrets Engines like key-value, ssh keys, certificates, etc.

To start with, let's use a KV secrets engine, click on the Enable New Engine+ button, select the KV engine and click on Next.

Give a path name and click on Enable Engine.

Now, let’s create a secret, click on Create secret and enter the details as shown in the below figure.

Once we click on save, we should be able to see that the secret has been successfully created and stored in a Vault.

Since Vault centrally stores, secures and controls access to the secrets, it is important to control the permissions before anyone can gain access. Vault supports Role Based Access Control to limit the access. Refer to the official Vault documentation on policies to understand more about the policies and roles.

Until this point, we have successfully deployed Vault in HA using Helm, learnt about configuring role based access and created a secret object. Let’s now inject this secret inside a pod.

Inject Vault secrets into pod

Since Vault has been deployed successfully let’s now jump into injecting the Vault secrets into the pods/application. Mainly there are two ways for a pod to use Secret object:

• As a file mounted in volume.
• As a container environment variable. Secrets can be injected into the application with the Vault agent and this introduces a new requirement that deployments install and configure Vault Agent alongside the application as a sidecar.

Now, we will be focusing on injecting a Vault secret into an application using External Secrets. The benefit of Kubernetes External Secrets is that it can be used with Vault, external secret management systems like AWS Secret Manager, Azure Key Vault to securely add secrets. Kubernetes External Secrets extends the Kubernetes API by adding an ExternalSecrets object using Custom Resource Definition.

An ExternalSecret object declares how to fetch the secret data, while the controller converts all ExternalSecrets to Secrets.

Just like the Hashicorp Vault, the recommended way to deploy Kubernetes-External-Secrets is using the official helm chart and use the below-mentioned values.yaml file.

env:
  VAULT_ADDR: http://vault:8200
  METRICS_PORT: 3001
  POLLER_INTERVAL_MILLISECONDS: 10000  
  WATCH_TIMEOUT: 60000
  WATCHED_NAMESPACES: ""  # Comma separated list of namespaces, empty or unset means ALL namespaces.
  LOG_LEVEL: info
  LOG_MESSAGE_KEY: "msg"
rbac:
  # Specifies whether RBAC resources should be created
  create: true

serviceAccount:
  # Specifies whether a service account should be created
  create: true
  # Specifies annotations for this service account
  annotations: {}
  # The name of the service account to use.
  # If not set and create is true, a name is generated using the fullname template
  name: kubernetes-external-secrets
replicaCount: 1
image:
  repository: ghcr.io/external-secrets/kubernetes-external-secrets
  tag: 8.5.1
  pullPolicy: IfNotPresent
resources: {}

Run the following command to deploy the external secrets.

$ helm repo add external-secrets 
https://external-secrets.github.io/kubernetes-external-secrets/
"external-secrets" has been added to your repositories

$ helm install k8s-external-secrets external-secrets/kubernetes-external-secrets -f values.yaml
NAME: k8s-external-secrets
LAST DEPLOYED: Wed Mar 23 22:50:35 2022
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
The kubernetes external secrets has been installed. Check its status by running:
$ kubectl --namespace default get pods -l "app.kubernetes.io/name=kubernetes-external-secrets,app.kubernetes.io/instance=k8s-external-secrets"

Visit https://github.com/external-secrets/kubernetes-external-secrets for instructions on how to use kubernetes external secrets

At this point, the kubernetes-external-secret has been deployed successfully and with the help of the ExternalSecrets object, now we can inject secrets inside the pods that are stored in Vault.

But the external-secrets controller needs to be authenticated with Vault before making any request to retrieve the secrets, so we need to enable the Kubernetes authentication method and attach a policy to it.

Enable Kubernetes Auth Method

Let's access the Vault pod's terminal and enable kubernetes auth method:

$ kubectl exec -it vault-0 sh
$ vault login s.d0LAlSnAerb4a7d6ibkfxrZy
$ vault auth enable kubernetes

Success! Enabled kubernetes auth method at: kubernetes/

The Kubernetes Auth Method can be used to authenticate with Vault using Kubernetes Service Account Token, as Vault accepts this token by any client within the Kubernetes cluster.

Configure the kubernetes authentication method by running below command from inside the active Vault pod:

$ vault write auth/kubernetes/config \
    token_reviewer_jwt="$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \
    kubernetes_host=https://${KUBERNETES_PORT_443_TCP_ADDR}:443 \
    kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt

Success! Data written to: auth/kubernetes/config

The token_reviewer_jwt and kubernetes_ca_cert are created for a pod by Kubernetes when it is started. The env variable KUBERNETES_PORT_443_TCP_ADDR is defined and references the internal Kubernetes API Server network address. When an application tries to authenticate with Vault using its Service Account Token, Vault uses the above configuration to verify the client application’s identity with Kubernetes API Server.

Attach a policy to Kubernetes auth method

Since we have already created a readonly policy, let’s create a role for kubernetes and attach this policy. In the below mentioned role all the service accounts in any namespaces are allowed to authenticate. We can also limit authentication to specific service accounts and namespaces as well.

$ vault write auth/kubernetes/role/k8s-role \
    bound_service_account_names=* \
    bound_service_account_namespaces=* \
    policies=readonly \
    ttl=1h

Success! Data written to: auth/kubernetes/role/k8s-role

Let’s now inject vault secrets inside a pod. We will be using the same example which we used at the start of this blog post. As discussed earlier, we just need to create an ExternalSecret object and the controller will convert it to a Secret object. Delete the secret object we created earlier and apply the below file.

apiVersion: 'kubernetes-client.io/v1'
kind: ExternalSecret
metadata:
 name: rabbitmq
 namespace: default
spec:
 kvVersion: 1
 backendType: vault
 vaultMountPoint: kubernetes
 vaultRole: k8s-role
 data:
 - name: RABBITMQ_USERNAME
   key: secret/Rabbitmq
   property: username

Save this file by name rabbitmq-external-secret.yaml and run:

$ kubectl delete secret rabbitmq
secret "rabbitmq" deleted

$ kubectl apply -f rabbitmq-external-secret.yaml 

Let’s run the below command to check the status of external-secret.

$ kubectl get externalsecrets.kubernetes-client.io
NAME       LAST SYNC   STATUS    AGE
rabbitmq   10s         SUCCESS   15s

$ kubectl get secrets
NAME       TYPE     DATA    AGE
rabbitmq   Opaque   1       15s

So here, External Secret has been created successfully and the External Secrets fetched the correct value from Vault, and created the Secret object for us. This Secret object is used by our application as we saw in the beginning of the post.

Summary

Since Kubernetes is widely used and to make a Kubernetes cluster production-grade, it is very important to make sure the secret objects are handled carefully. In this article, we learned why it is important not to use native secret objects due to their limitations and the importance of using a secret management tool like HashiCorp Vault.

We also learned about External Secrets, which help us to inject secrets inside a pod. With the help of external secrets, we can easily replace the Kubernetes secrets with Vault secrets inside any existing deployments.

That’s it, folks !! I hope this article was informative and engaging. I am looking forward to hearing your thoughts on this post, so let’s connect and start a conversation on LinkedIn.

Looking for help with cloud native security? do check out our capabilities how we’re helping startups & enterprises as an DevSecOps consulting services provider.