As more and more organizations making the shift to cloud native technologies, Kubernetes has become the de facto choice to orchestrate container based applications. As applications grow in size, the number of microservices increases and so does the data they process. Hence, handling data, especially sensitive data becomes critical. Out of the box, Kubernetes supports 'Secrets' objects to store sensitive information like passwords, tokens, ssh keys, etc. securely. Kubernetes secret eliminates the need of hard-coding sensitive data in the application code. Secrets provide this sensitive information as data mount or expose them as environment variables. Let’s take a quick example of how this secret object is getting used within the Kubernetes cluster. Create a secret object: ```yaml apiVersion: v1 kind: Secret metadata: name: rabbitmq namespace: default type: Opaque data: RABBITMQ_PASSWORD: cmFiYml0bXE= ``` Now, let's understand how to Inject this secret object inside a pod using an environment variable. ```yaml apiVersion: apps/v1 kind: Pod metadata: name: nginx spec: containers: - name: nginx image: nginx:latest env: - name: RABBITMQ_PASSWORD valueFrom: secretKeyRef: name: rabbitmq key: RABBITMQ_PASSWORD ``` That’s how we can easily inject a secret inside a pod and secure sensitive information. However, there is a challenge with this approach. Our sensitive information is still not fully secure because the yaml definition of secrets are base64 encoded, and anyone can easily decode base64 encoded secrets. To overcome this challenge with managing secrets within Kubernetes cluster, we need a better secret management system that provides a single source of credentials, secrets, security policies, etc. There are many solutions already available in the open source world like Bank-Vaults, AWS Secrets Manager, and Cloud KMS, but in this article, we will be focusing on HashiCorp Vault as it is currently widely adopted within the cloud native ecosystem. In this blog post, we will discuss how you can create a production grade secret management system using Hashicorp Vault. ## What is HashiCorp Vault? HashiCorp Vault is a secret management tool that is used to store sensitive values and access it securely. A secret can be anything, such as API encryption keys, passwords, or certificates. Vault provides encryption services and supports authentication and authorization. We can run Vault in high-availability (HA) mode and standalone mode. The standalone mode runs a single Vault server which is less secure and less resilient that is not recommended for production grade setup. ### Benefit of using HashiCorp Vault in HA using Raft Vault can run in the multi-server mode for high availability to protect against outages. To persist the encrypted data, Vault supports many storage backends like Consul, MySQL, DynamoDB, etc. But there are some challenges using these standard backend storages like: - Increased complexity with addition of storage backends. - Additional configuration needed to allow Vault to operate as a client. - Vault configuration to connect to the provider as a client. So we need a solution which minimizes these challenges and at the same time, allows us to [implement and manage security](/cloud-native-security-services/) efficiently. In this article we are focusing on Integrated storage to persist the data which has below benefits: - Integrated into Vault (reducing the total administration). - All configuration stays within Vault. - Supports failover and multi-cluster replication. - Get rid of additional network requests to backe. In a Kubernetes cluster, to deploy Vault in high availability mode, it would be deployed as a cluster of pods. When deploying Vault in HA mode, all the nodes in a Vault cluster will have a replicated copy of Vault's data. Under the hood, [Raft Consensus Algorithm](https://raft.github.io/) is used to replicate the data across all the nodes. ![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/o7p5ia1eqjf07vhx8vl8.png) Image Source ### Deploy Vault in HA and Vault Auto Unseal Before we jump into deploying Vault, it is important to understand what happens when we deploy a Vault pod. After deployment, the Vault pods start in a sealed state. That means until we unseal the Vault server, almost no operations are allowed. Here, unsealing is the process of decrypting the data inside Vault and allowing other services to access it. The data that is generally stored in Vault is encrypted and it needs an encryption key to decrypt the data. As we see in the below diagram, to decrypt the data, Vault must decrypt the encryption key first, which requires the master key. Unsealing is the process of getting access to this master key. The master key is also stored alongside all other Vault data, but it is encrypted by another mechanism: **the unseal key**. By default the Vault config uses [Shamir seals](https://www.vaultproject.io/docs/concepts/seal). ![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/8xusv87ojobleeg8okzu.png) Image Source ### Vault Auto Unseal To make the Vault server secure, there is also an API that seals it. This process puts away the master key in memory and requires another unseal process to restore it. So in case a pod gets restarted, every time someone has to manually unseal the Vault using **vault operator unseal** command. To avoid the manual step of unsealing, Vault supports automatic unsealing via cloud based key management services like Azure Key Vault, Amazon KMS, AliCloud KMS, and Google Cloud KMS which enables us to let trusted cloud providers take care of the unsealing process, as seen in the below diagram. ![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/fd6g3nubarvn4tjxhl5m.png) Image Source We are going to set up an auto-unsealing Vault on Kubernetes with Azure Key Vault. Let’s start by setting up the required Azure Services:\ Prerequisite: You need to have an Azure cloud account. If not, you can [create trial account](https://azure.microsoft.com/en-in/free/) for free. - Create an Azure Key Vault [by following official documentation](https://docs.microsoft.com/en-us/azure/key-vault/general/quick-create-portal) and note down the **Subscription ID** and **Directory ID**, which will be required in later steps. ![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/4u10bx5l2l32pe12xqzb.png) - Next step is to register an App by [following official documentation](https://docs.microsoft.com/en-us/azure/active-directory/develop/quickstart-register-app) which **establishes a trust relationship between our app and the Microsoft identity platform**. This trust is unidirectional: your app trusts the Microsoft identity platform, and not the other way around. Here we will create a Service Principal which will work as our access layer to the Azure Key Vault. ***It is very important to copy the value of Application (client ID) and client secret from register an App section mentioned above.*** - Once this is done, let's switch back to Azure Key Vault, and under IAM add a new Role Assignment as Key Vault Contributor for our newly created Service Principal: ![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/pg9siddfigt1rjayzlvu.png) - Once the Service Principal is assigned to the Azure Key Vault, we need to provide specific access permissions to it. Vault needs **Get** permissions at key level and Unwrap Key and Wrap Key at Cryptographic level, so under access policy, create a new one with those selected: ![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/xspg8tz9cb2ht0rkbfrd.png) - Select the Service Principal in the list, Click on Add and Save the configuration ![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ntnw5hae6nswhw14qw0x.png) - The last step on Azure portal is to create a key on Key Vault, which will be the unsealing key for Vault: ![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/pk9iivrxyqqw4hijot3w.png) Once the above steps are done, it's time to install the HashiCorp Vault. The recommended way to deploy a Vault in the Kubernetes cluster is using the Vault’s [official Helm chart](https://github.com/hashicorp/vault-helm). To deploy Vault in HA with auto unsealing use the below-mentioned `values.yml` file. ```yaml injector: enabled: false server: image: repository: "hashicorp/vault" tag: "1.9.0" # Overrides the default Image Pull Policy pullPolicy: IfNotPresent # Configure the Update Strategy Type for the StatefulSet updateStrategyType: "OnDelete" resources: requests: memory: 256Mi cpu: 250m limits: memory: 256Mi cpu: 250m ha: enabled: true replicas: 3 raft: enabled: true config: | ui = true listener "tcp" { tls_disable = 1 address = "[::]:8200" cluster_address = "[::]:8201" } seal "azurekeyvault" { tenant_id = "0fd16XXX-xxxx-xxxx-xxxx-xxxx" client_id = "ab509eca-xxxx-xxxx-xxxx-xxxxxxxxxxxx" client_secret = "UTb7Q~xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" vault_name = "vault-k8s-data" key_name = "vault-k8s-unsealer-key" subscription_id = "c83a96b1-xxxx-xxxx-xxxx-xxxxxxxxxxxx" } storage "raft" { path = "/vault/data" } service_registration "kubernetes" {} dataStorage: enabled: true size: 500Mi ``` The values mentioned under seal “azurekeyvault” in the above configuration are derived from Azure portal while setting up azure services. Note: | Config | Description | |:------------------|:-----------------------------------------------------------------------| | `tenant_id` | Key Vault's Directory ID | | `client_id` | Service Principal's Application ID | | `client_secret` | Service Principal's generated secret (the one that is not retrievable) | | `vault_name` | Name of Azure Key Vault instance | | `key_name` | Name of generated key on Azure Key Vault | | `subscription_id` | ID of the Azure Subscription | Proceed with the installation following the steps mentioned below: ```sh $ helm repo add hashicorp https://helm.releases.hashicorp.com "hashicorp" has been added to your repositories $ helm install vault hashicorp/vault -f values.yaml NAME: vault LAST DEPLOYED: Sat Mar 5 22:14:51 2022 NAMESPACE: default STATUS: deployed REVISION: 1 NOTES: Thank you for installing HashiCorp Vault! Now that you have deployed Vault, you should look over the docs on using Vault with Kubernetes available here: https://www.vaultproject.io/docs/ Your release is named vault. To learn more about the release, try: $ helm status vault $ helm get manifest vault ``` Let’s verify the installation by running below-mentioned command: ```sh $ kubectl get pods -n vault NAME READY STATUS RESTARTS vault-0 0/1 Running 0 vault-1 0/1 Running 0 vault-2 0/1 Running 0 ``` As discussed earlier, Vault starts in a sealed state, and that’s why all the Vault pods are in running status but none of the Vault pods are "ready". To unseal Vault, we need to initialize it. Just exec into pod vault-0 and initialize your Vault instance (and keep your generated keys safe) as shown below: ```sh $ kubectl exec -it vault-0 vault operator init Recovery Key 1: FKjt5wkzN5bUBIuR52KrPP1c2Il/f7RZdn5E+ipfNF8s Recovery Key 2: FCzUyduESPyavh6QtqWZpdnUDKa3bEEpBHbX3NgTrCiU Recovery Key 3: Tf7FVEpj5tdJLqQqNw/Jt0OytRI5FAZYig/yafSVz3Xg Recovery Key 4: duLpa/6IozTOR0mkO7sp0CwmnI+1DsC6d2+oZG/A1CIZ Recovery Key 5: pyVFs/rRFEk9rSn57Ru+KeuAQzW6eurl3j0/pS/JRpXD Initial Root Token: s.d0LAlSnAerb4a7d6ibkfxrZy Success! Vault is initialized Recovery key initialized with 5 key shares and a key threshold of 3. Please securely distribute the key shares printed above. ``` Now, we see the Vault unsealer in action: ```sh $ kubectl get pods -n vault NAME READY STATUS RESTARTS vault-0 1/1 Running 0 vault-1 0/1 Running 0 vault-2 0/1 Running 0 ``` However, our replica `vault-1` and `vault-2` are still not ready. They are follower pods of the leader `vault-0`. In order to make vault-0 visible, we need to login using our `Initial Root Token`. ```sh $ kubectl exec -it vault-0 -- vault login s.d0LAlSnAerb4a7d6ibkfxrZy Success! You are now authenticated. The token information displayed below is already stored in the token helper. You do NOT need to run "vault login" again. Future Vault requests will automatically use this token. Key Value --- ----- token s.d0LAlSnAerb4a7d6ibkfxrZy token_accessor hJEFibLTbUP4sgA8X4tMBqZ8 token_duration ∞ token_renewable false token_policies ["root"] identity_policies [] policies ["root"] ``` Next, let’s join vault-1 and vault-2 to Vault-0 to make the Vault setup Highly available by running commands as shown below: ```sh $ kubectl exec -it vault-1 -- vault operator raft join http://vault-0.vault-internal:8200 Key Value --- ----- Joined true $ kubectl exec -it vault-2 -- vault operator raft join http://vault-0.vault-internal:8200 Key Value --- ----- Joined true ``` Let’s check the pod status and this time you would be able to see them in running as well as Ready mode: ```sh $ kubectl get pods -n vault NAME READY STATUS RESTARTS vault-0 1/1 Running 0 vault-1 1/1 Running 0 vault-2 1/1 Running 0 ``` The last but important step is to verify whether the HA setup is correct or not, is by running the below command against each Vault pod and making sure that the value of HA Enabled parameters is true. ```sh $ kubectl exec -it -n vault vault-0 -- vault status Key Value --- ----- Recovery Seal Type shamir Initialized true Sealed false Total Recovery Shares 5 Threshold 3 Version 1.9.0 Storage Type raft Cluster Name vault-cluster-30882e80 Cluster ID 1afbe13a-e951-482d-266b-e31693d17e20 HA Enabled true HA Cluster https://vault-0.vault-internal:8201 HA Mode active Active Since 2022-01-19T04:39:37.586622342Z Raft Committed Index 61 Raft Applied Index 61 $ kubectl exec -it -n vault vault-1 -- vault status Key Value --- ----- … HA Enabled true HA Cluster https://vault-0.vault-internal:8201 HA Mode standby Active Node Address http://10.244.0.17:8200 Raft Committed Index 61 Raft Applied Index 61 $ kubectl exec -it -n vault vault-2 -- vault status Key Value --- ----- HA Enabled true HA Cluster https://vault-0.vault-internal:8201 HA Mode standby Active Node Address http://10.244.0.17:8200 Raft Committed Index 61 Raft Applied Index 61 ``` As you can see `vault-0` is `active` and `vault-1` and `vault-2` is `standby`. Now, Let’s try deleting vault-0 and run the same command, you will notice that vault-1 has become active and vault-0 becomes standby. ```sh kubectl -n vault delete pod vault-0 ``` ```sh $ kubectl exec -it -n vault vault-1 -- vault status Key Value --- ----- … HA Enabled true HA Cluster https://vault-1.vault-internal:8201 HA Mode active Active Node Address http://10.244.0.18:8200 Raft Committed Index 61 Raft Applied Index 61 $ kubectl exec -it -n vault vault-0 -- vault status Key Value --- ----- … HA Enabled true HA Cluster https://vault-1.vault-internal:8201 HA Mode standby Active Since 2022-01-19T04:39:37.586622342Z Raft Committed Index 61 Raft Applied Index 61 ``` ### Vault Web UI Finally to be able to access the Vault web interface; let’s port-forward the service. ```sh $ kubectl port-forward -n vault svc/vault 8200:8200` ``` Vault UI will be accessible on `http://localhost:8200` ![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/nipja0bkb10ri0838v23.png) To login in Vault, enter the token that we generated when we initialized Vault for the first time (`Initial Root Token`). ![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/5i680r6tqgf22ib21g34.png) Secrets Engine is a Vault component that stores or generates a secret. Vault supports different types of Secrets Engines like key-value, ssh keys, certificates, etc. To start with, let's use a KV secrets engine, click on the **Enable New Engine+** button, select the KV engine and click on **Next**. ![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/raiajbzwg6pepcpr4eul.png) Give a path name and click on **Enable Engine**. ![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/uiza3nekcwxt760rylba.png) ![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/9jq7th3lnhmznfazpx52.png) Now, let’s create a secret, click on Create secret and enter the details as shown in the below figure. ![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/yqurhawy148ej50ay3ky.png) Once we click on save, we should be able to see that the secret has been successfully created and stored in a Vault. Since Vault centrally stores, secures and controls access to the secrets, it is important to control the permissions before anyone can gain access. Vault supports Role Based Access Control to limit the access. Refer to the [official Vault documentation on policies](https://learn.hashicorp.com/tutorials/vault/policies) to understand more about the policies and roles. Until this point, we have successfully deployed Vault in HA using Helm, learnt about configuring role based access and created a secret object. Let’s now inject this secret inside a pod. ### Inject Vault secrets into pod Since Vault has been deployed successfully let’s now jump into injecting the Vault secrets into the pods/application. Mainly there are two ways for a pod to use Secret object: - As a file mounted in volume. - As a container environment variable. Secrets can be injected into the application with the Vault agent and this introduces a new requirement that deployments install and configure Vault Agent alongside the application as a sidecar. Now, we will be focusing on injecting a Vault secret into an application using **External Secrets**. The benefit of Kubernetes External Secrets is that it can be used with Vault, external secret management systems like AWS Secret Manager, Azure Key Vault to securely add secrets. Kubernetes External Secrets extends the Kubernetes API by adding an ExternalSecrets object using Custom Resource Definition. An ExternalSecret object declares how to fetch the secret data, while the controller converts all ExternalSecrets to Secrets. ![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/6tiw63joth3vuh9s633y.png) Image Source Just like the Hashicorp Vault, the recommended way to deploy Kubernetes-External-Secrets is using the official helm chart and use the below-mentioned values.yaml file. ```yaml env: VAULT_ADDR: http://vault:8200 METRICS_PORT: 3001 POLLER_INTERVAL_MILLISECONDS: 10000 WATCH_TIMEOUT: 60000 WATCHED_NAMESPACES: "" # Comma separated list of namespaces, empty or unset means ALL namespaces. LOG_LEVEL: info LOG_MESSAGE_KEY: "msg" rbac: # Specifies whether RBAC resources should be created create: true serviceAccount: # Specifies whether a service account should be created create: true # Specifies annotations for this service account annotations: {} # The name of the service account to use. # If not set and create is true, a name is generated using the fullname template name: kubernetes-external-secrets replicaCount: 1 image: repository: ghcr.io/external-secrets/kubernetes-external-secrets tag: 8.5.1 pullPolicy: IfNotPresent resources: {} ``` Run the following command to deploy the external secrets. ```sh $ helm repo add external-secrets https://external-secrets.github.io/kubernetes-external-secrets/ "external-secrets" has been added to your repositories $ helm install k8s-external-secrets external-secrets/kubernetes-external-secrets -f values.yaml NAME: k8s-external-secrets LAST DEPLOYED: Wed Mar 23 22:50:35 2022 NAMESPACE: default STATUS: deployed REVISION: 1 TEST SUITE: None NOTES: The kubernetes external secrets has been installed. Check its status by running: $ kubectl --namespace default get pods -l "app.kubernetes.io/name=kubernetes-external-secrets,app.kubernetes.io/instance=k8s-external-secrets" Visit https://github.com/external-secrets/kubernetes-external-secrets for instructions on how to use kubernetes external secrets ``` At this point, the kubernetes-external-secret has been deployed successfully and with the help of the ExternalSecrets object, now we can inject secrets inside the pods that are stored in Vault. But the external-secrets controller needs to be authenticated with Vault before making any request to retrieve the secrets, so we need to enable the Kubernetes authentication method and attach a policy to it. **Enable Kubernetes Auth Method** Let's access the Vault pod's terminal and enable kubernetes auth method: ```sh $ kubectl exec -it vault-0 sh $ vault login s.d0LAlSnAerb4a7d6ibkfxrZy $ vault auth enable kubernetes Success! Enabled kubernetes auth method at: kubernetes/ ``` The Kubernetes Auth Method can be used to authenticate with Vault using Kubernetes Service Account Token, as Vault accepts this token by any client within the Kubernetes cluster. Configure the kubernetes authentication method by running below command from inside the active Vault pod: ```sh $ vault write auth/kubernetes/config \ token_reviewer_jwt="$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \ kubernetes_host=https://${KUBERNETES_PORT_443_TCP_ADDR}:443 \ kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt Success! Data written to: auth/kubernetes/config ``` The token_reviewer_jwt and kubernetes_ca_cert are created for a pod by Kubernetes when it is started. The env variable KUBERNETES_PORT_443_TCP_ADDR is defined and references the internal Kubernetes API Server network address. When an application tries to authenticate with Vault using its Service Account Token, Vault uses the above configuration to verify the client application’s identity with Kubernetes API Server. **Attach a policy to Kubernetes auth method** Since we have already created a **readonly** policy, let’s create a role for kubernetes and attach this policy. In the below mentioned role all the service accounts in any namespaces are allowed to authenticate. We can also limit authentication to specific service accounts and namespaces as well. ![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ryftl2emt6npagdsegz9.png) ```sh $ vault write auth/kubernetes/role/k8s-role \ bound_service_account_names=* \ bound_service_account_namespaces=* \ policies=readonly \ ttl=1h Success! Data written to: auth/kubernetes/role/k8s-role ``` Let’s now inject vault secrets inside a pod. We will be using the same example which we used at the start of this blog post. As discussed earlier, we just need to create an ExternalSecret object and the controller will convert it to a Secret object. Delete the secret object we created earlier and apply the below file. ```yaml apiVersion: 'kubernetes-client.io/v1' kind: ExternalSecret metadata: name: rabbitmq namespace: default spec: kvVersion: 1 backendType: vault vaultMountPoint: kubernetes vaultRole: k8s-role data: - name: RABBITMQ_USERNAME key: secret/Rabbitmq property: username ``` Save this file by name rabbitmq-external-secret.yaml and run: ```sh $ kubectl delete secret rabbitmq secret "rabbitmq" deleted $ kubectl apply -f rabbitmq-external-secret.yaml ``` Let’s run the below command to check the status of external-secret. ```sh $ kubectl get externalsecrets.kubernetes-client.io NAME LAST SYNC STATUS AGE rabbitmq 10s SUCCESS 15s $ kubectl get secrets NAME TYPE DATA AGE rabbitmq Opaque 1 15s ``` So here, External Secret has been created successfully and the External Secrets fetched the correct value from Vault, and created the Secret object for us. This Secret object is used by our application as we saw in the beginning of the post. ## Summary Since Kubernetes is widely used and to make a Kubernetes cluster production-grade, it is very important to make sure the secret objects are handled carefully. In this article, we learned why it is important not to use native secret objects due to their limitations and the importance of using a secret management tool like HashiCorp Vault. We also learned about External Secrets, which help us to inject secrets inside a pod. With the help of external secrets, we can easily replace the Kubernetes secrets with Vault secrets inside any existing deployments. That’s it, folks !! I hope this article was informative and engaging. I am looking forward to hearing your thoughts on this post, so let’s connect and start a conversation on [LinkedIn](https://www.linkedin.com/in/nikhil-purva-a1b1b320/). Looking for help with cloud native security? do check out our capabilities how we’re helping startups & enterprises as an [DevSecOps consulting services provider](/cloud-native-security-services/).

nikhilpurva

Securing Kubernetes Secrets with HashiCorp Vault

Join Our Newsletter. No Spam, Only the good stuff.