Secrets are a fundamental building block of the modern Software Development Lifecycle (SDLC). Applications, CI/CD systems, API access, Databases, etc., all require some form of secrets/tokens/credentials. Keeping secrets out of source code, e.g. using the Twelve-Factor App methodology, is just one of many ways to ensure secrets are managed securely.

While Kubernetes secrets are relatively simple to understand, there are many factors to consider when deciding how they will be managed and injected into containers.

This blog post is a deep dive into the most popular approaches available for Kubernetes Secrets management in 2022, so without further adieu, let's get started!

Side Note: Enable Encryption of Kubernetes Secret Data

Because Kubernetes Secrets are stored as unencrypted base64-encoded strings, it's strongly recommended to enable encryption at rest for your cluster to provide additional protection in the case a malicious actor gains access to the underlying etcd data store.

Kubernetes Secrets Usage and Challenges

Kubernetes essentially provides two methods for injecting secrets into a container:

• Secrets as files
• Secrets as environment variables

The application can then either read the file contents of the secrets (in the case of mounting secrets as data volumes) or consume them as environment variables.

Secrets as Files

Accessing secrets from the file system is common in frameworks such as ASP.NET Core and Java Spring Boot; however, some consider it an outdated approach in preference to environment variables.

Because an application's config will likely vary between environments (e.g, development, staging, production, development), managing secrets in files pose several security questions:

• Should you check the secret files in your version control system?
  • If so, how to encrypt them (since they are secrets)?
  • If not, where to securely store the secret files?
• Should secrets be stored in CI/CD and shipped with the deployment artifact?
• What if secret files are scattered across different repositories?
• How is access to secret files managed and audited?
• Is there a single source of truth for all secrets?
• What if the secrets files are language/framework-specific, but an app in another language/framework needs access to the same secrets?

Kubernetes secrets can address most of these concerns by storing the contents of the secrets file in a single key and value and mounted as a single file inside a container.

To mount a Kubernetes secret into a Pod's container within a deployment:

...
    spec:
      containers:
        - name: web-app
          volumeMounts:
            - name: app-secrets
              # Mounted directory path in container
              mountPath: /usr/src/app/secrets
              readOnly: true
      volumes:
        - name: app-secrets
          secret:
            # Kubernetes secret name
            secretName: dotnet-webapp-appsettings
            items:
              # Key in secret containing appsettings.json contents
              - key: APP_SETTINGS
                # Results in file mounted at /usr/src/app/secrets/appsettings.json
                path: appsettings.json

The downside is that most secret files are framework specific, so environment variables may be preferable for teams wanting a consistent approach to secrets injection.

Secrets as Environment Variables

Environment variables from a Kubernetes secret are injected into a Pod's container process with precise control that allows for all or only a subset of secrets to be exposed.

The most straightforward approach is to expose every secret key-value pair using the envFrom property:

...
    spec:
      containers:
        - name: web-app
          envFrom:
            # Kubernetes secret name
            - secretRef:
                name: app-secrets

Or to only expose a specific list of key-value pairs, use the env property with one or more valueFrom items:

...
    spec:
      containers:
        - name: web-app
          env:
              # Environment variable name
            - name: MY_APP_SECRET
              valueFrom:
                secretKeyRef:
                  # Kubernetes secret name
                  name: doppler-test-secret
                  # Key name in Kubernetes secret
                  key: MY_APP_SECRET

Using environment variables for secrets comes with the benefit of not having to worry about teammates accidentally committing secrets to repositories. Unlike custom secret file formats like Java System Properties, environment variables are language and framework agnostic.

Kubernetes Secrets Source of Truth

So you've decided Kubernetes Secrets are the way to go! Awesome! But because Kubernetes doesn't provide a great experience managing and updating secret values, we'll need to devise our own strategy.

If we create Kubernetes Secrets using YAML files, where do we store those files? Encrypting them within a Git repository is one option. You'd then have the difficult task of managing encryption keys across different repositories and multiple environments and sharing secrets between teams with different needs and permissions. While tools such as Mozilla SOPS and Bitnami Sealed Secrets provide solutions for encrypted secrets, the operational overhead and complexity of managing secrets in version control is not the easiest solution to adopt and scale.

Suppose we choose not to use encrypted YAML files and instead use Kubernetes Secrets as the single truth source. This simplifies things as secrets are only ever stored within Kubernetes, but it also presents new management and access control challenges.

Imagine when you need to migrate to another Kubernetes cluster. You'd have to export and then import every secret, not to mention that multiple manual kubectl commands are required when updating a secret, and we know that manual operations are error-prone.

So, Kubernetes Secrets as the single source of truth is also a no-go.

We need an external and centralized single source of truth with encrypted storage and fine-grained access controls for managing application secrets that can then be synced to Kubernetes via automation.

Secrets Managers and Kubernetes

Secret managers provide a secure and encrypted source of truth for secrets storage at enterprise scale.

There are multiple choices on the market, such as Doppler, AWS Secrets Manager, HashiCorp Vault, etc. Although every secret manager has its quirks and features, they all meet the required security standards for storing and accessing sensitive data.

Secrets Managers solve the security, permissions, sharings, single source of truth, and operational overhead issues once and for all, leaving only one challenge: how to synchronize secrets from the Secrets Manager to Kubernetes.

Let's now look at the four most common secret sync approaches, working towards the ideal solution as we go.

1. Secrets Manager API or SDK

With this method, you don't need to use Kubernetes Secrets, as they are fetched directly from the Secrets Manager via its API or SDK.

The obvious downside is that this doesn't utilize Kubernetes Secrets at all. Plus, you are vendor locked-in (think of the tremendous overhead if you wanted to change to another Secrets Manager: you'd have to touch the code of all the apps).

Kubernetes Secrets provide a native and vendor-agnostic method for applications to access secrets, so this is the least preferred option.

2. Secrets Agent/Sidecar Injection

Using an agent or sidecar also accesses secrets at runtime via an API but provides a more native Kubernetes secrets access experience by dynamically mounting secrets into a Pod's container as part of the deployment process. It's emulating the same behavior as a Kubernetes mounted secret, but in a vendor-specific way and without actually using a Kubernetes Secret. Let's use HashiCorp Vault as an example to illustrate.

The HashiCorp Vault has an Agent Injector that dynamically alters pod specifications during deployment to include Vault Agent containers that render Vault secrets to a shared memory volume using Vault Agent Templates (phew!). The injector is a Kubernetes Mutation Webhook Controller. The controller intercepts pod events and applies mutations to the Pod if annotations exist within the request. This functionality is provided by the vault-Kubernetes project. For more info, refer to the Vault Kubernetes Injector documentation.

By rendering secrets to a shared volume, containers within the Pod can consume Vault secrets without being Vault-aware.

There are, however, a few downsides. The first and most significant is that installing and using this method requires in-depth knowledge of Kubernetes and HashiCorp Vault. The beauty of Kubernetes Secrets is that they're simple to understand and consume. In the quest to integrate a Secrets Manager more tightly into the application deployment process, we now have something much more complex to wrap our heads around.

The second is understanding how the agent injects the secrets as a volume. Every container in the Pod can be configured to mount a shared memory volume. This volume is mounted to /vault/secrets, meaning the containerized app must read its secrets from files, losing the benefits of environment variable injection.

Third, for the injection to work, the user must add various Vault-specific secret injection annotations, for example:

vault.hashicorp.com/agent-inject-secret-foo: database/roles/app
vault.hashicorp.com/agent-inject-secret-bar: consul/creds/app
vault.hashicorp.com/role: 'app'

The first annotation will be rendered to /vault/secrets/foo with the second annotation rendered to /vault/secrets/bar.

And this brings operational overhead as you'll need to update all your Deployments' annotation. Also, you are vendor locked in now, and if you want to move to another secret manager, you'd have to rework all of your Deployment manifests.

3. Kubernetes Secrets Operators

A Kubernetes Operator is a specific type of application designed to extend the functionality of Kubernetes, such as the Doppler Secrets Operator and External Secrets Operator. Here, we'll use the External Secrets Operator to show how to add new secrets sync functionality to your Kubernetes Cluster.

The External Secrets Operator integrates external secret managers such as Doppler, AWS Secrets Manager, HashiCorp Vault, Google Secrets Manager, Azure Key Vault, and many more. The operator is configured to sync specific secrets from one or more Secrets Managers to various Kubernetes Secrets while continuously syncing any secret changes on a set interval.

The Kubernetes External Secrets Operator has many advantages over the previous methods:

• It supports multiple secrets managers with no vendor-lockin.
• It syncs secrets to plain old Kubernetes Secrets, so there is no need to update your Deployment manifests.

It does introduce additional operational overhead in that even though you don't need to maintain Kubernetes Secrets anymore, you do have to manage the new Kubernetes objects specific to the External Secrets Operator.

The External Secrets Operator works by using a SecretStore or ClusterSecretStore to provide authenticated access to a secrets manager and an ExternalSecret that references one of these stores and controls how and which secrets are synced to Kubernetes:

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: doppler-auth-api
spec:
  provider:
    doppler:
      auth:
        secretRef:
          dopplerToken:
            name: doppler-token-auth-api
            key: dopplerToken
-
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: auth-api-all
spec:
  secretStoreRef:
    kind: SecretStore
    name: doppler-auth-api
  target:
    name: auth-api-all
  dataFrom:
    - find:
        name:
          regexp: .*

This is much better than maintaining native Kubernetes Secrets YAML and is significantly more straightforward than the Secrets Agent/Sidecar Injection method.

We'll also be covering the Doppler Secrets Operator later in this article.

4. Secrets Store Container Storage Interface (CSI)

A relatively new and emerging method uses a Kubernetes Secrets Store CSI Driver, which allows secrets stored in secret managers to be mounted into pods as a volume. Once the Volume is attached to the Pod, the data is mounted into the container's file system.

While this seems similar to the Agent/Sidecar method where you'd still have to read files, Secrets Store CSI also supports synchronizing secrets as native Kubernetes Secrets with an extra SecretProviderClass custom resource, allowing for secrets to be injected as environment variables:

apiVersion: secrets-store.csi.x-Kubernetes.io/v1
kind: SecretProviderClass
metadata:
  name: my-provider
spec:
  # accepted provider options: Azure, Vault, or GCP
  provider: vault
  # [OPTIONAL] SecretObject defines the desired state of synced Kubernetes secret objects
  secretObjects:
  - data:
      # data field to populate
    - key: username
      # name of the mounted content to sync. this could be the object name or the object alias
      objectName: foo1
    # name of the Kubernetes Secret object
    secretName: foosecret
    # type of the Kubernetes Secret object e.g., Opaque, kubernetes.io/tls
    type: Opaque

This is the same level of operational overhead as the External Secrets Operator, as you need to manage one YAML per secret.

Auto-Updating Deployments When Secrets Change

Now that we've covered the most common methods for syncing secrets to Kubernetes, let's get to a significant pain point. When a secret value changes, how do you reload the Kubernetes Deployments consuming those secrets?

The Secrets Store CSI Driver can periodically update the Pod mount and Kubernetes Secret with the latest content from external secrets managers (e.g., auto rotation of database credentials). However, the CSI driver does not trigger a reload for the affected deployments—it only handles updating the Pod mount and Kubernetes Secret, similar to how Kubernetes updates secrets mounted as volumes.

Unfortunately, there isn't one out-of-the-box, elegant solution for triggering deployments to be reloaded with the solutions we've shown. You'd need to rely on open source tools such as Reloader, which watches for changes in ConfigMap and Secrets and does rolling upgrades on Pods with their associated Deployment, StatefulSet, DaemonSet, and DeploymentConfig.

Kubernetes Secrets Management Essentials for 2023 and beyond

It's now easier than ever to effectively manage Kubernetes secrets, and whether you're just about to move your workloads to Kubernetes or you're revisiting how to manage secrets, here's the list of features that I'd prioritize:

• Easy to use, intuitive secrets management experience.
• Easy to sync secrets to Kubernetes Secrets without a complex setup and installation process.
• Environment variables for application config and secrets.
• Low operational overhead that makes best practices easy to implement.
• Multi-cloud capabilities to avoid vendor lock-in and siloed secrets.
• Last but not least, the killer feature: automatically reload a Deployment/Pod when secrets change.

How the Doppler SecretOps Platform Could Change the Game

Because traditional secrets managers are just key-value stores (e.g. HashiCorp Vault and AWS Secrets Manager), most teams still have a frustrating experience when it comes to managing secrets.

Managing and syncing secrets must be as easy as possible because as friction and difficulty increase, so does the number of developers and teams that implement their own (and likely insecure) solution instead.

A Secrets Manager like this should've existed years ago, but now, a new kid is on the block: Doppler.

Simply put, Doppler is trying to make it as easy as it should be to manage secrets on Kubernetes thanks to the Doppler Secrets Operator. Easy to use? Check. Easy to sync secrets as Kubernetes Secrets? Check. Twelve-Factor App, cloud-native? Check. No overhead? Check.

And the big one: Automatic reload upon secrets update? Check! And it only takes a few minutes to install and configure:

If you're interested in trying Doppler, you can check out my quick tutorial.

Summary

The evolving Kubernetes Secrets landscape now provides development teams with many choices for storing, managing, syncing, and injecting secrets into containers.

I hope this post has given you some new ideas for improving your secrets management workflows in Kubernetes.