Intention / Goal

When using an Infrastructure-as-Code approach to populate your Kubernetes cluster, e.g. with Terraform, a question is how secrets are handled that need to be injected into the cluster (The words secrets, sensistive or confidential data are used interchangeably here). An example would be a database password which is needed by a backend instance to be able to operate.

Usually the IaC code is stored in a remote git repository (e.g. on GitHub or GitLab), which could potentially leak the password to the public when the password is stored as plain text in code. Even if your git repository is not publicly available, it will drastically multiply the radius of your secrets.

Additionally to the git problem, Terraform also uses a State file where it keeps track of the deployed resources. Kubernetes Secrets will also be included, hence the State file will leak your sensitive data, too, when remote state is used (Storing the state e.g. in GitLab or Amazon S3).

Finding the right solution to keep your unencrypted secrets out of the git repo (and out of any other systems than the cluster itself) can be a tediuous research process.

In this article we will present a solution, storing secrets as encrypted data in your git repo and only decrypting it inside of your Kubernetes cluster. We have test-driven this approach under various circumstances and are convinced that it is one of the best solutions for most applications.

Only a small subset of your team, like some Site Realiability Engineers should be able to decrypt the credentials locally.

SOPS

How does SOPS(-Operator) work from a meta perspective:

SOPS is used to encrypt / decrypt data of type YAML, JSON, ENV, INI and BINARY (source: Mozilla sops Github) by using mechanisms like AWS KMS, GCP KMS, Azure Key Vault, age, and PGP. The SOPS-Operator was inspired by this mechanism and extends it into Kubernetes clusters.

SOPS Operator takes a Kubernetes Custom Resource Definition (CRD) called SopsSecret as input, whose sensitive components are encrypted, decrypts these and creates (updates, deletes) usual Kubernetes Secret Resources. These resources in turn can be consumed by your Kubernetes entities (like Pods).

Here we will use age for encryption and decryption (see Age Section). How these CRDs look like and how they are encrypted / decrypted will be explained in How to / Step-by-Step Guide.

Why SOPS(-Operator)?

You might ask yourself: why do I need to encrypt my secrets / sensitive data, when everything is stored in a private repo and it is not planned to change anything here anytime soon?

Also, not the entire team is allowed to access this repo, but just a small part (e.g. SREs). Who really needs access to these files? A look in the past, where incidents did occur (Adafruit, Slack, Github), shows that you should best keep your attack surface small.

By using SOPS-Operator (Secrets OPerationS - Kubernetes Operator), we add one more layer of security and also one more fine-grained control to our system. In addition we are able to hide sensitive information / secrets from cloud providers. Still everybody should be able to access a source code repository and use it normally.

Warning: Keep in mind that persons with cluster access can still (if not further restricted) list the clusters secrets and use them.

What encryption to use with SOPS?

AWS KMS, GCP KMS, Azure Key Vault can be used with sops, but they are not open source like age and tie your implementation to one provider only. Additionally, secrets in Vaults are handled centrally and do not live alongside your code. Hence, under some circumstances manipulations in the Vault need to be synced with code deployment. This is not a problem when using file-based encryption.

And, last but not least, Vaults usually do cost a small amount of money while file-based encryption is for free.

So GPG and age remain. Whereas in the past GPG was used to encrypt the entire repo and distribute the keys of each team member, it misses ease of use or at least it has room for improvement. GPG is a mature solution and is still widely used. But it is not very lightweight, due to the possibility to use it in a lot of scenarios like mail etc.

Flaws in / with GPG

GPG is the open-source alternative of Symantec´s PGP and has been on the market for more than 20 years. It is used on a variety of topics, and hence is prone to be being targeted. Of course GPG is constantly improved and secured, but however, the following security vulnerabilities exist(ed): Gnupg, Symantec PGP Desktop, Symantec PGP Universal Server.

AGE

Along comes age, which is written in GO and was invented by a Google Engineer in 2019. It tries to learn from past mistakes of other tools like GPG. At the same time, it tries to be very compact, lightweight and concise. This keeps the attack surface as small as possible. Hereby, it follows the typical UNIX approach solving only one single problem.
Age is a secure process for encrypting / decrypting any data regardless of the filetype.

To support its credibility, Mozilla recommends AGE over PGP.

Of the two available encryption approaches for age, asymmetric encryption based on X25519 and passphrase encryption type based on scrypt (see AGE Specification) are offered. For SOPS asymmetric encryption via X25519 is used, which is a mechanism that can also be choosen with e.g. ssh.

First of all a key-pair is need, which can be created with age-keygen (comparable to ssh-keygen) that creates a public and private key pair.

In Age the private key is called Identity. It allows to decrypt a file encrypted to its corresponding Recipient. The Recipient is like the public key of ssh that files can be encrypted to (see:
age man page)

The Identity / private key starts with AGE-SECRET-KEY- and must be kept secret / private, like your ssh private key.

The counterpart is the Recipient / public key that is written as a comment line in the file generated by age-keygen. It starts with 'age' and defines where a file is encrypted to.

The Password Vault, as shown in the image could be 1Password or similar. It is good practive that users should store the created key-pair file / AGE-SECRET-KEY in a password vault. In this example only an admin team is allowed to access the AGE Secret key file in an admin vault (blue rectangle) and other means to access the cluster itself.

Now SOPS / SOPS-Operator come into play. As explained in How does SOPS(-Operator) work from a meta perspective, SOPS-Operator is inspired by SOPS and is used to manage Kubernetes Secret Resources.

You first need to encrypt your sensitive information locally, like shown in the picture, using the AGE recipient Public Key and the unencrypted SopsSecret as input to sops. Sops hands this to the AGE process, which outputs the encrypted SopsSecret. The encrypted SopsSecret must then be supplied to your cluster.

If you receive a SopsSecret with encrypted keys, the decryption flow is quite similar, but instead of the public key the Identity of the AGE-SECRET-KEY file is used.

When we look at the cluster (see image above), the process is comparable. The cluster receives an encrypted SopsSecret Custom Resource Definition as input (see top left of the image). This SopsSecret CRD is created locally (it can be written as kubernetes_manifest with terraform). It has the same structure as usual Kubernetes Secret Resources. The values are the plain secrets, which are in the next step encrypted using SOPS. This encrypted manifest file is then applied to your cluster, via kubectl or e.g. terraform (it has to be converted to a terraform kubernetes_manifest Resource first in this case). Afterwards it is available in your cluster and the sops-secrets-operator Pod listens to state change (new or adjusted SopsSecret-Crd). It takes this CRD together with the sops-age-key-file Secret as input and decrypts the encrypted values via age, using the Identity from the sops-age-key-file. As a next step it creates a Kubernetes Secret from this SopsSecret CRD, so it can be consumed by other pods.

Data is hereby decrypted as close as possible to the resource that is using it (like the database password of your PostgreSQL database or similar). This resource does not need to know anything about SOPS, since it can consume the password via the Kubernetes Secret resource, as usual.

We now have established a solution that does not carry sensitive, plain secrets in any parts of the system where they shouldn´t be.

A typical workflow could look like it can be seen in the image above. You locally create an encrypted SopsSecret and push this to your git remote repository like Gitlab or Github. The pipeline picks up your changes and deploys this secret to your cluster.

How to / Step-by-Step Guide

Prepare your cluster and install SOPS-Operator

1. Encode your secret in base64 format: sed -n -e '/^AGE-SECRET-KEY/p' PATH_TO_YOUR_AGE_KEY_FILE | base64
2. Supply a secret (age identity) to your cluster (Needed to decrypt the SopsSecret CRDs)

cat <<EOF | kubectl apply -f -
apiVersion: v1
data:
  key: YOUR_BASE64_ENCODED_IDENTITY_PRIVATE_KEY
kind: Secret
metadata:
  name: sops-age-key-file
  namespace: YOUR_NAMESPACE
type: Opaque
EOF

Here we do not use Terraform on purpose in order to not make the secret part of your terraform state, as this state would leak your secrets (You can also generate a kubernetes_secret, but some regulatory might force you to not leak this secret to your cloud provider).

1. Add the sops-secrets-operator to your cluster, using helm (Alternative below):

helm repo add sops https://isindir.github.io/sops-secrets-operator/
helm upgrade --install --create-namespace sops sops/sops-secrets-operator --namespace YOUR_NAMESPACE

Or use e.g. Terraform:

resource "helm_release" "sops-secrets-operator" {
  name             = "sops-secrets-operator"
  chart            = "sops-secrets-operator"
  repository       = "https://isindir.github.io/sops-secrets-operator/"
  namespace        = "YOUR_NAMESPACE"
  version          = "0.14.0"
  create_namespace = true

  values = [
    <<EOF
extraEnv:
- name: SOPS_AGE_KEY_FILE
  value: /etc/sops-age-key-file/key
secretsAsFiles:
- mountPath: /etc/sops-age-key-file
  name: sops-age-key-file
  secretName: sops-age-key-file
EOF
  ]
}

Create SopsSecret & Deploy it to your cluster

1. Install sops, age and optionally tfk8s, if you want to use terraform, via a package manager. Here we use brew:

brew install \
sops \
age \
tk8s

1. Generate a public-private-keypair: age-keygen -o age-key.txt. [Optional: Store the age-key.txt in a password vault, like 1Password. We highly recommend to store it somewhere save.]
   1. Encrypt your secrets that should be used in your cluster

sops --encrypt --age 'YOUR_AGE_RECIPIENT_PUBLIC_KEY_STARTING_WITH_age' --encrypted-suffix Templates SOPS_SECRET_FILE_YOU_WANT_TO_ENCRYPT.yml > SOPS_SECRET_FILE_ENCRYPTED.yml.enc

apiVersion: isindir.github.com/v1alpha3
kind: SopsSecret
metadata:
  name: your-secrets-name
  namespace: YOUR_NAMESPACE
spec:
  secretTemplates:
    - name: your-secrets-name
      stringData:
         NAME_OF_YOUR_SECRET: SECRET_ITSELF
         NAME_OF_ANOTHER_SECRET: SECRET_ITSELF

1. Deploy this SopsSecret to your infrastructure via

  kubectl apply -f SOPS_SECRET_FILE_ENCRYPTED.enc.yml

or if you want to use terraform, you can convert it with

cat SOPS_SECRET_FILE_ENCRYPTED.yml.enc | tfk8s --strip -o your-secrets-name.tf

This produces a kubernetes_manifest resource. Afterwards you can use terraform plan and terraform apply as you would normally do or use your CI / CD pipeline to do so.

1. With the following Terraform datasource you are able to reference the secrets in other resources. Watch out for the values of your keys in the binary_data, they should be empty as they are populated by terraform itfself, without storing these inside your terraform state file .

data "kubernetes_secret" "your-secrets-name" {
  metadata {
    name      = "your-secrets-name"
    namespace = "YOUR_NAMESPACE"
  }

  binary_data = {
    NAME_OF_YOUR_SECRET                               = ""
}

You can e.g. define an output from it:

output "NAME_OF_YOUR_SECRET" {
  value       = base64decode(data.kubernetes_secret.your-secrets-name.binary_data.NAME_OF_YOUR_SECRET)
  sensitive   = true
  description = "FooBar"
}

and use this output in any other module module.secrets.NAME_OF_YOUR_SECRET.

1. Check if everything went fine, by checking if the Kubernetes Secret Resource was created: kubectl get secrets your-secrets-name -n YOUR_NAMESPACE -o yaml . If the output is No resources found in YOUR_NAMESPACE namespace., something went wrong. So best is to consult the logs of the operator first: kubectl logs -l app.kubernetes.io/name=sops-secrets-operator -n YOUR_NAMESPACE

Decrypt and edit SopsSecret

1. If you want to edit existing secrets that one of your colleagues did encrypt, then get the secret identity key file from your teams password vault.
2. Specify the location of your key file: export SOPS_AGE_KEY_FILE=./key.txt.
3. Decrypt the file sops -d SOPS_SECRET_FILE_ENCRYPTED.yml.enc.
4. Edit the contents.
5. Repeat the steps to encrypt and deploy the secret again.

Conclusion

Handling secrets in a safe way is one of the hardest tasks in software development.

Using SOPS Operator with age allows you to completely free your code versioning repositories, CICD pipelines and other systems involved in the deployment of Kubernetes clusters from secrets.

Our approach still uses whichever workflow you are used to deploy and does not interfer with any tooling you might be using for these steps. The usual Kubernetes Secret resources can be used within the cluster, enabling a high compatibility with existing mechanisms.