TheLazy.Dev
Portfolio GitHub

Blog

sops |kubernetes |helm |vscode

Goodbye Sealed Secrets, hello SOPS

docteurrs

Emilien Lancelot

Posted on July 30, 2022

Goodbye Sealed Secrets, hello SOPS

This tutorial will provide you with all the steps and commands to setup SOPS in your shell, Kubernetes, Helm and Visual Studio Code.

We assume that you are on Windows and using WSL as Linux environment. We will setup Sops in both.

Want to dive right in ? Click here to jump directly to the installation section.

What is SOPS

Sops is a binary able to encrypt configuration files. But rather than encrypting the whole file Sops understands format (JSON, YAML, INI, etc) and will only encrypt the values of each line (in a key/value pair).

The aim is to store encrypted configurations containing sensitive information into your versionning system (Git, SVN, etc) and then be able to work with these encrypted files effortlessly.

Sops provide the following advantages :

  1. Encrypting values but not keys
    • Preserves file structure even when encrypted
  2. Safely saving encrypted files to Git
  3. Editing encrypted files seemlessly
    • Sops will open the unencrypted version of the file in your favorite text editor
    • When saving, the file will be automatically reencrypted with its new content
  4. Encrypting specific keys and not the whole file
    • Perfect for long configuration files that have few pieces of sensitive information inside.
  5. Working with all configuration files and not only Kubernetes secrets
    • Can encrypt your Helm’s values.yaml files to safely store sensitive pieces of data inside it (passwords, etc)
  6. No backend in Kubernetes needed
    • The only requirement is to install Sops locally
    • Managing encrypted files will be done locally
  7. Plugins are available for all major tools to support working with Sops

Sops encryption example

Let’s say we have this classic Kubernetes secret. The secret value is not encrypted but only encoded to base64.



apiVersion: v1
data:
  secret_value: U29wcyBydWxlcw== 📖
kind: Secret
metadata:
  creationTimestamp: null
  name: my-secret
  namespace: default
Enter fullscreen mode Exit fullscreen mode

After being given to Sops this is what you get:



apiVersion: ENC[AES256_GCM,data:PFk=,iv:,[...]tag:G9mg==,type:str]🔒
data:
    secret_value: ENC[AES256_GCM,data:Ex2KhoAlCG4w==,[...]🔒
kind: ENC[AES256_GCM,data:eW1JrnrV,[...]🔒
metadata:
    creationTimestamp: null
    name: ENC[AES256_GCM,data:fkWXVtQS/C62,iv:GCTfamtu4ixP4w=[...]🔒
    namespace: ENC[AES256_GCM,data:J96kuhjMUA==,iv:QZMnvDGqX[...]🔒
sops:
[...]
Enter fullscreen mode Exit fullscreen mode

As you can see, all the values have been encrypted. But the keys are still in plain text.

Sops can also encrypt specific values based on keys that you give it.
Here, let’s say we want to ONLY encrypt the “secret_value” key :



apiVersion: v1📖
data:
    secret_value: ENC[AES256_GCM,data:9VBm+32QSCOaU+9exohSg==[...]🔒
kind: Secret📖
metadata:
    creationTimestamp: null
    name: my-secret📖
    namespace: default📖
sops:
[...]
Enter fullscreen mode Exit fullscreen mode

Only the “secret_value” value has been encrypted. The rest of the file remains untouched.

Why we stopped using Sealed Secrets

What does it do ?

We have been using SealedSecrets for a year but it’s time for a change. This (cool) technology allows you to encrypt a Kubernetes secret on disk and have it commited on GIT. This way, even if a hacker gets its hands on your repository, he won’t have access to the content of your secrets.

Why it’s not perfect ?

SealedSecret is great for a lot of usecase. But in our opinion it does come short here and there:

  • Must install a backend in the cluster
  • Certificate rotation might get complicated
  • Default encryption mode doesn’t allow to move secret from one namespace to another
  • On the Kubernetes cluster the generated secret is still in plain text But the most problematic part is…

We have sensitive data stored in plain text inside helm charts, for instance passwords inside the “values.yaml” of helm charts. This is not covered at all by sealed secrets.

Installation of Sops

Install the Sops binary on WSL

Download the binary for linux : https://github.com/mozilla/sops/releases



$ wget <URL_OF_THE_BIN_FROM_GITHUB>

$ sudo mv sops-<VERSION> /usr/local/bin/sops
Enter fullscreen mode Exit fullscreen mode

Install Age

Age tech logo

Sops doesn’t do the encryption itself. It leverages other encryption technologies like PGP, AGE, GCP KMS, Azure key vault, Hashicorp Vault, etc

As PGP is kind of deprecated, we’ll install the new kid on the block : Age

Age is a tool to encrypt files. It doesn’t care about the format of the file itself. To encrypt only the values or parts of a file we need to rely on Sops. Age is only a tool that Sops will use.

Let’s install Age…



$ AGE_VERSION=$(curl -s "https://api.github.com/repos/FiloSottile/age/releases/latest" | grep -Po '"tag_name": "(v.*)"' |grep -Po '[0-9].*[0-9]')

$ curl -Lo age.tar.gz "https://github.com/FiloSottile/age/releases/latest/download/age-v${AGE_VERSION}-linux-amd64.tar.gz"

$ tar xf age.tar.gz

$ sudo mv age/age /usr/local/bin

$ sudo mv age/age-keygen /usr/local/bin
Enter fullscreen mode Exit fullscreen mode

Generate public and private keys

Now that Age is installed, you must create a public and a private key.



$ age-keygen -o key.txt

$ cat key.txt
# created: 2022-07-30T17:02:43+02:00
# public key: age1rdje4gwnm2cc6uu6lvggzjj8gktu4cw5ng8yyjhrluwvqq387cls58dsy4
AGE-SECRET-KEY-18LMDYJRYZMRM52CYQA9MZW79M85CPR0HJHCASXF5ADT03KJ290HS88599E

$ mkdir ~/.sops

$ mv ./key.txt ~/.sops
Enter fullscreen mode Exit fullscreen mode

▶️The private and public keys created by Age are stored inside the file key.txt. We moved this file inside a .sops directory inside the home folder.

To tell Sops where everything is, add these ENV variables to your shell configuration (~/.bashrc or ~/.zshrc)



export SOPS_AGE_KEY_FILE=$HOME/.sops/key.txt
export EDITOR=nano # Replace by your favorite editor
Enter fullscreen mode Exit fullscreen mode

ℹ️ Dont forget to source your shell profile before continuing or at least open a new shell.

Testing Sops and Age are working

Let’s encrypt a file…



# Creating a test file
$ echo "my-key: my-value" > config.yaml

# Encrypting the file
sops --encrypt --age $(cat ~/.sops/key.txt |grep -oP "public key: \K(.*)") ./config.yaml
Enter fullscreen mode Exit fullscreen mode

▶️ we used the --age parameter to specify what public key to use. Here, we rely on bash to replace during runtime the public key present in key.txt.

Now to save the file to the disk a simple right redirection will do the trick :



$ sops --encrypt --age $(cat ~/.sops/key.txt |grep -oP "public key: \K(.*)") ./config.yaml > config_enc.yaml

$ cat config_enc.yaml
my-key: ENC[AES256_GCM,data:Jw1oRQcV8=,tag:Tzd/oUQ[...]==,type:str]
sops:
[...]
Enter fullscreen mode Exit fullscreen mode

Edit the file using Sops 



$ sops config_enc.yaml
<Your favorite editor should have opened>
<The file in front of you should be unencrypted>
<When saving, modifications you made will be encrypted>
Enter fullscreen mode Exit fullscreen mode

On the fly decryption and re-encryption is a fabulous feature. You don’t even know the file is encrypted. It’s all automatic !

Configure the tools

Add aliases to your interpretor

This should work effortlessly on bash and zsh.

As the syntax to encrypt a file is bit tedious (due to the fact that you must pass the Age public key to Sops) we have prepared an alias that can be added to your .bashrc or .zshrc



function cypher {
    filename=$(basename -- "$1")
    extension="${filename##*.}"
    filename="${filename%.*}"
    sops --encrypt --age $(cat ~/.sops/key.txt |grep -oP "public key: \K(.*)") $2 $3 $1 > "$filename.enc.$extension"
}
Enter fullscreen mode Exit fullscreen mode

Remember to always source your profile before trying new aliases.

Usage:



# Creating test files
$ echo "some_key: some_value" > test_alias.yaml

# Encrypting
$ cypher test_alias.yaml

$ ls
test_alias.enc.yaml
Enter fullscreen mode Exit fullscreen mode

You could also update the alias to edit the file in place and not generate a new one. This should look like this :



function cypher_inplace {
    sops --encrypt --in-place --age $(cat ~/.sops/key.txt |grep -oP "public key: \K(.*)") $2 $3 $1
}
Enter fullscreen mode Exit fullscreen mode

To encrypt only parts of a file you must explicit the keys using a regex. You won’t need to be a regex master to achieve this. Check out this example:



$ echo "key1: value1" >  test_regex.yaml
$ echo "key2: value2" >> test_regex.yaml
$ echo "key3: value3" >> test_regex.yaml

$ cat test_regex.yaml
key1: value1
key2: value2
key3: value3

$ cypher test_regex.yaml  --encrypted-regex='^(key1|key3)$'

$ cat test_regex.enc.yaml
key1: ENC[AES256_GCM,[...]tag:qHy9z22CIxcb9ykpPAWikQ==,type:str]
key2: value2
key3: ENC[AES256_GCM,data:3Faqn,[...]tag:1VB3/wrLekmg==,type:str]
Enter fullscreen mode Exit fullscreen mode

As you can see, only the values for key1 and key3 have been encrypted. key2 has been left untouched.

You can add any keys you want separated by | . For instance --encrypted-regex='^(<a_first_key>|<a_second_key>|<a_third_key>)$

Installing Sops for Visual Studio Code

As we have seen, Sops allows to edit encrypted file seemlessly. Visual Studio Code can do the same, provided it has a Sops plugin.

Download the Sops binary for Windows (or whatever OS your VS code runs on)

  • https://github.com/mozilla/sops/releases Add the binary into your OS path
  • On Windows that would be c:\Windows\system32
  • Rename it to sops.exe On VScode, download the Sops plugin @signageos/vscode-sops.
  • You shouldn’t have to, and yet, to configure the plugin in VScode go to file > preferences > settings > plugins > Sops
  • Copy the keys that were setup on linux (WSL) to your Windows partition: ```

$ mkdir /mnt/c/Users//AppData/Roaming/sops/age -p

$ cp ~/.sops/key.txt /mnt/c/Users//AppData/Roaming/sops/age/keys.txt

▶️ The path `AppData/Roaming/sops/age` is where VSCode expects your keys to be. It can be configured, in the plugin settings if needed.

### Installing Sops for Helm

A great Sops advantage is that you can leave plain text sensitive data inside the `values.yaml` of your helm chart and simply let Sops encrypt the file.

❗ Therefore the file cannot be used with helm directly as it won’t natively be able to read encrypted files…
▶️️ This is why you should install this Helm plugin. It allows to use all the classic helm commands but when encountering an encrypted file, it will automatically decrypt it and continue the installation process.
Enter fullscreen mode Exit fullscreen mode

$ helm plugin install https://github.com/jkroepke/helm-secrets --version v3.14.0

To use this plugin, you simply have to add the word `secrets` after the helm binary.
For instance when installing a helm chart with an encrypted values.yaml :
Enter fullscreen mode Exit fullscreen mode

$ helm secrets install minecraft-server itzg/minecraft

Installing Sops for Kubectl
There isn’t any good solution to mix kubectl and sops for now. But piping the output of Sops to kubectl apply is quite simple.
Enter fullscreen mode Exit fullscreen mode

$ sops -d .yaml | kubectl apply -f -


Thank you for following this tutorial. If it helped you setup your Sops installation please consider giving a 👏.

Written by [Emilien Lancelot](www.linkedin.com/in/emilien-lancelot-62943389)

[Stack Overflow](https://stackoverflow.com/users/5512455/doctor)

In need of Function As A Service on Kubernetes ? Say no more ! [Checkout Gitfaas now !](https://github.com/rememberSoftwares/gitfaas) Free for all, opensource for everyone.
Enter fullscreen mode Exit fullscreen mode
💖 💪 🙅 🚩
docteurrs
Emilien Lancelot

Posted on July 30, 2022

Join Our Newsletter. No Spam, Only the good stuff.

Sign up to receive the latest update from our blog.

Related

Goodbye Sealed Secrets, hello SOPS
sops Goodbye Sealed Secrets, hello SOPS

July 30, 2022