Helm: helm-secrets — sensitive data encryption with AWS KMS and use it from Jenkins

Helm: helm-secrets — sensitive data encryption with AWS KMS and use it with Jenkins

So, as a follow-up to the Helm: Kubernetes package manager — an overview, getting started post — let’s discuss about sensitive data in our Helm charts.

What I want is to store a chart files in a repository, but even if such a repo will be a private Github repo — I still don’t want to store passwords in a plaintext way there.

The solution here might be to use a well-known helm-secrets plugin. The plugin itself is just a collection if bash-scripts using Mozilla SOPS (SOPS - Secrets OPerationS), and is a wrapper around helm secrets command.

It allows us to encrypt strings in a specified file using GPG/AWS KMS/GCP KMS keys and decrypt such a data on the fly to embed it as a usual value in the same way as we doing it using the common values.yaml file.

So, in this post, we will install the plugin on Arch Linux, will create an AWS KMS key, then will see how to encrypt/decrypt data in a Helm chart, and then will use it in a Jenkins job which is used in my job’s project to deploy a real working application.

Did a nice type during writing this post, btw:

The helm-secrets plugin install

Actually, the installation must be done just by typing “helm plugin install", but its installation script a bit awkward and may not work properly on some operating systems.

helm-secrets & sops on Arch Linux

For example, on my Arch Linux first I got permissions issue:

$ helm plugin install [https://github.com/futuresimple/helm-secrets](https://github.com/futuresimple/helm-secrets)
which: no dpkg in (/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/lib/jvm/default/bin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl:/home/setevoy/go/bin)
mv: cannot create regular file ‘/usr/local/bin/sops’: Permission denied
Error: plugin install hook for “secrets” exited with error

And then again permissions error, and anyway I have no willingness to install it to the /root directory using the sudo:

$ sudo helm plugin install [https://github.com/futuresimple/helm-secrets](https://github.com/futuresimple/helm-secrets)
which: no dpkg in (/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/lib/jvm/default/bin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl:/home/setevoy/go/bin)
/root/.local/share/helm/plugins/helm-secrets/install-binary.sh: line 62: /tmp/sops: Permission denied
Error: plugin install hook for “secrets” exited with error

So — just install the sops package from AUR:

$ yay -S sops

Or, if you are macOS user — use the brew:

$ brew install sops

Then repeat the helm plugin install https://github.com/futuresimple/helm-secrets command and check the documentation - let's take a look at what it installed for us.

It has to use the $XDG_DATA_HOME/helm/plugins directory, and $XDG_DATA_HOME has to be pointed to the $HOME/.local/share location:

$ ll $HOME/.local/share/helm/plugins/
total 4
lrwxrwxrwx 1 setevoy setevoy 76 May 15 08:30 helm-secrets -> /home/setevoy/.cache/helm/plugins/https-github.com-futuresimple-helm-secrets

And the plugin’s directory content is the following:

$ ls -l /home/setevoy/.cache/helm/plugins/https-github.com-futuresimple-helm-secrets/
total 68
-rw-r — r — 1 setevoy setevoy 11337 May 15 08:30 LICENSE
-rw-r — r — 1 setevoy setevoy 20057 May 15 08:30 README.md
drwxr-xr-x 4 setevoy setevoy 4096 May 15 08:30 example
-rwxr-xr-x 1 setevoy setevoy 2346 May 15 08:30 install-binary.sh
-rw-r — r — 1 setevoy setevoy 338 May 15 08:30 plugin.yaml
-rwxr-xr-x 1 setevoy setevoy 12388 May 15 08:30 secrets.sh
-rwxr-xr-x 1 setevoy setevoy 4621 May 15 08:30 test.sh

Here are the bash-scripts mentioned above and in the Moving parts of project documentation.

Also, we can inspect the plugin.yaml file to see which actions will be performed by the plugin:

$ cat /home/setevoy/.cache/helm/plugins/https-github.com-futuresimple-helm-secrets/plugin.yaml
name: “secrets”
version: “2.0.2”
usage: “Secrets encryption in Helm for Git storing”
description: |-
This plugin provides secrets values encryption for Helm charts secure storing
command: “$HELM_PLUGIN_DIR/secrets.sh”
useTunnel: true
hooks:
install: “$HELM_PLUGIN_DIR/install-binary.sh”
update: “$HELM_PLUGIN_DIR/install-binary.sh”

command: "$HELM_PLUGIN_DIR/secrets.sh" - well, yes - the secrets.sh is the main working script of the plugin.

You can also check it’s content — but it’s really big enough:

$ cat /home/setevoy/.cache/helm/plugins/https-github.com-futuresimple-helm-secrets/secrets.sh | wc -l
534

Encryption configuration

To find which keys are used for encryption and decryption process — the.sops.yaml file will be used, which is stored in the root of your chart's directory.

Actually, this file is used by the SOPS utility, so you can find it’s configuration examples in the Using .sops.yaml conf to select KMS/PGP for new files doc.

Now we can go ahead and create an AWS KMS key which will be used to protect our data.

AWS KMS — a key creation

If you didn’t use the AWS KMS service before — check its docs here — https://aws.amazon.com/ru/kms/getting-started.

Go to the AWS Console and create a new, symmetric, key (see the Using symmetric and asymmetric keys):

Set its Administrator:

Now, we need to add it’s users.

As I’ll use the key from my Jenkins instance which is running on an AWS EC2, and this EC2 instance has the AWS EC2 Instance Profile attached to authenticate for the AWS services — then I need to add its IAM Role as a User for the key.

Go to the Jenkins EC2 and find its IAM Role (at the bottom of the screenshot below):

Attach it to the Users of the key we are creating:

Confirm the policy generated:

By the way — pay attention here:

...
            "Principal": {
                "AWS": "arn:aws:iam::534***385:root"
            },
            "Action": "kms:*",
            "Resource": "*"
...

The “arn:aws:iam::534***385:root” ARN refers to the all the users of the AWS account, as I remember from the AWS Elastic Kubernetes Service: RBAC-авторизация через AWS IAM и RBAC группы post (in Russian only for now), so keep it in mind.

Press the Finish button and the key is ready:

The .sops.yaml config

Now, go to your chart’s directory and create the .sops.yaml file to configure the key used for our secrets.

Let’s use the only one default rule here to be applied for any secrets file with the AWS KMS key we created above:

---
creation_rules:
  - kms: 'arn:aws:kms:eu-west-1:534***385:key/620b89fe-***-25b435611e8b'

A secrets file

Next, create a new file named secrets.yaml.

Here is a small trap — the script will look for the secrets.*.yaml files by default, see the Usage and examples, and you can’t just use a name like “my-passwords.yaml” at least without some additional configuration:

By convention, files containing secrets are named secrets.yaml, or anything beginning with “secrets.” and ending with “.yaml”. E.g. secrets.test.yaml and secrets.prod.yaml.

So, for example, I have the following strings in my existing values.yaml - and here is a "password" key with a plaintext value "pass":

...
image:
  registry: "docker.io"
  username: "bttrm"
  password: "pass"
  repository: "bttrm"
  name: "bttrm-apps"
  tag: "120"
...

Cut the image.password from the values.yaml and move it to the secrets.yaml:

image:
  password: "pass"

Or, to make this example more clear and simple — let’s also add a new key and value to this secrets.yaml:

test:
    secret: testecret

And then create a manifest for Kubernetes Secrets, in my case all the Secrets are described in the bttrm-apps-backend/templates/bttrm-apps-secrets.yaml file, where bttrm-apps-backend is the chart's directory:

---
apiVersion: v1
kind: Secret
metadata:
  name: test-secret
type: Opaque
stringData:
  example-secret: {{ .Values.test.secret }}

As you can see — in the stringData field we are using the .Values as usually for the common values.yaml file.

helm secrets - AccessDeniedException

Now, we can encrypt the file using the helm secrets enc command:

$ helm secrets enc secrets.yaml
Encrypting secrets.yaml
Could not generate data key: [failed to encrypt new data key with master key “arn:aws:kms:eu-west-1:534***385:key/620b89fe-8365–45a6-aad6–25b435611e8b”: Failed to call KMS encryption service: AccessDeniedException:
status code: 400, request id: 699332aa-492b-47b4-b9ab-50f83dbcc2a4]
Error: plugin “secrets” exited with error

Ah, okay.

The issue is obvious enough — the SOPS is trying to access the AWS KMS key specified in the .sops.yaml config using the default AWS CLI profile from the ~/.aws/credentials, where I have my personal AWS account configured, while the key is located in my work account.

So, in this case, you can specify a profile using the aws_profile option in the .sops.yaml:

---
creation_rules:
  - kms: 'arn:aws:kms:eu-west-1:534***385:key/620b89fe-***-25b435611e8b'
    aws_profile: 'arseniy'

But then the same account will be used from a Jenkins job.

Another way is to use the $AWS_PROFILE environment variable:

$ AWS_PROFILE=arseniy

Repeat the secrets.yaml file encryption in the bttrm-apps-backend directory:

$ helm secrets enc bttrm-apps-backend/secrets.yaml

Check its content now:

image:
    password: ENC[AES256_GCM,data:1t7 .. PyY,iv:9mzTyB ... 9KG7+Hg=,tag:6KJ ... gvA==,type:str]
test:
    secret: ENC[AES256_GCM,data:l4c ... vCy,iv:9riw ... 0fvQ=,tag:G3p ... oSw==,type:str]
sops:
    kms:
    - arn: arn:aws:kms:eu-west-1:534***385:key/620b89fe- ... -25b435611e8b
        created_at: '2020-05-15T06:33:44Z'
        enc: AQICAHj9F0HBsgu ... kb6GOxJkiOMZSSxOtA==
...

As you can see, the values are encrypted now, and then there is information for the SOPS about configuration applied during the encryption process.

You can see the initial data using the secrets view command:

$ helm secrets view bttrm-apps-backend/secrets.yaml
image:
password: pass
test:
secret: testecret

And edit the file with the secrets edit:

$ helm secrets edit bttrm-apps-backend/secrets.yaml

And decrypt with the secrets dec command - this will create a new secrets.yaml.dec file near the original secrets.yaml.

Test deploy

To deploy the chart with the encrypted data — call helm with the install or upgrade as usual, or as in my case with the Jenkins job (will speak about it a bit later) - upgrade --install, but now add the secrets command - helm will execute the secrets.sh script which will perform all necessary operations.

In the example below I’m installing the bttrm-apps-backend/ chart as a bttrm-apps-backend release using the values.yaml and secrets.yaml files as the source of values (actually - you can skip the default values.yaml file and specify the only secrets.yaml as an additional file):

$ helm secrets upgrade — install bttrm-apps-backend bttrm-apps-backend/ -f bttrm-apps-backend/values.yaml -f bttrm-apps-backend/secrets.yaml
Release “bttrm-apps-backend” has been upgraded. Happy Helming!
…

Here might be interesting to check the processes started during the encryption process after you executed the helm secrets upgrade:

$ ps aux | grep helm
setevoy 74906 0.0 0.3 1270888 49984 pts/1 Sl+ 17:05 0:00 helm secrets upgrade — install — namespace bttrm-apps-dev-1-ns — create-namespace — atomic bttrm-apps-backend bttrm-apps-backend/ -f bttrm-apps-backend/values.yaml -f bttrm-apps-backend/secrets.yaml
setevoy 74914 0.0 0.0 4436 3672 pts/1 S+ 17:05 0:00 bash /home/setevoy/.local/share/helm/plugins/helm-secrets/secrets.sh upgrade — install — create-namespace — atomic bttrm-apps-backend bttrm-apps-backend/ -f bttrm-apps-backend/values.yaml -f bttrm-apps-backend/secrets.yaml
setevoy 74963 0.8 0.5 1271768 96876 pts/1 Sl+ 17:06 0:00 helm upgrade bttrm-apps-backend bttrm-apps-backend/ — install — create-namespace — atomic -f bttrm-apps-backend/values.yaml -f bttrm-apps-backend/secrets.yaml.dec

E.g:

we ran the helm secrets upgrade
it called the the bash /home/setevoy/.local/share/helm/plugins/helm-secrets/secrets.sh upgrade
which executed the helm upgrade

Now — check the secret in the Kubernetes:

$ kubectl -n bttrm-apps-dev-1-ns get secret test-secret -o yaml
apiVersion: v1
data:
example-secret: dGVzdGVjcmV0
…

And value from the dGVzdGVjcmV0 base64-encoded string is:

$ echo dGVzdGVjcmV0 | base64 — decode
testecret

Go to the Jenkins job.

Jenkins

Now, I need to embed the things above to a Jenkinsfile, which I wrote in the Helm: пошаговое создание чарта и деплоймента из Jenkins post (also still in Russian, unfortunately) — the script will build a Docker image and will deploy the bttrm-apps-backend chart as bttrm-apps-backend release to a Dev-cluster.

If delete the release now:

$ helm -n bttrm-apps-dev-1-ns delete bttrm-apps-backend
release “bttrm-apps-backend” uninstalled

And run the Jenkins job it obviously will fail as helm will not be able to find the data which we moved from the values.yaml into the secrets.yaml file:

+ helm upgrade — install bttrm-apps-backend bttrm-apps-backend-20.tgz

Release “bttrm-apps-backend” does not exist. Installing it now.

Error: unable to build kubernetes objects from release manifest: error validating “”: error validating data: [unknown object type “nil” in Secret.stringData.backend-apple-cert-passphrase, unknown object type “nil” in Secret.stringData.backend-apple-sigin-key-id, unknown object type “nil” in Secret.stringData.backend-db-password, unknown object type “nil” in Secret.stringData.backend-private-key, unknown object type “nil” in Secret.stringData.backend-secret]

So to make it great again working again need to add the SOPS tool and the helm-secrets plugin to my Docker image used for the build, and then update the stage("Helm install") in the script to add the secrets command and -f secrets.yaml.

Jenkins Docker build-image

Currently, my Docker image is built using the next Dockerfile:

FROM bitnami/minideb:stretch

RUN apt update && install_packages ca-certificates wget
RUN install_packages curl python-pip python-setuptools jq git

RUN curl -LO [https://storage.googleapis.com/kubernetes-release/release/`curl](https://storage.googleapis.com/kubernetes-release/release/%60curl) -s [https://storage.googleapis.com/kubernetes-release/release/stable.txt`/bin/linux/amd64/kubectl](https://storage.googleapis.com/kubernetes-release/release/stable.txt%60/bin/linux/amd64/kubectl)
RUN chmod +x ./kubectl
RUN mv ./kubectl /usr/local/bin/kubectl

WORKDIR /tmp
RUN curl --location "https://github.com/weaveworks/eksctl/releases/latest/download/eksctl_$(uname -s)_amd64.tar.gz" | tar xz -C /tmp
RUN mv /tmp/eksctl /usr/local/bin

RUN pip install ansible boto3 awscli

WORKDIR /tmp
RUN curl -fsSL -o get_helm.sh [https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3](https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3)
RUN /bin/bash get_helm.sh

USER root

Add the plugin install — there is the bitnami/minideb:stretch image used, so I hope that SOPS will be installed without any issues as it was with my Arch Linux:

...
RUN curl -fsSL -o get_helm.sh [https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3](https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3)
RUN /bin/bash get_helm.sh
RUN helm plugin install [https://github.com/futuresimple/helm-secrets](https://github.com/futuresimple/helm-secrets) 

USER root

Try to build:

$ docker build -t bttrm/kubectl-aws:4.1 .
…
Step 14/15 : RUN helm plugin install [https://github.com/futuresimple/helm-secrets](https://github.com/futuresimple/helm-secrets)
 — -> Running in 9c8b1e891e86
/usr/bin/dpkg
/root/.local/share/helm/plugins/helm-secrets/install-binary.sh: line 57: sudo: command not found
Error: plugin install hook for “secrets” exited with error
The command ‘/bin/sh -c helm plugin install [https://github.com/futuresimple/helm-secrets'](https://github.com/futuresimple/helm-secrets') returned a non-zero code: 1

Well — no :-)

Okay — we can install it using PIP:

...
RUN mv /tmp/eksctl /usr/local/bin 

RUN pip install ansible boto3 awscli sops

WORKDIR /tmp
...

Although during build Helm told that:

…
sops is already installed:
INFO: You’re using Sops 1 written in Python. Sops 2 was rewritten in Go. Consider installing it with: $ go get -u go.mozilla.org/sops/cmd/sops
sops 1.18
…

But let’s see if this will work in a Jenkins build.

Jenkinsfile and deploy job

The last step is to update the pipeline’s Jenkinsfile — add the secrets and -f bttrm-apps-backend/secrets.yaml:

...
        stage("Helm install") {
                    ...
                    sh "helm secrets upgrade --install --namespace ${AWS_EKS_NAMESPACE} --create-namespace --atomic ${APP_CHART_NAME} ${APP_CHART_NAME}-${RELEASE_VERSION}.tgz -f bttrm-apps-backend/secrets.yaml"
                }
            }   
        }
...

Run:

And it’s ready.

Useful links

Originally published at RTFM: Linux, DevOps, and system administration.

Blog