Mostly for personal/learning experiences, I have created quite a few Kubernetes clusters, such as the one on my Raspberry Pi rack. I also created two clusters for a production and a staging environment on ultra-cheap cloud servers from Hetzner Cloud. Luckily, none of those environments where serious business.

Disclaimer: I'm not a Kubernetes expert, nor am I a security expert, so make sure that you second-source the information you find on this post before you rely on them. I just wanted to publish the experience and insights that I made during this trip - thanks!

Why was that lucky?

Because I accidentally leaked the certificates for my admin access to the staging cluster. I was trying to set up a CI/CD pipeline for an open source project using CircleCI. While I was testing out the steps one by one, I dumped the content of ${HOME}/.kube/config that has been created from a BASE64-encoded environment variable, like described on this blog post. That was fatal, though, since a) the job logs of open source projects are publicly visible and b) jobs and their logs can not be deleted manually, I had to reach out to the support for this. Ouch!

So let's dig into what happened here.

Creating a cluster

First of all, I created the cluster manually using kubeadm, following the official docs. Doing so, I created a cluster with RBAC enabled and a kube-config has been created for me that includes a user that is identified by a certificate.

Accessing the cluster

After kubeadm created the cluster successfully, it instructs you what to do to access your cluster:

"Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:"

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

The admin.conf that kubeadm creates includes a user identified by a certificate:

- name: kubernetes-admin
  user:
    client-certificate-data: <BASE64 ENCODED X509 CERTIFICATE>
    client-key-data:  <BASE64 ENCODED PRIVATE KEY FOR THE CERTIFICATE>

Following these instructions is your only way to access this cluster using kubectl as of now, so you should go ahead and do this now. After you copied the admin.conf, you have cluster-admin access. You are root, so to say.

How is that?

What kubeadm did is that it created a new CA (Certificate Authority) root certificate that is the master certificate for your cluster. It looks something like this:

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 0 (0x0)
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN = kubernetes
        Validity
            Not Before: May 19 11:11:04 2019 GMT
            Not After : May 16 11:11:04 2029 GMT
        Subject: CN = kubernetes
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    <REDACTED>
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment, Certificate Sign
            X509v3 Basic Constraints: critical
                CA:TRUE
    Signature Algorithm: sha256WithRSAEncryption
         <REDACTED>

So... it doesn't really contain much besides the info that it is a CA and it's CN (= Common Name) is Kubernetes. That's because this cert only acts as a root for other certs that are used for different purposes on the cluster. You can have a look at /etc/kubernetes/pki to take a peek at some of the certs that are used in your cluster and have been signed by the CA:

daniel@kube-box:~# ls /etc/kubernetes/pki/ -1
apiserver.crt
apiserver-etcd-client.crt
apiserver-etcd-client.key
apiserver.key
apiserver-kubelet-client.crt
apiserver-kubelet-client.key
ca.crt
ca.key
etcd
front-proxy-ca.crt
front-proxy-ca.key
front-proxy-client.crt
front-proxy-client.key
sa.key
sa.pub

It is possible to allow access to clients that authenticate themselves using certificates that are trusted by the CA. This is enabled by passing this ca.crt to kube-controller-manager in the --client-ca-file parameter. This is what the docs have to say about it:

--client-ca-file string
If set, any request presenting a client certificate signed by one of the
authorities in the client-ca-file is authenticated with an identity 
corresponding to the CommonName of the client certificate.

Back to your kube-config: The certificate that is included in BASE64 in your admin.conf is signed by that exact CA. This is why it is trusted by the cluster. Let's have a look at the certificate:

grep 'client-certificate-data: ' ${HOME}/.kube/config | \
   sed 's/.*client-certificate-data: //' | \
   base64 -d | \
   openssl x509 --in - --text

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 3459994011761527671 (0x30045e38cc064b77)
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN = kubernetes
        Validity
            Not Before: May 12 10:54:39 2019 GMT
            Not After : May 11 10:54:42 2020 GMT
        Subject: O = system:masters, CN = kubernetes-admin
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    <REDACTED>
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Extended Key Usage:
                TLS Web Client Authentication
    Signature Algorithm: sha256WithRSAEncryption
         <REDACTED>

What does this cert tell us (and the cluster)?

a) It is issued and trusted by our kubernetes cluster
b) It identifies the Organisation (O) system:masters, which is interpreted as a group by kubernetes
c) It identifies the Common Name (CN) kubernetes-admin, which is interpreted as a user by kubernetes

In other words: This certificate logs in as the user kubernetes-admin with the group system:masters. This is the reason why you don't need to provide the group name in the kube-config, and why you can change the user's name at will in the kube-config, without this changing the actual user that is being logged in.

Where are the permissions defined?

In RBAC-enabled clusters, permissions are defined in Roles (per namespace) or ClusterRoles (for all namespaces). These permissions are then granted to objects using RoleBindings and ClusterRoleBindings. So what you have to look for are RoleBindings and ClusterRoleBindings that grant permissions to the group system:masters or the user kubernetes-admin. You can do this by having a look at the output of

kubectl -A=true get rolebindings && kubectl -A=true get clusterrolebindings

The default setup that kubeadm created for me yielded one hit for that search, the ClusterRoleBinding named cluster-admin, which grants permissions to a ClusterRole with the same name. Here's the definition:

kind: ClusterRole
metadata:
  name: cluster-admin
rules:
- apiGroups:
  - '*'
  resources:
  - '*'
  verbs:
  - '*'
- nonResourceURLs:
  - '*'
  verbs:
  - '*'
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: cluster-admin
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: Group
  name: system:masters

So there we have it! The group system:masters, which the certificate authorizes as, grants '*' permissions to all resources using all verbs, hence full access.

Unvalidated assumption: I think that users and groups in this case are only defined in the certificates, and new users can be created by issuing a new certificate with the CommonName set to the desired username and the Organisation set to the desired group. This username and group can then, without further ado, be used in ClusterRoleBindings and RoleBindings. I did not take the time to validate this, though, which would be possible by issuing a new certificate using openssl, signed with the cluster's CA. It'd be great if someone could confirm or debunk this assumption in a comment!

The mystery

So I got this far and found out how the user and group are identified and how permissions are granted to this user and group. My guess then was that when I delete the ClusterRoleBinding, or rather remove the group system:masters from it, that the certificate should not have access to the cluster anymore. If I did that, and it had the expected result, I would lose all access to the cluster and would have successfully logged me out for good. So I first added a serviceaccount and created a kube-config that logged in using a token for that serviceaccount and verified that the access worked. We will see later how to do this. Then, after setting the safety net in place, I removed the system:masters subject from the ClusterRoleBinding. To my surprise, this did not lock the user out. I could still fully access the cluster using the old kube-config… maybe someone can explain this behaviour in a comment?

Alternative 1: Replace the CA

One sure-as-hell way to make the leaked certificate useless is to replace the CA in the cluster. This would require a restart of the cluster, though. And it would require to re-issue all the certificates that we have seen above, and maybe some more. I rated the possibility to totally fuck everything up and waste multiple hours on the trip at about 99%, so I abandoned the plan. :-)

Alternative 2: Rebuild the whole cluster

Luckily, it was a staging cluster, so I had plenty of freedom. Before starting my investigations, I powered off all nodes. Then, after not finding a proper solution to only make the leaked certificate useless, I killed the whole cluster using

kubeadm reset
rm -rf /etc/kubernetes
rm -rf /var/lib/kubelet

And recreated from scratch with kubeadm. (Which is so great, by the way!!)

Then I went ahead and made a few dozen more commits reading 'NOT printing the content of the kube-config anymore', 'Getting CI/CD to work', 'Maybe now it works', 'Uhm what?', 'That gotta work', 'fuck CI/CD', … :-)

Lessons learned: Use service-accounts with tokens

(Or other authentication methods like OpenID, as recommended in this awesome post.)

So my lesson learned is to do what I've seen at the big managed kubernetes providers: Use a service-account and it's access token for authorization. Here I'll show how to set up a super-user that uses a token instead of a cert:

kubectl -n kube-system create serviceaccount admin

To grant super-user permissions, the easiest way is to create a new ClusterRoleBinding to bind this service-account to the cluster-admin ClusterRole:

kubectl create clusterrolebinding add-on-cluster-admin \
  --clusterrole=cluster-admin \
  --serviceaccount=kube-system:admin

Use your new service-account

Your admin user is now ready and armed. Now we need to log in with this user. I assume that you have the admin.conf in ${HOME}/.kube/conf. We now want to add the new user, identified by it's token, and add a new context that uses this user:

TOKENNAME=`kubectl -n kube-system get serviceaccount/admin -o jsonpath='{.secrets[0].name}'`
TOKEN=`kubectl -n kube-system get secret $TOKENNAME -o jsonpath='{.data.token}' | base64 -d`
kubectl config set-credentials admin --token=$TOKEN
kubectl config set-context admin@kubernetes --cluster kubernetes --user admin

Now go ahead and try your new, shiny service-account:

kubectl config use-context admin@kubernetes
kubectl -n kube-system get all

If this went well, you should go ahead and delete the certificate-based user and the corresponding context:

kubectl config unset users.kubernetes-admin
kubectl config delete-context kubernetes-admin@kubernetes

Yay! Now we have a kube-conf that only includes token-based access. This is great, because it is very easy to revoke that token if this config might be leaked or published.

How to invalidate a leaked token

This is easy! Just delete the secret that corresponds to the user's token. We already saw how to find out which is the correct secret:

kubectl -n kube-system get serviceaccount/admin -o yaml

You will see a field "name" in the "secrets" array. This is a name of a secret that holds this service-account's token. Now go ahead and simply delete it:

kubectl -n kube-system delete secrets/token-admin-xyz123

Then wait a few seconds, and try to access your cluster:

dainel@kube-box:~# kubectl -n kube-system get all
error: You must be logged in to the server (Unauthorized)

Wohoo!

But how do you regain access? Well, if you're on your master node, simply copy the admin.conf back to your ${HOME}/.kube/conf and repeat the steps from "Use your new service-account". Kubernetes will have created and assigned a new token by now.

I hope that this helped, and I'd love to hear feedback, errata, etc. in the comments!

Also make sure to read the VERY comprehensible and awesome post "Kubernetes Security Best-Practices" by Peter Benjamin.

And a big thank you to Andreas Antonsson, vaizki and Alan J Castonguay, who have helped me on the official Kubernetes Slack channel to get a better understanding of what is going on.