How to make exclusive locks in Kubernetes

Klock: Exclusive Lock Primer

There is an application running in a Kubernetes cluster. Goal is to protect this application from any modifications,
except if those modifications are coming from predefined actor.

Setup Requirements

To start let's set up the Kind cluster and Klock. I assume you already have installed Kind.
Now let's create a cluster and install all requirements:

Create cluster: kind create cluster
Install cert-manager: kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.8.2/cert-manager.yaml
Install Klock :

helm repo add rnemet https://rnemet.dev/helm-charts
helm repo update
helm install klock rnemet/klock

Setup Scenario

My application has one Pod and one ConfigMap. By default the Klock supports locking Deployments, Pods, Secrets, and ConfigMaps. So, I'm covered there.

Let's deploy our application, one Pod and one ConfigMap:

kubectl run my-pod --image nginx --labels aura=red
kubectl create configmap my-cm --from-literal nikola=tesla
kubectl label cm my-cm aura=red

Add one more Pod:

kubectl run other-pod --image nginx

Check what we have in default namespace:

❯ kubectl get pods,cm
NAME            READY   STATUS    RESTARTS   AGE
pod/my-pod      1/1     Running   0          5m52s
pod/other-pod   1/1     Running   0          112s

NAME                         DATA   AGE
configmap/kube-root-ca.crt   1      18m
configmap/my-cm              1      4m20s

While our application is:

❯ kubectl get pod,cm -A --selector aura=red
NAMESPACE   NAME         READY   STATUS    RESTARTS   AGE
default     pod/my-pod   1/1     Running   0          7m2s

NAMESPACE   NAME              DATA   AGE
default     configmap/my-cm   1      5m30s

Let's add one service account(SA) to be used for our so-called operator to manage our application:

kubectl create serviceaccount jonny-op

If you try to list pods with jonny-op:

❯ kubectl run operator  --image roffe/kubectl --overrides='{"apiVersion":"v1","spec":{"serviceAccount":"jonny-op"}}' -it --restart Never -- /bin/sh -c "kubectl -n default get pods"
No resources found.
Error from server (Forbidden): pods is forbidden: User "system:serviceaccount:default:jonny-op" cannot list resource "pods" in API group "" in the namespace "default"
pod default/operator terminated (Error)

Here I'm trying to list all pods in default namespace with our fake operator. As operator run with SA jonny-op it can not list pods. What about our application:

kubectl delete pod operator
❯ kubectl run operator  --image roffe/kubectl --overrides='{"apiVersion":"v1","spec":{"serviceAccount":"jonny-op"}}' -it --restart Never -- /bin/sh -c "kubectl get pod,cm  --selector aura=red"
No resources found.
Error from server (Forbidden): pods is forbidden: User "system:serviceaccount:default:jonny-op" cannot list resource "pods" in API group "" in the namespace "default"
Error from server (Forbidden): configmaps is forbidden: User "system:serviceaccount:default:jonny-op" cannot list resource "configmaps" in API group "" in the namespace "default"
pod default/operator terminated (Error)

Doh... Let's fix this by creating Role and RoleBinding:

kubectl create role app-op --verb list --verb get --verb create --verb update --verb delete --verb patch --resource pods --resource cm
kubectl create rolebinding jonny-app --role app-op --serviceaccount default:jonny-op

And check:

❯ kubectl delete pod operator
pod "operator" deleted
❯ kubectl run operator  --image roffe/kubectl --overrides='{"apiVersion":"v1","spec":{"serviceAccount":"jonny-op"}}' -it --restart Never -- /bin/sh -c "kubectl get pod,cm  --selector aura=red"
If you don't see a command prompt, try pressing enter.
NAME         READY     STATUS    RESTARTS   AGE
pod/my-pod   1/1       Running   0          26m

NAME              DATA      AGE
configmap/my-cm   1         25m

Notice: you need to delete operator pod each time because it can not be updated. That I simulate operator with a Pod.

Run Scenario

Me, as external actor can modify both Pod my-pod and ConfigMap my-cm. Well, I created the cluster and I'm an admin in this case so that is natural.
As shown, we can do the same in the default namespace running a workload with SA jonny-op, or operator. But my goal is just and only jonny_op
can do it. And only for my application.

All resources in my application are grouped with the label aura=red. So, let's create a lock:

kubectl apply -f - <<EOF
apiVersion: klock.rnemet.dev/v1
kind: Lock
metadata:
  name: lockred
spec:
  operations:
    - UPDATE
    - DELETE
  matcher:
    aura: red
  exclusive:
    name: jonny-op
EOF

Now let's try to update Pod my-pod:

❯ kubectl label pod my-pod aura=blue --overwrite
Error from server (denied, there is a lock: map[aura:red]): admission webhook "klocks.rnemet.dev" denied the request: denied, there is a lock: map[aura:red]

Ok... What about Pod other-pod:

❯ kubectl label pod other-pod aura=blue --overwrite
pod/other-pod labeled

So, I as admin can label(UPDATE) Pod that is not labeled with aura:red. What about SA jonny-op:

❯ kubectl run operator  --image roffe/kubectl --overrides='{"apiVersion":"v1","spec":{"serviceAccount":"jonny-op"}}' -it --restart Never -- /bin/sh -c "kubectl label pod my-pod here=was-jonny"
pod/my-pod labeled
❯ k get pods --show-labels
NAME        READY   STATUS      RESTARTS   AGE   LABELS
my-pod      1/1     Running     0          91m   aura=red,here=was-jonny
operator    0/1     Completed   0          25s   run=operator
other-pod   1/1     Running     0          87m   aura=blue,run=other-pod

What about ConfigMap my-cm?

❯ kubectl label cm my-cm admin=was-here
Error from server (denied, there is a lock: map[aura:red]): admission webhook "klocks.rnemet.dev" denied the request: denied, there is a lock: map[aura:red]
❯ k delete pods operator
pod "operator" deleted
❯ kubectl run operator  --image roffe/kubectl --overrides='{"apiVersion":"v1","spec":{"serviceAccount":"jonny-op"}}' -it --restart Never -- /bin/sh -c "kubectl label cm my-cm here=was-jonny"
configmap/my-cm labeled
❯ kubectl get cm --show-labels
NAME               DATA   AGE    LABELS
kube-root-ca.crt   1      108m   <none>
my-cm              1      94m    aura=red,here=was-jonny

Same. So, I managed to protect my application from other actors in the cluster, while SA jonny-op can operate with it.

Conclusion

Even if this is more manual example, it shows that is possible to make exclusive lock in Kubernetes with Klock.

Blog

How to make exclusive locks in Kubernetes

Robert Nemet

Setup Requirements

Setup Scenario

Run Scenario

Conclusion

Join Our Newsletter. No Spam, Only the good stuff.

Related