With the deprecation of PSP on Kubernetes v1.21, we will have to migrate to other methods to control the resource permissions in a cluster.

One case that I wanted to handle was running an untrusted job in a cluster, to help review student’s homework and leveraging kubernetes for resource allocation and security. Here we will explore how to develop a new admission controller, that will verify the fields of new jobs on a given namespace are secure enough to run untrusted code in a safe way.

Recommended policy enforcement applications

If you want to start defining policies for a production cluster, you will probably want to use a ready to use application, which have predefined policies, and setup custom policies easily by using custom resources. Some of them are:

Cluster requirements

The api-server must have the plugin ValidatingAdmissionWebhook enabled (If you want to modify the resources, you also need MutatingAdmissionWebhook). Note that these plugins are disabled by default in a kind cluster.

Application goal

In our example, we will write a validating admission webhook (So we will not modify the resource) that will check new jobs created in a namespace, verifying as many security options of the pod as we can (running as non-root, using gvisor as sandbox, and many others). The target container image that we are targeting is an untrusted job that can be potentially malicious.

Writing the admission controller

Our admission controller will be written in Go, but you can use any language you know as the api use normal https json requests.

I will be trimming some of the code to make it more readable. The full source code can be found at https://github.com/fdns/simple-admission

Listening to admission requests

First, we will need to create a HTTPS listener (TLS is mandatory). You can use any http path to serve the requests, but you must update the manifest afterwards with the correct location when we define the ValidatingAdmissionWebhook.

func main() {
    // ...
    certs, err := tls.LoadX509KeyPair(certFile, keyFile)

    server := &http.Server{
        Addr: fmt.Sprintf(":%v", port),
        TLSConfig: &tls.Config{
            Certificates: []tls.Certificate{certs},
        },
    }

    // Define server  handler
    handler := AdmissionHandler{
        RuntimeClass: runtimeClass,
    }
    mux := http.NewServeMux()
    mux.HandleFunc("/validate", handler.handler)
    server.Handler = mux

    go func() {
        log.Printf("Listening on port %v", port)
        if err := server.ListenAndServeTLS("", ""); err != nil {
            log.Printf("Failed to listen and serve webhook server: %v", err)
            os.Exit(1)
        }
    }()

    // Listen to the shutdown signal
    signalChan := make(chan os.Signal, 1)
    signal.Notify(signalChan, syscall.SIGINT, syscall.SIGTERM)
    <-signalChan

    log.Printf("Shutting down webserver")
    server.Shutdown(context.Background())
}

Handling admission request

When receiving the request, you must load the body as an AdmissionReview object. This object contains all the information of the objects that is being created.

import (
    admission "k8s.io/api/admission/v1beta1"
    batchv1 "k8s.io/api/batch/v1"
    k8meta "k8s.io/apimachinery/pkg/apis/meta/v1"
)

func (handler *AdmissionHandler) handler(w http.ResponseWriter, r *http.Request) {
    var body []byte
    if r.Body != nil {
        data, err := ioutil.ReadAll(r.Body)
        if err == nil {
            body = data
        } else {
            log.Printf("Error %v", err)
            http.Error(w, "Error reading body", http.StatusBadRequest)
            return
        }
    }

    request := admission.AdmissionReview{}
    if err := json.Unmarshal(body, &request); err != nil {
        log.Printf("Error parsing body %v", err)
        http.Error(w, "Error parsing body", http.StatusBadRequest)
        return
    }

    result, err := checkRequest(request.Request, handler)
    // ...
}

Validating the request

In the checkRequest function, we will check if we can handle the resource, verifying the resource group, kind, operation and namespace.

func checkRequest(request *admission.AdmissionRequest, handler *AdmissionHandler) (bool, error) {
    if request.RequestKind.Group != "batch" || request.RequestKind.Kind != "Job" || request.Operation != "CREATE" {
        log.Printf("Skipped resource [%v,%v,%v], check rules to exclude this resource", request.RequestKind.Group, request.RequestKind.Kind, request.Operation)
        return true, nil
    }
    // ...
}

The resource body (In our case, a Job) must un unmarshal again before we can verify the parameters.

    var job *batchv1.Job
    err := json.Unmarshal(request.Object.Raw, &job)
    if err != nil {
        log.Printf("Error parsing job %v", err)
        return true, nil
    }

    return checkJob(job, handler)

Checking the resource

On the checkJob, we will have full access to the resource parameters. Most of the parameters that are not defined will be nil, so you must verify that the parameters is defined before getting its value.

I will copy some of the rules as an example, and the full list that I defined can be found here.

func checkJob(request *batchv1.Job, handler *AdmissionHandler) (bool, error) {
    if request.Spec.ActiveDeadlineSeconds == nil || *request.Spec.ActiveDeadlineSeconds == 0 {
        return false, fmt.Errorf("activeDeadlineSeconds must be set")
    }

    spec := request.Spec.Template.Spec
    if spec.RuntimeClassName == nil || *spec.RuntimeClassName != handler.RuntimeClass {
        return false, fmt.Errorf("wrong RuntimeClass %v is set for job %v, must be %v", spec.RuntimeClassName, request.Name, handler.RuntimeClass)
    }

    if spec.HostNetwork != false {
        return false, fmt.Errorf("HostNetwork must not be set")
    }

    if spec.SecurityContext != nil && len(spec.SecurityContext.Sysctls) > 0 {
        return false, fmt.Errorf("Sysctls must be empty")
    }

    // ...

    for _, container := range spec.Containers {
        if container.SecurityContext == nil {
            return false, fmt.Errorf("SecurityContext must be set for the container")
        }
        context := *container.SecurityContext

        if context.RunAsNonRoot == nil || *context.RunAsNonRoot != true {
            return false, fmt.Errorf("RunAsNonRoot must be set per container")
        }

        // ...
    }
    return true, nil
}

Returning to the api-server

After doing all the validations, you must return an AdmissionResponse object that is json encoded. In this object we will define if the objects is allowed or not in our cluster. We can also append a message that will be displayed when the resource is not allowed, so the developer can fix the resource according to the conditions you define.

    result, err := checkRequest(request.Request, handler)
    response := admission.AdmissionResponse{
        UID:     request.Request.UID,
        Allowed: result,
    }
    if err != nil {
        response.Result = &k8meta.Status{
            Message: fmt.Sprintf("%v", err),
            Reason:  k8meta.StatusReasonUnauthorized,
        }
    }

    outReview := admission.AdmissionReview{
        TypeMeta: request.TypeMeta,
        Request:  request.Request,
        Response: &response,
    }
    json, err := json.Marshal(outReview)

    if err != nil {
        http.Error(w, fmt.Sprintf("Error encoding response %v", err), http.StatusInternalServerError)
    } else {
        w.Header().Set("Content-Type", "application/json")
        if _, err := w.Write(json); err != nil {
            log.Printf("Error writing response %v", err)
            http.Error(w, fmt.Sprintf("Error writing response: %v", err), http.StatusInternalServerError)
        }
    }

Building our project

As this is a standard go project, you can use a very simple Dockerfile to create the image. This image can be built by running docker build . --tag fdns/simple-admission:latest (You can change the tag to the one you like).

FROM golang:1.16.2 as builder

WORKDIR $GOPATH/src/github.com/fdns/simple-admission
COPY go.mod .
COPY go.sum .

RUN go mod download

COPY . .

RUN CGO_ENABLED=0 go build -o /go/bin/simple-admission

FROM scratch
COPY --from=builder /go/bin/simple-admission /go/bin/simple-admission
ENTRYPOINT ["/go/bin/simple-admission"]

The only thing left is uploading it to our cluster.

Uploading controller to a kubernetes cluster

Create TLS certificates

As the webhook require the use of HTTPS to work, we can create our own CA and certificate for the controller. The CA keys can be dropped as soon as we sign the client certificate, as the CA bundle is included in the ValidatingAdmissionWebhook object.

As the requests will come from a service object, you will want to define as altnames in the certificate all the variations to call the services. In the configuration, this will look as something like the following.

[alt_names]
DNS.1 = ${service}
DNS.2 = ${service}.${namespace}
DNS.3 = ${service}.${namespace}.svc

To stop copying so much code, you can find a simple script to generate the certificate at https://github.com/fdns/simple-admission/blob/master/generate_certs.sh, which we will call with the service name and namespace of our admission controller (For example, ./generate_certs.sh simple-admission default).

The generated certificates must be mounted as a secret, as we will need to mount them in our application (save the ca.pem file as we will need it later).

apiVersion: v1
kind: Secret
metadata:
  creationTimestamp: null
  name: admission-certs
  namespace: default
data:
  server-key.pem: $(cat certs/server-key.pem | base64 | tr -d '\n')
  server.pem: $(cat certs/server.crt | base64 | tr -d '\n')

Creating the service and webhook

You can create the deployment and services the same way as any other deployment in your cluster. Here it is recommended to increase the replica count to increase the availability.

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: simple-admission
  name: simple-admission
spec:
  replicas: 1
  selector:
    matchLabels:
      app: simple-admission
  strategy: {}
  template:
    metadata:
      labels:
        app: simple-admission
    spec:
      containers:
      - name: simple-admission
        image: fdns/simple-admission:latest
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 8443
        volumeMounts:
        - name: admission-certs
          mountPath: /certs
          readOnly: true
      volumes:
      - name: admission-certs
        secret:
          secretName: admission-certs
---
apiVersion: v1
kind: Service
metadata:
  creationTimestamp: null
  labels:
    app: simple-admission
  name: simple-admission
spec:
  ports:
  - name: 443-8443
    port: 443
    protocol: TCP
    targetPort: 8443
  selector:
    app: simple-admission
  type: ClusterIP

Creating the ValidatingAdmissionWebhook

Finally, we will create the ValidatingAdmissionWebhook. We can define multiple webhooks, where in each one we must tell kubernetes the service, path and CA to send the request to the admission controller. For each one we can define the rules to filter the requests that are sent to our controller, where in this case we will filter for jobs resources in namespaces labeled with the name default (the namespace MUST be labeled in our example).

In case you want to audit your webook before applying it to your cluster, you can change the failurePolicy from Fail to Ignore

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
 name: simple-admission.default.cluster.local
 namespace: default
webhooks:
- name: simple-admission.default.cluster.local
  clientConfig:
    service:
      name: simple-admission
      namespace: default
      path: "/validate"
    caBundle: $(cat certs/ca.pem | base64 | tr -d '\n')
  rules:
  - apiGroups: ["batch"]
    apiVersions: ["v1"]
    resources: ["jobs"]
    operations: ["CREATE"]
    scope: "*"
  namespaceSelector:
    matchExpressions:
    - key: name
      operator: In
      values: ["default"]
  admissionReviewVersions: ["v1"]
  sideEffects: None
  failurePolicy: Fail

Testing our admission controller

To the newly applied admission controller, you can simply try to create a basic job running kubectl create job test --image busybox, which in our case will output the following message:

error: failed to create job: admission webhook "simple-admission.default.cluster.local" denied the request: activeDeadlineSeconds must be set

Conclusions

Creating an admission controller is not difficult, but making sure all the parameters to make your containers secure is a difficult task, as not all fields are generally known, and new fields must be taken into account when kubernetes release a new version.

When creating a new admission controller, you should try to target a single problem, like image verification or single fields of the resources like runtimeClass over your cluster. In case you need more complex rules, the use of the already available admission controllers is recommended, as you can define the rules in your own CRD and allow you to iterate faster (some of them have audit mode so you can check your cluster before enforcing a rule).

Blog

Writing a Kubernetes Admission Controller

Felipe Espinoza