Build your Service Mesh: Admission Controller

Creating the Admission Controller

Before kube-apiserver persists the object and later be scheduled to a node,
the Admission Controller can validate and mutate the object.

The Mutation Admission Controller is going to mutate the pods that have the the
annotation diy-service-mesh: true.

Let's dig into how the Admission Controller works.

Admission Controller Flow

kube-controller or kubectl sends a request to the kube-apiserver to create a pod.
The kube-apiserver sends the request to the Admission Controller. In this case the proxy-injector.
The proxy-injector returns the mutated patch to the kube-apiserver.
Kube-apiserver persists the object in the etcd if the object is valid.
Kube-scheduler will schedule the pod to a node.
Kube-scheduler returns an available node to the kube-apiserver or an error if the pod can't be scheduled.
The kube-apiserver will store the object in the etcd with the selected node.
The kubelet in the selected node will create the pod in the container runtime.

Admission Controller Code

Full code of the Admission Controller: injector

The mutate function processes the AdmissionReview object and returns the AdmissionResponse object with the mutated patch.

func mutate(ar *admissionv1.AdmissionReview) *admissionv1.AdmissionResponse {
    req := ar.Request
    // Ignore all requests other than pod creation.
    if req.Operation != admissionv1.Create || req.Kind.Kind != "Pod" {
        return &admissionv1.AdmissionResponse{
            UID:     req.UID,
            Allowed: true,
        }
    }

    var pod v1.Pod
    // Unmarshal the raw object to the pod.
    if err := json.Unmarshal(req.Object.Raw, &pod); err != nil {
        return &admissionv1.AdmissionResponse{
            UID: req.UID,
            Result: &metav1.Status{
                Message: err.Error(),
            },
        }
    }

    // Check if the pod contains the inject annotation.
    if v, ok := pod.Annotations["diy-service-mesh/inject"]; !ok || strings.ToLower(v) != "true" {
        return &admissionv1.AdmissionResponse{
            UID:     req.UID,
            Allowed: true,
        }
    }

    // Add the initContainer to the pod.
    pod.Spec.InitContainers = append(pod.Spec.InitContainers, v1.Container{
        Name:            "proxy-init",
        Image:           os.Getenv("IMAGE_TO_DEPLOY_PROXY_INIT"),
        ImagePullPolicy: v1.PullAlways,
        SecurityContext: &v1.SecurityContext{
            Capabilities: &v1.Capabilities{
                Add:  []v1.Capability{"NET_ADMIN", "NET_RAW"},
                Drop: []v1.Capability{"ALL"},
            },
        },
    })

    // Add the sidecar container to the pod.
    pod.Spec.Containers = append(pod.Spec.Containers, v1.Container{
        Name:            "proxy",
        Image:           os.Getenv("IMAGE_TO_DEPLOY_PROXY"),
        ImagePullPolicy: v1.PullAlways,
        SecurityContext: &v1.SecurityContext{
            RunAsUser:    func(i int64) *int64 { return &i }(1337),
            RunAsNonRoot: func(b bool) *bool { return &b }(true),
        },
    })

    patch := []map[string]any{
        {
            "op":    "replace",
            "path":  "/spec/initContainers",
            "value": pod.Spec.InitContainers,
        },
        {
            "op":    "replace",
            "path":  "/spec/containers",
            "value": pod.Spec.Containers,
        },
    }

    podBytes, err := json.Marshal(patch)
    if err != nil {
        return &admissionv1.AdmissionResponse{
            UID: req.UID,
            Result: &metav1.Status{
                Message: err.Error(),
            },
        }
    }

    patchType := admissionv1.PatchTypeJSONPatch
    return &admissionv1.AdmissionResponse{
        UID:     req.UID,
        Allowed: true,
        AuditAnnotations: map[string]string{
            "proxy-injected": "true",
        },
        Patch:     podBytes,
        PatchType: &patchType,
    }
}

IMAGE_TO_DEPLOY_PROXY_INIT and IMAGE_TO_DEPLOY_PROXY are environment variables that tilt will update with the last proxy-init and proxy image respectively.

For complex patch use thi library: https://github.com/evanphx/json-patch

Deploying the Admission Controller

MutatingWebhookConfiguration tells the kube-apiserver to send the pod creation requests to the injector.

apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
metadata:
  name: service-mesh-injector-webhook
webhooks:
- name: service-mesh-injector.service-mesh.svc
  clientConfig:
    service:
      name: service-mesh-injector
      namespace: service-mesh
      path: "/inject"
  rules:
  - operations: ["CREATE"]
    apiGroups: [""]
    apiVersions: ["v1"]
    resources: ["pods"]
  admissionReviewVersions: ["v1"]
  sideEffects: None
  timeoutSeconds: 5

Some important points:

caBundle in the clientConfig is missing. This is a necessary field because the kube-apiserver only calls the webhook if the certificate is valid.
Two jobs, injector-admission-create injector-admission-patch are going generate the certificates and patch the MutatingWebhookConfiguration with the caBundle.
The rules options allow to filter the objects that are going to be sent to the injector.

The file injector.yaml contains the nesesary resources
including the Service Account, Role, RoleBinding, ClusterRole, ClusterRoleBinding,
Service, Deployment and the Job to generate the certificates.

Testing the Admission Controller

Let's modify the http-client and http-server deployments to add the annotation diy-service-mesh/inject: "true".

spec:
  replicas: 1
  selector:
    matchLabels:
      app: http-client
  template:
    metadata:
      labels:
        app: http-client
      annotations:
        diy-service-mesh/inject: "true"
    spec:

Important: the annotation needs to be added to the pod template and not to the deployment.

Blog

Build your Service Mesh: Admission Controller

Ramón Berrutti

Creating the Admission Controller

Admission Controller Flow

Admission Controller Code

Deploying the Admission Controller

Testing the Admission Controller

Join Our Newsletter. No Spam, Only the good stuff.

Related