Working with Kubernetes is fun when your Kubernetes cluster is up and your nodes are working as you expected. But as a DevOps enthusiast who follows the principles of Continuous Integration and Continuous Delivery, you are aware that new changes or feature requests are always coming through the Jenkins Pipeline, and eventually, you’ll have to deploy changes into the Kubernetes cluster which feeds into your production environment.

In most cases, you will need to update your Kubernetes manifest YAMLs or Helm charts (Deployment, Service, or Pod) and apply those changes either directly into the Production or Staging or into the Test environment.

During a new release or deployment, I am generally troubled by the question: Is this new Kubernetes manifest YAML change I am deploying in the production environment actually going to work?

No matter how seasoned or experienced a developer you are, if you are not sure about your changes, this will be a scary experience. So, is there a way to verify your Kubernetes manifest YAML and Helm charts before they reach production?

Luckily, the answer is yes. You can indeed verify your Kubernetes manifest before making any changes, by using the Datree tool.

Datree allows developers to verify the Kubernetes configuration before applying it directly into the Production, Stage, Test, or even the Development environment.
Datree is a command-line tool that is installed locally onto the developer’s machine so that he or she can run the CLI commands that verify the Kubernetes manifests (YAML files).

Here is a short command snippet of Datree -

datree test my-kubernetes-manifest.yaml

In this post, I will show you how to use Datree step by step, starting with installing the tool and then verifying the Kubernetes manifest file. (If you need to check your helm charts instead, this can easily be done with the Datree Helm plugin.)

Table of Content

1. How to Install Datree
2. How to test Kubernetes Manifest using Datree
3. How to understand the validation errors thrown by Datree
4. View and edit Datree Policy via browser
5. Define Policy as Code(PaC)
6. Conclusion

1. How to Install Datree

Let’s start with the first step: Installation. Datree installation is really easy. Generally, I prefer Linux distros as my development machine.

On Linux, you can simply install it by running the following command -

Linux

curl https://get.datree.io | /bin/bash

If you are using another Operating System, it’s just as easy to install -

MacOS

curl https://get.datree.io | /bin/bash

Windows

iwr -useb https://get.datree.io/windows_install.ps1 | iex 

2. How to test Kubernetes Manifest using Datree

After installing the Datree, the next step is to test the Kubernetes manifest (YAML). In order to do this, you must have a copy of your Kubernetes manifest (YAML) available locally where you have already installed Datree.

You can run the following command to start testing your Kubernetes manifest (YAML) (Replace the Kubernetes manifest YAML name in the following command) -

datree test <your_kubernetes_manifest_YAML_NAME>

Here is my Kubernetes manifest(YAML) of my Spring Boot application which I am trying to test by deploying it on Kubernetes Cluster

(Click here to clone my GitHub Repo for Spring Boot Application)

k8s-spring-boot-deployment.yml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: jhooq-springboot
spec:
  replicas: 3
  selector:
    matchLabels:
      app: jhooq-springboot
  template:
    metadata:
      labels:
        app: jhooq-springboot
    spec:
      containers:
        - name: springboot
          image: rahulwagh17/jhooq-docker-demo:jhooq-docker-demo
          ports:
            - containerPort: 8080
          env:
            - name: PORT
              value: "8080"
---
apiVersion: v1
kind: Service
metadata:
  name: jhooq-springboot
spec:
  type: NodePort
  ports:
    - port: 80
      targetPort: 8080
  selector:
    app: jhooq-springboot 

Here is a command to test my Kubernetes manifest -

datree test k8s-spring-boot-deployment.yml

When you run the above command, you should see the following validation error messages -

3. How to understand the validation errors thrown by Datree

What I have discovered is that my Kubernetes manifest (YAML) has a lot of problems. it will work fine inside my Kubernetes cluster, it is still far from ready for the production environment.

Let's start with the errors -

3.1 Ensure each container has a configured memory request

❌  Ensure each container has a configured memory request  [2 occurrences]
— metadata.name: RELEASE-NAME-helloworld (kind: Deployment)
— metadata.name: RELEASE-NAME-helloworld-test-connection (kind: Pod)
💡  Missing property object `requests.memory` - value should be within the accepted boundaries recommended by the organization 

This error tells me that I have neglected to add request.memory size inside the Kubernetes deployment manifest.

This error can easily be fixed by adding the following request.memory attribute -

resources:
  requests:
    memory: "128Mi"
    cpu: "512m"

3.2 Ensure each container has a configured CPU request

❌  Ensure each container has a configured CPU request  [2 occurrences]
— metadata.name: RELEASE-NAME-helloworld (kind: Deployment)
— metadata.name: RELEASE-NAME-helloworld-test-connection (kind: Pod)
💡  Missing property object `requests.cpu` - value should be within the accepted boundaries recommended by the organization 

Similar to the previous error 3.1, we need to add the limit.cpu attribute.

Here is the missing limit.cpu attribute along with request.memory -

resources:
  requests:
    memory: "128Mi"
    cpu: "512m"

3.3 Ensure each container has a configured memory limit

❌  Ensure each container has a configured memory limit  [2 occurrences]
— metadata.name: RELEASE-NAME-helloworld-test-connection (kind: Pod)
— metadata.name: RELEASE-NAME-helloworld (kind: Deployment)
💡  Missing property object `limits.memory` - value should be within the accepted boundaries recommended by the organization

After fixing the memory and cpu requests, let’s try to fix the memory limit.

Add the following memory snippet inside your deployment manifest of Kubernetes under the resource limits -

limits:
  memory: "128Mi"

3.4 Ensure each container has a configured CPU limit

❌  Ensure each container has a configured CPU limit  [2 occurrences]
— metadata.name: RELEASE-NAME-helloworld-test-connection (kind: Pod)
— metadata.name: RELEASE-NAME-helloworld (kind: Deployment)
💡  Missing property object `limits.cpu` - value should be within the accepted boundaries recommended by the organization 

You also need to update the memory for the cpu, which can be done by adding the following attributes along with the previous one 3.4 -

limits:
  memory: "128Mi"
  cpu: "512m"

3.5 Ensure each container has a configured liveness probe

❌  Ensure each container has a configured liveness probe  [1 occurrence]
— metadata.name: RELEASE-NAME-helloworld-test-connection (kind: Pod)
💡  Missing property object `livenessProbe` - add a properly configured livenessProbe to catch possible deadlocks 

Next on the list are the liveness and readiness probes. They go hand in hand, but to keep the things simple and easy to understand, we will look at each error individually.

To fix the liveness probe, you must add the following attribute inside your Kubernetes manifest (YAML) -

livenessProbe:
  httpGet:
     path: /hello
     port: 8080
  initialDelaySeconds: 15
  periodSeconds: 10

3.6 Ensure each container has a configured readiness probe

❌  Ensure each container has a configured readiness probe  [1 occurrence]
— metadata.name: RELEASE-NAME-helloworld-test-connection (kind: Pod)
💡  Missing property object `readinessProbe` - add a properly configured readinessProbe to notify kubelet your Pods are ready for traffic

After fixing the liveness probe, let’s fix the readiness probe by adding the following attribute to the Kubernetes manifest (YAML) -

readinessProbe:
  httpGet:
    path: /hello
    port: 8080
  initialDelaySeconds: 15
  periodSeconds: 10

3.7 Ensure Deployment has more than one replica configured

❌  Ensure Deployment has more than one replica configured  [1 occurrence]
— metadata.name: RELEASE-NAME-helloworld (kind: Deployment)
💡  Incorrect value for key `replicas` - running 2 or more replicas will increase the availability of the service

Lastly, we need to fix the replica count.This can be done easily by updating the replicas attribute.

It is not recommended to have a replica count of 1. Update the replica count to a number greater than 1 -

spec:
  replicas: 2

Here is the final k8s-spring-boot-deployment.yml after all 7 fixes have been made:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: jhooq-springboot
spec:
  replicas: 2
  selector:
    matchLabels:
      app: jhooq-springboot
  template:
    metadata:
      labels:
        app: jhooq-springboot
    spec:
      containers:
        - name: springboot
          image: rahulwagh17/kubernetes:jhooq-k8s-springboot

          resources:
            requests:
              memory: "128Mi"
              cpu: "512m"
            limits:
              memory: "128Mi"
              cpu: "512m"

          ports:
            - containerPort: 8080

          readinessProbe:
            httpGet:
              path: /hello
              port: 8080
            initialDelaySeconds: 15
            periodSeconds: 10

          livenessProbe:
            httpGet:
              path: /hello
              port: 8080
            initialDelaySeconds: 15
            periodSeconds: 10

          env:
            - name: PORT
              value: "8080"
---
apiVersion: v1
kind: Service
metadata:
  name: jhooq-springboot
spec:
  type: ClusterIP
  ports:
    - port: 80
      targetPort: 8080
  selector:
    app: jhooq-springboot 

I recommend updating your Kubernetes manifest and re-running the Datree test command.

datree test k8s-spring-boot-deployment.yml

Here is the screenshot of the results, where you can see that all the validation errors are now gone.

4. View and edit Datree Policy via browser

In addition to providing CLI to troubleshoot the errors associated with your manifest, Datree also allows you to view the policies on a browser. This is one of the coolest features offered by Datree for developers and other team members.

How to access the datree policies in a browser?

This too is pretty simple. Run the test command datree test k8s-spring-boot-deployment.yml and then look for the following message box into your terminal -

Copy the URL from the terminal and paste it in your browser. Authenticate yourself using your GitHub account and then you will see the following dashboard -

4.1 Edit Datree Policies

This is important because most of the time you will be working with Datree on CLI, but having a web portal where policies are accessible via browser makes the policy handling really easy.

4.1.1. Toggle ON/OFF policy

You can very easily toggle the policy which you want to enforce and revoke for your Kubernetes cluster.

4.1.2. Edit Policy Message

You can also edit policy messages so that you can add text which can help the developer troubleshoot the policy validation errors.

Finally, save your custom message -

4.2 Datree policies invocation history

Along with edit features, you can also see the previous invocation history and the results. Simply click on the History option on the left navigation -

5. Define Policy as Code(PaC)

One more important feature of Datree which I like the most is Policy As Code (PaC). If you have worked with Terraform before, then you might recognize this feature, because it helps you create and save policy as code.

Benefits-

1. You can easily version your policies under the version control
2. Easy to share polices with version control
3. Each policy change can be tracked and roll back with the help of version control.

5.1 Enabled Datree Policy As Code

Under your account settings, you can find the toggle option to enable the Policy As Code -

5.2 Download the policies.yaml

You can download the policies.yaml from the settings page.

5.3 How to update the policies.yaml

Keep these key components in mind before updating the policies -

1. name - Here you can specify the name of the policy which you want to keep
2. isDefault - If you want to keep the policies as default, then you can keep the value of the policy as true
3. identifier - It should be unique in nature
4. messageOnFailure - The message which you want to show upon failure

Here is an example of policies.yaml -

apiVersion: v1
policies:
  - name: Default
    isDefault: true
    rules:
      - identifier: INGRESS_INCORRECT_HOST_VALUE_PERMISSIVE
        messageOnFailure: Incorrect value for key `host` - specify host instead of using a wildcard character ("*") 

The above YAML is just an example in which you can either keep no policies or more than one policy, as per your need.

6. Conclusion

Datree is a really good framework for a DevOps person who wants to build a stable Kubernetes manifest (YAML) which is very well tested and secured with custom policies.

Apart from testing the Kubernetes manifest (YAML) locally, Datree provides some out of the box integrations -

1. Git Hooks - With Git Hooks, you can trigger the testing of Kubernetes manifest (YAML) as soon as you commit and push some new updates into your repo.
2. **Helm Plugin - If you are using Helm Chart to manage your Kubernetes cluster, then you can install the Datree helm plugin at https://github.com/datreeio/helm-datree which can help you to achieve the same result which we saw earlier in the post.
3. Datree has really good integration with CicleCi, TravisCi, GitHub Actions, GitLab also.