The idea of writing this article was born by the author after getting acquainted with two entities: this and this repository. 😎

So, noone wants to throw poop in the pot

As part of the GitOps approach, increasingly used today, for the majority of the time, you will need to update your Helm charts (Deployment, Service, or Pod) or YAML manifests for Kubernetes and then either immediately apply the changes to the Production, Staging, or Test environments, depending on the situation.

Worrying about whether the new Kubernetes manifest YAML modification which you're deploying in the production environment? Will it actually work during a release or deployment?

Well, a slightly nervous situation, illustrated (the author just remembered this) about six years ago.

No matter how skilled or experienced you are as an engineer, this will be a scary experience if you are unsure about your changes.

So, is there a way to validate your YAML manifests and Helm charts before they are used in production? Fortunately, the answer is “yes”. A tool named Datree can be used to validate your Kubernetes manifest before making any mods!

What makes Datree special

While Kubescape is a utility that improves Kubernetes security by scanning clusters and detecting already deployed YAML files that are not compliant with security standards, or subject to vulnerabilities (that the engineer might not have known about), Datree was made 💡to prevent Kubernetes misconfigurations from reaching production with Datree’s automated policy checks for your pipeline.

Datree, the 5.8k⭐ open-source CLI tool empowers engineers to write more stable configurations, so they can actually sleep at night. It allows you to integrate it into any CI flow and trigger it whenever you want, such as each time team make a change or submit a pull request, for example.

🎯 Is it safe in terms of data leaks?

Creators' statements can be interpreted in such a way that the appraisal of Datree's policies is purely local.

🖼️ A larger picture is here. Also, you may open picture in a new browser's tab to zoom in.

Since the CLI runs the policy check on your system, your files and their contents are not transferred to their backend. To their backend, which is used to show your policy check history on your dashboard, the tool only sends metadata.

What is not less important: to do checks, Datree does not need to be connected to the cluster. It has an offline mode as well.

🎯 What to verify?

The following three assertions are verified by Datree:

• Is the file a well-composed YAML file?
• Schema for Kubernetes: is this a valid Kubernetes file?
• Is the file compliant with your Kubernetes policy?

🎯 Policy as code

Policy-as-a-code, similar to Infrastructure-as-a-Code, is the concept of using declarative code to replace actions that require using a user interface. Proven software development best practices, such as version control, collaboration, and automation, can be applied by encoding policies in code.

🎯 Centralized policy

This idea refers to managing distributed policy execution from a single point. This gives the policy owner simple control over the rules that Datree evaluates during each run without adding more work to the operation. You can control the central policy by logging into the dashboard.

🎯 Rules' flexity

To fit your preferences, you can toggle any of the 50+ built-in rules "ON" or "OFF" in the dashboard. When a rule is turned on or off, any policy checks run against that policy will automatically be updated (via account token). This eliminates the need for the policy owner to manually update every device (cluster host) linked to the policy.

You can pick from dozens of tried-and-true rules that address various Kubernetes resources and use cases, which are tied to: containers, cron jobs, workloads (running apps), networking, security, API deprecation, ArgoCD rollouts, CVEs described by the NSA and to other Kubernetes syntax nuances.

🎯 Custom rules

You can write any tests you like and run them against your Kubernetes setups to check for rule violations in addition to the tool's built-in rules. The built-in rule engine supports both YAML and JSON declarative syntax because it is based on JSON Schema.

Installation and general usage

All you need to do is run the command below:

curl https://get.datree.io | /bin/bash

Once installed, you can easily use Datree to check the security of Kubernetes manifests.

The syntax is as follows:

datree test [k8s-manifest-file]

When a check is started, it goes through 3 main stages:

• YAML validation;
• checking Kubernetes charts;
• checking Kubernetes policies.

For example, when checking demo manifest, the command would look like this:

datree test ~/.datree/k8s-demo.yaml

Let's see the output:

$ datree test ~/.datree/k8s-demo.yaml
>>  File: .datree/k8s-demo.yaml

[V] YAML validation
[V] Kubernetes schema validation

[X] Policy check

❌  Ensure each container image has a pinned (tag) version  [1 occurrence]
    - metadata.name: rss-site (kind: Deployment)
💡  Incorrect value for key `image` - specify an image version to avoid unpleasant "version surprises" in the future

❌  Ensure each container has a configured liveness probe  [1 occurrence]
    - metadata.name: rss-site (kind: Deployment)
💡  Missing property object `livenessProbe` - add a properly configured livenessProbe to catch possible deadlocks

❌  Ensure each container has a configured memory limit  [1 occurrence]
    - metadata.name: rss-site (kind: Deployment)
💡  Missing property object `limits.memory` - value should be within the accepted boundaries recommended by the organization

❌  Ensure workload has valid label values  [1 occurrence]
    - metadata.name: rss-site (kind: Deployment)
💡  Incorrect value for key(s) under `labels` - the vales syntax is not valid so the Kubernetes engine will not accept it


(Summary)

- Passing YAML validation: 1/1

- Passing Kubernetes (1.20.0) schema validation: 1/1

- Passing policy check: 0/1

+-----------------------------------+-----------------------+
| Enabled rules in policy "Default" | 21                    |
| Configs tested against policy     | 1                     |
| Total rules evaluated             | 21                    |
| Total rules skipped               | 0                     |
| Total rules failed                | 4                     |
| Total rules passed                | 17                    |
| See all rules in policy           | https://app.datree.io |
+-----------------------------------+-----------------------+

From the output, you can see detailed information about the violations present in the manifest. This gives engineers the necessary guidance to resolve them.

Each Datree policy check is performed using the default policy, which includes 50+ built-in rules.

To configure the policy, you must return to the terminal and register by clicking on the link provided at the end of the output.

kubectl plugin

This kubectl plugin extends the Datree CLI's capabilities to allow scanning resources within your cluster for misconfigurations.

The Kubectl plugin can be installed using the command:

kubectl krew install datree

What you should see:

$ kubectl krew install datree
Updated the local copy of plugin index.
Installing plugin: datree
Installed plugin: datree
\
 | Use this plugin:
 |  kubectl datree
 | Documentation:
 |  https://github.com/datreeio/kubectl-datree
 | Caveats:
 | \
 |  | Before using this plugin, the Datree CLI needs to be installed.
 |  | See https://hub.datree.io/ for quick and easy installation.
 | /
/

Let's try to check our cluster's namespace (having this release by Bitnami deployed in it).

Right now, the default K8s schema version that is checked when you run a policy check is 1.2. To check for deprecated APIs before deploying you K8s manifest, change the default K8s version in your dashboard to match your culster server version.

Author's cluster playground is k3s/k3d, so API version is

$ k3d version
k3s version v1.22.7-k3s1 (default)

Another (and more common) approach:

$ kubectl version --short
Server Version: v1.22.7+k3s1

🥁 Let's perform a scan routine with Datree now.

$ kubectl datree test -s "1.22.7" -- service my-release-drupal

(Summary)

- Passing YAML validation: 1/1

- Passing Kubernetes (1.22.7) schema validation: 1/1

- Passing policy check: 1/1

+-----------------------------------+-----------------------+
| Enabled rules in policy "Default" | 21                    |
| Configs tested against policy     | 1                     |
| Total rules evaluated             | 21                    |
| Total rules skipped               | 0                     |
| Total rules failed                | 0                     |
| Total rules passed                | 21                    |
| See all rules in policy           | https://app.datree.io |
+-----------------------------------+-----------------------+

The following cluster resources in namespace 'default' were checked:

service/my-release-drupal

You may see a very similar report on Datree's SaaS dashboard:

If there are no rules failed, then you get a green mark! Well done.

helm plugin

This plugin is used to check charts against Datree policy. The mentioned plugin can be installed using the command:

helm plugin install https://github.com/datreeio/helm-datree

To run a Datree policy check on Helm charts, run a command with the following syntax.

helm datree test [CHART_DIRECTORY]

If you need to pass arguments to your template, add -- in front of them, as shown below:

helm datree test [CHART_DIRECTORY] -- --values ​​values.yaml --set name=prod

CI integration

To integrate Datree into CI/CD, you can follow the example below. You need to follow these steps:

• Get your account token (you can find it in the dashboard's Settings).
• Set DATREE_TOKEN as secret/environment variable
• Add Datree to your CI flow via token as shown (i.e., for GitHub).

🖼️ A larger picture is here. Here is the example of involving action:

on:
  push:
    branches: [ main ]
  pull_request:
    branches: [ main ]

env:
  DATREE_TOKEN: ${{ secrets.DATREE_TOKEN }} 

jobs:
  k8sPolicyCheck:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout
        uses: actions/checkout@v2

      - name: Run Datree Policy Check
        uses: datreeio/action-datree@main
        with:
          path: '**/*.yaml'
          cliArguments: '--only-k8s-files'

Instead of a conclusion

This brief article on using Datree to perform security checks on Helm charts and Kubernetes manifests is now complete. The author hopes readers can all agree that Datree can be used to prevent configuration problems in Kubernetes that can result in production cluster failure.

Wishing you to have clean YAMLs and green marks, folks! ✅

The author expresses his sincere gratitude to Noaa 🐦@BarkiNoaa Barki, Eyar 🐦@eyarzilb Zilberman, Anais 🐦@urlichsanais Urlich and Scott 🐦@chiefmartec Brinker.