🚀 Automating Image Vulnerability Patching in Kubernetes with Trivy Operator, Copacetic, and GitHub Actions

1. Installing and Using Copacetic (Copa) CLI

Copacetic is a CLI tool (copa) designed to help automate the patching of vulnerabilities in your container images. Here’s how to install it:

1.1. Clone the Copacetic Repository

Start by cloning the Copacetic repository to your local machine:

git clone https://github.com/project-copacetic/copacetic
cd copacetic

1.2. Build Copacetic

Inside the cloned directory, build the Copacetic CLI:

make

1.3. Install Copa CLI (Optional)

If you want to install the copa command to a directory in your system's PATH, move the binary to a pathed folder:

sudo mv dist/linux_amd64/release/copa /usr/local/bin/

This allows you to run copa from anywhere in your terminal.

2. Scanning and Patching Images with Trivy and Copa

Once you have installed both Trivy and Copacetic, you can start scanning your images and patching vulnerabilities.

2.1. Scan the Container Image for Patchable OS Vulnerabilities

Use Trivy to scan the container image for operating system vulnerabilities. The results will be output to a JSON file:

trivy image --vuln-type os --ignore-unfixed -f json -o $(basename $IMAGE).json $IMAGE

Replace $IMAGE with the name of the container image you want to scan (e.g., nginx:latest).
The command will create a JSON file with the same name as the image, which contains details about the vulnerabilities found.

2.2. Patch the Image Using Copa

After generating the Trivy report, use Copa to patch the image based on the vulnerabilities reported:

copa patch -r $(basename $IMAGE).json -i $IMAGE

This command reads the vulnerability report from the JSON file and patches the specified image.
By default, Copa will produce a new image with a -patched suffix (e.g., nginx-patched), tagging it with the original image tag suffixed with -patched.
The new patched image will be exported to your local Docker daemon.

3. Installing Trivy Operator with Helm

Once you’ve patched your container images, you can use Trivy Operator to continuously monitor your AKS cluster for any vulnerabilities.

3.1. Add the Aqua Security Helm Repository

First, add the Aqua Security Helm repository:

helm repo add aqua https://aquasecurity.github.io/helm-charts/

For more details, visit the Aqua Security Helm Charts documentation.

3.2. Update Your Helm Repositories

Update your Helm repositories to get the latest charts:

helm repo update

3.3. Install Trivy Operator

Finally, install the Trivy Operator:

helm install trivy-operator aqua/trivy-operator --namespace trivy-system --create-namespace

Trivy Operator will now be running in your AKS cluster, ready to scan for vulnerabilities. You can find more detailed instructions in the Trivy Operator documentation.

4. Exporting Trivy Metrics

Once Trivy Operator is up and running, it’s important to monitor the health of your cluster by exporting Trivy metrics. These metrics provide valuable insights into the security posture of your container images.

4.1. Enable Metrics Export

Trivy Operator can export metrics to Prometheus, allowing you to monitor the security status of your AKS cluster over time.

Edit the Trivy Operator ConfigMap:

Enable the metrics exporter in the Trivy Operator's ConfigMap:

   kubectl edit configmap trivy-operator-config -n trivy-system

Add or Modify the Following Configuration:

Ensure the following settings are enabled in the ConfigMap:

   data:
     trivy.severity: "HIGH,CRITICAL"
     trivy.resources.requests.cpu: "100m"
     trivy.resources.requests.memory: "100Mi"
     trivy.resources.limits.cpu: "500m"
     trivy.resources.limits.memory: "500Mi"
     trivy.metrics.enabled: "true"

This configuration ensures that Trivy Operator will export security metrics to Prometheus.

4.2. Access Trivy Metrics

With metrics enabled, you can scrape them using Prometheus and visualize them in Grafana or another monitoring tool of your choice. This allows you to track the number of vulnerabilities over time, monitor the effectiveness of your security patches, and respond to new threats promptly.

5. Setting Up a Custom Webhook to Trigger a GitHub Action for Copa

To fully automate the process of patching images, you can set up a custom webhook that triggers a GitHub Action when a vulnerability is detected by Trivy Operator.

5.1. Create a Custom Webhook

First, set up a custom webhook in your Trivy Operator configuration that triggers when a vulnerability report is generated.

Edit the Trivy Operator ConfigMap:

   kubectl edit configmap trivy-operator-config -n trivy-system

Add the Webhook Configuration:

Add the webhook URL that points to your GitHub repository's webhook endpoint:

   data:
     trivy.webhook.enabled: "true"
     trivy.webhook.url: "https://api.github.com/repos/your-username/your-repo/dispatches"
     trivy.webhook.method: "POST"
     trivy.webhook.payload: |
       {
         "event_type": "trivy_vulnerability_detected",
         "client_payload": {
           "image": "$IMAGE",
           "report_url": "$REPORT_URL"
         }
       }

Replace "your-username/your-repo" with your GitHub username and repository name.
This webhook sends a payload to GitHub whenever Trivy detects a vulnerability.

5.2. Set Up a GitHub Action to Patch the Image Using Copa

Now, create a GitHub Action in your repository that uses the copa-action to patch the image.

Create a GitHub Action Workflow:

In your GitHub repository, create a new file at .github/workflows/patch-image.yml:

   name: Patch Docker Image

   on:
     repository_dispatch:
       types: [trivy_vulnerability_detected]

   jobs:
     patch:
       runs-on: ubuntu-latest

       steps:
         - name: Checkout repository
           uses: actions/checkout@v2

         - name: Download Trivy Report
           run: curl -L ${{ github.event.client_payload.report_url }} -o report.json

         - name: Patch the Docker Image with Copa
           uses: project-copacetic/copa-action@v1
           with:
             report: report.json
             image: ${{ github.event.client_payload.image }}

         - name: Push Patched Image to Docker Hub
           run: |
             docker tag ${{ github.event.client_payload.image }}:patched your-docker-repo/${{ github.event.client_payload.image }}:patched
             docker push your-docker-repo/${{ github.event.client_payload.image }}:patched

Replace your-docker-repo with your Docker registry's name.
This workflow will automatically run when the webhook from Trivy Operator triggers it, patching the vulnerable image using copa-action and pushing the patched image to your Docker registry.

🎉 Wrapping Up

By integrating Trivy Operator with Copacetic and GitHub Actions, you've created a fully automated workflow for detecting, patching, and monitoring vulnerabilities in your Kubernetes environment. Trivy Operator scans and exports metrics, while the custom webhook and GitHub Action using copa-action handle the patching process. This setup ensures that your container images are always up-to-date and secure, with minimal manual intervention.

This approach provides an efficient, automated solution for managing security in your Kubernetes environment, allowing you to focus on innovation with peace of mind! 🚀🔒

For further reading and more detailed guides, you can refer to the following resources:

Happy clustering and stay safe !

Blog