Filebeat config on k8s after switching to containerd

You can not ignore it, dockershim (layer for using Docker Runtime in Kubernetes) will be removed starting 1.24. Do not worry, it's a change pretty seamless and your images built with Docker will still be fully functional.

But it's pretty sure that if your current cluster nodes are running through Docker Runtime, then you have some hardcoded configuration tight to Docker.

In this article we will focus on a filebeat configuration originally setup for Docker Runtime, and what needs to be done after the switch to containerd in order to keep getting your precious logs.

The main steps are updating your filebeat config file :

activating symlinks option
update the path of the logs files
use together dissect and drop_fields processor to only parse and keep the necessary

Then after that update the volumeMounts section of your filebeat DaemonSet definition :

each existing mountPath or path with value /var/lib/docker/containers will need to be changed to /var/log/containers

Here is a snippet of a filebeat config file that worked for me, do not hesitate to let us know if it helped you in some way or if you have a suggestion for improvement :

apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-config
  namespace: kube-system
data:
  filebeat.yml: |-
    setup.ilm.enabled: false
    filebeat.inputs:
    - type: log
      symlinks: true
      paths:
        - /var/log/containers/*.log
      processors:
        - add_kubernetes_metadata:
            host: ${NODE_NAME}
            in_cluster: true
            default_matchers.enabled: false
            matchers:
            - logs_path:
                logs_path: /var/log/containers/

    processors:
      - add_cloud_metadata:
      - drop_event:
          when:
            equals:
              kubernetes.namespace: "kube-system"
      - dissect:
          tokenizer: "%{timestamp} %{std} %{capital-letter} %{parsed-message}"
          field: "message"
          target_prefix: ""
      - decode_json_fields:
          fields: ["message","log","logs.log","parsed-message"]
          target: "logs"
          process_array: true
      - drop_fields:
          when:
            regexp:
              message: "^{\""
          fields: ["message"]
          ignore_missing: true
      - drop_fields:
          fields: ["log.file.path","timestamp","std","capital-letter","parsed-message"]
          ignore_missing: true

...

Have a great day!

Blog

Filebeat config on k8s after switching to containerd

Lucien Boix

Join Our Newsletter. No Spam, Only the good stuff.

Related