Unlock Native Fluent Bit Power in EKS Fargate: Seamless Log Shipping to CloudWatch

Amazon EKS Fargate now supports a built-in Fluent Bit-based log router, eliminating the need for running sidecar containers to collect and ship logs. This simplifies log management by reducing overhead and complexity while providing an efficient way to ship logs from EKS pods to Amazon CloudWatch.

This article will guide you through configuring Fluent Bit on EKS Fargate to ship logs to CloudWatch, both globally for all pods and specifically for individual deployments.

Built-in Logging with Fluent Bit

EKS Fargate uses Fluent Bit for log routing. By applying a ConfigMap in the aws-observability namespace, you can configure log routing for all pods in the cluster. The logging ConfigMap affects applications cluster-wide, regardless of their namespace.

Step 1 : Grant Required IAM Permissions

Ensure the Fargate pod execution role includes the following permissions to allow Fluent Bit to send logs to CloudWatch:

{
 "Version": "2012-10-17",
 "Statement": [{
  "Effect": "Allow",
  "Action": [
   "logs:CreateLogStream",
   "logs:CreateLogGroup",
   "logs:DescribeLogStreams",
   "logs:PutLogEvents",
   "logs:PutRetentionPolicy"
  ],
  "Resource": "*"
 }]
}

Step 2 : Create the aws-observability Namespace

The Fluent Bit configuration must reside in the aws-observabilitynamespace.

apiVersion: v1
kind: Namespace
metadata:
  name: aws-observability
  labels:
    aws-observability: enabled

Apply this namespace using:

kubectl apply -f namespace.yaml

Step 3 : Configure Fluent Bit

Send All Logs to One CloudWatch Log Group

To route all logs to a single CloudWatch log group, create the following ConfigMap:

apiVersion: v1
kind: ConfigMap
metadata:
  name: aws-logging
  namespace: aws-observability
data:
  output.conf: |
    [OUTPUT]
        Name cloudwatch_logs
        Match   *
        region <region-code>
        log_group_name fluent-bit-cloudwatch
        log_stream_prefix from-fluent-bit-
        auto_create_group true

Use Dedicated Log Groups for Each Deployment

If you wish to route logs to specific CloudWatch log groups per deployment, use a more detailed ConfigMap:

kind: ConfigMap
apiVersion: v1
metadata:
  name: aws-logging
  namespace: aws-observability
data:
  output.conf: |
    [OUTPUT]
        Name cloudwatch_logs
        Match kube.var.log.containers.deployment1*
        region eu-west-1
        log_group_name fluent-bit-cloudwatch-deployment1
        log_stream_prefix from-fluent-bit-
        auto_create_group true

    [OUTPUT]
        Name cloudwatch_logs
        Match kube.var.log.containers.deployment2*
        region eu-west-1
        log_group_name fluent-bit-cloudwatch-deployment2
        log_stream_prefix from-fluent-bit-
        auto_create_group true

  filters.conf: |
    [FILTER]
        Name                kubernetes
        Match               kube.*
        Merge_Log           On
        Merge_Log_Key       log_processed
        K8S-Logging.Parser  On
        K8S-Logging.Exclude On

  parsers.conf: |
    [PARSER]
        Name                docker
        Format              json
        Time_Key            time
        Time_Format         %Y-%m-%dT%H:%M:%S.%LZ


#Replace the "deployment1" with your deployment name, and re-use the output conf as per the number of deployments you have.

Step 4 : Restart Fargate Deployments

After applying the ConfigMap, restart your Fargate pods to apply the new logging configuration:

Validation

Check if Logging is Enabled

To verify that logging is enabled for a pod, describe the pod using kubectl:

kubectl -n <namespace> describe po <pod-name>

Look for the following annotation:

Annotations:       Logging: LoggingEnabled

Verify Logs in CloudWatch:

Open the CloudWatch console.
Navigate to Log groups.
Confirm that the logs are being shipped to the configured log groups.

That’s it. Thank you for taking the time to read this article! Keep up the great work, and happy deploying! 🚀 😊

Blog

Unlock Native Fluent Bit Power in EKS Fargate: Seamless Log Shipping to CloudWatch

Aakash Sai Raj

Built-in Logging with Fluent Bit

Join Our Newsletter. No Spam, Only the good stuff.

Related