Stack

• K8s
• FluxCD
• AWS Load Balancer Controller
• Prometheous
• Grafana

Problem

Prometheous has CRDs for K8s known as ServiceMonitor and PodMonitor. I created a ServiceMonitor for AWS Load Balancer Controller, but I had no clue if Prometheous was collecting it's data.

AI Interactions

During troubleshooting, I was starting to doubt that I even had an AWS Load balancer controller service.

I asked:

should the aws-load-balancer-controller from eks-charts deploy a aws-load-balancer-controller that would show up when I run kubectl get service -n kube-system

ChatGPT: Terraform Engineer

[no] ... AWS Load Balancer Controller primarily operates as a controller deployment in your Kubernetes cluster. It does not typically expose a service that you can see with kubectl get service -n kube-system.

Wow! ChatGPT knew better than me that there is a difference between a K8s Service-Resource and a Service referenced by a ServiceMonitor. Now reassured that my Load Balancer Controller was deployed to my cluster just fine as a Deployment and not a Service I asked:

how can I connect the aws-load-balancer-controller to prometheous

Along with a lot of suggestions, ChatGPT Terraform Engineer answered:

... Ensure that the AWS Load Balancer Controller is configured to expose metrics. This is usually done through command-line arguments in the controller's deployment configuration. Look for arguments like --metrics-bind-addr=ADDRESS where ADDRESS specifies the host and port where metrics are exposed (e.g., 0.0.0.0:8080) ...

ChatGPT had also provided a lot of details relevant to other deployment solutions, so I re-asked while giving it more context to my context.

how can I connect the aws-load-balancer-controller to prometheous ... given I'm currently deploying with a HelmRelease in FluxCD

ChatGPT Terraform Engineer instructed:

... Update Your HelmRelease Resource: ... with args - --metrics-bind-addr=0.0.0.0:8080

apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  name: aws-load-balancer-controller
  namespace: kube-system
spec:
  chart:
    spec:
      chart: aws-load-balancer-controller
      sourceRef:
        kind: HelmRepository
        name: eks
        namespace: flux-system
      version: <chart version>
  values:
    serviceAccount:
      create: true
    clusterName: <your-cluster-name>
    region: <your-region>
    vpcId: <your-vpc-id>
    args:
      - --metrics-bind-addr=0.0.0.0:8080

... After FluxCD applies the changes, you can verify that the AWS Load Balancer Controller is exposing metrics by checking the logs of the controller pods or using kubectl port-forward

So I deployed these updates then asked specifically>

provide the cli command for kubectl port-forward to access the metrics endpoint locally.

ChatGPT Terraform Engineer answered:

Find the Pod Name: ...

kubectl get pods -n kube-system -l app.kubernetes.io/name=aws-load-balancer-controller

Port Forward to Access Metrics: ...

kubectl port-forward -n kube-system <pod-name> 8080:8080

Access Metrics Locally ...

curl http://localhost:8080/metrics

Eureka! Metrics are available. Now to work on other 1/2.

I was embolden to finish the rest on my own, and after fumbling around and getting back to ChatGPT it suggested to run:

kubectl get prometheus kube-prometheus-stack-prometheus -n monitoring -o yaml

Which ultimately lead to finding it's matchLabels

 serviceMonitorSelector:
    matchLabels:
      release: kube-prometheus-stack

and adding them to the ServiceMonitor:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: autoscaler-monitor
  namespace: kube-system
  labels:
    release: kube-prometheus-stack
    app.kubernetes.io/part-of: flux
    app.kubernetes.io/component: monitoring

    ...

But this still didn't quite fix it. So it recommended looking at the prometheus logs:

kubectl logs deployment/kube-prometheus-stack-operator -n monitoring

Ultimately the issue was that the ServiceMonitor needed to be in the monitoring name space.

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: autoscaler-monitor
  namespace: monitoring
  labels:
    release: kube-prometheus-stack
    app.kubernetes.io/part-of: flux
    app.kubernetes.io/component: monitoring

    ...