Hello, I hope you are well and safe, I don't know if you scrape Metrics from your Kubernetes workloads dynamically using Consul Service Catalog as Service Discovery Method.

I would like to mention that all the workloads are running on Kubernetes, so in order to monitor them I am using Prometheus Operator, as a result, I have configured it to identify the pods that match specific criteria in the metadata fields.

Finally, in the following example, I will try to show you how you can define a Prometheus scraper that uses Consul Service Catalog as a Service Registry.

Let's Dive in...

          - job_name: "test-consul-sd-backend"
            consul_sd_configs:
              - server: "consul-server.consul-system.svc.cluster.local:8500"
            relabel_configs:
              - source_labels: [__meta_consul_service]
                regex: .*backend.*
                action: keep
              - source_labels: [__meta_consul_service]
                target_label: job
            scrape_interval: 60s
            metrics_path: "/api/metrics"

of course, you can any of the fields found in the Prometheus documentation: https://prometheus.io/docs/prometheus/latest/configuration/configuration/#consul_sd_config

I hope you like the tutorial, if you do give it a thumps up! and follow me on Twitter, also you can subscribe to my Newsletter in order to avoid missing any of the upcoming tutorials.

Media Attribution

I would like to thank Clark Tibbs for designing the awesome photo I am using in my posts.

Thank you, Cheers!!!