I have running prometheus in monitoring namespace. prometheus installed with helm chart. NATS also. NATS making connection between components specially between each application pods messaging to other pods.

NATS is a popular open source messaging system that is used in Kubernetes applications to provide reliable, secure, and high-performance communication between components. It enables the decentralized components of an application to communicate with each other in a distributed manner, allowing for faster and more reliable communication while providing scalability and fault-tolerance. Nats also supports service discovery, authentication, authorization, logging, and monitoring, making it a powerful tool for managing applications in a Kubernetes environment.

this post will let you configure how to get metrics of NATS. there we have to run agent on cluster with nats. simply we can export metrics with Prometheus NATS exporter.

• We have already prometheus is running

• Now installed Nats with Helm charts.

Pre requisites (if nats earliest version or service monitor configuration) we need to enable Monitoring CRD for service monitoring configuration until nats wont be install if exporter enabled with service monitoring.

1. apply this CRD :

2. Create service monitor spec:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: staging-nats-service-monitor
  namespace: staging-nats
spec:
  endpoints:
    - interval: 15s
      port: metrics
      scrapeTimeout: 14s
  namespaceSelector:
    matchNames:
      - staging-nats
  selector:
    matchLabels:
      app.kubernetes.io/name: prometheus #here you should put a labels of prometheus server
      app.kubernetes.io/instance: staging-prometheus#this also refers running prometheus

1. then configure the helm values file to enable metrics

exporter:
  enabled: true
  image: natsio/prometheus-nats-exporter:0.10.0
  portName: metrics
  pullPolicy: IfNotPresent
  securityContext: {}
  resources: {}
  # Prometheus operator ServiceMonitor support. Exporter has to be enabled
  serviceMonitor:
    enabled: true
    ## Specify the namespace where Prometheus Operator is running
    ##
    namespace: staging-monitoring
    labels: {}
    # annotations: 
    #   prometheus.io/port: "metrics"
    #   prometheus.io/scrape: "true"
    path: /metrics

now you have done NATS pod will running along with Exporter container. but is this enough?

No because you have to tell prometheus to scrap my metrics !!!!

1. prometheus values file have configuration line call "scrap_configs."

scrape_configs:
 - job_name: "Nats_metrics"
   scrape_interval: 5s
   static_configs:
      - targets:
         ["localhost:7777"] #if you enable nodeport if not use kubernetes dns name config for example "staging-nats.nats-svc.svc.cluster.local:7777"

1. then restart prometheus server. ( go to the prometheus-server/metrics page then you can see all the metrics)

find the use full dashboards if you use grafana :

**

more advanced: Engineers likely to go wil Terraform implementation below explain how we can able to deploy NATS with Terraform

**

why prefer to go with this document error below :

• error: resource mapping not found for name: "servicemonitors.monitoring.coreos.com" namespace: "" from ".\service-monitor.yaml": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1beta1" ensure CRDs are installed first

please clone terraforming NATS repo with metrics export configuration for prometheus Terraform build : (https://github.com/sham97/Helm-Nats_terraform.git
)

Reference : (https://stackoverflow.com/questions/61681450/how-to-connect-nats-streaming-cluster)