TheLazy.Dev
Portfolio GitHub

Blog

Install Prometheus and Grafana on Kubernetes Cluster.

jasper475

Jasper Rodda

Posted on January 22, 2024

Install Prometheus and Grafana on Kubernetes Cluster.

Prometheus is an open-source monitoring and alerting system that helps you collect and store metrics about your software systems and infrastructure and analyze that data to gain insights into their health and performance. It provides a powerful query language, a flexible data model, and a range of integrations with other tools and systems. With Prometheus, you can easily monitor metrics such as CPU usage, memory usage, network traffic, and application-specific metrics, and use that data to troubleshoot issues, optimize performance, and create alerts to notify you when things go wrong.

Install Prometheus via 2 ways.

  1. Helm Charts
  2. Operators

Using operators, using Operator life cycle management (OLM) one can automatically have latest updates without worrying to update manually and therefore securing cluster with zero-day vulnerabilities.

Install using Helm.

1. Add helm repo 

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
Enter fullscreen mode Exit fullscreen mode

2. Update helm repo 

helm repo update
Enter fullscreen mode Exit fullscreen mode

3. Install helm 

helm install prometheus prometheus-community/prometheus
Enter fullscreen mode Exit fullscreen mode 
PS C:\WINDOWS\system32> helm install prometheus prometheus-community/prometheus
NAME: prometheus
LAST DEPLOYED: Mon Jan 22 12:55:19 2024
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
The Prometheus server can be accessed via port 80 on the following DNS name from within your cluster:
prometheus-server.default.svc.cluster.local


Get the Prometheus server URL by running these commands in the same shell:
  export POD_NAME=$(kubectl get pods --namespace default -l "app.kubernetes.io/name=prometheus,app.kubernetes.io/instance=prometheus" -o jsonpath="{.items[0].metadata.name}")
  kubectl --namespace default port-forward $POD_NAME 9090


The Prometheus alertmanager can be accessed via port 9093 on the following DNS name from within your cluster:
prometheus-alertmanager.default.svc.cluster.local


Get the Alertmanager URL by running these commands in the same shell:
  export POD_NAME=$(kubectl get pods --namespace default -l "app.kubernetes.io/name=alertmanager,app.kubernetes.io/instance=prometheus" -o jsonpath="{.items[0].metadata.name}")
  kubectl --namespace default port-forward $POD_NAME 9093
#################################################################################
######   WARNING: Pod Security Policy has been disabled by default since    #####
######            it deprecated after k8s 1.25+. use                        #####
######            (index .Values "prometheus-node-exporter" "rbac"          #####
###### .          "pspEnabled") with (index .Values                         #####
######            "prometheus-node-exporter" "rbac" "pspAnnotations")       #####
######            in case you still need it.                                #####
#################################################################################


The Prometheus PushGateway can be accessed via port 9091 on the following DNS name from within your cluster:
prometheus-prometheus-pushgateway.default.svc.cluster.local


Get the PushGateway URL by running these commands in the same shell:
  export POD_NAME=$(kubectl get pods --namespace default -l "app=prometheus-pushgateway,component=pushgateway" -o jsonpath="{.items[0].metadata.name}")
  kubectl --namespace default port-forward $POD_NAME 9091

For more information on running Prometheus, visit:
https://prometheus.io/
Enter fullscreen mode Exit fullscreen mode

- Verify Prometheus Pods Running 

$ kubectl get pods
NAME                                                 READY   STATUS    RESTARTS        AGE
prometheus-alertmanager-0                            1/1     Running   0               3m8s
prometheus-kube-state-metrics-745b475957-ngqm7       1/1     Running   0               3m8s
prometheus-prometheus-node-exporter-698nc            1/1     Running   0               3m8s
prometheus-prometheus-pushgateway-6ccd698d79-ld86d   1/1     Running   0               3m8s
prometheus-server-bc7ccb595-jpl7p                    2/2     Running   0               3m8s
sample-python-deployment-99fcf4b6f-rb59w             1/1     Running   2 (5m40s ago)   8d
sample-python-deployment-99fcf4b6f-vcnwt             1/1     Running   2 (5m40s ago)   8d
Enter fullscreen mode Exit fullscreen mode

- Verify Prometheus services via Kubectl get svc 

$ kubectl get svc
NAME                                  TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)        AGE
kubernetes                            ClusterIP      10.96.0.1        <none>        443/TCP        8d
prometheus-alertmanager               ClusterIP      10.100.6.213     <none>        9093/TCP       7m38s
prometheus-alertmanager-headless      ClusterIP      None             <none>        9093/TCP       7m38s
prometheus-kube-state-metrics         ClusterIP      10.103.166.42    <none>        8080/TCP       7m38s
prometheus-prometheus-node-exporter   ClusterIP      10.104.211.162   <none>        9100/TCP       7m38s
prometheus-prometheus-pushgateway     ClusterIP      10.100.179.109   <none>        9091/TCP       7m38s
prometheus-server                     ClusterIP      10.105.170.132   <none>        80/TCP         7m38s
sample-python-service                 LoadBalancer   10.101.160.8     <pending>     80:30007/TCP   8d
Enter fullscreen mode Exit fullscreen mode

4. Expose Prometheus Service

This is required to access prometheus-server using your browser.

kubectl expose service prometheus-server --type=NodePort --target-port=9090 --name=prometheus-server-ext
Enter fullscreen mode Exit fullscreen mode

  • Notice below is created prometheus-server-ext NodePort 10.99.142.160 <none> 80:31401/TCP 3s

  • Get svc

 kubectl get svc
NAME                                  TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)        AGE
kubernetes                            ClusterIP      10.96.0.1        <none>        443/TCP        8d
prometheus-alertmanager               ClusterIP      10.100.6.213     <none>        9093/TCP       10m
prometheus-alertmanager-headless      ClusterIP      None             <none>        9093/TCP       10m
prometheus-kube-state-metrics         ClusterIP      10.103.166.42    <none>        8080/TCP       10m
prometheus-prometheus-node-exporter   ClusterIP      10.104.211.162   <none>        9100/TCP       10m
prometheus-prometheus-pushgateway     ClusterIP      10.100.179.109   <none>        9091/TCP       10m
prometheus-server                     ClusterIP      10.105.170.132   <none>        80/TCP         10m
prometheus-server-ext                 NodePort       10.99.142.160    <none>        80:31401/TCP   3s
sample-python-service                 LoadBalancer   10.101.160.8     <pending>     80:30007/TCP   8d
Enter fullscreen mode Exit fullscreen mode

Output:

$ kubectl expose service prometheus-server --type=NodePort --target-port=9090 --name=prometheus-server-ext
service/prometheus-server-ext exposed
Enter fullscreen mode Exit fullscreen mode

Get Minikube IP 

minikube ip
192.168.59.107
Enter fullscreen mode Exit fullscreen mode

Open Browser and access Prometheus

- Go to http://192.168.59.107:31401/

Prometheus Dashboard

Install Grafana

1. Install using Helm

  • Add helm repo
  • _Ignore is you already have help and grafana repo added _ helm repo add grafana https://grafana.github.io/helm-charts 
$helm repo add grafana https://grafana.github.io/helm-charts
"grafana" has been added to your repositories
Enter fullscreen mode Exit fullscreen mode

2. Update helm repo

  • helm repo update 
$ helm repo update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "eks" chart repository
...Successfully got an update from the "istio" chart repository
...Successfully got an update from the "grafana" chart repository
...Successfully got an update from the "prometheus-community" chart repository
Update Complete. ⎈Happy Helming!⎈
Enter fullscreen mode Exit fullscreen mode

3. Install Grafana via helm

  • helm install grafana grafana/grafana 
$ helm install grafana grafana/grafana
NAME: grafana
LAST DEPLOYED: Mon Jan 22 13:20:52 2024
NAMESPACE: default
STATUS: deployed
REVISION: 1
NOTES:
1. Get your 'admin' user password by running:

   kubectl get secret --namespace default grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo


2. The Grafana server can be accessed via port 80 on the following DNS name from within your cluster:

   grafana.default.svc.cluster.local

   Get the Grafana URL to visit by running these commands in the same shell:
     export POD_NAME=$(kubectl get pods --namespace default -l "app.kubernetes.io/name=grafana,app.kubernetes.io/instance=grafana" -o jsonpath="{.items[0].metadata.name}")
     kubectl --namespace default port-forward $POD_NAME 3000

3. Login with the password from step 1 and the username: admin
#################################################################################
######   WARNING: Persistence is disabled!!! You will lose your data when   #####
######            the Grafana pod is terminated.                            #####
#################################################################################
Enter fullscreen mode Exit fullscreen mode

4. Get Grafana Login Details

  • Get password: $ kubectl get secret --namespace default grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo 
$ kubectl get secret --namespace default grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo
XPhzARnKv93uhRGt8Nu9cimhuAHZoj64ZtCHDeQd
Enter fullscreen mode Exit fullscreen mode
  • Get Grafana services kubectl get svc Notice its running on Cluster IP 
kubectl get svc
NAME                                  TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)        AGE
grafana                               ClusterIP      10.109.8.93      <none>        80/TCP         8m12s
kubernetes                            ClusterIP      10.96.0.1        <none>        443/TCP        8d
prometheus-alertmanager               ClusterIP      10.100.6.213     <none>        9093/TCP       33m
prometheus-alertmanager-headless      ClusterIP      None             <none>        9093/TCP       33m
prometheus-kube-state-metrics         ClusterIP      10.103.166.42    <none>        8080/TCP       33m
prometheus-prometheus-node-exporter   ClusterIP      10.104.211.162   <none>        9100/TCP       33m
prometheus-prometheus-pushgateway     ClusterIP      10.100.179.109   <none>        9091/TCP       33m
prometheus-server                     ClusterIP      10.105.170.132   <none>        80/TCP         33m
prometheus-server-ext                 NodePort       10.99.142.160    <none>        80:31401/TCP   23m
sample-python-service                 LoadBalancer   10.101.160.8     <pending>     80:30007/TCP   8d
Enter fullscreen mode Exit fullscreen mode

5. Expose Grafana Service to Access IT 

kubectl expose service grafana --type=NodePort --target-port=3000 --name=grafana-ext
Enter fullscreen mode Exit fullscreen mode
  • Verify exposed Grafana service 
kubectl expose service grafana --type=NodePort --target-port=3000 --name=grafana-ext
service/grafana-ext exposed
Enter fullscreen mode Exit fullscreen mode
  • Get Grafana-Ext Service at port 30197 
 kubectl get svc
NAME                                  TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)        AGE
grafana                               ClusterIP      10.109.8.93      <none>        80/TCP         11m
grafana-ext                           NodePort       10.111.66.82     <none>        80:30197/TCP   51s
kubernetes                            ClusterIP      10.96.0.1        <none>        443/TCP        8d
prometheus-alertmanager               ClusterIP      10.100.6.213     <none>        9093/TCP       37m
prometheus-alertmanager-headless      ClusterIP      None             <none>        9093/TCP       37m
prometheus-kube-state-metrics         ClusterIP      10.103.166.42    <none>        8080/TCP       37m
prometheus-prometheus-node-exporter   ClusterIP      10.104.211.162   <none>        9100/TCP       37m
prometheus-prometheus-pushgateway     ClusterIP      10.100.179.109   <none>        9091/TCP       37m
prometheus-server                     ClusterIP      10.105.170.132   <none>        80/TCP         37m
prometheus-server-ext                 NodePort       10.99.142.160    <none>        80:31401/TCP   26m
sample-python-service                 LoadBalancer   10.101.160.8     <pending>     80:30007/TCP   8d
Enter fullscreen mode Exit fullscreen mode

- Login via user=admin and password=XPhzARnKv93uhRGt8Nu9cimhuAHZoj64ZtCHDeQd

Grafana login page

  • Grafana Dashboard

Grafana Dashboard

  • Configure Prometheus as Datasource to Grafana
  • Go to Data sources --> Add Prometheus --> Provide IP Address - Save and Test

Save and Test configuration

Configure Prometheus as Datasource to Grafana

  • Import an existing dashboard

  • Dashboard --> Import --> Enter ID 3662 --> Load

Import Dashboard 3662

configure Dashboard 3662

  • Dashboard

Grafana Dashboard description

6. Expose prometheus-kube-state-metrics

  • How do we do it by following the steps below. 
kubectl expose service prometheus-kube-state-metrics --type=NodePort --target-port=8080 --name=prometheus-kube-state-metrics-ext
Enter fullscreen mode Exit fullscreen mode
  • Verify kube-state-metrics exposed 
kubectl get svc
NAME                                  TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)          AGE
grafana                               ClusterIP      10.109.8.93      <none>        80/TCP           54m
grafana-ext                           NodePort       10.111.66.82     <none>        80:30197/TCP     43m
kubernetes                            ClusterIP      10.96.0.1        <none>        443/TCP          8d
prometheus-alertmanager               ClusterIP      10.100.6.213     <none>        9093/TCP         79m
prometheus-alertmanager-headless      ClusterIP      None             <none>        9093/TCP         79m
prometheus-kube-state-metrics         ClusterIP      10.103.166.42    <none>        8080/TCP         79m
prometheus-kube-state-metrics-ext     NodePort       10.100.12.19     <none>        8080:30229/TCP   15s
prometheus-prometheus-node-exporter   ClusterIP      10.104.211.162   <none>        9100/TCP         79m
prometheus-prometheus-pushgateway     ClusterIP      10.100.179.109   <none>        9091/TCP         79m
prometheus-server                     ClusterIP      10.105.170.132   <none>        80/TCP           79m
prometheus-server-ext                 NodePort       10.99.142.160    <none>        80:31401/TCP     69m
sample-python-service                 LoadBalancer   10.101.160.8     <pending>     80:30007/TCP     8d
Enter fullscreen mode Exit fullscreen mode
  • Open browser and access via port 30229

- http://192.168.59.107:30229/

Kube state metrics external service

  • Kubectl get cm or configmap 
kubectl get cm
NAME                      DATA   AGE
grafana                   1      61m
kube-root-ca.crt          1      8d
prometheus-alertmanager   1      86m
prometheus-server         6      86m
Enter fullscreen mode Exit fullscreen mode
  • Edit config map of prometheus-server

kubectl edit cm prometheus-server

change scrpe confi from local-host to prometheus-kube-state-metrics-ext service

  • BEFORE: 
scrape_configs:
    - job_name: prometheus
      static_configs:
      - targets:
        - localhost:9090
Enter fullscreen mode Exit fullscreen mode
  • AFTER: Add config to http://192.168.59.107:30229/ state-metrics-ext 
scrape_configs:
    - job_name: prometheus
      static_configs:
      - targets:
        - localhost:9090
    - job_name: prometheus
      static_configs:
      - targets:
        - http://192.168.59.107:30229
Enter fullscreen mode Exit fullscreen mode

Credits:-
Thanks to Abhishek Veeramalla

💖 💪 🙅 🚩
jasper475
Jasper Rodda

Posted on January 22, 2024

Join Our Newsletter. No Spam, Only the good stuff.

Sign up to receive the latest update from our blog.

Related

What was your win this week?
weeklyretro What was your win this week?

November 29, 2024

If you're a beginner, definitely check this open source guide. I've explained almost everything you need to know.
undefined If you're a beginner, definitely check this open source guide. I've explained almost everything you need to know.

November 28, 2024

Exploring Typesafe design tokens in Tailwind 4
tailwindcss Exploring Typesafe design tokens in Tailwind 4

November 29, 2024

Where GitOps Meets ClickOps
devops Where GitOps Meets ClickOps

November 29, 2024

The best way to get better at writing code is...
development The best way to get better at writing code is...

November 28, 2024