Microservices architecture is going to be one of the essential features in software development in the coming years. Packing a large monolithic application into small containers poses various advantages. One of the key advantages I can think of is that if certain things fail, then a part of the application will be down and it can auto-heal rather than crashing the whole application. This is one of the reasons that when Instagram, Facebook, and Whatsapp crashes, only certain functionalities stop working. The application, as a whole, is still working. This serves a great advantage. Also, when the demand for a particular service or component increases, the number of components can be increased or decreased easily. The small containers which I am referring here are the Docker containers and the application is deployed on Kubernetes. Kubernetes is a container orchestration software which manages the containers at various levels and enables the management of connections and endpoints of these containers.

Kubernetes or K8s is an open-source self-healing application which manages the deployment, scaling, and operation of these containers. It was originally developed by Google but later on, donated to CNCF. Since Kubernetes provides tons of services, there needs to be an easier way to monitor the activities in the Kubernetes cluster. This is possible by Prometheus. It is an open-source solution to monitor the metrics and manage the alerts in the system. It was developed by SoundCloud, but later joined CNCF as the second hosted project after Kubernetes. Prometheus provided a rich set of monitoring metrics and alert management system which helps the developers to monitor and get notified about any unusual activity or consumption.

Prometheus Operator

CoreOs launched Prometheus operator to ease the process of integrating K8s with Prometheus. It preserves the configuration of both the K8s and Prometheus while installing and configuring the cluster. It provides easy monitoring for K8s services and deployments, along with managing Prometheus, Grafana and Alertmanager configuration.
When a new version of the application is deployed, K8s manages the creation of new pod and deletes the older one. Prometheus, on the other hand, constantly watches the K8s API and creates a new Prometheus configuration whenever it detects a change, based on the services/pods changes. It uses a ServiceMonitor, a CRD(Custom Resource Definition), to abstract the configuration to target.

Installation

Prerequisites

• Kubernetes
• Helm (Package installer for K8s)

Steps

Install Prometheus operator in a different namespace. It is preferable to keep your monitoring containers in a separate namespace.

$ helm install stable/prometheus-operator --name prometheus-operator --namespace monitor

If everything got installed perfectly, you can see these pods available:

$ kubectl get pods -n monitor
NAME                                                     READY   STATUS    RESTARTS   AGE
alertmanager-prometheus-operator-alertmanager-0          2/2     Running   0          13d
prometheus-operator-grafana-749b598b6c-t4r48             2/2     Running   0          13d
prometheus-operator-kube-state-metrics-d7b8b7666-zfqg5   1/1     Running   0          13d
prometheus-operator-operator-667dd7cbb7-hjbl6            1/1     Running   0          13d
prometheus-operator-prometheus-node-exporter-mgsqb       1/1     Running   0          13d
prometheus-prometheus-operator-prometheus-0              3/3     Running   1          13d

To run the dashboard, enter the following command and go to http://localhost:9000.

$ kubectl port-forward -n monitor prometheus-prometheus-operator-prometheus-0 9090

Prometheus Dashboard

You can enter your query to get the results about any particular instance or even a graph of it as shown in the figure above.
To see the visual representation at each level, we use Grafana. It provides some great visual insights regarding the usage, health and other metrics. We can also add more custom metrics. We will get real-time analysis of the data.

$ kubectl port-forward $(kubectl get  pods --selector=app=grafana -n  monitor --output=jsonpath="{.items..metadata.name}") -n monitor  3000

Go to http://localhost:3000 and enter ‘admin’ as username and ‘prom-operator’ as password. These are the available options:

You can get visual graphs by selecting any one of the options. Node-level metrics are shown here:

We can configure alerts in many ways. We can access the dashboard to configure the AlertManager by going to http://localhost:9093 after executing this command:

$ kubectl port-forward -n monitor alertmanager-prometheus-operator-alertmanager-0 9093

It’ll look like this. Here you can add more alerts and can see the Slack API URL under the status tab. We can set the notifications on Slack, HipChat or even email. Some of the templates are available here.

Conclusion

The use of microservices is going to increase soon and monitoring the metrics and alert notifications are going to be an essential part of it. Prometheus provides optimal monitoring services with easy installation services.