Server Monitoring with Prometheus and Grafana setup in Docker and Portainer
Huynh-Chinh
Posted on February 25, 2022
I-Introduction
Portainer is a free Docker Container management tool with compact size and intuitive management interface, simple to deploy and use, allowing users to easily manage Docker host or Swarm cluster. This tool works on a container deployed on Docker Engine.
Grafana is a leading time-series, an open-source platform for visualization and monitoring. It allows you to query, visualize, set alerts, and understand metrics no matter where they are stored. You can create amazing dashboards in Grafana to visualize and monitor the metrics.
Prometheus is an open-source time-series monitoring system for machine-centric and highly dynamic service-oriented architectures. It can literally monitor everything. It integrates with Grafana very smoothly as Grafana also offers Prometheus as one of its data sources.
Prometheus Architecture: At its core, Prometheus has a main component called Prometheus Server, responsible for the actual monitoring work.
The Prometheus server consists of:
- Time Series Database that stores all the metric data like current CPU usage, memory usage etc.
- Data Retrieval Worker is responsible for all the data pulling activities from applications, services, servers etc. and pushing them into the database.
- HTTP Server API meant to accept queries for the stored data. The Server API is used to display the data in a dashboard or a Web UI.
II-Install Portainer with Docker Swarm on Linux
1. Introduction
Portainer can be deployed on top of any K8s, Docker or Swarm environment. It works seamlessly in the cloud, on prem and at the edge to give you a consolidated view of all your containers.
Portainer consists of two elements, the Portainer Server and the Portainer Agent. Both elements run as lightweight Docker containers on a Docker engine. This document will help you deploy the Portainer Server and Agent containers on your Linux environment. To add a new Linux Swarm environment to an existing Portainer Server installation, please refer to the Portainer Agent installation instructions.
To get started, you will need:
- The latest version of Docker installed and working
- Swarm mode enabled and working, including the overlay network for the swarm service communication
- sudo access on the manager node of your swarm cluster
- By default, Portainer will expose the UI over port 9443 and expose a TCP tunnel server over port 8000.
- The manager and worker nodes must be able to communicate with each other over port 9001.
2. Deployment
Portainer can be directly deployed as a service in your Docker cluster. Note that this method will automatically deploy a single instance of the Portainer Server, and deploy the Portainer Agent as a global service on every node in your cluster.
First, retrieve the stack YML manifest:
curl -L https://downloads.portainer.io/portainer-agent-stack.yml \
-o portainer-agent-stack.yml
Then use the downloaded YML manifest to deploy your stack:
docker stack deploy -c portainer-agent-stack.yml portainer
Portainer Server and the Agents have now been installed. You can check to see whether the Portainer Server and Agent containers have started by running docker ps
:
root@manager01:~# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
59ee466f6b15 portainer/agent:2.11.1 "./agent" About a minute ago Up About a minute portainer_agent.xbb8k6r7j1tk9gozjku7e43wr.5sa6b3e8cl6hyu0snlt387sgv
2db7dd4bfba0 portainer/portainer-ce:2.11.1 "/portainer -H tcp:/…" About a minute ago Up About a minute 8000/tcp, 9443/tcp portainer_portainer.1.gpuvu3pqmt1m19zxfo44v7izx
3. Logging In
Now that the installation is complete, you can log into your Portainer Server instance by opening a web browser and going to:
https://localhost:9443
Accessing the Portainer dashboard page, at the first interface after successful setup, the user will see the information of the successfully connected endpoint: the number of stacks, the number of containers, the number of volumes, the number of images and one Docker and host information.
...readmore...
III-Server Monitoring with Prometheus and Grafana setup in Docker and Portainer
Ref
1. Deploy Prometheus and Grafana
Create a new stack and define or paste the content of your docker-compose file in a box web editor.
image1
We need to deploy Prometheus and Grafana so the full content of docker-compose will look like this:
version: '3'
volumes:
prometheus-data:
driver: local
grafana-data:
driver: local
services:
prometheus:
image: prom/prometheus:latest
container_name: prometheus
ports:
- "9090:9090"
volumes:
- /etc/prometheus:/etc/prometheus
- prometheus-data:/prometheus
restart: unless-stopped
command:
- "--config.file=/etc/prometheus/prometheus.yml"
grafana:
image: grafana/grafana:latest
container_name: grafana
ports:
- "3000:3000"
volumes:
- grafana-data:/var/lib/grafana
restart: unless-stopped
2. Configure Prometheus
Configuring Prometheus to monitor itself
$sudo mkdir /etc/prometheus
sudo vim /etc/prometheus/prometheus.yml
global:
scrape_interval: 15s # By default, scrape targets every 15 seconds.
# Attach these labels to any time series or alerts when communicating with
# external systems (federation, remote storage, Alertmanager).
# external_labels:
# monitor: 'codelab-monitor'
# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
# The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
- job_name: 'prometheus'
# Override the global default and scrape targets from this job every 5 seconds.
scrape_interval: 5s
static_configs:
- targets: ['localhost:9090']
3. Third-Party Exporters
Node_exporter is one of the exports that help prometheus collect metrics of the machine to be monitored and installed on the target machine (the machine being monitored).
cAdvisor is to analyze usage, performance, and many other metrics from Container applications, providing users with an overview of all running containers.
At stack monitoring we need to add the content config of cadvisor and node_exporter to the docker-compose file then update the stack.
node_exporter:
image: quay.io/prometheus/node-exporter:latest
container_name: node_exporter
command:
- '--path.rootfs=/host'
pid: host
restart: unless-stopped
volumes:
- '/:/host:ro,rslave'
cadvisor:
image: google/cadvisor:latest
container_name: cadvisor
# ports:
# - "8080:8080"
volumes:
- /:/rootfs:ro
- /var/run:/var/run:ro
- /sys:/sys:ro
- /var/lib/docker/:/var/lib/docker:ro
- /dev/disk/:/dev/disk:ro
devices:
- /dev/kmsg
restart: unless-stopped
In the Prometheus config file we need to add node-exporter and cadvisor to the scrape configuration as follows:
# Example job for node_exporter
- job_name: 'node_exporter'
static_configs:
- targets: ['node_exporter:9100']
# Example job for cadvisor
- job_name: 'cadvisor'
static_configs:
- targets: ['cadvisor:8080']
4. Visualize data with Grafana
In the Grafana page we built above at localhost:3000
Here we log in with the user and password as admin.
At the Home homepage, we need to add the data source, here we will add the data source from Prometheus because what we need to do is visualize on the Grafana dashboard to receive the data poured in from Prometheus.
At the setting we need to reconfigure the HTTP tab at the URL is http://prometheus:9090
After connecting your data source, at Home we create a new dashboard => Add an empty panel.
The Metrics browser helps us query the data to visualize.
5. Import Grafana Dashboards
In the Grafana homepage https://grafana.com
at the Dashboards tab we will see there are many dashboards. In this article we will learn about node exporters and cadvisor.
Get node exporter full by copying the ID and pasting it into the dashboards server we built.
Get a Cadvisor exporter by copying the ID and pasting it into the dashboards server we built.
IV-Alertmanager with Slack
1.Introduction
The Alertmanager handles alerts sent by client applications such as the Prometheus server. It takes care of deduplicating, grouping, and routing them to the correct receiver integration such as email, PagerDuty, or OpsGenie. It also takes care of silencing and inhibition of alerts.
Readmore...
Conclusion
Thank you very much for taking time to read this. I would really appreciate any comment in the comment section.
Enjoy🎉
Posted on February 25, 2022
Join Our Newsletter. No Spam, Only the good stuff.
Sign up to receive the latest update from our blog.
Related
February 25, 2022