TheLazy.Dev
Portfolio GitHub

Blog

prometheus |observability |cloudnative |opensource

Prometheus - Installation on Amazon EC2 (Ubuntu) - Part 3

iamunnip

Unni P

Posted on August 21, 2023

Prometheus - Installation on Amazon EC2 (Ubuntu) - Part 3

In my previous article, we looked at how we can quickly set up Prometheus and Node Exporter.

Prometheus - Quick Setup on Amazon EC2 (Ubuntu) - Part 2

In this article, we will look at how we can quickly setup Prometheus and Node Exporter on a Ubuntu instance on Amazon EC2

favicon iamunnip.hashnode.dev

But in that setup, Prometheus and Node Exporter processes are running in the foreground and it won't start the process after a reboot of the VM.

We are going to fix that issue in this article.

Prerequisites

  • Setup an EC2 instance of type t2.small

  • Ubuntu 22.04 LTS as AMI

  • 30 GB of hard disk space

  • Open ports 22 for SSH, 9090 for Prometheus and 9100 for Node Exporter

Installation

Prometheus

Login to your EC2 instance

$ ssh -i prometheus.pem ubuntu@3.80.133.119
Enter fullscreen mode Exit fullscreen mode 
$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 22.04.2 LTS
Release:        22.04
Codename:       jammy
Enter fullscreen mode Exit fullscreen mode

Create a new user for managing the Prometheus process

Upon executing the command it will create a user called prometheus without a home directory and shell as /bin/false

$ sudo useradd --shell /bin/false --no-create-home prometheus
Enter fullscreen mode Exit fullscreen mode

Create a new directory for storing our Prometheus configuration file and update the ownership of the directory

$ sudo mkdir -p /etc/prometheus

$ sudo chown prometheus:prometheus /etc/prometheus
Enter fullscreen mode Exit fullscreen mode

Create another directory for storing Prometheus time series data and update the ownership of the directory

$ sudo mkdir -p /var/lib/prometheus

$ sudo chown prometheus:prometheus /var/lib/prometheus
Enter fullscreen mode Exit fullscreen mode

Download and extract the latest Prometheus release from their GitHub page

$ wget https://github.com/prometheus/prometheus/releases/download/v2.46.0/prometheus-2.46.0.linux-amd64.tar.gz
Enter fullscreen mode Exit fullscreen mode 
$ tar -xzvf prometheus-2.46.0.linux-amd64.tar.gz
Enter fullscreen mode Exit fullscreen mode

Change the directory and install Prometheus and Promtool binaries

$ cd prometheus-2.46.0.linux-amd64

$ sudo install prometheus /usr/local/bin

$ sudo install promtool /usr/local/bin
Enter fullscreen mode Exit fullscreen mode

Change the permission of these binaries to prometheus user

$ sudo chown prometheus:prometheus /usr/local/bin/prometheus

$ sudo chown prometheus:prometheus /usr/local/bin/promtool
Enter fullscreen mode Exit fullscreen mode

Check the version of prometheus or promtool binaries

$ prometheus --version
prometheus, version 2.46.0 (branch: HEAD, revision: cbb69e51423565ec40f46e74f4ff2dbb3b7fb4f0)
  build user:       root@42454fc0f41e
  build date:       20230725-12:31:24
  go version:       go1.20.6
  platform:         linux/amd64
  tags:             netgo,builtinassets,stringlabels
Enter fullscreen mode Exit fullscreen mode 
$ promtool --version
promtool, version 2.46.0 (branch: HEAD, revision: cbb69e51423565ec40f46e74f4ff2dbb3b7fb4f0)
  build user:       root@42454fc0f41e
  build date:       20230725-12:31:24
  go version:       go1.20.6
  platform:         linux/amd64
  tags:             netgo,builtinassets,stringlabels
Enter fullscreen mode Exit fullscreen mode

Copy the consoles and console_libraries directory which is used for dashboarding and visualization

$ sudo cp -rv consoles /etc/prometheus

$ sudo cp -rv console_libraries /etc/prometheus
Enter fullscreen mode Exit fullscreen mode

Change the ownerships of the directories recursively to prometheus user

$ sudo chown -R prometheus:prometheus /etc/prometheus/consoles

$ sudo chown -R prometheus:prometheus /etc/prometheus/console_libraries
Enter fullscreen mode Exit fullscreen mode

Copy the configuration file to the above-mentioned directory and change its ownership to prometheus user

$ sudo cp prometheus.yml /etc/prometheus

$ sudo chown prometheus:prometheus /etc/prometheus/prometheus.yml
Enter fullscreen mode Exit fullscreen mode

Create a systemd unit file for the Prometheus service

$ sudo vi /etc/systemd/system/prometheus.service

[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target

[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus \
    --config.file /etc/prometheus/prometheus.yml \
    --storage.tsdb.path /var/lib/prometheus \
    --web.console.templates /etc/prometheus/consoles \
    --web.console.libraries /etc/prometheus/console_libraries

[Install]
WantedBy=multi-user.target
Enter fullscreen mode Exit fullscreen mode

Start the prometheus process and enable the process at boot

$ sudo systemctl daemon-reload

$ sudo systemctl start prometheus

$ sudo systemctl enable prometheus
Enter fullscreen mode Exit fullscreen mode

Check the prometheus service status

$ sudo systemctl is-active prometheus
active

$ sudo systemctl is-enabled prometheus
enabled
Enter fullscreen mode Exit fullscreen mode

Open http://3.80.133.119:9090 in the browser and you can see the Prometheus expression browser

prom-1

Node Exporter

Create a new user for managing the node exporter process

$ sudo useradd --shell /bin/false --no-create-home node_exporter
Enter fullscreen mode Exit fullscreen mode

Download and extract the latest Node Exporter release from their GitHub page

$ wget https://github.com/prometheus/node_exporter/releases/download/v1.6.1/node_exporter-1.6.1.linux-amd64.tar.gz

$ tar -xzvf node_exporter-1.6.1.linux-amd64.tar.gz
Enter fullscreen mode Exit fullscreen mode

Change the directory and install the node_exporter binary

$ cd node_exporter-1.6.1.linux-amd64

$ sudo install node_exporter /usr/local/bin
Enter fullscreen mode Exit fullscreen mode

Change the ownership of the binary to node_exporter user

$ sudo chown node_exporter:node_exporter /usr/local/bin/node_exporter
Enter fullscreen mode Exit fullscreen mode

Check the version of the node_exporter binary

$ node_exporter --version
node_exporter, version 1.6.1 (branch: HEAD, revision: 4a1b77600c1873a8233f3ffb55afcedbb63b8d84)
  build user:       root@586879db11e5
  build date:       20230717-12:10:52
  go version:       go1.20.6
  platform:         linux/amd64
  tags:             netgo osusergo static_build
Enter fullscreen mode Exit fullscreen mode

Create a systemd unit file for the Node Exporter service

$ sudo vi /etc/systemd/system/node_exporter.service

[Unit]
Description=Node Exporter
Wants=network-online.target
After=network-online.target

[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter

[Install]
WantedBy=multi-user.target
Enter fullscreen mode Exit fullscreen mode

Start the node_exporter process and enable the process at boot

$ sudo systemctl daemon-reload

$ sudo systemctl start node_exporter

$ sudo systemctl enable node_exporter
Enter fullscreen mode Exit fullscreen mode

Check the status of the node_exporter process

$ sudo systemctl is-active node_exporter
active

$ sudo systemctl is-enabled node_exporter
enabled
Enter fullscreen mode Exit fullscreen mode

Open http://3.80.133.119:9100/metrics in your browser to view all the metrics

node-1

Configuration

Prometheus

Now we have installed Prometheus and Node Exporter on the server and we need to modify the Prometheus configuration file to add new targets to scrape metrics

In the installation step, we already copied a default Prometheus configuration file to the /etc/prometheus location. You can edit this file or replace it with the below contents.

Here I have added a new job called "node_exporter" to scrape metrics from the same VM, instead of using localhost I have used the private IP of the instance

$ sudo vi /etc/prometheus/prometheus.yml

global:
  scrape_interval: 15s
  scrape_timeout: 10s

scrape_configs:
  - job_name: "prometheus"
    static_configs:
      - targets: ["localhost:9090"]
  - job_name: "node_exporter"
    static_configs:
      - targets: ["172.31.93.117:9100"]
Enter fullscreen mode Exit fullscreen mode

Validate the configuration file using the promtool

$ promtool check config /etc/prometheus/prometheus.yml

Checking /etc/prometheus/prometheus.yml
 SUCCESS: /etc/prometheus/prometheus.yml is valid prometheus config file syntax
Enter fullscreen mode Exit fullscreen mode

Restart the Prometheus service to effect new configuration changes

$ sudo systemctl restart prometheus

$ sudo systemctl status prometheus
Enter fullscreen mode Exit fullscreen mode

Open the expression browser and navigate to Status -> Targets to view the newly added "node_exporter" target

prom-2

We can view an example metric called "node_memory_Active_bytes" using the expression browser

prom-3

Select the Graph option to view it as a graph

prom-4

That's all for now

Reference

https://prometheus.io/docs/prometheus/latest/getting_started/

https://kodekloud.com/courses/prometheus-certified-associate-pca/

💖 💪 🙅 🚩
iamunnip
Unni P

Posted on August 21, 2023

Join Our Newsletter. No Spam, Only the good stuff.

Sign up to receive the latest update from our blog.

Related

Prometheus - HTTPS & Authentication - Part 4
prometheus Prometheus - HTTPS & Authentication - Part 4

August 22, 2023

Prometheus - Installation on Amazon EC2 (Ubuntu) - Part 3
prometheus Prometheus - Installation on Amazon EC2 (Ubuntu) - Part 3

August 21, 2023