As a SysAdmin, DevOps, or cybersecurity analyst, the moment will inevitably come in your work when you will need to consult the logs to investigate an incident or a bug.

Imagine a scenario where one of your collaborators often allows himself to connect in ssh as 'root' on the servers, or a scenario in which your main database replication server is down, wouldn't it be interesting to have a place where you have all these information without having to log in on each server?

Log aggregation

In the previous scenarios, the best way to quickly have all these information is by log aggregation. So what is log aggregation?

Log aggregation is the process of collecting and standardizing log events from various sources across your IT infrastructure for faster log analysis. Log aggregation is one of the early stages in the log management process. With log aggregation, you have the assurance of having centralized the logs and of having a tool for analyzing the logs.
Using log aggregation gives you a lot of advantages. Indeed, you can:

• Perform real-time monitoring
• Troubleshoot production incidents
• Centralize logs in the same location
• Collaborate with others on log analysis
• Enhanced Security in your infrastructure

Elasticsearch-Logstash-Kibana

ELK is a log analysis suite composed of 3 open source tools, developed by the company Elastic: Elasticsearch, Logstash and Kibana.

ElasticSearch is a search and analysis engine that uses the JSON format. Its goal is to efficiently extract data from structured or unstructured data sources in real-time. Elasticsearch uses Lucene to provide the most powerful full-text search capabilities available in any open-source product.

Logstash is a tool for entering, processing, and outputting log data. Its function is to analyze, filter, and cut the logs to transform them into formatted documents for Elasticsearch.

Kibana is an interactive and configurable dashboard that allows you to visualize the data stored in ElasticSearch. Kibana provides insight into trends and patterns in all forms of diagrams and curves. This dashboard can be shared and combined with data visualizations for quick and smart communication.

For the collection of data, Elastic has planned the beats which are agents that you install on your machines to monitor.

With this stack, you, therefore, have the possibility of setting up a centralized system (SIEM) that offers total visibility on the activity of your infrastructure and which thus allows you to react to threats in real-time.

Image source

In the rest of this article, we will therefore make a demo where we will set up the suite for elk to collect logs from a machine and visualize the dashboards on kibana.

Demo

Disclaimer: In this article, we will not cover logstash. Logstash is not essential for what we want to do in this series.

To realize our work here, we will use a server to install elk. The information-gathering agents (Filebeat, metricbeat...) will be installed on a host.

The server used to install elk is a GCP server with the following specifications:

• 25GB of disk space
• 4GB of memory

Although these are fairly minimalist features, they should suffice for this item.

Prerequisites

Make sure you have java installed and that ports 9200(elasticsearch) and 5601(kibana) are open on your server.

Elasticsearch

We will start by importing the Elasticsearch public GPG key.

wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -

After that, you have to get OK

Next, we add the Elastic source list to the sources.list.d directory :

echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee –a /etc/apt/sources.list.d/elastic-7.x.list

Install and configure elasticsearch

After that, you have to update your system and install elasticsearch

sudo apt-get update

sudo apt-get install elasticsearch

To configure elasticsearch, we will modify the file /etc/elasticsearch/elasticsearch.yml

sudo nano /etc/elasticsearch/elasticsearch.yml

The most important things to change:

network.host
http.port
discovery.type

After that, we start the service:

sudo systemctl start elasticsearch

You also need to allow Elasticsearch to start every time the server starts.

sudo systemctl enable elasticsearch

After that, we check if everything works well with an HTTP request. In my case, since I used my IP address, I use it for the request:

curl -X GET "your_ip:9200"

If you used localhost, it comes out to :

curl -X GET "localhost:9200"

Kibana

Install and configure kibana

To install kibana :

sudo apt install kibana

sudo systemctl enable kibana

sudo systemctl start kibana

After that, we configure kibana by modifying the file /etc/kibana/kibana.yml. The most important elements that we modify here :

server.port: 5601

server.host: "0.0.0.0"

elasticsearch.hosts: ["http://localhost:9200"]

For elasticsearch.hosts replace localhost if you have configured elasticsearch on another IP address.

Let's restart the service

systemctl restart kibana

You should normally have access to the kibana dashboard with http://localhost:5601 ou http://your-ip:5601 :

As you may have noticed when logging into kibana, no authentication is required. While this doesn't particularly bother our work here, in a real environment, we can't let the dashboard be accessible so easily. So we will add authentication to our dashboard. To do this, elastic has prepared some tools that we can use.

First, we need to stop the two services

systemctl stop elasticsearch kibana

In the file /etc/elasticsearch/elasticsearch.yml, let's add :

xpack.security.enabled: true

To communicate with our cluster, we need to configure a username for the embedded users. For this we have a program elasticsearch in the folder /usr/share/elasticsearch/. So start by stopping the elasticsearch service and running the following command from the folder :

./bin/elasticsearch

Then in another terminal :

./bin/elasticsearch-setup-passwords auto

You must answer with "y" here for this step

After that, you will get the passwords for the different users as shown here :

Then we have to go back to the kibana configuration file to authenticate the user kibana_system. Otherwise, kibana and elasticsearch will not be able to communicate.

The following information must be added to the configuration by filling in the credentials obtained earlier. The user to indicate here is kibana_system

elasticsearch.username: "kibana_system"
elasticsearch.password: "your_password_for_kibana_system"

You have to restart elasticsearch and kibana and voila! 😎

As you can see kibana requires authentication. You have to connect with the credential of the user elastic

Beats

If you browse a bit on kibana you will notice that we don't have any dashboard currently.

This is quite normal since we have not created any dashboard and especially we have no data in elasticsearch for that.

To solve this problem, we will use metricbeat.

Metricbeat

Elastic define Metricbeat as a lightweight shipper that you can install on your servers to periodically collect metrics from the operating system and from services running on the server. Metricbeat takes the metrics and statistics that it collects and ships them to the output that you specify, such as Elasticsearch or Logstash.

The installation of Metricbeat is quite simple. You have to install it on the host you want to monitor and then configure it to send information to elasticsearch. We use version 7.17.5 of elasticsearch in this tutorial. So we have to install the same version of metricbeat.

So we start by downloading the deb package here

Then we proceed to the installation with :

sudo dpkg -i metricbeat-7.17.5-amd64.deb

After that, you have to configure the file /etc/metricbeat/metricbeat.yml

In the basic configuration that we exploit here, we change the following elements:

setup.kibana:

  # Kibana Host
  host: "your_ip:5601"


output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["localhost:9200"]
  #username: "elastic"
  #password: "changeme"

Take care to change the information according to your infrastructure. For the username and password, these are the credentials of the user elastic.

What is interesting with beats is that there are dashboards that are pre-loaded, which allows us to have a basic global view of the data transferred by the agent. You will be able to create your dashboards if you wish later on.

To initialize metricbeat and load the dashboards, use the following command:

sudo metricbeat setup -e

Normally this command will take some time to finalize the loading of the data and dashboards.

After that, you have to start the service.

sudo service metricbeat start

Then go to kibana to see the change

You can see that we now have a list of dashboards proposed by metricbeat. We are going to browse some dashboards concerning us to see the feedback. (It should be noted that some dashboards may be empty if your server to monitor does not have the metrics that concern this dashboard)

Let's take a look at [Metricbeat System] Host overview ECS dashboard

As the name of the dashboard indicates, we have an overview of the host. There is more information :

Let's take a look at another dashboard : [Metricbeat System] Containers overview ECS

As you can see, there is much less information than in the previous dashboard. This is due to the very few containers running on the host. The list of dashboards offered by metricbeat is quite large. You can also create your own dashboard.

Filebeat

The installation of filebeat is done following the same process as for metricbeat. Download the deb package here

You can also renew the basic configuration, taking care to change the modified information for metricbeat.

Here we can also indicate the folders where filebeat can fetch the logs.

- type: filestream
  # Unique ID among all inputs, an ID is required.
  id: my-filestream-id
  # Change to true to enable this input configuration.
  enabled: true
  paths:
    - /var/log/*.log
    #- c:\programdata\elasticsearch\logs\*

After that, we perform the setup and start the service.

sudo filebeat setup -e

sudo service filebeat start

Note that Metricbeat and Filebeat have modules that can be activated to get more information. Indeed if you have tools such as apache, Nginx, docker... you can activate the modules related to these tools to have data on the corresponding dashboard. You can consult the list of dashboards in the folder: /etc/filebeat/modules.d

To enable MySQL module for example :

sudo metricbeat modules enable mysql

sudo  metricbeat modules enable apache

You will need to re-setup and restart filebeat

We can see that you now have dashboards for filebeat.

Take the time to explore the dashboards. The basic dashboard that you will have with enough information will be that of Syslog.

You have to enable the service first with

sudo metricbeat modules enable system

Summary

In this tutorial we have covered the following points:

• Installation and configuration of elasticsearch
• Installation and configuration of kibana
• Activation of authentication on kibana
• Installation and configuration of metricbeat and filebeat
• Exploration of some dashboards

What next?

So in this first tutorial, we have covered the interesting points concerning the implementation of the ELK stack but there is still a lot to discover. In the next part of the series, we are going to focus on setting up security alerts with a very great tool (Elastalert). It will also be an opportunity to come back to the possibilities that ELK offers us.