TheLazy.Dev
Portfolio GitHub

Blog

Installing and Configuring Elasticsearch/Kibana 8.x with Security

balajivedagiri

balajivedagiri

Posted on May 10, 2023

Installing and Configuring Elasticsearch/Kibana 8.x with Security

We will be installing,configuring elasticsearch and kibana 8.4, but steps should be same for most versions.

Our cluster will have 3 master nodes, 3 hot data nodes, 3 warm data nodes and 1 machine learning node.

1) pre-requisites 

1a) create /var/lib/elasticsearch mount point on all the nodes.
1b) turn off swap on OS(to ensure JVM heap is not swapped out).
1c) since we are using packages to install elasticsearch, ulimits are enforced in systemd unit file /usr/lib/systemd/system/elasticsearch.service.
1d) settings like file descriptors, max processes, max virtual memory size , max file size, etc are controlled from the systemd unit file.
1e) change default value of TCP retransmission timeout value, update the net.ipv4.tcp_retries2 setting in /etc/sysctl.conf to 5, and sysctl -w net.ipv4.tcp_retries2=5.
Enter fullscreen mode Exit fullscreen mode

2) Installing elasticsearch

Our cluster will have 3 master nodes, 3 hot data nodes, 3 warm data nodes and 1 machine learning node.

2a) Import elasticsearch PGP key.

wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo gpg --dearmor -o /usr/share/keyrings/elasticsearch-keyring.gpg
Enter fullscreen mode Exit fullscreen mode

2b) Install apt-transport-https package

sudo apt install apt-transport-https
Enter fullscreen mode Exit fullscreen mode

2c) save the repo,

echo "deb [signed-by=/usr/share/keyrings/elasticsearch-keyring.gpg] https://artifacts.elastic.co/packages/8.x/apt stable main" | sudo tee /etc/apt/sources.list.d/elastic-8.x.list
Enter fullscreen mode Exit fullscreen mode

2d) update the repo and install the package,

apt update && apt install elasticsearch
Enter fullscreen mode Exit fullscreen mode 
apt-get install elasticsearch
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following NEW packages will be installed:
  elasticsearch
0 upgraded, 1 newly installed, 0 to remove and 119 not upgraded.
Need to get 0 B/566 MB of archives.
After this operation, 1,170 MB of additional disk space will be used.
Selecting previously unselected package elasticsearch.
(Reading database ... 111616 files and directories currently installed.)
Preparing to unpack .../elasticsearch_8.4.3_amd64.deb ...
Creating elasticsearch group... OK
Creating elasticsearch user... OK
Unpacking elasticsearch (8.4.3) ...
Setting up elasticsearch (8.4.3) ...
--------------------------- Security autoconfiguration information ------------------------------

Authentication and authorization are enabled.
TLS for the transport and HTTP layers is enabled and configured.

The generated password for the elastic built-in superuser is : B25meUI2L6WcfTWBNvNp

If this node should join an existing cluster, you can reconfigure this with
'/usr/share/elasticsearch/bin/elasticsearch-reconfigure-node --enrollment-token <token-here>'
after creating an enrollment token on your existing cluster.

You can complete the following actions at any time:

Reset the password of the elastic built-in superuser with
'/usr/share/elasticsearch/bin/elasticsearch-reset-password -u elastic'.

Generate an enrollment token for Kibana instances with
 '/usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s kibana'.

Generate an enrollment token for Elasticsearch nodes with
'/usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s node'.
Enter fullscreen mode Exit fullscreen mode

2e) Ansible playbook to install the package.

---
- hosts: elasticsearch
  become: true
  gather_facts: true
  tasks:
  - name: Import the Elasticsearch PGP key
    apt_key:
      url: https://artifacts.elastic.co/GPG-KEY-elasticsearch
      keyring: /usr/share/keyrings/elasticsearch-keyring.gpg
      state: present
  - name: Install apt-transport-https
    apt:
      name: apt-transport-https
      state: present
# Add elasticsearch repo into sources list file /etc/apt/sources.list.d/elastic-8.x.list, after adding it will also run apt update or apt-get update by default
  - apt_repository:
      repo: 'deb [signed-by=/usr/share/keyrings/elasticsearch-keyring.gpg] https://artifacts.elastic.co/packages/8.x/apt stable main'
      state: present
      filename: elastic-8.x.list
  - name: Install a specific version of elasticsearch
    apt:
      name: elasticsearch=8.4.3
      state: present
      update_cache: yes
Enter fullscreen mode Exit fullscreen mode

2f) enable the service to start automatically on boot

sudo systemctl daemon-reload
sudo systemctl enable elasticsearch
Enter fullscreen mode Exit fullscreen mode

3) Generating certificates to enable TLS for transport and http.

3a) Generate CA certificate.

Login to one of the node where you installed elasticsearch , and issue below command to generate CA certificate. For higher protection, ensure you are setting password the certificate when it prompts below at the end and ensure you save that password in a secure location to use it later.

/usr/share/elasticsearch/bin/elasticsearch-certutil ca --out /root/elasticsearch_certs/elasticsearch-test-ca.p12
Enter fullscreen mode Exit fullscreen mode 
/usr/share/elasticsearch/bin/elasticsearch-certutil ca --out /root/elasticsearch_certs/elasticsearch-test-ca.p12
This tool assists you in the generation of X.509 certificates and certificate
signing requests for use with SSL/TLS in the Elastic stack.

The 'ca' mode generates a new 'certificate authority'
This will create a new X.509 certificate and private key that can be used
to sign certificate when running in 'cert' mode.

Use the 'ca-dn' option if you wish to configure the 'distinguished name'
of the certificate authority

By default the 'ca' mode produces a single PKCS#12 output file which holds:
    * The CA certificate
    * The CA's private key

If you elect to generate PEM format certificates (the -pem option), then the output will
be a zip file containing individual files for the CA certificate and private key

Enter password for elasticsearch-test-ca.p12 :

root@jumperserver:~/elasticsearch_certs# ls
elasticsearch-test-ca.p12
Enter fullscreen mode Exit fullscreen mode

3b) Generate node certificates

We use node certificates to join nodes to cluster and for transport layer encrytion. add all of your node details with dns name and ip into an yaml file like below,

root@jumperserver:~# cat /root/elasticsearch_certs/instances.yaml
instances:
  - name: "test-elastic-master01"
    ip: "10.10.4.6"
    dns: "test-elastic-master01"
  - name: "test-elastic-master02"
    ip: "10.10.4.7"
    dns: "test-elastic-master02"
  - name: "test-elastic-master03"
    ip: "10.10.4.8"
    dns: "test-elastic-master03"
  - name: "test-elastic-hotdata01"
    ip: "10.10.4.2"
    dns: "test-elastic-hotdata01"
  - name: "test-elastic-hotdata02"
    ip: "10.10.4.3"
    dns: "test-elastic-hotdata02"
  - name: "test-elastic-hotdata03"
    ip: "10.10.4.4"
    dns: "test-elastic-hotdata03"
  - name: "test-elastic-warmdata01"
    ip: "10.10.4.11"
    dns: "test-elastic-warmdata01"
  - name: "test-elastic-warmdata02"
    ip: "10.10.4.12"
    dns: "test-elastic-warmdata02"
  - name: "test-elastic-warmdata03"
    ip: "10.10.4.13"
    dns: "test-elastic-warmdata03"
  - name: "test-elastic-ml01"
    ip: "10.10.4.10"
    dns: "test-elastic-ml01"
Enter fullscreen mode Exit fullscreen mode

below you need to enter CA certificate password that you entered in step 3a, and ensure you set password for each and every node certificate ( you can set same password for all the nodes or different password as per security compliance)

/usr/share/elasticsearch/bin/elasticsearch-certutil cert --in /root/elasticsearch_certs/instances.yaml --out /root/elasticsearch_certs/server-cert-bundle.zip --ca /root/elasticsearch_certs/elasticsearch-test-ca.p12
Enter fullscreen mode Exit fullscreen mode 
root@elasticsearch-jumperserver:~/elasticsearch_certs# /usr/share/elasticsearch/bin/elasticsearch-certutil cert --in /root/elasticsearch_certs/instances.yaml --out /root/elasticsearch_certs/server-cert-bundle.zip --ca /root/elasticsearch_certs/elasticsearch-test-ca.p12
This tool assists you in the generation of X.509 certificates and certificate
signing requests for use with SSL/TLS in the Elastic stack.

The 'cert' mode generates X.509 certificate and private keys.
    * By default, this generates a single certificate and key for use
       on a single instance.
    * The '-multiple' option will prompt you to enter details for multiple
       instances and will generate a certificate and key for each one
    * The '-in' option allows for the certificate generation to be automated by describing
       the details of each instance in a YAML file

    * An instance is any piece of the Elastic Stack that requires an SSL certificate.
      Depending on your configuration, Elasticsearch, Logstash, Kibana, and Beats
      may all require a certificate and private key.
    * The minimum required value for each instance is a name. This can simply be the
      hostname, which will be used as the Common Name of the certificate. A full
      distinguished name may also be used.
    * A filename value may be required for each instance. This is necessary when the
      name would result in an invalid file or directory name. The name provided here
      is used as the directory name (within the zip) and the prefix for the key and
      certificate files. The filename is required if you are prompted and the name
      is not displayed in the prompt.
    * IP addresses and DNS names are optional. Multiple values can be specified as a
      comma separated string. If no IP addresses or DNS names are provided, you may
      disable hostname verification in your SSL configuration.


    * All certificates generated by this tool will be signed by a certificate authority (CA)
      unless the --self-signed command line option is specified.
      The tool can automatically generate a new CA for you, or you can provide your own with
      the --ca or --ca-cert command line options.


By default the 'cert' mode produces a single PKCS#12 output file which holds:
    * The instance certificate
    * The private key for the instance certificate
    * The CA certificate

If you specify any of the following options:
    * -pem (PEM formatted output)
    * -multiple (generate multiple certificates)
    * -in (generate certificates from an input file)
then the output will be be a zip file containing individual certificate/key files

Enter password for CA (/root/elasticsearch_certs/elasticsearch-test-ca.p12) :
Enter password for test-elastic-master01/test-elastic-master01.p12 :
Enter password for test-elastic-master02/test-elastic-master02.p12 :
Enter password for test-elastic-master03/test-elastic-master03.p12 :
Enter password for test-elastic-hotdata01/test-elastic-hotdata01.p12 :
Enter password for test-elastic-hotdata02/test-elastic-hotdata02.p12 :
Enter password for test-elastic-hotdata03/test-elastic-hotdata03.p12 :
Enter password for test-elastic-warmdata01/test-elastic-warmdata01.p12 :
Enter password for test-elastic-warmdata02/test-elastic-warmdata02.p12 :
Enter password for test-elastic-warmdata03/test-elastic-warmdata03.p12 :
Enter password for test-elastic-ml01/test-elastic-ml01.p12 :

Certificates written to /root/elasticsearch_certs/server-cert-bundle.zip

This file should be properly secured as it contains the private keys for
all instances
After unzipping the file, there will be a directory for each instance.
Each instance has a single PKCS#12 (.p12) file containing the instance
certificate, instance private key and the CA certificate
For each Elastic product that you wish to configure, you should copy
this '.p12' file to the relevant configuration directory
and then follow the SSL configuration instructions in the product guide.

For client applications, you may only need to copy the CA certificate and
configure the client to trust this certificate.
root@elasticsearch-jumperserver:~/elasticsearch_certs#
Enter fullscreen mode Exit fullscreen mode

below we are checking the generated certificates,

root@elasticsearch-jumperserver:~/elasticsearch_certs# ls
elasticsearch-test-ca.p12  instances.yaml  server-cert-bundle.zip
root@elasticsearch-jumperserver:~/elasticsearch_certs#

root@elasticsearch-jumperserver:~/elasticsearch_certs# unzip server-cert-bundle.zip
Archive:  server-cert-bundle.zip
   creating: test-elastic-master01/
  inflating: test-elastic-master01/test-elastic-master01.p12
   creating: test-elastic-master02/
  inflating: test-elastic-master02/test-elastic-master02.p12
   creating: test-elastic-master03/
  inflating: test-elastic-master03/test-elastic-master03.p12
   creating: test-elastic-hotdata01/
  inflating: test-elastic-hotdata01/test-elastic-hotdata01.p12
   creating: test-elastic-hotdata02/
  inflating: test-elastic-hotdata02/test-elastic-hotdata02.p12
   creating: test-elastic-hotdata03/
  inflating: test-elastic-hotdata03/test-elastic-hotdata03.p12
   creating: test-elastic-warmdata01/
  inflating: test-elastic-warmdata01/test-elastic-warmdata01.p12
   creating: test-elastic-warmdata02/
  inflating: test-elastic-warmdata02/test-elastic-warmdata02.p12
   creating: test-elastic-warmdata03/
  inflating: test-elastic-warmdata03/test-elastic-warmdata03.p12
   creating: test-elastic-ml01/
  inflating: test-elastic-ml01/test-elastic-ml01.p12
root@elasticsearch-jumperserver:~/elasticsearch_certs#
root@elasticsearch-jumperserver:~/elasticsearch_certs# ls
elasticsearch-test-ca.p12  test-elastic-hotdata02  test-elastic-master01  test-elastic-master03  test-elastic-warmdata01  test-elastic-warmdata03  server-cert-bundle.zip
test-elastic-hotdata01      test-elastic-hotdata03  test-elastic-master02  test-elastic-ml01      test-elastic-warmdata02  instances.yaml
root@elasticsearch-jumperserver:~/elasticsearch_certs#
root@elasticsearch-jumperserver:~/elasticsearch_certs# ls -ltr *
-rw-r--r-- 1 root root   876 Oct 26 18:49 instances.yaml
-rw------- 1 root root  2672 Oct 26 18:55 elasticsearch-test-ca.p12
-rw------- 1 root root 39406 Oct 26 18:56 server-cert-bundle.zip

test-elastic-master01:
total 4
-rw-r--r-- 1 root root 3700 Oct 26 18:56 test-elastic-master01.p12

test-elastic-master02:
total 4
-rw-r--r-- 1 root root 3700 Oct 26 18:56 test-elastic-master02.p12

test-elastic-master03:
total 4
-rw-r--r-- 1 root root 3700 Oct 26 18:56 test-elastic-master03.p12

test-elastic-hotdata01:
total 4
-rw-r--r-- 1 root root 3702 Oct 26 18:56 test-elastic-hotdata01.p12

test-elastic-hotdata03:
total 4
-rw-r--r-- 1 root root 3702 Oct 26 18:56 test-elastic-hotdata03.p12

test-elastic-hotdata02:
total 4
-rw-r--r-- 1 root root 3702 Oct 26 18:56 test-elastic-hotdata02.p12

test-elastic-warmdata01:
total 4
-rw-r--r-- 1 root root 3704 Oct 26 18:56 test-elastic-warmdata01.p12

test-elastic-warmdata03:
total 4
-rw-r--r-- 1 root root 3704 Oct 26 18:56 test-elastic-warmdata03.p12

test-elastic-warmdata02:
total 4
-rw-r--r-- 1 root root 3704 Oct 26 18:56 test-elastic-warmdata02.p12

test-elastic-ml01:
total 4
-rw-r--r-- 1 root root 3676 Oct 26 18:56 test-elastic-ml01.p12
root@elasticsearch-jumperserver:~/elasticsearch_certs#
Enter fullscreen mode Exit fullscreen mode

3c) Generate http certificate.

generate http certificates for http encryption, ensure you enter hostnames and ip's of the machines from which you would like you to communicate with elaticsearch over http, e.g jumpservers, kibana, elasticsearch nodes , so on.

/usr/share/elasticsearch/bin/elasticsearch-certutil http
Enter fullscreen mode Exit fullscreen mode 
root@elasticsearch-jumperserver:~/elasticsearch_certs/http# /usr/share/elasticsearch/bin/elasticsearch-certutil http

## Elasticsearch HTTP Certificate Utility

The 'http' command guides you through the process of generating certificates
for use on the HTTP (Rest) interface for Elasticsearch.

This tool will ask you a number of questions in order to generate the right
set of files for your needs.

## Do you wish to generate a Certificate Signing Request (CSR)?

A CSR is used when you want your certificate to be created by an existing
Certificate Authority (CA) that you do not control (that is, you don't have
access to the keys for that CA).

If you are in a corporate environment with a central security team, then you
may have an existing Corporate CA that can generate your certificate for you.
Infrastructure within your organisation may already be configured to trust this
CA, so it may be easier for clients to connect to Elasticsearch if you use a
CSR and send that request to the team that controls your CA.

If you choose not to generate a CSR, this tool will generate a new certificate
for you. That certificate will be signed by a CA under your control. This is a
quick and easy way to secure your cluster with TLS, but you will need to
configure all your clients to trust that custom CA.

Generate a CSR? [y/N]n

## Do you have an existing Certificate Authority (CA) key-pair that you wish to use to sign your certificate?

If you have an existing CA certificate and key, then you can use that CA to
sign your new http certificate. This allows you to use the same CA across
multiple Elasticsearch clusters which can make it easier to configure clients,
and may be easier for you to manage.

If you do not have an existing CA, one will be generated for you.

Use an existing CA? [y/N]y

## What is the path to your CA?

Please enter the full pathname to the Certificate Authority that you wish to
use for signing your new http certificate. This can be in PKCS#12 (.p12), JKS
(.jks) or PEM (.crt, .key, .pem) format.
CA Path: /root/elasticsearch_certs/elasticsearch-test-ca.p12
Reading a PKCS12 keystore requires a password.
It is possible for the keystore's password to be blank,
in which case you can simply press <ENTER> at the prompt
Password for elasticsearch-test-ca.p12:

## How long should your certificates be valid?

Every certificate has an expiry date. When the expiry date is reached clients
will stop trusting your certificate and TLS connections will fail.

Best practice suggests that you should either:
(a) set this to a short duration (90 - 120 days) and have automatic processes
to generate a new certificate before the old one expires, or
(b) set it to a longer duration (3 - 5 years) and then perform a manual update
a few months before it expires.

You may enter the validity period in years (e.g. 3Y), months (e.g. 18M), or days (e.g. 90D)

For how long should your certificate be valid? [5y] 10y

## Do you wish to generate one certificate per node?

If you have multiple nodes in your cluster, then you may choose to generate a
separate certificate for each of these nodes. Each certificate will have its
own private key, and will be issued for a specific hostname or IP address.

Alternatively, you may wish to generate a single certificate that is valid
across all the hostnames or addresses in your cluster.

If all of your nodes will be accessed through a single domain
(e.g. node01.es.example.com, node02.es.example.com, etc) then you may find it
simpler to generate one certificate with a wildcard hostname (*.es.example.com)
and use that across all of your nodes.

However, if you do not have a common domain name, and you expect to add
additional nodes to your cluster in the future, then you should generate a
certificate per node so that you can more easily generate new certificates when
you provision new nodes.

Generate a certificate per node? [y/N]N

## Which hostnames will be used to connect to your nodes?

These hostnames will be added as "DNS" names in the "Subject Alternative Name"
(SAN) field in your certificate.

You should list every hostname and variant that people will use to connect to
your cluster over http.
Do not list IP addresses here, you will be asked to enter them later.

If you wish to use a wildcard certificate (for example *.es.example.com) you
can enter that here.

Enter all the hostnames that you need, one per line.
When you are done, press <ENTER> once more to move on to the next step.

test-elastic-master01
test-elastic-master02
test-elastic-master03
test-elastic-kibana01
test-elastic-clustmon01
elasticsearch-jumpserver

You entered the following hostnames.

 - test-elastic-master01
 - test-elastic-master02
 - test-elastic-master03
 - test-elastic-kibana01
 - test-elastic-clustmon01
 - elasticsearch-jumpserver

Is this correct [Y/n]Y

## Which IP addresses will be used to connect to your nodes?

If your clients will ever connect to your nodes by numeric IP address, then you
can list these as valid IP "Subject Alternative Name" (SAN) fields in your
certificate.

If you do not have fixed IP addresses, or not wish to support direct IP access
to your cluster then you can just press <ENTER> to skip this step.

Enter all the IP addresses that you need, one per line.
When you are done, press <ENTER> once more to move on to the next step.

10.10.4.6
10.10.4.7
10.10.4.8
10.10.4.5
10.10.4.16
10.10.4.17
10.10.4.18
10.10.4.1
10.10.4.31

You entered the following IP addresses.

 - 10.10.4.6
 - 10.10.4.7
 - 10.10.4.8
 - 10.10.4.5
 - 10.10.4.16
 - 10.10.4.17
 - 10.10.4.18
 - 10.10.4.1
 - 10.10.4.31

Is this correct [Y/n]Y

## Other certificate options

The generated certificate will have the following additional configuration
values. These values have been selected based on a combination of the
information you have provided above and secure defaults. You should not need to
change these values unless you have specific requirements.

Key Name: test-elastic-master01
Subject DN: CN=test-elastic-master01
Key Size: 2048

Do you wish to change any of these options? [y/N]N

## What password do you want for your private key(s)?

Your private key(s) will be stored in a PKCS#12 keystore file named "http.p12".
This type of keystore is always password protected, but it is possible to use a
blank password.

If you wish to use a blank password, simply press <enter> at the prompt below.
Provide a password for the "http.p12" file:  [<ENTER> for none]
Repeat password to confirm:

## Where should we save the generated files?

A number of files will be generated including your private key(s),
public certificate(s), and sample configuration options for Elastic Stack products.

These files will be included in a single zip archive.

What filename should be used for the output zip file? [/usr/share/elasticsearch/elasticsearch-ssl-http.zip]

Zip file written to /usr/share/elasticsearch/elasticsearch-ssl-http.zip
root@elasticsearch-jumperserver:~/elasticsearch_certs/http#
Enter fullscreen mode Exit fullscreen mode 
root@elasticsearch-jumperserver:~/elasticsearch_certs/http# ls
elasticsearch  elasticsearch-ssl-http.zip  kibana
root@elasticsearch-jumperserver:~/elasticsearch_certs/http# mv elasticsearch-ssl-http.zip elasticsearch-ssl-http.zip_old
root@elasticsearch-jumperserver:~/elasticsearch_certs/http# mv elasticsearch elasticsearch_old
root@elasticsearch-jumperserver:~/elasticsearch_certs/http# mv kibana kibana_old
root@elasticsearch-jumperserver:~/elasticsearch_certs/http# pwd
/root/elasticsearch_certs/http
root@elasticsearch-jumperserver:~/elasticsearch_certs/http# cp /usr/share/elasticsearch/elasticsearch-ssl-http.zip .
root@elasticsearch-jumperserver:~/elasticsearch_certs/http# unzip elasticsearch-ssl-http.zip
Archive:  elasticsearch-ssl-http.zip
   creating: elasticsearch/
  inflating: elasticsearch/README.txt
  inflating: elasticsearch/http.p12
  inflating: elasticsearch/sample-elasticsearch.yml
   creating: kibana/
  inflating: kibana/README.txt
  inflating: kibana/elasticsearch-ca.pem
  inflating: kibana/sample-kibana.yml
root@elasticsearch-jumperserver:~/elasticsearch_certs/http#
root@elasticsearch-jumperserver:~/elasticsearch_certs/http# ls
elasticsearch  elasticsearch_old  elasticsearch-ssl-http.zip  elasticsearch-ssl-http.zip_old  kibana  kibana_old
root@elasticsearch-jumperserver:~/elasticsearch_certs/http#
Enter fullscreen mode Exit fullscreen mode

4) Copy the generated certificates

Copy the node certificate and http certificate to respective nodes to the path /etc/elasticsearch/certs/

Note: Node certificate is different for each and every elasticsearch node, http certificate is common for all the nodes.

*5) Setting keystore and trustore for transport and http *

Transport Truststore password is the password of CA certificate.
Transport Keystore password is the password of node certificates.

Transport http password is the password of http certificate.

set transport truststore/keystore and http keystore with below commands on all the nodes, you need run below commands on each and every elasticsearch nodes,

/usr/share/elasticsearch/bin/elasticsearch-keystore add xpack.security.transport.ssl.keystore.secure_password
/usr/share/elasticsearch/bin/elasticsearch-keystore add xpack.security.transport.ssl.truststore.secure_password
/usr/share/elasticsearch/bin/elasticsearch-keystore add xpack.security.http.ssl.keystore.secure_password
Enter fullscreen mode Exit fullscreen mode

6) Configuring elasticsearch parameters

Setting the configuration in /etc/elasticsearch/elasticsearch.yml, comment all the existing lines and append below after changing ip and hostnames to your node ip's and hostnames,

6a) Master nodes

path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
xpack.security.enabled: true
xpack.security.enrollment.enabled: true
cluster.name: test-elasticsearch
node.name: test-elastic-master01
network.host: 10.10.4.6
discovery.seed_hosts: ["test-elastic-master01", "test-elastic-master02", "test-elastic-master03"]
cluster.initial_master_nodes: ["test-elastic-master01", "test-elastic-master02", "test-elastic-master03"]
node.roles: [ master ]
xpack.watcher.enabled: true


# transport SSL/TLS
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: full
xpack.security.transport.ssl.client_authentication: required
xpack.security.transport.ssl.keystore.path: certs/test-elastic-master01.p12
xpack.security.transport.ssl.truststore.path: certs/test-elastic-master01.p12

# http SSL/TLS
http.host: 0.0.0.0
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.keystore.path: certs/http.p12
Enter fullscreen mode Exit fullscreen mode

6b) Hot nodes

path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
xpack.security.enabled: true
xpack.security.enrollment.enabled: true
cluster.name: test-elasticsearch
node.name: test-elastic-hotdata01
network.host: 10.10.4.2
discovery.seed_hosts: ["test-elastic-master01", "test-elastic-master02", "test-elastic-master03"]
cluster.initial_master_nodes: ["test-elastic-master01", "test-elastic-master02", "test-elastic-master03"]
node.roles: [ data,ingest ]
node.attr.box_type: hot
xpack.watcher.enabled: true


# transport SSL/TLS
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: full
xpack.security.transport.ssl.client_authentication: required
xpack.security.transport.ssl.keystore.path: certs/test-elastic-hotdata01.p12
xpack.security.transport.ssl.truststore.path: certs/test-elastic-hotdata01.p12

# http SSL/TLS
http.host: 0.0.0.0
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.keystore.path: certs/http.p12
Enter fullscreen mode Exit fullscreen mode

6c) Warm nodes

path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
xpack.security.enabled: true
xpack.security.enrollment.enabled: true
cluster.name: test-elasticsearch
node.name: test-elastic-warmdata01
network.host: 10.10.4.11
discovery.seed_hosts: ["test-elastic-master01", "test-elastic-master02", "test-elastic-master03"]
cluster.initial_master_nodes: ["test-elastic-master01", "test-elastic-master02", "test-elastic-master03"]
node.roles: [ data,ingest ]
node.attr.box_type: warm
xpack.watcher.enabled: true


# transport SSL/TLS
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: full
xpack.security.transport.ssl.client_authentication: required
xpack.security.transport.ssl.keystore.path: certs/test-elastic-warmdata01.p12
xpack.security.transport.ssl.truststore.path: certs/test-elastic-warmdata01.p12
# http SSL/TLS
http.host: 0.0.0.0
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.keystore.path: certs/http.p12
Enter fullscreen mode Exit fullscreen mode

6d) ML nodes

path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
xpack.security.enabled: true
xpack.security.enrollment.enabled: true
cluster.name: test-elasticsearch
node.name: test-elastic-ml01
network.host: 10.10.4.10
discovery.seed_hosts: ["test-elastic-master01", "test-elastic-master02", "test-elastic-master03"]
cluster.initial_master_nodes: ["test-elastic-master01", "test-elastic-master02", "test-elastic-master03"]
node.roles: [ ml ]
xpack.watcher.enabled: true


# transport SSL/TLS
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: full
xpack.security.transport.ssl.client_authentication: required
xpack.security.transport.ssl.keystore.path: certs/test-elastic-ml01.p12
xpack.security.transport.ssl.truststore.path: certs/test-elastic-ml01.p12
# http SSL/TLS
http.host: 0.0.0.0
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.keystore.path: certs/http.p12
Enter fullscreen mode Exit fullscreen mode

7) Starting elasticsearch

Start the nodes one by one using systemctl start elasticsearch, you can monitor the logs in /var/log/elasticsearch/test-elasticsearch.log

We need to ensure we remove the paramter once cluster is formed in /etc/elasticsearch/elasticsearch.yml

cluster.initial_master_nodes: ["test-elastic-master01", "test-elastic-master02", "test-elastic-master03"]

8) resetting elastic user password

you can also do this once you start the first node,

root@test-elastic-master01:/var/log/elasticsearch# /usr/share/elasticsearch/bin/elasticsearch-reset-password -u elastic
This tool will reset the password of the [elastic] user to an autogenerated value.
The password will be printed in the console.
Please confirm that you would like to continue [y/N]y


Password for the [elastic] user successfully reset.
New value: xxxxxxxxxxxxxxxxxxxxxxxx
root@test-elastic-master01:/var/log/elasticsearch#
Enter fullscreen mode Exit fullscreen mode

9) Check the status of cluster and list nodes 

root@test-elastic-master01:/var/log/elasticsearch# curl -X GET "https://10.10.4.2:9200/_cluster/health?pretty"  -u elastic -k
Enter host password for user 'elastic':
{
  "cluster_name" : "test-elasticsearch",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 6,
  "number_of_data_nodes" : 3,
  "active_primary_shards" : 2,
  "active_shards" : 4,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}


root@test-elastic-master01:/var/log/elasticsearch# curl -X GET "https://10.10.4.2:9200/_cat/nodes?pretty"  -u elastic -k
Enter host password for user 'elastic':
10.10.4.2  2 63 0 0.04 0.05 0.02 di - test-elastic-hotdata01
10.10.4.3  2 63 0 0.00 0.06 0.06 di - test-elastic-hotdata02
10.10.4.8  7 97 1 0.00 0.10 0.09 m  - test-elastic-master03
10.10.4.7 11 96 2 0.00 0.03 0.01 m  * test-elastic-master02
10.10.4.6 10 97 2 0.00 0.04 0.02 m  - test-elastic-master01
10.10.4.4  2 62 0 0.00 0.06 0.05 di - test-elastic-hotdata03
root@test-elastic-master01:/var/log/elasticsearch#
Enter fullscreen mode Exit fullscreen mode

10) Install and Configure Kibana

10a) Installing kibana

root@test-elastic-kibana01:~# wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo gpg --dearmor -o /usr/share/keyrings/elasticsearch-keyring.gpg

root@test-elastic-kibana01:~# apt-get install apt-transport-https
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages will be upgraded:
  apt-transport-https
1 upgraded, 0 newly installed, 0 to remove and 118 not upgraded.
Need to get 1,704 B of archives.
After this operation, 0 B of additional disk space will be used.
Get:1 http://us.archive.ubuntu.com/ubuntu focal-updates/universe amd64 apt-transport-https all 2.0.9 [1,704 B]
Fetched 1,704 B in 1s (3,407 B/s)
(Reading database ... 111616 files and directories currently installed.)
Preparing to unpack .../apt-transport-https_2.0.9_all.deb ...
Unpacking apt-transport-https (2.0.9) over (2.0.8) ...
Setting up apt-transport-https (2.0.9) ...
root@test-elastic-kibana01:~#

root@test-elastic-kibana01:~# echo "deb [signed-by=/usr/share/keyrings/elasticsearch-keyring.gpg] https://artifacts.elastic.co/packages/8.x/apt stable main" | sudo tee /etc/apt/sources.list.d/elastic-8.x.list
deb [signed-by=/usr/share/keyrings/elasticsearch-keyring.gpg] https://artifacts.elastic.co/packages/8.x/apt stable main
root@test-elastic-kibana01:~#

root@test-elastic-kibana01:~# sudo apt-get update && sudo apt-get install kibana
0% [Working]
Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease
Get:2 https://artifacts.elastic.co/packages/8.x/apt stable InRelease [10.4 kB]
Hit:3 http://us.archive.ubuntu.com/ubuntu focal InRelease
Hit:4 http://us.archive.ubuntu.com/ubuntu focal-updates InRelease
Get:5 https://artifacts.elastic.co/packages/8.x/apt stable/main amd64 Packages [34.0 kB]
Hit:6 http://us.archive.ubuntu.com/ubuntu focal-backports InRelease
Get:7 https://artifacts.elastic.co/packages/8.x/apt stable/main i386 Packages [3,556 B]
Fetched 48.0 kB in 1s (33.1 kB/s)
Reading package lists... Done
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following NEW packages will be installed:
  kibana
0 upgraded, 1 newly installed, 0 to remove and 118 not upgraded.
Need to get 285 MB of archives.
After this operation, 680 MB of additional disk space will be used.
Get:1 https://artifacts.elastic.co/packages/8.x/apt stable/main amd64 kibana amd64 8.4.3 [285 MB]
Fetched 285 MB in 3s (83.2 MB/s)
Selecting previously unselected package kibana.
(Reading database ... 111616 files and directories currently installed.)
Preparing to unpack .../kibana_8.4.3_amd64.deb ...
Unpacking kibana (8.4.3) ...
Setting up kibana (8.4.3) ...
Creating kibana group... OK
Creating kibana user... OK
Created Kibana keystore in /etc/kibana/kibana.keystore
root@test-elastic-kibana01:~#
Enter fullscreen mode Exit fullscreen mode

10b) Copy the ca certificate to kibana server

Copy the ca certificate that was generated from the step 3c kibana/elasticsearch-ca.pem to /etc/kibana/elasticsearch-ca.pem

10c) Reset kibana_system password

To do below, login into one of the elasticsearch node which is added to http certificate.

root@test-elastic-master01:/var/log/elasticsearch# /usr/share/elasticsearch/bin/elasticsearch-reset-password -u kibana_system
This tool will reset the password of the [kibana_system] user to an autogenerated value.
The password will be printed in the console.
Please confirm that you would like to continue [y/N]y

Password for the [kibana_system] user successfully reset.
New value: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
root@test-elastic-master01:/var/log/elasticsearch#

10d) Configuring kibana

set below parameters in /etc/kibana/kibana.yml, we are pointing to hot data nodes below

elasticsearch.hosts: ["https://10.10.4.2:9200","https://10.10.4.3:9200","https://10.10.4.4:9200"]
elasticsearch.username: "kibana_system"
elasticsearch.password: "xxxxxxxxxxxxxxxxxxxxxxxx"
elasticsearch.ssl.certificateAuthorities: [ "/etc/kibana/elasticsearch-ca.pem" ]
Enter fullscreen mode Exit fullscreen mode

10f) Start kibana and enable the service

systemctl start kibana
systemctl enable kibana
Enter fullscreen mode Exit fullscreen mode

Access kibana using elastic user using url http://kibana-hostname:5601

Image description

💖 💪 🙅 🚩
balajivedagiri
balajivedagiri

Posted on May 10, 2023

Join Our Newsletter. No Spam, Only the good stuff.

Sign up to receive the latest update from our blog.

Related

I am purposefully super nice to AI because I imagine they are trained on data where nice people get better help. Doing this, though, has an existential uneasiness to it.
ai I am purposefully super nice to AI because I imagine they are trained on data where nice people get better help. Doing this, though, has an existential uneasiness to it.

November 28, 2024

If you're a beginner, definitely check this open source guide. I've explained almost everything you need to know.
undefined If you're a beginner, definitely check this open source guide. I've explained almost everything you need to know.

November 28, 2024

The best way to get better at writing code is...
development The best way to get better at writing code is...

November 28, 2024

AI Innovations at Microsoft Ignite 2024 What You Need to Know (Part 2)
githubcopilot AI Innovations at Microsoft Ignite 2024 What You Need to Know (Part 2)

November 29, 2024

GIT Interview Questions & Answers
git GIT Interview Questions & Answers

November 29, 2024