TheLazy.Dev
Portfolio GitHub

Blog

vault |security |ubuntu |aws

Production-Ready Vault Deployment on EC2: A Detailed Guide

ifedayo

Ifedayo Adesiyan

Posted on July 31, 2024

Production-Ready Vault Deployment on EC2: A Detailed Guide

HashiCorp Vault is a powerful tool designed for securely managing secrets and protecting sensitive data. It provides a unified interface to access secrets and protect sensitive data using a variety of methods such as encryption, access control, and dynamic secrets generation. Vault is widely used in modern infrastructure to ensure that sensitive information, like API keys, passwords, and certificates, is handled securely and accessed only by authorized entities.
This guide aims to walk you through the process of deploying a production-ready HashiCorp Vault server on an AWS EC2 instance. The goal is to help you set up a secure, highly available, and scalable Vault environment that meets the needs of a production system.

Prerequisites

Before deploying HashiCorp Vault on an AWS EC2 instance, several prerequisites need to be addressed:
AWS Account Setup: Ensure you have an active AWS account. This is necessary for creating and managing EC2 instances and other AWS resources.
SSH Key Pair Creation: Create an SSH key pair to securely access your EC2 instances. This key pair will be used to establish an SSH connection to your instance.
Security Group Configuration: Configure a security group for your EC2 instance to allow inbound traffic on necessary ports, such as port 8200 for Vault and port 22 for SSH access.

Installing Vault

ec2 dashboard
To manage your EC2 instance, you'll need to connect to it via SSH. Open your terminal and use the following command, replacing /path/to/your-key.pem with the path to your SSH key file and your-ec2-public-dns with the public DNS address of your EC2 instance:

# ssh -i /path/to/your-key.pem ubuntu@your-ec2-public-dns
ssh -i tutorial-key.pem ubuntu@ec2-34-254-186-220.eu-west-1.compute.amazonaws.com
Enter fullscreen mode Exit fullscreen mode

This command establishes a secure connection to your EC2 instance using the provided key for authentication.

ec2 ssh connection on terminal

Now you have gained entry into your ec2 instance.
First, ensure your package index is up-to-date to avoid installation issues using:

sudo apt-get update
Enter fullscreen mode Exit fullscreen mode

This refreshes the list of available packages and their versions.

Installing Necessary Packages

Vault installation requires some basic tools like unzip. Install them using:

sudo apt-get install -y unzip
Enter fullscreen mode Exit fullscreen mode

This command installs the unzip utility, which is needed to extract the Vault binary.
Next, download the latest version of Vault from the official HashiCorp release page. As of July 31, 2024, the latest version is 1.17.2. Replace the version number with the latest release if necessary:

wget https://releases.hashicorp.com/vault/1.17.2/vault_1.17.2_linux_amd64.zip
Enter fullscreen mode Exit fullscreen mode

This command fetches the Vault binary in a zip archive.
Extract the downloaded zip file and move the Vault binary to a directory included in your system's PATH:

unzip vault_1.17.2_linux_amd64.zip
Enter fullscreen mode Exit fullscreen mode 
sudo mv vault /usr/local/bin/
Enter fullscreen mode Exit fullscreen mode

This extracts the vault binary and moves it to /usr/local/bin, making it accessible from anywhere in your terminal.
To ensure Vault is installed correctly, check its version:

vault --version
Enter fullscreen mode Exit fullscreen mode

vault --version output

Configuring Vault

To start configuring Vault, first create a directory to store the configuration file:

sudo mkdir /etc/vault
Enter fullscreen mode Exit fullscreen mode

Next, create and edit the Vault configuration file:

sudo nano /etc/vault/vault.hcl
Enter fullscreen mode Exit fullscreen mode

This opens a new file in the Nano text editor where you can define your Vault configuration.
In the vault.hcl file, specify the storage backend. For a simple setup, use the file storage backend, which stores data locally:

storage "file" {
  path = "/mnt/vault/data"
}
Enter fullscreen mode Exit fullscreen mode

This configuration tells Vault to store its data in the /mnt/vault/data directory. Make sure this directory exists and is writable by the Vault process.
We will verify the directory /mnt/vault exists and that the Vault process has the correct permissions to write to it using the following commands:

sudo mkdir -p /mnt/vault
sudo chown vault:vault /mnt/vault
sudo chmod 700 /mnt/vault
Enter fullscreen mode Exit fullscreen mode

Next, set up a listener to handle incoming requests. 
For an EC2 instance, port 8200 is not open by default, you'll need to check and modify the security group settings to ensure that port 8200 is open.
Find your instance and select the security group; click on the Inbound rules tab, and then click the Edit inbound rules button.

Inbound rules tab

To add a rule for port 8200, click Add rule, set Type to Custom TCP Rule, set Port range to 8200, set Source to 0.0.0.0/0 (for open access) or specify a custom IP range for more restricted access, and click Save rules

Edit Inbound rules

For production environments, it's crucial to use TLS to secure communications. To set up TLS certificates for Vault, you can either generate self-signed certificates or obtain certificates from a Certificate Authority (CA). Here's an example configuration for a TLS listener:

listener "tcp" {
  address     = "0.0.0.0:8200"
  tls_cert_file = "/etc/vault/vault-cert.pem"
  tls_key_file  = "/etc/vault/vault-key.pem"
}
Enter fullscreen mode Exit fullscreen mode

Replace /etc/vault/vault-cert.pem and /etc/vault/vault-key.pem with the paths to your TLS certificate and key files. This setup ensures that Vault communicates securely over HTTPS.

To enable the Vault web UI, add the following line to the configuration file:
ui = true
This setting allows access to Vault's user interface via a web browser, which can be accessed at https://:8200.
The disable_mlock = true setting in Vault's configuration disables the use of memory locking, which normally prevents Vault's sensitive data from being swapped to disk, potentially reducing security but improving compatibility in environments where memory locking is not available
Here is a minimal example of a vault.hcl configuration file:

# vault.hcl
storage "file" {
  path = "/mnt/vault/data"
}

listener "tcp" {
  address     = "0.0.0.0:8200"
  # tls_cert_file = "/etc/vault/vault-cert.pem"
  # tls_key_file  = "/etc/vault/vault-key.pem"
  tls_disable = 1

ui = true
disable_mlock = true
Enter fullscreen mode Exit fullscreen mode

This configuration includes the basic setup for storage, disabled TLS connection, enabling the UI, and disabling memory locking. Ensure that all file paths and other settings match your actual environment.
With these configurations in place, Vault will be set up to run securely and provide its secret management capabilities through a web interface.

Starting Vault as a Service

To manage Vault as a systemd service, create a service file for Vault:

sudo nano /etc/systemd/system/vault.service
Enter fullscreen mode Exit fullscreen mode

This will open a new file in the Nano text editor where you'll define how Vault should be managed.
Add the following content to the vault.service file:

[Unit]
Description=HashiCorp Vault Service- A tool for managing secrets
Documentation=https://www.vaultproject.io/
Requires=network-online.target
After=network-online.target

[Service]
User=vault
Group=vault
ExecStart=/usr/local/bin/vault server -config=/etc/vault/vault.hcl
ExecReload=/bin/kill -HUP $MAINPID
KillMode=process
Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
Enter fullscreen mode Exit fullscreen mode

Here's what each section does:
[Unit]: Defines the service description and dependencies.
[Service]: Configures the service to run as the vault user, start Vault with the specified configuration file, and handle restarts and reloads.
[Install]: Configures the service to start at boot.

Ensure that the vault user and group exist or adjust the User and Group fields accordingly. You might need to create these if they don't exist:

sudo useradd --system --home /etc/vault vault
Enter fullscreen mode Exit fullscreen mode

After creating the service file, reload the systemd configuration to recognize the new service:

sudo systemctl daemon-reload
Enter fullscreen mode Exit fullscreen mode

Enable the Vault service to start automatically at boot and then start it:

sudo systemctl enable vault
sudo systemctl start vault
Enter fullscreen mode Exit fullscreen mode

Verify that the Vault service is running correctly:

sudo systemctl status vault
Enter fullscreen mode Exit fullscreen mode

This displays the current status of the Vault service, including whether it is active, and provides logs for troubleshooting if necessary.
By following these steps, you will have Vault running as a managed systemd service, making it easier to handle restarts, automatic starts, and other service management tasks.

Initializing and Unsealing Vault

Initialization is the first step in setting up Vault. It generates the encryption keys and the initial root token. 
Before initializing, we need to run:

export VAULT_ADDR='http://<your private IPv4 address>:8200'
export DBUS_SESSION_BUS_ADDRESS=$XDG_RUNTIME_DIR/bus
Enter fullscreen mode Exit fullscreen mode

export VAULT_ADDR='http://172.31.4.123:8200' sets the environment variable VAULT_ADDR to the URL of the Vault server, specifying where Vault commands should be directed.
export DBUS_SESSION_BUS_ADDRESS=$XDG_RUNTIME_DIR/bus sets the DBUS_SESSION_BUS_ADDRESS environment variable to the path of the DBus session bus address, helping avoid issues with runaway DBus processes.
To initialize Vault, run:

vault operator init
Enter fullscreen mode Exit fullscreen mode

This will output several pieces of important information:

vault operator init output

Unseal Keys: You will receive multiple unseal keys. These keys are used to unseal the Vault server after it has been started.
Root Token: The initial root token allows you to authenticate and configure Vault.
Important: Save these unseal keys and root token in a secure location. They are critical for accessing and managing your Vault instance.

Once Vault is initialized, it will be sealed. You need to unseal it using the unseal keys provided during initialization. Use the following command to unseal Vault:

vault operator unseal <Unseal Key 1>
vault operator unseal <Unseal Key 2>
vault operator unseal <Unseal Key 3>
Enter fullscreen mode Exit fullscreen mode

You need to enter a majority of the unseal keys to unseal Vault. The exact number depends on your configuration, but typically it's three out of five.
You can also do this via the Vault web UI. Open a web browser and navigate to:

http://<your-ec2-public-dns>:8200
Enter fullscreen mode Exit fullscreen mode

Replace <your-ec2-public-dns> with the public DNS name or IP address of your EC2 instance:

unseal page on vault web

Ensure that the unseal keys and root token are stored securely:
Unseal Keys: Store them in a safe place, such as a secure password manager or a hardware security module (HSM).
Root Token: This token provides full access to Vault, so it should be treated with the highest level of security. Store it in a secure, encrypted location and use it only as needed.

Once you put in three correct keys, you'll be redirected to sign in. Use the root token obtained during initialization to sign in:
Enter the root token in the login prompt.
Click "Sign in" to access the Vault dashboard.

vault sign in page

The Vault UI provides a user-friendly interface for managing secrets, policies, and configurations. From here, you can perform administrative tasks and configure Vault according to your needs.

vault dashboard

Securing Vault Deployment

Enabling TLS for secure communication

To ensure that communication with Vault is secure, you should enable TLS (Transport Layer Security). This encrypts the data transmitted between clients and the Vault server, protecting it from eavesdropping and tampering.

  1. Obtain TLS Certificates: Acquire or generate a TLS certificate and private key. You can use a certificate authority (CA) or a self-signed certificate for testing purposes.
  2. Configure TLS in Vault: Update the Vault configuration file (/etc/vault/vault.hcl) to enable TLS: 
listener "tcp" {
  address     = "0.0.0.0:8200"
  tls_cert_file = "/etc/vault/vault-cert.pem"
  tls_key_file  = "/etc/vault/vault-key.pem"
}
Enter fullscreen mode Exit fullscreen mode

Replace the file paths with the locations of your TLS certificate and key. This ensures that Vault uses HTTPS for secure communication.

  1. Restart Vault: Apply the new configuration by restarting the Vault service: 
sudo systemctl restart vault
Enter fullscreen mode Exit fullscreen mode

Using Consul or Another Backend for High Availability (HA)

For production environments, it's crucial to ensure high availability and fault tolerance. Consul is a popular choice for Vault's high-availability backend, but other options like etcd or a cloud-based database can also be used.

  1. Install Consul: Follow the Consul installation guide to set up Consul on your infrastructure.
  2. Configure Vault to Use Consul: Update the Vault configuration file to use Consul as the storage backend: 
storage "consul" {
  address = "127.0.0.1:8500"
  path    = "vault/"
}
Enter fullscreen mode Exit fullscreen mode

Replace 127.0.0.1:8500 with your Consul server address.

  1. Restart Vault: Restart Vault to apply the new backend configuration: 
sudo systemctl restart vault
Enter fullscreen mode Exit fullscreen mode

Configuring Auto-Unseal with AWS KMS

Auto-unseal is a feature that allows Vault to automatically unseal itself using a cloud provider's key management service (KMS). For AWS, this means using AWS KMS to manage the encryption keys.

  1. Create a KMS Key: In the AWS Management Console, create a new KMS key for Vault.
  2. Configure Vault for Auto-Unseal: Update the Vault configuration file to use AWS KMS: 
seal "awskms" {
  kms_key_id = "arn:aws:kms:region:account-id:key/key-id"
}
Enter fullscreen mode Exit fullscreen mode

Replace arn:aws:kms:region:account-id:key/key-id with the ARN of your KMS key.

  1. Restart Vault: Apply the configuration by restarting Vault: 
sudo systemctl restart vault
Enter fullscreen mode Exit fullscreen mode

Setting Up Regular Backups

Regular backups are essential for disaster recovery and data protection.

  1. Plan Backup Strategy: Decide on the frequency and method of backups (e.g., daily snapshots).
  2. Backup Storage: Store backups in a secure and durable location, such as Amazon S3 or another cloud storage service.
  3. Automate Backups: Use cron jobs or other scheduling tools to automate the backup process. For example, create a script to copy the Vault data directory to your backup location and schedule it with cron.
  4. Test Restores: Regularly test backup restores to ensure that backups are valid and that you can recover data as needed.

Implementing Monitoring and Alerting

Monitoring and alerting are crucial for maintaining the health and security of your Vault deployment.

  1. Enable Vault Metrics: Vault provides a built-in metrics endpoint that you can enable in the configuration file: 
telemetry {
  dogstatsd_addr = "127.0.0.1:8125"
  disable_hostname = true
}
Enter fullscreen mode Exit fullscreen mode

This configuration sends metrics to a monitoring service like Datadog.

  1. Set Up Monitoring: Use a monitoring tool such as Prometheus, Grafana, or Datadog to collect and visualize metrics from Vault.
  2. Configure Alerts: Set up alerts for critical metrics, such as high error rates or low disk space, to ensure timely responses to potential issues.
  3. Review Logs: Regularly review Vault logs for any unusual activity or errors. Configure log rotation and retention policies to manage log files efficiently. By implementing these security measures, you ensure that your Vault deployment is robust, highly available, and resilient against potential issues, providing reliable secret management for your organization.

Conclusion

In this guide, we've covered the essential steps for deploying a production-ready HashiCorp Vault server on AWS EC2, including initialization, unsealing, and configuring Vault for secure and high-availability operations. Emphasizing security and reliability is crucial for protecting sensitive data and maintaining system integrity. We encourage you to explore Vault's advanced features and best practices further to optimize and secure your secret management infrastructure effectively.

Additional Resources

How to Connect an EC2 Instance Using SSH
Vault CLI tutorial
Auto-Unseal with AWS KMS
A Guide on Configuring Backup for Amazon EC2 Instances using AWS Backup Service

If you encounter any issues following this guide or setting up Vault in production, feel free to contact me on LinkedIn

💖 💪 🙅 🚩
ifedayo
Ifedayo Adesiyan

Posted on July 31, 2024

Join Our Newsletter. No Spam, Only the good stuff.

Sign up to receive the latest update from our blog.

Related

Production-Ready Vault Deployment on EC2: A Detailed Guide
vault Production-Ready Vault Deployment on EC2: A Detailed Guide

July 31, 2024