Introduction

In today’s digital landscape, the need to protect sensitive information has become paramount. Secrets management is the practice of securely storing, managing and distributing critical data such as passwords, API keys, database credentials, and encryption keys. By implementing robust practices and utilizing specialized tools, secrets management aims to safeguard these secrets from unauthorized access and misuse.

HashiCorp Vault

HashiCorp Vault is a popular open-source tool designed for secure secrets management and data protection. It offers a comprehensive solution for storing, accessing, and managing sensitive information, such as passwords, API keys, certificates, and encryption keys.

Key Features and Capabilities

• Centralized Secrets Management : Vault provides a central repository for securely storing secrets, eliminating the need to store sensitive information in configuration files or source code. This centralized approach improves security by reducing the risk of accidental exposure or unauthorized access
• Secure Storage: Vault encrypts secrets at rest using industry-standard encryption algorithms. It ensures that sensitive information remains encrypted when stored in backend storage systems, adding an extra layer of protection.
• Dynamic Secrets: Vault can generate dynamic secrets on-demand for various resources such as databases, cloud providers, and more. These dynamic secrets have short lifetimes and are automatically revoked after a certain period or when no longer needed. This approach minimizes the risk of credentials being compromised or misused.
• Auditing and Logging : Vault keeps detailed logs of all operations, including access attempts, secret retrieval, and modification. These audit logs enable organizations to track and monitor secret usage, aiding in compliance efforts and troubleshooting
• Integration and Extensibility : Vault integrates with various authentication systems, identity providers, and cloud platforms. It provides a flexible API and SDKs for developers to build custom integrations and automate secrets management processes.
• Encryption as a Service : Vault provides encryption as a service, allowing users to encrypt and decrypt data using various encryption algorithms. This feature ensures the confidentiality and integrity of secrets
• Dynamic Secrets Revocation : Vault supports automatic revocation of dynamic secrets after a predefined period or when no longer needed. This helps mitigate the risk of unauthorized access to secrets.
• Secrets Encryption and Decryption : Vault allows users to encrypt and decrypt secrets using encryption keys managed within the system. This feature ensures that secrets are protected and can be securely transmitted across different systems.

Common Use Cases For Vault

• Database Credential Management : Vault can securely store and manage credentials for databases, such as usernames, passwords, and connection strings. Applications can retrieve these credentials dynamically from Vault, reducing the risk of hardcoded credentials and improving security
• API Key Management: Vault can be used to store and distribute API keys securely. By centralizing API key management in Vault, organizations can enforce access control, monitor usage, and rotate keys when needed.
• Encryption Key Management : Vault can generate, store, and manage encryption keys used for data encryption. This ensures that encryption keys are securely stored and protected, reducing the risk of unauthorized access to sensitive data.
• Token Management: Vault can generate and manage tokens used for authentication and authorization. It provides fine-grained control over token creation, revocation, and expiration, allowing organizations to enforce security policies and manage access to resources.
• SSH Key Management : Vault can act as an SSH key manager, providing a secure repository for storing and distributing SSH keys. This centralizes the management of SSH keys, simplifying access control and auditing.

Vault Installation

Note : Please note that the installation instructions provided below are specific to Linux. If you are using a different operating system, we recommend referring to the official documentation for the appropriate installation steps

Using the Debian package repository, HashiCorp provides a variety of Official Release Channels. see more here

GPG is required for the package signing key

sudo apt update && sudo apt install gpg

Download the signing key to a new keyring

wget -O- https://apt.releases.hashicorp.com/gpg | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg

Verify the key’s fingerprint, add the HashiCorp repo, update and install the vault

gpg --no-default-keyring --keyring /usr/share/keyrings/hashicorp-archive-keyring.gpg --fingerprint

echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list

sudo apt update

sudo apt install vault

Verify if everything is all right (vault installed correctly)

vault

Usage: vault <command> [args]

Common commands:
    read Read data and retrieves secrets
    write Write data, configuration, and secrets
    delete Delete secrets and configuration
    list List data or secrets
    login Authenticate locally
    agent Start a Vault agent
    server Start a Vault server
    status Print seal and HA status
    unwrap Unwrap a wrapped secret

Other commands:
    audit Interact with audit devices
    auth Interact with auth methods
    debug Runs the debug command
    kv Interact with Vault's Key-Value storage
    lease Interact with leases
    monitor Stream log messages from a Vault server
    namespace Interact with namespaces
    operator Perform operator-specific tasks
    path-help Retrieve API help for paths
    plugin Interact with Vault plugins and catalog
    policy Interact with policies
    print Prints runtime configurations
    secrets Interact with secrets engines
    ssh Initiate an SSH session
    token Interact with tokens

With Vault installed, the next step is to start a Vault server.

Starting the Server

you can test working with Vault with the “dev” server, which automatically unseals Vault, sets up in-memory storage, etc. (it's not covered here ) ⇒ see doc

Deploy vault in prod env

The /var/lib/vault/data directory that the vault uses must exist. (you can choose other placements)

sudo mkdir -p /var/lib/vault/data

Vault is configured using HCL files

Create the Vault configuration in the file config.hcl

listener "tcp" {
  address = "127.0.0.1:8200"
  tls_disable = true
}

api_addr = "http://127.0.0.1:8200"

storage "file" {
  path = "/var/lib/vault/data"
}

Let’s break down the configuration file:

• The listener block defines the network interface and port where Vault will listen for API requests. In this example, it's set to 127.0.0.1:8200 , which means Vault will listen on the loopback address on port 8200.
• tls_disable is set to true to disable TLS (Transport Layer Security) encryption for simplicity. In a production environment, it's recommended to use TLS.
• The api_addr configuration specifies the base URL for accessing the Vault API. In this case, it's set to http://127.0.0.1:8200 to match the listener address.
• The storage block configures the storage backend for Vault. In this example, it's set to a file storage backend with the path /var/lib/vault/data. You can modify this path based on your needs or use a different storage backend like "consul" or "etcd" for production environments.

Set the -config flag to point to the proper path where you saved the configuration above.

vault server -config=/etc/vault/config.hcl

Initializing the Vault

Initialization is the process of configuring Vault.

This only happens once when the server is started against a new backend that has never been used with Vault before.

Launch a new terminal session, and set VAULT_ADDR environment variable.

export VAULT_ADDR='http://127.0.0.1:8200'

To initialize Vault use vault operator init. This is an unauthenticated request, but it only works on brand new Vaults without existing data:

vault operator init

output:

Unseal Key 1: LhMnWcFuP/alHmtIlWTf8cp7ONbz0WFLiy28FG1IcQBK
Unseal Key 2: VU2lbINIzkovgTmHihSiTgxXUGsHnrJwLb/Q2V9FJ1Vc
Unseal Key 3: vo2lwvK8AqUHq87ASUDYoIpjO7jJnI2QJrOrQnRO9FGJ
Unseal Key 4: XMHENMUX8RPoI8AF2dIPx7/pJS+EeTywmJSHF/kt970B
Unseal Key 5: lE8hetFDRFoQ61NKJ3nNLRH/QKhD+sno5zlnKU/+z5sv

Initial Root Token: hvs.57tWqWNKEVgbyF1zPcETh4Qt

Vault initialized with 5 key shares and a key threshold of 3. Please securely
distribute the key shares printed above. When the Vault is re-sealed,
restarted, or stopped, you must supply at least 3 of these keys to unseal it
before it can start servicing requests.

Vault does not store the generated root key. Without at least 3 keys to
reconstruct the root key, Vault will remain permanently sealed!

It is possible to generate new unseal keys, provided you have a quorum of
existing unseal keys shares. See "vault operator rekey" for more information.

Initialization outputs two incredibly important pieces of information: the unseal keys and the initial root token. This is the only time ever that all of this data is known by Vault, and also the only time that the unseal keys should ever be so close together.

For the purpose of this tutorial, save all of these keys somewhere, and continue. In a real deployment scenario, you would never save these keys together. Instead, you would likely use Vault’s PGP and Keybase.io support to encrypt each of these keys with the users’ PGP keys. This prevents one single person from having all the unseal keys. Please see the documentation here

Seal/Unseal

Every initialized Vault server starts in the sealed state. From the configuration, Vault can access the physical storage, but it can’t read any of it because it doesn’t know how to decrypt it. The process of teaching Vault how to decrypt the data is known as unsealing the Vault.

Unsealing has to happen every time Vault starts. It can be done via the API and via the command line. To unseal the Vault, you must have the threshold number of unseal keys. In the output above, notice that the “key threshold” is 3. This means that to unseal the Vault, you need 3 of the 5 keys that were generated.

vault operator unseal

To unseal the vault you must use three different unseal keys, the same key repeated will not work.

As you use keys, as long as they are correct, you should soon see output like this:

Key Value
--- -----
Seal Type shamir
Initialized true
Sealed false
Total Shares 5
Threshold 3
Version 1.7.0
Storage Type raft
Cluster Name vault-cluster-0ba62cae
Cluster ID 7d49e5fd-a1a4-c1d1-55e2-7962e43006a1
HA Enabled true
HA Cluster n/a
HA Mode standby
Active Node Address <none>
Raft Committed Index 24
Raft Applied Index 24

When the value for Sealed changes to false, the Vault is unsealed.

login

Now authenticate as the initial root token (it was included in the output with the unseal keys).

vault login

While token-based authentication is enabled by default, Vault also supports various other authentication methods, such as username/password, LDAP, OIDC, GitHub, etc. These methods provide additional flexibility for authenticating users and applications with Vault.

Using the HTTP APIs with Authentication

All of Vault’s capabilities are accessible via the HTTP API in addition to the CLI.

Start a new Vault instance using the newly created configuration. (stop the previous instance using ctrl+c )

vault server -config=/etc/vault/config.hcl

Launch a new terminal session, and use curl to initialize Vault with the API.

curl \
    --request POST \
    --data '{"secret_shares": 1, "secret_threshold": 1}' \
    http://127.0.0.1:8200/v1/sys/init | jq

output

{
  "keys": [
    "ff27b63de46b77faabba1f4fa6ef44c948e4d6f2ea21f960d6aab0eb0f4e1391"
  ],
  "keys_base64": [
    "/ye2PeRrd/qruh9Ppu9EyUjk1vLqIflg1qqw6w9OE5E="
  ],
  "root_token": "s.Ga5jyNq6kNfRMVQk2LY1j9iu"
}

This response contains your initial root token. It also includes the unseal key. You can use the unseal key to unseal the Vault and use the root token perform other requests in Vault that require authentication.

To make this tutorial easy to copy-and-paste, you will be using the environment variable $VAULT_TOKEN to store the root token.

export VAULT_TOKEN="s.Ga5jyNq6kNfRMVQk2LY1j9iu"

Using the unseal key (not the root token) from above, you can unseal the Vault via the HTTP API.

Example:

curl \
    --request POST \
    --data '{"key": "/ye2PeRrd/qruh9Ppu9EyUjk1vLqIflg1qqw6w9OE5E="}' \
    http://127.0.0.1:8200/v1/sys/unseal | jq

output

{
  "type": "shamir",
  "initialized": true,
  "sealed": false,
  "t": 1,
  "n": 1,
  "progress": 0,
  "nonce": "",
  "version": "1.7.0",
  "migration": false,
  "cluster_name": "vault-cluster-1b34e68e",
  "cluster_id": "2cccf342-091a-b060-900b-04c29bb71ed4",
  "recovery_seal": false,
  "storage_type": "file"
}

You can invoke the Vault API to validate the initialization status.

curl http://127.0.0.1:8200/v1/sys/init

output

{ "initialized": true }

Creating secrets

Vault’s secret engines are components that enable the secure storage, generation, and management of secrets within Vault. They provide a way to interact with and manage different types of sensitive data, such as passwords, API keys, certificates, and more. Each secret engine has its own purpose and functionality

some of the common secret engines in Vault:

• Key/Value (KV) Secret Engine
• Database Secret Engine
• AWS Secret Engine:
• PKI (Public Key Infrastructure) Secret Engine
• Transit Secret Engine

for the sake of simplicity, we will go with Key/Value (KV) Secret Engine :

• The KV secret engine is the most basic and versatile secret engine in Vault.
• It allows you to store key-value pairs of arbitrary secrets within Vault.
• It provides a simple CRUD (Create, Read, Update, Delete) API to interact with secrets.
• It is often used for storing application configuration, database credentials, or other general-purpose secrets.

You can check the list of enabled secret engines by running the following cURL command:

curl --header "X-Vault-Token: $VAULT_TOKEN" http://127.0.0.1:8200/v1/sys/mounts

To enable the KV secret engine and mount it at the appropriate path, you can use the following cURL command :

curl --header "X-Vault-Token: $VAULT_TOKEN" --request POST --data '{"type": "kv-v2"}' http://127.0.0.1:8200/v1/sys/mounts/secret

Create your first secret : create a simple secret with name mysecret and object {“key”:”value”}

curl --header "X-Vault-Token: $VAULT_TOKEN" --request POST --data '{"data": {"key": "value"}}' http://127.0.0.1:8200/v1/secret/data/mysecret

output

{"request_id":"ad28e5de-5eac-34bd-9252-fc9894da95cd","lease_id":"","renewable":false,"lease_duration":0,"data":{"created_time":"2023-06-23T00:27:21.7933063Z","custom_metadata":null,"deletion_time":"","destroyed":false,"version":1},"wrap_info":null,"warnings":null,"auth":null}

Retreive the created secret

curl --header "X-Vault-Token: $VAULT_TOKEN" http://127.0.0.1:8200/v1/secret/data/mysecret

output

{"request_id":"f3ec354c-cf71-b499-a8f4-90cc30d0eab6","lease_id":"","renewable":false,"lease_duration":0,"data":{"data":{"key":"value"},"metadata":{"created_time":"2023-06-23T00:27:21.7933063Z","custom_metadata":null,"deletion_time":"","destroyed":false,"version":1}},"wrap_info":null,"warnings":null,"auth":null}

Conclusion

In conclusion, this tutorial provided a straightforward and concise guide on how to initiate, initialize, and access the Harishcop vault using simple HTTP requests. By following these steps, you can easily gain access to the vault and leverage its benefits. We hope that you found this tutorial useful and that it assists you in your endeavors.