Environment variables play a crucial role in configuring and customizing software applications. They allow developers to store sensitive information securely and adjust the behavior of their code without modifying the underlying source files. In Python, retrieving environment variables is typically a straightforward process.

However, when dealing with remote environments or secrets management, the process can become more complex. This article presents a way to retrieve environment variables, with a particular focus on integrating with AWS Secrets Manager.

Content

• Code to be discussed
• The env function
• The get_env function
• The remote_environment_variable function
• Usage examples

Code to be discussed

The code provided down below consists of three main functions: env, get_env, and remote_environment_variable, read carefully each function's implementation, try to understand its purpose and its role in retrieving environment variables.

import json
import os
from base64 import b64decode

from botocore.exceptions import (ClientError, NoCredentialsError,
                                 ParamValidationError)

from configs import aws
from configs.logging import logger

"""
Retrieves the value of an env variable from a remote source or a .env file.

:type key: str
:type default_value: str
:return: str
"""


def env(key: str, default_value: str = ""):
    value = get_env(key, default_value)

    if value.lower() == "true":
        return True
    if value.lower() == "false":
        return False

    return value


"""
Retrieves env variable from .env and AWS Secrets Manager, giving precedence to
remote value. If both sources exist, the remote value is returned; otherwise,
.env value or default_value is used.

:type key: str
:type default_value: str
:return: str
"""


def get_env(key: str, default_value: str = "") -> str:
    local_value = str(os.getenv(key, default_value))
    remote_value = str(remote_environment_variable(key))
    return remote_value if remote_value else local_value


"""
Retrieves env variable from AWS Secrets Manager.

:type key: str
:return: str
"""


def remote_environment_variable(key: str) -> str:
    secret_id = os.getenv("AWS_SECRETS_MANAGER_SECRET_ID")

    if not secret_id:
        return ""

    try:
        response = aws.secrets_manager.get_secret_value(SecretId=secret_id)
    except (ClientError, NoCredentialsError, ParamValidationError) as error:
        if isinstance(error, (NoCredentialsError, ParamValidationError)):
            logger.debug("AWS Secrets Manager: %s", error)
        else:
            message = f"{error.response['Error']['Code']} to secret"
            logger.error(f"{message} {secret_id}: {error}")

        return ""

    if "SecretString" in response:
        secret_dictionary = json.loads(response["SecretString"])
    else:
        secret_dictionary = json.loads(b64decode(response["SecretBinary"]))

    return secret_dictionary.get(key)

Now, that the code has been read, let's explore the code in detail.

The env function

This function is designed to retrieve the value of an environment variable from a remote source or a .env file. It provides a practical interface for accessing environment variables, with additional handling for boolean values. By utilizing the get_env function internally, it ensures consistent retrieval of variables from the appropriate source.

def env(key: str, default_value: str = "") -> str:
    value = get_env(key, default_value)

    if value.lower() == "true":
        return True
    if value.lower() == "false":
        return False

    return value

The env function takes two parameters: key, which represents the name of the environment variable to retrieve, and an optional default_value that is used if the variable is not found.

Within the function, it calls the get_env function, passing the key and default_value parameters. The return value is assigned to the value variable.

To provide additional flexibility, the function checks if the retrieved value is either “true” or “false”. If the value is “true” (case-insensitive), it returns a boolean True. If the value is “false” (case-insensitive), it returns a boolean False. This allows for more practical handling of boolean environment variables.

If the value is not a boolean, the function returns the original value as a string.

Overall, the env function simplifies the retrieval of environment variables by utilizing the get_env function internally and providing a consistent and flexible interface for accessing and interpreting their values.

The get_env function

This function serves as a central piece in retrieving environment variables. It handles the logic to fetch variables from either a local .env file or AWS Secrets Manager. It gives precedence to remote values when available, and falls back to local values or default values if necessary.

def get_env(key: str, default_value: str = "") -> str:
    local_value = str(os.getenv(key, default_value))
    remote_value = str(remote_environment_variable(key))
    return remote_value if remote_value else local_value

The get_env function is responsible for retrieving the environment variable from either the local .env file or AWS Secrets Manager.

Within the function, it first attempts to retrieve the value from the local environment using os.getenv(key, default_value). The os.getenv() function retrieves the value of the environment variable with the provided key. The retrieved value, whether from the environment or the default value, is then converted to a string using the str() function and stored in the local_value variable.

Next, the function calls the remote_environment_variable(key) function, which retrieves the value from AWS Secrets Manager. The result is stored in the remote_value variable, also converted to a string representation.

Finally, the function returns the remote_value if it exists (i.e., is not an empty string) or falls back to the local_value if the remote value is not available. This ensures that the function gives precedence to the remote value when both sources exist.

In summary, the get_env function provides a unified approach to retrieve environment variables, combining values from a local .env file and AWS Secrets Manager. It prioritizes remote values when available, offering flexibility and consistency in handling environment variable retrieval.

The remote_environment_variable function

This function is responsible for retrieving environment variables from AWS Secrets Manager. It uses the AWS_SECRETS_MANAGER_SECRET_ID environment variable to identify the secret in AWS Secrets Manager and fetches the corresponding value.

def remote_environment_variable(key: str) -> str:
    secret_id = os.getenv("AWS_SECRETS_MANAGER_SECRET_ID")

    if not secret_id:
        return ""

    try:
        response = aws.secrets_manager.get_secret_value(SecretId=secret_id)
    except (ClientError, NoCredentialsError, ParamValidationError) as error:
        if isinstance(error, (NoCredentialsError, ParamValidationError)):
            logger.debug("AWS Secrets Manager: %s", error)
        else:
            message = f"{error.response['Error']['Code']} to secret"
            logger.error(f"{message} {secret_id}: {error}")

        return ""

    if "SecretString" in response:
        secret_dictionary = json.loads(response["SecretString"])
    else:
        secret_dictionary = json.loads(b64decode(response["SecretBinary"]))

    return secret_dictionary.get(key)

The remote_environment_variable function retrieves the value of an environment variable from AWS Secrets Manager. It takes one parameter, key, which represents the name of the environment variable to retrieve.

First, the function retrieves the value of the AWS_SECRETS_MANAGER_SECRET_ID environment variable using os.getenv(). This value represents the ID of the secret in AWS Secrets Manager that contains all the desired environment variables stored as a JSON.

If the secret_id is not found or empty, indicating that the required environment variable is not available, the function returns an empty string.

Next, the function attempts to fetch the secret value using the aws.secrets_manager.get_secret_value() method, passing the SecretId parameter with the retrieved secret_id. Any potential exceptions, such as ClientError, NoCredentialsError, or ParamValidationError, are caught and handled within the except block.

If the retrieval is successful, the function checks if the response contains a “SecretString” key. If present, it assumes the secret value is in string format and parses it using json.loads(), storing the resulting dictionary in the secret_dictionary variable. Otherwise, if the response contains a “SecretBinary” key, it assumes the secret value is in base64-encoded binary format and decodes it using b64decode() before parsing it as a dictionary.

Finally, the function attempts to retrieve the value associated with the provided key from the secret_dictionary using the get() method. If the key is found in the dictionary, the corresponding value is returned. If not found, None is returned.

In summary, the remote_environment_variable function provides a way to retrieve environment variables from AWS Secrets Manager by utilizing the AWS_SECRETS_MANAGER_SECRET_ID environment variable to identify the secret. It handles potential errors and exceptions during the retrieval process, ensuring robustness and reliable access to sensitive configuration values.

Usage examples

This function demonstrates the usage of the env function in different scenarios.

from example_module import env

def example_function():
    # Example 1
    database_url = env('DATABASE_URL')
    print(f"The database URL is: {database_url}")

    # Example 2
    api_key = env('API_KEY', default_value='DEFAULT_API_KEY')
    print(f"The API key is: {api_key}")

    # Example 3
    debug_mode = env('DEBUG_MODE')
    if debug_mode:
        print("Debug mode is enabled.")
    else:
        print("Debug mode is disabled.")

In the first example it's possible to see a basic environment variable retrieval, it retrieves the value of the 'DATABASE_URL' environment variable and prints it. In the second example, retrieves the value of the 'API_KEY' environment variable with a default value and prints it. The third and last example handles boolean environment variables, it retrieves the value of the 'DEBUG_MODE' environment variable as a boolean and prints the debug mode status accordingly.

Final thoughts

Using this approach combined with the appropriated AWS EC2 ↔ AWS IAM configuration, it’s possible to centralize all environment variables into AWS Secrets Manager, eliminating the need to deal with an .env file during the deployment.

Therefore, this solution presents two upsides to the application lifecycle; 1) simplification of the deployment due to the elimination of the .env file outside the local environment; 2) better security since you can establish IAM access polices in a way that just specific members of the team can add and modify the Secrets Manager entries.

I'm Jack Miras, a Brazilian engineer who works mostly with infrastructure and backend development, for more of my stuff click here.

Happy coding!