AWS Security Hub detected several security risks related to ECS task definitions. Among them, we addressed a significant issue: "Database passwords were stored in plaintext in ECS task definition environment variables."

Issue Overview

Background

In our deployment workflow, we read the database password (DB_PASSWORD) from GitHub Secrets and passed it to the environment section of the ECS task definition using ecspresso. Initially, we assumed this approach was safe since the secret wasn't hardcoded into the source code.

However, AWS Security Hub flagged this configuration as a risk, highlighting that "plaintext secrets were included in the ECS task definition."

Example of the Vulnerable Configuration

GitHub Actions Workflow

The workflow set DB_PASSWORD as an environment variable and passed it to ecspresso during deployment:

- name: Setup environment
  run: |
    echo "DB_PASSWORD=${{ secrets.DB_PASSWORD }}" >> $GITHUB_ENV

- name: Deploy to Company ECS service
  run: ecspresso deploy --config ecspresso.yml
  env:
    ENV: ${{ env.DB_PASSWORD }}

ECS Task Definition

In the task definition template, the DB_PASSWORD was passed directly as an environment variable:

{
  "containerDefinitions": [
    {
      "environment": [
        {
          "name": "DB_PASSWORD",
          "value": "{{ must_env `DBPASSWORD` }}"
        }
      ]
    }
  ]
}

Why This Is a Problem

1. Plaintext Secrets in Task Definitions
   ECS task definitions are stored within AWS and are generally secure. However, if task definition details are exposed (e.g., during debugging or through misconfigured access controls), secrets can be leaked.

2. Violation of AWS Security Best Practices
   AWS recommends managing sensitive information using Secrets Manager or SSM Parameter Store, and referencing them dynamically rather than including plaintext values in task definitions.

Solution

To eliminate the risk of storing plaintext secrets in the ECS task definition, we transitioned to using AWS Secrets Manager to securely manage database passwords.

Secrets Manager Configuration

Step 1: Register Secrets in Secrets Manager

We created a secret in AWS Secrets Manager to store the database password.

• Secret Name: dev/app/mysql/db
• Key/Value Format:

  {
    "DB_PASSWORD": "secure-password"
  }

Registering via CLI

You can register the secret using the following AWS CLI command:

aws secretsmanager create-secret \
  --name dev/app/mysql/db \
  --secret-string '{"DB_PASSWORD":"secure-password"}'

Updating ECS Task Definition

Step 1: Use secrets Section to Reference Secrets Manager

We updated the ECS task definition to dynamically retrieve the password from Secrets Manager using the secrets section.

Updated Task Definition Example

{
  "containerDefinitions": [
    {
      "name": "app-container",
      "secrets": [
        {
          "name": "DB_PASSWORD",
          "valueFrom": "arn:aws:secretsmanager:<region>:<account-id>:secret:dev/app/mysql/db:DB_PASSWORD::"
        }
      ]
    }
  ]
}

If Using ecspresso

With ecspresso, the Secrets Manager ARN can be referenced dynamically using the secretsmanager_arn function, simplifying configuration.

ecspresso Task Definition Template

{
  "containerDefinitions": [
    {
      "name": "app-container",
      "secrets": [
        {
          "name": "DB_PASSWORD",
          "valueFrom": "{{ secretsmanager_arn `dev/app/mysql/db` }}:DB_PASSWORD::"
        }
      ]
    }
  ]
}

Using this approach, there's no need to manually write the ARN, improving maintainability.

Configuring the IAM Role

To retrieve secrets from Secrets Manager, the Task Execution Role specified in the ECS task definition must have the appropriate permissions.

Example IAM Policy

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "secretsmanager:GetSecretValue",
        "secretsmanager:DescribeSecret"
      ],
      "Resource": "arn:aws:secretsmanager:<region>:<account-id>:secret:dev/app/mysql/db"
    }
  ]
}

Task Execution Role vs. Task Role

• Task Execution Role:

  • This is required for ECS to interact with AWS resources like Secrets Manager or ECR.
  • Secrets Manager permissions must be granted here for this use case.

• Task Role:

  • Used by the application itself to access AWS resources like DynamoDB or S3.
  • It is unrelated to Secrets Manager access in this scenario.

Outcome

1. Resolved Security Hub Alert
   By eliminating plaintext secrets from the task definition, we addressed the risk flagged by AWS Security Hub.

2. Compliance with Best Practices
   The solution aligns with AWS's recommendation to manage sensitive information securely using Secrets Manager.

3. Improved Maintainability
   Password changes can now be handled directly in Secrets Manager without modifying the ECS task definition or redeploying the service.

Conclusion

Storing plaintext secrets in ECS task definitions is a security risk that should be avoided. AWS Secrets Manager provides a secure and efficient way to manage sensitive information, ensuring better security and maintainability.

If you're managing ECS tasks and haven't yet adopted Secrets Manager or SSM Parameter Store, now is a great time to enhance your security practices!