The Problem

Best practices dictate that you should not SSH into containers; instead, you should implement proper observability mechanisms for monitoring, debugging, and log analysis. So it was impossible with ECS, at least not until the 16th of March 2021. After that, it was a highly requested feature on the AWS Container roadmap. AWS eventually gave in because they understood that engineers must be able to debug and test quickly in the early stages of development. It can also be helpful to debug high severity issues in production.

The Solution

Before getting into your ECS container, you would have to open the SSH ports, get keys/passwords to the host instance, SSH into that instance, and then docker exec into the container. If you were running the serverless Fargate model, you couldn't even do this because you do not have access to the host machine, so you'd have to redeploy it to an EC2 instance.

Then ECS exec came to the rescue, and it allows you to docker exec right into the container you want without the overhead of worrying about security issues with distributing SSH keys or passwords

Implementation

I will use the Fargate container from this post and make modifications to specific parts of the CloudFormation template.

• IAM Policy for the Task Role
• KMS key to encrypt the ECS Exec data channel
• Enable Execute command
• Place cluster in public subnets The template already has the cluster in public subnets, so first, we'll add the IAM Policy. ``` yaml TaskRole: Type: AWS::IAM::Role Properties: RoleName: !Join ["-", [!Ref ServiceName, TaskRole]] AssumeRolePolicyDocument: Statement: - Effect: Allow Principal: Service: ecs-tasks.amazonaws.com Action: "sts:AssumeRole" Policies: - PolicyName: "systems-manager" PolicyDocument: Version: "2012-10-17" Statement: - Effect: "Allow" Action: - "ssmmessages:CreateControlChannel" - "ssmmessages:CreateDataChannel" - "ssmmessages:OpenControlChannel" - "ssmmessages:OpenDataChannel" Resource: "*" - Effect: "Allow" Action: - "kms:Decrypt" Resource: !Ref KMSKey

Then the KMS key:
```yaml


  ############################################################
  #              KMS KEY
  ############################################################
  KMSKey:
    Type: AWS::KMS::Key
    Properties:
      Description: Key needed to encrypt docker exec data channel
      KeyPolicy:
        Version: "2012-10-17"
        Statement:
          - Effect: "Allow"
            Principal:
              AWS: !Sub "arn:aws:iam::${AWS::AccountId}:root"
            Action:
              - "kms:*"
            Resource: "*"

Now we enable the exec command on the service, that can be done by adding EnableExecuteCommand: true

  ############################################################
  #              ECS SERVICE
  ############################################################
  Service:
    Type: AWS::ECS::Service
    # This dependency is needed so that the load balancer is setup correctly in time
    DependsOn:
      - ApplicationLoadBalancer
      - ALBHTTPListener
      - ListenerRule
      - TargetGroup
    Properties:
      ServiceName: !Ref ServiceName
      Cluster: !Ref Cluster
      TaskDefinition: !Ref TaskDefinition
      EnableExecuteCommand: true        
      DeploymentConfiguration:
        MinimumHealthyPercent: 100
        MaximumPercent: 200
      DesiredCount: !Ref MinContainers
      HealthCheckGracePeriodSeconds: 300
      LaunchType: FARGATE
      NetworkConfiguration:
        AwsvpcConfiguration:
          AssignPublicIp: ENABLED
          Subnets:
            - !Ref PublicSubnetOne
            - !Ref PublicSubnetTwo
          SecurityGroups:
            - !Ref ContainerSecurityGroup
      LoadBalancers:
        - ContainerName: !Ref ServiceName
          ContainerPort: !Ref ContainerPort
          TargetGroupArn: !Ref TargetGroup

You can deploy the template:

aws cloudformation deploy --template-file ecs.yml --stack-name ecs-infrastructure --capabilities CAPABILITY_NAMED_IAM --profile default

Then you can get the Task Id by running:

aws ecs list-tasks --cluster rest-api-cluster

The task id is the part highlighted in red below:

This is an alpine linux docker image, so instead of "/bin/bash", to docker exec into it, "/bin/sh" will have to be used instead.

aws ecs execute-command     --region us-east-1    --cluster rest-api-cluster     --task INSERT_YOUR_TASK_ID_HERE     --container rest-api     --command "/bin/sh"     --interactive

Sample output:

You can use this tool to troubleshoot issues.

Cleanup

Always remember to delete resources you do not need, run:

aws cloudformation delete-stack --stack-name ecs-infrastructure

You can check the status of the stack to confirm if it was deleted successfully:

aws cloudformation describe-stacks --stack-name ecs-infrastructure

Confirm the stack was deleted successfully; remember to clean up the ECR repository if you implemented that.