Applications running in EKS often require access to AWS resources, including S3 buckets, DynamoDB tables, Secrets Manager secrets, KMS keys, SQS queues, and other resources. As a security auditor, mapping EKS pods in a cluster to assigned IAM policy permissions can be challenging. In this post, we will review three different ways to audit EKS pod permissions. ## EKS Node IAM Role Permissions EKS avoids provisioning long-lived access keys to the cluster’s nodes by using an instance profile to pass IAM role credentials to the EC2 instance. Services running on the EKS node can request the role’s temporary access keys using the [EC2 Instance Metadata Service (IMDS)](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-metadata.html). By default, all pods running on the node can also communicate with the IMDS. This allows pods to request temporary access keys for the node's IAM role and access AWS resources. Consider the EKS node in the diagram below. The node’s *kubelet* makes AWS API calls for the Kubernetes control plane to function, which typically requires the *AmazonEKSWorkerNodePolicy*, *AmazonEKS_CNI_Policy*, *AmazonEC2ContainerRegistryReadOnly*, and *CloudWatchAgentServerPolicy* managed policies. Knowing the *web* pod needs access to an S3 bucket and a Secrets Manager secret, the appropriate permissions are attached to the node’s IAM Role as well. ![AWS EKS Node Image Permissions](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/e8t1xb293isqcyo3zhmg.png) The *web* pod's path to accessing the AWS APIs is as follows: 1. The development team deploys the *web* pod in the *dm* namespace on the EKS node. 2. The process running in the *web* container requests the IAM role's temporary credentials from the instance metadata service (*http://169.254.169.254/latest/meta-data/iam/security-credentials/dm-eks-node-role*). The response includes a temporary AWS Access Key Id, Secret Access Key, and Session Token for authenticating to AWS APIs using the EKS node's IAM Role. 3. The process running in the web container invokes the S3 and Secrets Manager API using the temporary credentials. So, what’s the problem? Mixing the *kubelet’s* control plane permissions with permissions for other workloads can create privilege escalation issues. To start, the EKS node has a number of AWS managed policies, including the *AmazonEKS_CNI_Policy* managed policy, which could allow the *web* pod to create, delete, and attach network interfaces from the nodes in the cluster. These permissions are unlikely to be required for the *web* pod to operate. Now, let's consider a scenario where a new pod, *evil*, is deployed into a different namespace called *sabre*. The next diagram shows the *evil* pod requesting temporary credentials from the node's instance metadata service. The temporary credentials will also include the web pod's permission to the S3 and Secrets Manager! ![AWS EKS Node Privilege Escalation](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/3vr1j83nc6o8z00p8039.png) ### Blocking Pod IMDS Access Given the privilege escalation issues granting pod permissions at the node level, ensure your EKS administrators disable pod access to the node’s IMDS endpoint. This will avoid mixing node and pod level permissions, and make it much easier for auditors to understand an individual pod’s permissions. This can be done using either of the following methods: #### IMDSv2 Hop Limit Enforce IMDS version (IMDSv2) on the cluster nodes and set the response hop limit to one (1). This ensures that IMDS responses cannot be received by the pod’s network interface. The following Terraform configuration sets the appropriate IMDSv2 configuration on the node’s *aws_launch_template* resource. ```terraform # IMDSv2 w/ hop limit set to 1 disables pod access to the IMDS metadata_options { http_endpoint = "enabled" http_tokens = "required" http_put_response_hop_limit = 1 instance_metadata_tags = "enabled" } ``` #### Network Policy Blocking IMDS access cluster-wide could cause problems if pods running in the cluster actually need access to the IMDS. Applying a *NetworkPolicy* to namespaces in the cluster that should not have access to the IMDS can provide a more flexible alternative. The following Calico *NetworkPolicy* allows the *evil* pod outbound access to networks outside the cluster, but explicitly blocks access to the IMDS endpoint: ```yaml apiVersion: crd.projectcalico.org/v1 kind: NetworkPolicy metadata: name: egress-all namespace: sabre spec: types: - Egress egress: - action: Deny destination: nets: - "169.254.169.254/32" - action: Allow destination: nets: - "0.0.0.0/0" ``` ## IAM Roles for Service Accounts (IRSA) Now that the pods cannot inherit permissions from the EKS node, workloads will need an IAM Role assigned directly to the pod. In AWS EKS, the first option for granting pod-level access to the AWS APIs is called IAM Roles for Service Accounts (IRSA). IRSA uses the EKS cluster’s OpenID Connect (OIDC) identity provider to federate into an AWS identity provider and assume a role. The updated diagram below shows the new authentication flow: ![AWS EKS Node IRSA](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/g6u9xetigh9zsgtr5ltd.png) The *web* pod's path to accessing the AWS APIs is now: 1. The development team deploys a new Kubernetes ServiceAccount *web-sa* in the *dm* namespace. A special annotation *eks.amazonaws.com/role-arn* is set on the service account mapping the Kubernetes service account to an IAM Role ARN. 2. The development team adjusts the *web* deployment to assign the new *web-sa* service account to the *web* pods. 3. As EKS creates the *web* pod, a mutating web hook automatically injects two new environment variables: *AWS_ROLE_ARN* and *AWS_WEB_IDENTITY_TOKEN_FILE*. The process running in the web container uses the new environment variables to call the *sts:AssumeRoleWithWebIdentity* API using the service account identity token (JWT) and obtain temporary access keys. ### Auditing IRSA Pod Permissions All EKS service accounts using IRSA will contain the *eks.amazonaws.com/role-arn* annotation. ```yaml apiVersion: v1 kind: ServiceAccount metadata: annotations: eks.amazonaws.com/role-arn: "arn:aws:iam::123456789012:role/dm-web-eks-pod-role" name: "web-sa" namespace: "dm" ``` Auditing IAM policies assigned to pods using IRSA can be done by enumerating all of the EKS cluster’s service accounts with the *eks.amazonaws.com/role-arn* annotation, parsing the IAM Role ARN, and then enumerating the IAM Role’s policy attachments. The *[aws-eks-audit-irsa-pods](https://gist.github.com/ejohn20/928c6de2fa7659062c8299f49c6b6884)* script uses the *kubectl* and *aws* command line interfaces to discover IRSA pod permissions: ```bash #!/bin/bash while IFS= read -r sa_metadata; do service_account=$(jq -r .name <<<"${sa_metadata}") namespace=$(jq -r .namespace <<<"${sa_metadata}") role_arn=$(jq -r .rolearn <<<"${sa_metadata}") role_name=$(jq -r '.rolearn | split("/") | .[1]' <<<"${sa_metadata}") echo "Service Account: system:serviceaccount:${namespace}:${service_account}" echo "Role ARN: ${role_arn}" echo "Policy Attachments:" aws iam list-attached-role-policies --role-name ${role_name} | jq -r .'AttachedPolicies[].PolicyArn' echo "" done < <(kubectl get serviceaccounts -A -o json | jq -c '.items[] | select(.metadata.annotations."eks.amazonaws.com/role-arn" != null) | {name: .metadata.name, namespace: .metadata.namespace, rolearn: .metadata.annotations."eks.amazonaws.com/role-arn"}' ) ``` The output shows the Kubernetes service account, IAM Role ARN, and the IAM policy attachments: ```bash Service Account: system:serviceaccount:dm:web-sa Role ARN: arn:aws:iam::123456789012:role/dm-web-eks-pod-role Policy Attachments: arn:aws:iam::123456789012:policy/dm-web-eks-pod-policy Service Account: system:serviceaccount:kube-system:aws-load-balancer-controller Role ARN: arn:aws:iam::123456789012:role/dm-eks-alb-controller-role Policy Attachments: arn:aws:iam::123456789012:policy/dm-eks-alb-controller-policy Service Account: system:serviceaccount:kube-system:ebs-csi-controller-sa Role ARN: arn:aws:iam::123456789012:role/dm-eks-ebs-csi-role Policy Attachments: arn:aws:iam::aws:policy/service-role/AmazonEBSCSIDriverPolicy ``` ## EKS Pod Identity Launched in late 2023, EKS Pod Identity offers an additional option for assigning permissions to pods. EKS Pod Identity is specific to EKS, and does not translate into other managed Kubernetes offerings (e.g., AKS, GKE). After installing the EKS Pod Identity Agent cluster add on, IAM Role to Kubernetes service *associations* are defined on the EKS cluster instead of the Kubernetes service account resource. For more details, see Christophe Tafani-Dereeper’s post titled [Deep dive into the new Amazon EKS Pod Identity feature](https://securitylabs.datadoghq.com/articles/eks-pod-identity-deep-dive/). The following Terraform configuration sets an EKS pod identity association using the *aws_eks_pod_identity_association* resource. The resource creates an association between the *sabre* namespace’s *web-sa* service account and the *dm_web_eks_pod* IAM Role. As the pod is being created, the EKS Pod Identity mutating webhook will automatically inject the *AWS_CONTAINER_CREDENTIALS_FULL_URI* and *AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE* environment variables. Processes running in the container can use the AWS_CONTAINER_CREDENTIALS_FULL_URI, which will have a value similar to *http://169.254.170.23/v1/credentials*, to request temporary credentials for the IAM Role. Invoking the endpoint requires the *Authorization* header to have the identity token found in the *AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE* volume mount (/var/run/secrets/pods.eks.amazonaws.com/serviceaccount/eks-pod-identity-token). ```terraform resource "aws_eks_pod_identity_association" "dm_web_pod_identity" { cluster_name = aws_eks_cluster.dm.name namespace = "sabre" service_account = "web-sa" role_arn = aws_iam_role.dm_web_eks_pod.arn } ``` ### Auditing Pod Identity Permissions Auditing IAM policies assigned using EKS Pod Identity can be done by enumerating all of the EKS cluster’s associations, finding the IAM Role, and then enumerating the IAM Role’s policy attachments. The *[aws-eks-audit-pod-identity-pods](https://gist.github.com/ejohn20/212ef7ea95d6dd92bda8b77e8b3b3881)* script uses only the *aws* command line interfaces to discover pod permissions. The *kubectl* command line interface is not needed because the associations live at the cluster level, not at the Kubernetes service account level. ```bash #!/bin/bash CLUSTER_NAME="$1" while IFS= read -r pod_identity_assn; do association_id=$(jq -r .associationId <<<"${pod_identity_assn}") service_account=$(jq -r .serviceAccount <<<"${pod_identity_assn}") namespace=$(jq -r .namespace <<<"${pod_identity_assn}") association=$(aws eks describe-pod-identity-association --cluster "${CLUSTER_NAME}" --association-id "${association_id}") role_arn=$(jq -r '.association.roleArn' <<<"${association}") role_name=$(jq -r '.association.roleArn | split("/") | .[1]' <<<"${association}") echo "Kubernetes Service Account: system:serviceaccount:${namespace}:${service_account}" echo "Role ARN: ${role_arn}" echo "Policy Attachments:" aws iam list-attached-role-policies --role-name "${role_name}" | jq -r .'AttachedPolicies[].PolicyArn' echo "" done < <(aws eks list-pod-identity-associations --cluster-name "${CLUSTER_NAME}" | jq -c '.associations[]') ``` The output shows the Kubernetes service account, IAM Role ARN, and the IAM policy attachments: ```bash Kubernetes Service Account: system:serviceaccount:sabre:web-sa Role ARN: arn:aws:iam::192792859965:role/dm-web-eks-pod-identity-role Policy Attachments: arn:aws:iam::192792859965:policy/dm-web-eks-pod-policy ``` ## Conclusions Auditing permissions assigned directly to EKS pods can be challenging and confusing with multiple options for creating associations between IAM Roles and Kubernetes service accounts. The audit scripts above are two helper scripts that live in my local bin directory for easily enumerating Kubernetes service accounts with access to the AWS APIs. Datadog also maintains the [Managed Kubernetes Auditing Toolkit](https://github.com/DataDog/managed-kubernetes-auditing-toolkit) (MKAT), which can be installed to perform similar permission checks. ### References * [EKS Node IAM Role](https://docs.aws.amazon.com/eks/latest/userguide/create-node-role.html) * [EKS Restrict Use of IMDS](https://docs.aws.amazon.com/whitepapers/latest/security-practices-multi-tenant-saas-applications-eks/restrict-the-use-of-host-networking-and-block-access-to-instance-metadata-service.html) * [IAM Roles for Service Accounts](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html) * [EKS Pod Identity](https://docs.aws.amazon.com/eks/latest/userguide/pod-identities.html) * [EKS Auditing Scripts](https://gist.github.com/ejohn20) * [Datadog Managed Kubernetes Auditing Toolkit](https://github.com/DataDog/managed-kubernetes-auditing-toolkit) ### About the Author [Eric Johnson](https://www.linkedin.com/in/eric-m-johnson/) is a Co-founder and Principal Security Engineer at Puma Security and a Senior Instructor with the SANS Institute. His experience includes cloud security assessments, cloud infrastructure automation, cloud architecture, static source code analysis, web and mobile application penetration testing, secure development lifecycle consulting, and secure code review assessments. Eric is the lead author and an instructor for [SEC540: Cloud Security and DevSecOps Automation](sans.org/sec540) and a co-author and instructor for both [SEC549: Cloud Security Architecture](sans.org/sec549), and [SEC510: Cloud Security Controls and Mitigations](sans.org/sec510).

ejohn20

Auditing AWS EKS Pod Permissions

Join Our Newsletter. No Spam, Only the good stuff.