Accessing S3 Buckets from CloudShell

teresafds

Teresa N. Fontanella De Santis

Posted on December 26, 2021

Accessing S3 Buckets from CloudShell

After one year since AWS CloudShell was released, it worth to comment about a connection issue between this technology and S3 Buckets.

Issue

We have a S3 bucket (in this case, named mytestbucket0123) that we need to access through AWS CloudShell.
But when trying to list all objects on a bucket from CloudShell, executing aws s3 ls s3://mytestbucket0123 we’re getting the following error

"An error occurred (AccessDenied) when calling the ListObjectsV2 operation: Access Denied"

AWS CloudShell error when listing s3 objects

Verification steps

As a starting point of this situation, we’ll need to analyze the scene and do the following verification questions:

  1. Has the user got the required permissions to list objects of our S3 Bucket? To answer this we have several ways: first check on IAM that the user has assigned those permissions.
    User on IAM with required S3 Permissions
    The user has attached the AmazonS3ReadOnlyAccess Policy, so it has ListObjects required permission. So let’s verify that the user can already list the s3 bucket objects (from the AWS console for example). Listing objects on the bucket seems to work fine from the AWS Console with the same user (from server machine) as per screenshot below.
    User can list S3 Objects from AWS Console correctly
    So, it doesn’t seem to be user IAM permissions at least. Which leads us to the following step.

  2. Has the S3 Bucket got any Bucket policy enabled? We need to search, on the AWS console, the S3 bucket and look the “Bucket policy” section in the “Permissions” tab.
    In this case, the S3 Bucket has the following Bucket Policy

    
    
    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Sid": "Statement1",
                "Effect": "Deny",
                "Principal": "*",
                "Action": "s3:*",
                "Resource": "arn:aws:s3:::mytestbucket0123",
                "Condition": {
                    "NotIpAddress": {
                        "aws:SourceIp": "<Public IP Address>"
                    },
                    "StringEqualsIfExists": {
                        "aws:SourceVpc": "<VPC-Id>"
                    }
                }
            }
        ]
    }
    
    This bucket policy denies access to all users (no matter they have the required IAM permissions), except they access from a specific IP Address or connect from our VPC (which, in this case is the AWS Account’s default VPC). That means the CloudShell is not accessing to the S3 Bucket from the VPC… So let’s ask the next question.

3. **Does CloudShell terminal connect to the S3 Bucket through a public IP Address?** To answer this, we need to execute the following command into the CloudShell terminal: `curl ifconfig.me`![The result of executing  curl ifconfig.me is 18.224.171.123, which means Public IP Address](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ycs8gjesegoplzoizvlg.png) It seems that CloudShell is trying to access to the S3 Bucket from a non whitelisted Public Address (which belongs to AWS Reserved IP Addresses).

### Cause explanation
After the verification steps, we can realize this is a CloudShell’s current limitation: it cannot connect to AWS Resources using VPC Endpoints or with VPC restricted access. Besides, the IP Address allocated per each CloudShell terminal may change over the time. 

### Solution 
To solve this issue we need to whitelist the Cloudshell’s user agent. Although this may not be a 100% secure resolution, it can be a temporary fix until AWS CloudShell can improve (or no CloudShell access is required). At the time of this writing, the user agent is "[aws-cli/2.4.5 Python/3.8.8 Linux/4.14.248-189.473.amzn2.x86_64 exec-env/CloudShell exe/x86_64.amzn.2 prompt/off command/s3.ls]”. Then edit the S3 Bucket Policy to the following (excluding also the CloudShell user agent). 
Enter fullscreen mode Exit fullscreen mode

{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "Statement1",
"Effect": "Deny",
"Principal": "",
"Action": "s3:
",
"Resource": "arn:aws:s3:::mytestbucket0123",
"Condition": {
"StringEquals": {
"aws:UserAgent": "[aws-cli/2.4.5 Python/3.8.8 Linux/4.14.248-189.473.amzn2.x86_64 exec-env/CloudShell exe/x86_64.amzn.2 prompt/off command/s3.ls]"
},
"NotIpAddress": {
"aws:SourceIp": ""
},
"StringEqualsIfExists": {
"aws:SourceVpc": ""
}
}
}
]
}



Finally, when trying to perform on the terminal the command  `aws s3 ls s3://mytestbucket0123`  finally works!

![aws s3 ls working correctly from CloudShell after S3 Bucket Policy update](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/1eylou7y1pq6kr50ckd7.png)



Enter fullscreen mode Exit fullscreen mode
💖 💪 🙅 🚩
teresafds
Teresa N. Fontanella De Santis

Posted on December 26, 2021

Join Our Newsletter. No Spam, Only the good stuff.

Sign up to receive the latest update from our blog.

Related

Accessing S3 Buckets from CloudShell
cloudshell Accessing S3 Buckets from CloudShell

December 26, 2021