Hello Dev's, in this article, I will show you how can you use the S3 Gateway endpoint to connect to S3 from a private subnet.

You might be thinking what is the difference between accessing S3 from a public subnet vs accessing it from a private subnet.

If you are using S3 you might already know that S3 is a public service and you can access it using the public endpoint, so the URL is publicly resolvable. What I exactly mean here, if you are in VPC which is a private environment and you want to connect to S3 you have to go out to the internet. So there will be an internet gateway your request will go through that internet gateway to access the S3.

Hope you might get the answer to the above question, in the public subnet you can access the internet via the internet gateway, and in the private subnet, you don’t have internet so you cant reach out to the internet and request time out.

Imagine due to some security policies you don’t want to go over the public internet to access the S3. Then how can you access the S3, here comes the “S3 Gateway Endpoint” to rescue.

What is “VPC Endpoint”?

As per AWS official documentation:

A VPC endpoint enables connections between a virtual private cloud (VPC) and supported services, without requiring that you use an internet gateway, NAT device, VPN connection, or AWS Direct Connect connection. Therefore, you control the specific API endpoints, sites, and services that are reachable from your VPC.

AWS provides the following types of VPC Endpoint

• Interface endpoints
• Gateway load balancer endpoints
• Gateway endpoints

S3 Gateway endpoint provides a way for the request to be routed through AWS Network to S3, without going out to the internet.

Let's understand what problem exactly the S3 gateway solves for us by doing a small Lab.

Prerequisites

• AWS Account - Click here to create one
• Basic understanding of EC2, VPC & S3

Let's Setup a Lab

To set up our lab we will create the below resources in our AWS account

• Public & Private Subnet
• Public & Private route table
• EC2 instances in both subnets
• IAM Role to access S3 from EC2
• S3 Gateway Endpoint

Public & Private Subnets

A public subnet is a subnet that has a route to the internet gateway, due to that the EC2 instances launched in a public subnet can access the internet. On the other hand, a private subnet is a subnet that does not have an internet gateway attached to it.

To create a subnet go to VPC and from the left side menu select Subnets and click on the Create Subnet button, fill in the form and click on Create Subnet.

Follow the same process to create a private subnet, once you created two subnets that will look like

Public & Private route table

Before creating a route table we will create an Internet Gateway to attach to our public route table

Once our internet gateway is created we will attach it to our VPC

To create a route table click on the Route Tables menu from the left sidebar and click Create route table

Once our route table is created we will add a route to an internet gateway

and associate the public subnet to our public route table

Launch EC2 instances

We will launch two EC2 instances each in a public and private subnet. Navigate to Services -> EC2 and click Launch Instances, it will open the launch EC2 instance wizard,

Once AMI is selected, on the next screen select the instance type, and then click Next

On the Configure Instance step select the VPC & Subnet, as we want to put this EC2 instance in the public subnet we will select the public subnet we created from the subnet dropdown

As we will stick with the default setting for other steps click Review and Launch and then Launch. The popup will show up, select the existing key pair or create a new key pair for your EC2 instance, this key will be used for SSH access

Follow the same steps again to launch our second instance, just make sure you select the private subnet this time.

Now you should have the two instances, as we launch one instance in the public subnet hence that instance got the public IP.

Create IAM Role

To access the S3 service securely from our EC2 instances, we will create one IAM role which we will then assign to EC2 instances.

To create an IAM role go to IAM -> Roles and click the Create Role button

Select the type of trusted entity and select the service to which this role will be assigned, select AWS services and then EC2 from the use case

Next, we will select which permissions we want to assign to this new role, as we want to access the S3 service we will search for S3 and select AmazonS3FulllAccess

Go to the review step and give a name to the newly created role and click Create Role

Assign Role to EC2

Now we will assign this role to our EC2 instances, (you can create this role before launching the EC2 and then select the role from IAM Role dropdown in EC2 instance launch wizard)

To assign the role go to EC2 and select one of the EC2 instances then click on
Action -> Security -> Modify IAM role

It will open a new screen, select the role that we created in the above step, and click save,

Demo 1: Fall Down - Access S3 from the private subnet

In this demonstration, we will test the scenario of what exactly happens if we don't have the S3 Gateway assigned to the private subnet.

Let's SSH into our public instance, then from the public instance, we will SSH into our private instance using the ssh-agent forwarding.

Use below command to add SSH key in ssh-agent,

❯ ssh-add -K SW-Demo-Key.pem 

Once we added the key, use the below command to ssh into the public EC2 instance,

❯ ssh -A ec2-user@13.127.220.246

Now we will ssh to our private instance using private IP

❯ ssh ec2-user@10.10.1.245

Next, we will try to access the S3 from a private EC2 instance using the S3 command-line tool,

[ec2-user@ip-10-10-1-245 ~]$ aws s3 ls --region ap-south-1

This will not work because, the EC2 instance in the private subnet doesn't have access to the internet, therefore S3 CLI cant reach the S3 service.

Create S3 Gateway Endpoint

To create an S3 gateway endpoint, navigate to services ->VPC->Endpoints and click the Create Endpoint button,

As we want to create an endpoint for S3 search for S3 in the search box and select the service of type Gateway

Once we choose which type of endpoint we want to create now we need to apply that endpoint to our private subnet, to do that select the VPC and then select the private route table.

Next, click the Create Endpoint button to create the endpoint,

AWS will automatically add a route in the selected route table in the background.

Demo 2:

Now let's try to access the S3 one more time from our private EC2 instance, SSH into the private EC2 instance as shown above using the SSH agent forwarding, and re-run the S3 CLI command

[ec2-user@ip-10-10-1-245 ~]$ aws s3 ls --region ap-south-1
2022-01-18 18:31:44 sw-s3-endpoint-demo

Hurry !! this time we can successfully connect to the S3 using the S3 Gateway endpoint.

And that’s it, guys, don't forget to delete all the resources created during this lab.

Hope you find this tutorial helpful. Don't forget to share and leave a comment if it helps you.