If you're anything like me, you prefer to avoid logging into the AWS console as much as possible. Did you set up your IAM root user with 2FA and correctly configure the CORS and ACL settings on your S3 bucket?

🤷‍♂️ nah.

I also prefer to keep my local development environment as close as possible to how it's going to work in production. Additionally, I'm always looking for new ways to fill up my small hard drive. I can't think of a better away to achieve all of the above than putting a bunch of S3 servers inside my computer.

This tutorial will cover setting up Localstack within a node app. Localstack allows you to emulate a number of AWS services on your computer, but we're just going to use S3 in this example. Also, Localstack isn't specific to Node - so even if you aren't working in Node, a good portion of this tutorial will still be relevant. This also covers a little bit about Docker - if you don't really know what you're doing with Docker or how it works, don't worry. Neither do I.

You can see the demo repo for the finished code.

A few benefits of this approach are:

• You can work offline
• You don't need a shared 'dev' bucket that everyone on your team uses
• You can easily wipe & replace your local buckets
• You don't need to worry about paying for AWS usage
• You don't need to log into AWS 😛

Initial Setup

First, we'll need to install a few things.

1. Install Docker if you haven't already.
2. Install the AWS CLI. Even though we aren't going to be working with "real" AWS, we'll use this to talk to our local docker containers.
3. Once the AWS CLI is installed, run aws configure to create some credentials. Even though we're talking to our "fake" local service, we still need credentials. You can enter real credentials (as described here), or dummy ones. Localstack requires that these details are present, but doesn't actually validate them. Thanks to @alexiswilke for pointing out in the comments that I missed this step!
4. Make a few files. Create a new directory for your project, and within it: touch index.js docker-compose.yml .env && mkdir .localstack
5. Add an image to your project directory and rename it to test-upload.jpg
6. npm init to set up a package.json, then npm install aws-sdk dotenv

Docker

(disclaimer: I'm not a docker expert. If anyone has any suggestions on how to improve or better explain any of this, please let me know in the comments!)

Docker Config

You can run Localstack directly from the command line, but I like using Docker because it makes me feel smart. It's also nice because you don't need to worry about installing Localstack on your system. I prefer to use docker-compose to set this up. Here's the config:

docker-compose.yml

version: '3.2'
services:
  localstack:
    image: localstack/localstack:latest
    container_name: localstack_demo
    ports:
      - '4563-4599:4563-4599'
      - '8055:8080'
    environment:
      - SERVICES=s3
      - DEBUG=1
      - DATA_DIR=/tmp/localstack/data
    volumes:
      - './.localstack:/tmp/localstack'
      - '/var/run/docker.sock:/var/run/docker.sock'

(August 10, 2019 edit: LocalStack now has a larger range of ports, the yaml above has been updated to reflect this. Thanks to @arqez for mentioning this in the comments)

Breaking some of these lines down:

image: localstack/localstack:latest

Use the latest Localstack image from Dockerhub

container_name: localstack_demo:

This gives our container a specific name that we can refer to later in the CLI.

ports: '4563-4599:4563-4599' and '8055:8080':

When your docker container starts, it will open up a few ports. The number on the left binds the port on your localhost to the port within the container, which is the number on the right. In most cases, these two numbers can be the same, i.e. 8080:8080. I often have some other things running on localhost:8080, so here, I've changed the default to 8055:8080. This means that when I connect to http://localhost:8055 within my app, it's going to talk to port 8080 on the container.

The line '4563-4584:4563-4584' does the same thing, but binds a whole range of ports. These particular port numbers are what Localstack uses as endpoints for the various APIs. We'll see more about this in a little bit.

environment

These are environment variables that are supplied to the container. Localstack will use these to set some things up internally:

• SERVICES=s3: You can define a list of AWS services to emulate. In our case, we're just using S3, but you can include additional APIs, i.e. SERVICES=s3,lambda. There's more on this in the Localstack docs.
• DEBUG=1: 🧻 Show me all of the logs!
• DATA_DIR=/tmp/localstack/data: This is the directory where Localstack will save its data internally. More in this next:

volumes

'./.localstack:/tmp/localstack'

Remember when set up the DATA_DIR to be /tmp/localstack/data about 2 seconds ago? Just like the localhost:container syntax we used on the ports, this allows your containers to access a portion of your hard drive. Your computer's directory on the left, the container's on the right.

Here, we're telling the container to use our .localstack directory for its /tmp/localstack. It's like a symlink, or a magical portal, or something.

In our case, this makes sure that any data created by the container will still be present once the container restarts. Note that /tmp is cleared frequently and isn't a good place to store. If you want to put it in a more secure place

• '/var/run/docker.sock:/var/run/docker.sock'

Starting our Container

Now that we have our docker-compose.yml in good shape, we can spin up the container: docker-compose up -d.

To make sure it's working, we can visit http://localhost:8055 to see Localstack's web UI. Right now it will look pretty empty:

Similarly, our S3 endpoint http://localhost:4572 will show some basic AWS info:

(If you don't see something similar to these, check the logs for your docker containers)

Working with Localstack

AWS is now inside our computer. You might already be feeling a little bit like you are the richest person in the world. (If not, don't worry, just keep reading 😛)

Before we start uploading files, we need to create and configure a bucket. We'll do this using the AWS CLI that we installed earlier, using the --endpoint-url flag to talk to Localstack instead.

1. Create a bucket: aws --endpoint-url=http://localhost:4572 s3 mb s3://demo-bucket
2. Attach an ACL to the bucket so it is readable: aws --endpoint-url=http://localhost:4572 s3api put-bucket-acl --bucket demo-bucket --acl public-read

Now, when we visit the web UI, we will see our bucket:

If you used volumes in your docker settings, let's pause for a moment to look at what's going on in ./.localstack/data.

Here, we can see that Localstack is recording all API calls in this JSON file. When the container restarts, it will re-apply these calls - this is how we are able to keep our data between restarts. Once we start uploading, we won't see new files appear in this directory. Instead, our uploads will be recorded in this file as raw data. (You could include this file in your repo if you wanted to share the state of the container with others - but depending on how much you upload, it's going to become a pretty big file)

If you want to be able to "restore" your bucket later, you can make a backup of this file. When you're ready to restore, just remove the updated s3_api_calls.json file, replace it with your backup, and restart your container.

Uploading from our app

There are a lot of S3 uploading tutorials out there, so this section won't be as in-depth. We'll just make a simple upload function and try uploading an image a few times.

Copy these contents into their files:

.env, our environment variables

AWS_ACCESS_KEY_ID='123'
AWS_SECRET_KEY='xyz'
AWS_BUCKET_NAME='demo-bucket'

Note: it doesn't matter what your AWS key & secret are, as long as they aren't empty.

aws.js, the module for our upload function

const AWS = require('aws-sdk')
require('dotenv').config()

const credentials = {
   accessKeyId: process.env.AWS_ACCESS_KEY_ID,
   secretAccessKey: process.env.AWS_SECRET_KEY,
}

const useLocal = process.env.NODE_ENV !== 'production'

const bucketName = process.env.AWS_BUCKET_NAME

const s3client = new AWS.S3({
   credentials,
   /**
    * When working locally, we'll use the Localstack endpoints. This is the one for S3.
    * A full list of endpoints for each service can be found in the Localstack docs.
    */
   endpoint: useLocal ? 'http://localhost:4572' : undefined,
   /**
     * Including this option gets localstack to more closely match the defaults for
     * live S3. If you omit this, you will need to add the bucketName to the `Key`
     * property in the upload function below.
     *
     * see: https://github.com/localstack/localstack/issues/1180
     */
   s3ForcePathStyle: true,
})


const uploadFile = async (data, fileName) =>
   new Promise((resolve) => {
      s3client.upload(
         {
            Bucket: bucketName,
            Key: fileName,
            Body: data,
         },
         (err, response) => {
            if (err) throw err
            resolve(response)
         },
      )
   })

module.exports = uploadFile

Thanks to @mcmule for the hint about the s3ForcePathStyle option above. If you're getting an ECONNREFUSED error, take a look at his comment

test-upload.js, which implements the upload function

const fs = require('fs')
const path = require('path')
const uploadFile = require('./aws')

const testUpload = () => {
   const filePath = path.resolve(__dirname, 'test-image.jpg')
   const fileStream = fs.createReadStream(filePath)
   const now = new Date()
   const fileName = `test-image-${now.toISOString()}.jpg`
   uploadFile(fileStream, fileName).then((response) => {
      console.log(":)")
      console.log(response)
   }).catch((err) => {
      console.log(":|")
      console.log(err)
   })
}

testUpload()

the testUpload() function gets the file contents, gives it a unique name based on the current time, and uploads it. Let's give it a shot:

node test-upload.js

Copy the URL in the Location property of the response and paste it into your browser. The browser will immediately download the image. If you want to see it in your browser, you can use something like JS Bin:

Then, if you look at .localstack/data/s3_api_calls.json again, you'll see it filled up with the binary data of the image:

Finally, let's restart the container to make sure our uploads still work. To do this, run docker restart localstack_demo. After it has restarted, run docker logs -f localstack_demo. This will show you the logs of the container (the -f flag will "follow" them).

After it initializes Localstack, it will re-apply the API calls found in s3_api_calls.json:

When you reload your browser, you should see the image appear just as before.

🎉 That's it! Thanks for sticking around. This is my first tutorial and I'd love to know what you think. If you have any questions or suggestions, let me know in the comments!