--- title: Using Docker on Lambda for Postgres to S3 Backups published: true description: Exploring AWS Lambda's new container image support by implementing a cron job that backs up my Postgres database to S3. tags: lambda, docker, postgres, aws cover_image: https://i.postimg.cc/N0c38Ns6/dia3.png --- ### Intro If you don't care about any context you can just skip down to the [Ship it](#ship-it) section. I've never written a tech blog post before, but I figured out how to leverage AWS Lambda's [newly announced container image support (Dec 1, 2020)](https://aws.amazon.com/blogs/aws/new-for-aws-lambda-container-image-support/) to back up a database I'm maintaining and it wasn't as straightforward as I'd hoped, so I figured I'd write something on it. For context, as a cost-cutting measure™, I have just a single EC2 instance with some docker containers running my application, and that same EC2 instance houses my database! Sorry if you hate it! If you *also* didn't feel like dealing with the costs and complexities of RDS & snapshotting and just want a nice way to back up your data, you're in the right place! ### My reasons 1. I have 2- count 'em- 2 users for an app I made and I want to make sure their data is less ephemeral than the reliability of a database-on-EC2-instance setup would suggest. I imagine this scales to databases much larger than what I have, but I haven't tested it out personally. I suppose you're up against the max timeout (15 minutes), and the max memory capacity (10GB) of a Lambda execution. 1. It's super cheap- for S3, all you're paying for is the storage, since data transfer *into* S3 is free, and you shouldn't be exporting from S3 very often. And of course Lambda is very cheap- I've calculated that for my setup, backing up an admittedly very small amount of data once every hour, it will cost approximately $0.05/mo. 1. It's super configurable- you can snapshot as frequently or infrequently as you'd like, and I guess if you wanted to you could backup only certain tables or something- I don't know I'm not super familiar with RDS' snapshotting capabilities, maybe they're similar. 1. I can pull my production data down for local testing/manipulating very easily! It's just a single docker command! ### High-Level 1. For the actual database backups I adapted a Docker-based script I found on GitHub [here](https://github.com/schickling/dockerfiles/tree/master/postgres-backup-s3). 1. For running that script I'm using AWS Lambda's new "Container Image" option. I borrowed the Dockerfile from the **Building a Custom Image for Python** section of [Amazon's announcement](https://aws.amazon.com/blogs/aws/new-for-aws-lambda-container-image-support/) to help configure things in a way that'd make sense. 1. For triggering the Lambda on a cron I'm using [Amazon EventBridge](https://console.aws.amazon.com/events/home). I've never heard of it before but it was really easy to create a rule that just says, "run this Lambda once an hour," so I'm recommending it. 1. I'm storing the SQL dump files in a private S3 bucket with a retention policy of 14 days. ### Let's go lower now So in my head I was like, "Oh cool I can just chuck this Dockerfile on a Lambda and run a command on an image with some environment variables once an hour and we're golden." It doesn't really work that way though. Lambda requires that your Docker image's entrypoint be a function, in some programming language, that gets called when triggered. So rather than just being able to trigger a docker run command (which could in my case run a script, `backup.sh`) like so: ```bash docker run schickling/postgres-backup-s3 ``` It's more that you're setting up a Docker environment for a program (in my case a Python program), that'll have an entrypoint function, that'll run whatever you need to run (again, in my case `backup.sh`). What's that entrypoint function look like? Pretty simple: ```python import json import subprocess def handler(event, context): print("Received event: " + json.dumps(event, indent=2)) subprocess.run("sh backup.sh".split(" ")) print("Process complete.") return 0 ``` The mangling of the Dockerfile they provide as an example in the **Building a Custom Image for Python** section of [Amazon's announcement](https://aws.amazon.com/blogs/aws/new-for-aws-lambda-container-image-support/), to look more like [the Dockerfile of the Postgres to S3 backup script](https://github.com/schickling/dockerfiles/blob/master/postgres-backup-s3/Dockerfile) was the more complicated part. I'll let you take a look at that what that final Dockerfile looks like [here](https://github.com/cd17822/lambda-s3-pg-backup/blob/main/db-backup.Dockerfile). ### Some other gotchas #### AWS Environment variables By far the most annoying part of this whole thing was finding some [GitHub issue comment](https://github.com/jschneier/django-storages/issues/606#issuecomment-426136721) that mentioned that Lambdas automatically set the `AWS_SESSION_TOKEN` and `AWS_SECURITY_TOKEN` environment variables, and it turns out it was causing a hard-to-track-down error in the backup script's invocation of the aws client along the lines of: ``` An error occurred (InvalidToken) when calling the PutObject operation: The provided token is malformed or otherwise invalid. ``` If all this article accomplishes is that someone stumbles upon this section after Googling that error, I will consider it astoudingly successful. Anyway, I just had to edit the `backup.sh` file to add these two lines and the complaining stopped: ```bash unset AWS_SECURITY_TOKEN unset AWS_SESSION_TOKEN ``` #### Writing to files For some reason Lambda didn't like that the `backup.sh` script was writing to a file. After a couple minutes of researching with no luck, I decided to change: ```bash echo "Creating dump of ${POSTGRES_DATABASE} database from ${POSTGRES_HOST}..." pg_dump $POSTGRES_HOST_OPTS $POSTGRES_DATABASE | gzip > dump.sql.gz echo "Uploading dump to $S3_BUCKET" cat dump.sql.gz | aws $AWS_ARGS s3 cp - s3://$S3_BUCKET/$S3_PREFIX/${POSTGRES_DATABASE}_$(date +"%Y-%m-%dT%H:%M:%SZ").sql.gz || exit 2 ``` to: ```bash echo "Creating dump of ${POSTGRES_DATABASE} database from ${POSTGRES_HOST} and uploading dump to ${S3_BUCKET}..." pg_dump $POSTGRES_HOST_OPTS $POSTGRES_DATABASE | gzip | aws $AWS_ARGS s3 cp - s3://$S3_BUCKET/$S3_PREFIX/${POSTGRES_DATABASE}_$(date +"%Y-%m-%dT%H:%M:%SZ").sql.gz || exit 2 ``` There might be a better way around this but I couldn't find one, so here we are, just piping away. #### Lambda timeouts The Lamba I had was timing out by default after 3 seconds. Make sure you jack that up in the function configuration's **General Configuration**. ### Test it out locally Testing this out locally is really easy because the example Dockerfile that Amazon provided in their announcement has "Lambda Runtime Interface Emulator" support [built in](https://github.com/cd17822/lambda-s3-pg-backup/blob/main/db-backup.Dockerfile#L55-L57). Build the image: ```bash docker build -t db-backup -f db-backup.Dockerfile . ``` Run it in Terminal Window 1: ```bash docker run \ -e POSTGRES_DATABASE=ms -e POSTGRES_HOST= \ -e POSTGRES_PASSWORD= \ -e POSTGRES_USER= \ -e S3_ACCESS_KEY_ID= \ -e S3_BUCKET= \ -e S3_REGION= \ -e S3_PREFIX= \ -e S3_SECRET_ACCESS_KEY= \ -p 9000:8080 \ db-backup:latest ``` Trigger it in Terminal Window 2: ```bash curl -XPOST "http://localhost:9000/2015-03-31/functions/function/invocations" -d '{}' ``` ### Ship it Sure, the right™ way to ship it can be debated by whoever, but the simplest way is probably: 1. Clone the code: ```bash git clone https://github.com/cd17822/lambda-s3-pg-backup.git ``` 1. Build the Docker image: ```bash docker build -t db-backup -f db-backup.Dockerfile ``` 1. Tag the built image to be pushed to a private [ECR](https://aws.amazon.com/ecr/) repository: ```bash docker tag db-backup .dkr.ecr..amazonaws.com/:latest ``` 1. Push the image up to ECR: ```bash docker push .dkr.ecr..amazonaws.com/:latest ``` 1. Create a private S3 bucket that we'll be storing the backups in (I'd also recommend setting up a retention policy unless you want to keep around these files forever). 1. Create a Lambda function by selecting `Create Function` in the [Lambda Console](https://console.aws.amazon.com/lambda/home). 1. Select `Container Image`, name it whatever you want, find the Docker image in ECR in `Browse Images`, leave everything else as default and finally select `Create Function`. 1. Scroll down to **Environment variables** and set values for the following environment variables: ``` POSTGRES_DATABASE POSTGRES_HOST POSTGRES_PASSWORD POSTGRES_USER S3_ACCESS_KEY_ID S3_BUCKET S3_PREFIX S3_SECRET_ACCESS_KEY ``` 1. Scroll down further and make sure you edit the **General Configuration** such that the Timeout is bumped up to something like 5 minutes. 1. At this point you can select `Test` on the top right and check to make sure your function's working. 1. Finally, you can set up a scheduled trigger by selecting `Add trigger` in the **Designer**. I'd recommend setting up a simple EventBridge trigger that runs on a cron (`cron(13 * * * *)`) or with a set frequency (`rate(1 hour)`). ### Restoring from backup You could set up a Lambda to restore your database from backup that's triggered by emailing a photo of yourself crying to an unsolicited email address using AWS Computer Vision, but for the sake of this article I figured I'd just include the easy way to do it. In the same repo that the backup script is in lies a [restore script](https://github.com/schickling/dockerfiles/tree/master/postgres-restore-s3). It's hosted on [DockerHub](https://registry.hub.docker.com/r/schickling/postgres-restore-s3) making it really easy to pull and run locally: ```bash docker pull schickling/postgres-restore-s3 docker run -e S3_ACCESS_KEY_ID=key -e S3_SECRET_ACCESS_KEY=secret -e S3_BUCKET=my-bucket -e S3_PREFIX=backup -e POSTGRES_DATABASE=dbname -e POSTGRES_USER=user -e POSTGRES_PASSWORD=password -e POSTGRES_HOST=localhost schickling/postgres-restore-s3 ```

cd17822

Using Docker on Lambda for Postgres to S3 Backups

Join Our Newsletter. No Spam, Only the good stuff.