Scaffolding Serverless Web Application on AWS

In this article, I'm going to setup a serverless based web application on AWS Cloud. Firstly I will explain its AWS cloud architecture and infrastructure that used in the application. Then, dive into the details of AWS services. Finally, go through the source code, and guide you to setup your own application from scratch. In the end, you should know:

How to design a serverless web application using API Gateway, Lambda and DynamoDB.
How to provision and organize cloud infrastructure using Terraform.
How to use Lambda PowerTools to organize and fulfill HTTP/HTTPS requests/responses.
How to use PynamoDB to interact with DynamoDB using an ORM-like interface.

Demo Application
AWS Infrastructure
Source Code
Development & Deployment
Summary
References

Let's get started!

Demo Application

The demo application provides a simple “to-do list” backend interface that supports CURD operations. Here is the screenshot of Swagger UI.

AWS Infrastructure

The serverless web application is built with several AWS services together. Below diagram shows how these AWS services integrate with each other. It contains four layers:

Core Layer: includes API Gateway and Lambda function, layers.
Database Layer: DynamoDB tables.
Access Control Layer: IAM roles for Lambda function.
Monitoring Layer: CloudWatch Log groups for API Gateway and Lambda function.

Core Layer - API Gateway & Lambda

Create a Restful API in API Gateway, and a function and layers in Lambda.
API Gateway provide APIs that act as the "front door" for applications to access data, business logic, or functionality from your backend services.

From the screenshot, each API method has a Lambda integration . Any GET /todos request flows into integrated Lambda function. The function fulfills the request and get todo items from DynamoDB table.

Database Layer - DynamoDB

Amazon DynamoDB is a serverless, NoSQL database service that enables you to develop modern applications at any scale. Two tables are created.

Access Control Layer - IAM

With AWS Identity and Access Management (IAM), you can specify who or what can access services and resources in AWS, centrally manage fine-grained permissions, and analyze access to refine permissions across AWS.
Two IAM roles are created: API Gateway logging role and Lambda function execution role.

API Gateway Logging Role

In order to collect logs from API Gateway, I grant API Gateway permission to read and write logs to CloudWatch in the account. The role takes effect on the account level, so I created manually from AWS console before the demo. Follow the official tutorial to create the role.

Lambda Function Execution Role

A Lambda function's execution role is an AWS IAM role that grants the function permission to access AWS services and resources. For example, you might create an execution role that has permission to send logs to Amazon CloudWatch. In the demo, we should grant DynamoDB permissions to the role.

Monitoring Layer - CloudWatch

As we grant API Gateway and Lambda function permissions to write logs to CloudWatch Logs, we still need to create CloudWatch Log groups for them. You can find the logs for investigation or debugging.

Next section, I'll introduce the source code of Lambda function, and AWS infrastructure. You can find the source code from https://github.com/camillehe1992/scaffolding-serverless-project-on-aws.

Source Code

As I mentioned in the beginning, although we don't need to provision and manage servers, we still need to create AWS infrastructure manually from AWS console, or manage infrastructure as code (IaC) using CDK, CloudFormation or Terraform, etc. I use Terraform to define all AWS infrastructure and deploy them into AWS via Terraform CLI in GitHub Actions workflow. Except for Terraform code, the Lambda function source code is important as well.
I will introduce Terraform and Lambda function code structure and python dependencies I used. These dependencies make the code readable, maintainable and be organized. Greatly reduce the workload of development in a real project of your work.

Terraform

Terraform is an infrastructure as code tool that lets you build, change, and version infrastructure safely and efficiently. Terraform code is in the terraform directory.



.
├── deployments     # Define AWS resources and data with Terraform scripts     
│   ├── api
│   ├── common_infra
│   └── dynamodb
├── modules        # Define Terraform modules for AWS resources
│   ├── README.md
│   ├── api_gateway
│   ├── dynamodb
│   ├── iam
│   ├── lambda_function
│   ├── lambda_layer
│   └── vpc_endpoint
└── settings.     # Define Terraform variables for each environment
    ├── dev
    └── prod

I won't dive into the details of Terraform scripts here, which are not in the scope. The key point I want to share, from my experience, you should organize Terraform code according to the cloud architecture. For example, separate AWS infrastructure into several groups, so that they can be managed and provisioned individually, especially when there is a bunch of infrastructure in the cloud architecture. Use Terraform modules to reduce code redundancy and operational and maintenance cost.

Terraform Modules are containers for multiple resources that are used together in a configuration. https://developer.hashicorp.com/terraform/language/modules#modules

Lambda Code



# File structure in src directory
.
├── __init__.py
├── portal                 # Lambda function source code
│   ├── __init__.py
│   ├── app
│   └── requirements.txt
└── tests                  # All test related source code
    ├── e2e
    ├── local
    ├── thunder
    └── unit

The core lambda logic code is in src.portal.app directory.



# In src.portal
.
├── __init__.py
├── app
│   ├── __init__.py
│   ├── database
│   ├── enum.py
│   ├── logging.py
│   ├── main.py
│   ├── models
│   └── routers
└── requirements.txt

The main Python dependencies I use in the serverless web application are Lambda PowerTools and PynamoDB.

Lambda PowerTools

https://docs.powertools.aws.dev/lambda/python/latest/
Lambda PowerTools provides many amazing features, such as validation, logging, event handler, enable swagger ui etc. It makes the serverless code work as web application framework that you are more familiar with, for example Flask or FastAPI. The entry of the application is lambda_handler function in app.main.py file.

PynamoDB

https://github.com/pynamodb/PynamoDB
A pythonic interface to Amazon's DynamoDB, that provide an ORM-like interface with query and scan filters. It supports many features which makes it more comfortable to interact with DynamoDB API.

Others

Except for Lambda and Terraform source code, there are some development configurations.

cloudformation/infra.yaml: Define the S3 bucket and DynomoDB lock table for Terraform backend configuration.
Makefile: Define make script to simplify your local deployment using shell scripts.
.pylinrc, .pytest.ini, .pre-commit-config.yaml etc : Testing, linting and formatting configuration for python code.
.github/actions: automate workflows to deploy/destroy infrastructure to/from AWS account.

Development & Deployment

For local development, you can follow the Development documentation to setup environment on your local machine.
You can also deploy the serverless web application from your local machine or via GitHub Actions following Deployment documentation.

Summary

In the end, you should learned:

Design a serverless web application architecture using API Gateway, Lambda and DynamoDB.
Provision and organize cloud infrastructure using Terraform.
Use Lambda PowerTools to fulfill core logic.
Use PynamoDB to interact with DynamoDB using an ORM-like interface.

Besides, there are tips you'd better to know.

AWS officially provides Lambda layers for Lambda PowerTools https://docs.powertools.aws.dev/lambda/python/2.26.1/#lambda-layer
API Gateway API methods and resource can be generated from swagger.yaml. https://github.com/camillehe1992/scaffolding-serverless-project-on-aws/blob/main/terraform/deployments/api/swagger.yaml

References

Thanks for reading and looking forward to your ideas and advices.

Blog