TheLazy.Dev
Portfolio GitHub

Blog

monorepo |serverless |aws |cicd

Serverless AWS With Monorepos

tomfern

Tomas Fernandez

Posted on November 30, 2021

Serverless AWS With Monorepos

Learn how to build and deploy a face-recognition application using serverless functions in AWS. Along the way, we’ll discover the Serverless Application Model, how to test Lambdas, and optimize monorepo CI/CDs.

Image description

Deploying a serverless application

We'll be deploying a face-indexing application built with a serverless backend that consumes AWS services. The components we need are:

  • AWS Lambda: runs serverless functions that process and index faces.
  • API Gateway: presents a single endpoint for the Lambda functions.
  • Rekognition: does all the face recognition.
  • DynamoDB: stores face metadata.
  • Cloudfront: is a global, low-latency CDN to speed up website load times.
  • S3: stores image files and hosts the frontend code.

Image description

How does the application work? After populating the database with faces processed with Rekognition, a user may submit an image to look up its name. If the face has already been scanned, his or her name is returned.

The complete code for the project is located here:

The demo is a monorepo, a repository holding several projects. Monorepos increase transparency across the team and help us keep code consistent. They can, however, present some scalability issues but they shouldn’t affect us as long as we use Semaphore monorepo workflows.

Shall we start?

Preparations

To complete the tutorial, you will need:

First, ensure you’ve run aws configure to connect your machine with the AWS account.


 bash
$ aws configure

AWS Access Key ID: <Type Your Access Key>
AWS Secret Access Key: <Type Your Secret Key>
Enter fullscreen mode Exit fullscreen mode

While you’re setting things up, also head to Semaphore and create a secret with the AWS access key. Semaphore will need this information to deploy on your behalf. Be sure to check the guided tour if this is your first time using Semaphore.

Image description

Finally, pick a region for the deployment. All deployed functions and services should run in the same region when possible. In this tutorial, we’re using eu-west-1 (Ireland).

Building and deploying serverless Lambdas

AWS popularized serverless functions with their Lambda service back in 2014. Serverless keep us focused solely on the code without worrying about servers, infrastructure, or containers. There is no maintenance and we enjoy reduced runtime costs (at least at the beginning).

The flip side is that we’re limited to the supported languages and must work within the runtime limits imposed by the platform. Also, their event-driven nature force us to rethink how applications are built.

Amazon published SAM (Serverless Application Model) as a way of standardizing and easing Lambda development. SAM is an open-source framework that provides shorthand expressions for interacting with the AWS ecosystem. Not only can it deploy serverless functions, but it can also test them locally, debug them while running, access remote logs, and even help us perform canary deployments.

Let’s see how it works.

Deploying the backend

We’ll start with the backend. The demo includes two alternatives for this; we have serverless functions in Java and Python. Both implement the same entrypoints:

  • Upload: adds a new face to the database.
  • Recognize: scans a face and looks it up on the database.

We’ll work with the Java version. I’ll leave the other for you to figure out.

Build the backend. Run sam build inside the backend folder in the monorepo.


 bash
$ cd java-app-backend/
$ sam build
Enter fullscreen mode Exit fullscreen mode

The build artifacts are stored in the .aws-sam folder, so you may want to gitignore it.

The template file describes everything needed to run the functions in AWS: the API paths to expose, the permissions required, and what services it depends on.

Test serverless functions. SAM uses a Docker-based testing environment for rapid development. Try running sam local start-api to start the development environment. SAM creates a local HTTP server that hosts all your functions.

Alternatively, you can run sam local start-lambda coupled with sam local invoke to invoke a specific serverless function. In most cases, you’ll need to provide an event payload during the invocation, which you can generate with sam local generate-event.

For examples of how it works, check out the developer guide.

Deploy the backend. SAM packages and uploads the code to S3. Then, it creates the AWS Lambda and API Gateway definitions so the functions can run on demand. Internally, SAM generates CloudFormation templates to provision everything.

To begin deployment, run the following command.


 bash
$ sam deploy --guided \
    --stack-name serverless-web-application-java-backend \
    --capabilities CAPABILITY_AUTO_EXPAND CAPABILITY_IAM
Enter fullscreen mode Exit fullscreen mode

The --guided option will walk you through all the steps. With --capabilities you’re granting extra permissions to CloudFormation, specifically to change IAM permissions and expand macros in the template.

When asked “X may not have authorization defined, Is this okay? type Y. This is SAM’s roundabout way of warning you that the function will be publicly-accessible.

In some cases, SAM may fail to create the S3 bucket (the error is that it can’t find bucket X). In that case, you’ll need to manually create it:


 bash
$ aws s3api create-bucket --bucket BUCKET_NAME --region eu-west-1
Enter fullscreen mode Exit fullscreen mode

If you run into any errors, You can restart the deployment process from scratch by deleting the samconfig.toml file and rerunning sam deploy.

The deployment will take a few minutes. Once done, run the following command and check the id property the function has been created. Take note of it as we'll need it later on.


 bash
$ aws apigateway get-rest-apis
{
    "items": [
        {
            "id": "iaadyd31lk", // <-- this is the unique id of the API
            "name": "serverless-web-application-java-backend",
            ...
Enter fullscreen mode Exit fullscreen mode

Backend build stage

Commit all changes made so far into the repository as we’ll set up continuous deployment with Semaphore next.

Before continuing, add your repository to Semaphore and complete the basic project setup. Once that's done, create a block with a single job in the build stage for the Java backend.

Take a look at these commands:


 bash
sem-version java 8
wget https://github.com/aws/aws-sam-cli/releases/latest/download/aws-sam-cli-linux-x86_64.zip
unzip aws-sam-cli-linux-x86_64.zip -d sam-installation
sudo ./sam-installation/install
checkout
cd java-app-backend
sam build
cache store build-java-app-backend-$SEMAPHORE_GIT_BRANCH .aws-sam
Enter fullscreen mode Exit fullscreen mode

The build job begins by switching the active Java version with sem-version and installing SAM in the CI machine. After doing checkout to clone the repository, we build the artifact, located in the .aws-sam folder, and save it in the cache.

Image description

Next, scroll down the right pane until you find the Skip/Run conditions section. Change the condition to “Run this block when conditions are met” and type the following in the When field:


 bash
change_in('/java-app-backend')
Enter fullscreen mode Exit fullscreen mode

change_in is at the center of gravity of monorepo workflows because it allows us to tie blocks to folders and files in the repository. The function scans the Git history and figures out what parts of the code had recently changed and run or skip CI/CD jobs, cutting down build times and costs.

📙 You can learn all about continuous integration for monorepos with our free ebook: CI/CD for Monorepos

Backend continuous deployment

Then, click on Add First Promotion and enable automatic promotions. Type this condition in the When? field, which trigger the deployment when all tests pass on the master branch and the backend code has been modified.


 bash
branch = 'master' AND result = 'passed' AND change_in('/java-app-backend')
Enter fullscreen mode Exit fullscreen mode

Image description

Create a deployment job in the new pipeline. By now, most commands should sound familiar. The only change is that we’re using --no-confirm-changeset, so the process is non-interactive.


 bash
sem-version java 8
wget https://github.com/aws/aws-sam-cli/releases/latest/download/aws-sam-cli-linux-x86_64.zip
unzip aws-sam-cli-linux-x86_64.zip -d sam-installation
sudo ./sam-installation/install
checkout
cd java-app-backend
cache restore build-java-app-backend-$SEMAPHORE_GIT_BRANCH
sam deploy --no-confirm-changeset --stack-name serverless-web-application-java-backend --capabilities CAPABILITY_AUTO_EXPAND CAPABILITY_IAM
Enter fullscreen mode Exit fullscreen mode

Image description

Finish setting up the CI/CD block by enabling the AWS secret you created at the beginning. Finally, save the changes into the master branch by clicking on Run the workflow > Start.

Image description

Build and deploy the frontend

The frontend is a React SPA (Single Page Application) that interacts with the Python and Java backends. We’ll run it directly from an S3 Bucket.

In the file frontend/src/GlobalConstants.ts, you’ll find the URLs for both backends. Remember the id property for the serverless functions from earlier? You need it now. Update the URLs so they point to the Lambda functions.

Image description

Now build the application with:


 bash
$ cd frontend/
$ yarn install
$ yarn build
Enter fullscreen mode Exit fullscreen mode

Create the bucket either with the S3 console or with the CLI. Pay attention to the bucket name; it should be unique, so you may need to experiment until you find one that works for you:


 bash
$ aws s3api create-bucket --bucket myfrontend-aws-monorepo --region eu-west-1 --create-bucket-configuration LocationConstraint=eu-west-1
Enter fullscreen mode Exit fullscreen mode

Copy the files and then set the bucket to host the website with the following commands:


 bash
$ aws s3 cp build s3://myfrontend-aws-monorepo --recursive --acl public-read
$ aws s3 website s3://myfrontend-aws-monorepo --index-document index.html
Enter fullscreen mode Exit fullscreen mode

The website can be visited with this URL (change the region as needed):


 bash
http://BUCKET_NAME.s3-website.eu-west-1.amazonaws.com
Enter fullscreen mode Exit fullscreen mode

To complete the deployment, provision Cloudfront and point it to the S3 website. The CDN will help you deliver the application fast to end-users all over the world. Replace the URL of the website as needed:

$ aws cloudfront create-distribution --origin-domain-name myfrontend-aws-monorepo.s3-website.eu-west-1.amazonaws.com --default-root-object index.html

{
    "Location": "https://cloudfront.amazonaws.com/2020-05-31/distribution/E26Q8XKJFH6K4G",
    "ETag": "E1E0WE2I8CVJPH",
    "Distribution": {
        "Id": "E26Q8XKJFH6K4G",
        "ARN": "arn:aws:cloudfront::890702391356:distribution/E26Q8XKJFH6K4G",
        "Status": "InProgress",
        "LastModifiedTime": "2021-11-09T20:32:02.716000+00:00",
        "InProgressInvalidationBatches": 0,
        "DomainName": "dk4ef7pcvfhnv.cloudfront.net",  // <--- Cloudfront enabled URL

        . . .
Enter fullscreen mode Exit fullscreen mode

Open a browser at the CloudFront-enabled site (use the returned DomainName value). In the upper right corner of your application, you can switch between the Java and Python backends.

Try registering some faces to see if everything is working as expected.

Image description

You may check the Lambda remote execution logs with sam logs -n LAMBDA_FUNCTION_NAME --stack STACK_NAME.

Continuous deployment for the frontend

We’ll update our CI/CD pipeline to test and deploy the frontend every time it changes.

Open the workflow editor, add a new block to build the frontend and cache the build files.


 bash
sem-version node 16.13.0
checkout
cd frontend
cache restore
yarn install
cache store
yarn build
cache store build-frontend-$SEMAPHORE_GIT_BRANCH build
Enter fullscreen mode Exit fullscreen mode

Image description

Create a test block to run the React unit tests.


 bash
sem-version node 16.13.0
checkout
cd frontend
cache restore
yarn test
Enter fullscreen mode Exit fullscreen mode

Image description

Then, set the Skip/Run conditions on both new blocks to the following:


 bash
change_in('/frontend')
Enter fullscreen mode Exit fullscreen mode

Now, create a promotion and a deploy job. The condition for deployment is:


 text
branch = 'master' AND result = 'passed' AND change_in('/frontend')
Enter fullscreen mode Exit fullscreen mode

Finally, create a deployment job to copy the build files into the S3 bucket.


 bash
checkout
cd frontend
cache restore build-frontend-$SEMAPHORE_GIT_BRANCH
aws s3 cp build s3://${BUCKET_NAME} --recursive --acl public-read
Enter fullscreen mode Exit fullscreen mode

Image description

Enable the AWS secret, then click on Run the workflow > Start.

Image description

Try the backend and frontend deployments once more to check everything is in order.

Conclusion

In this post, we’ve learned a few tools to help us work with monorepos and AWS. We’ve seen how SAM can help us quickly develop and test serverless functions and how to individually deploy services in a monorepo with Semaphore.

Read Next:

💖 💪 🙅 🚩
tomfern
Tomas Fernandez

Posted on November 30, 2021

Join Our Newsletter. No Spam, Only the good stuff.

Sign up to receive the latest update from our blog.

Related

Serverless AWS With Monorepos
monorepo Serverless AWS With Monorepos

November 30, 2021