How to add custom resources to AWS Amplify. Add an SQS queue that triggers lambda.

AWS Amplify is allowing you to build cloud-connected apps very easily. It has a broad range of categories it supports natively which you can use to build (see my post here). But there are still some major components missing, such as SQS and RDS. In this post, I want to show you how to add AWS services that are not currently supported in Amplify. We start with the general steps you need to take for adding these resources. In the second part, I show you how to add an SQS queue that triggers a lambda.

There are many ways of doing this but I think the easiest is to use CloudFormation. It is also possible to use frameworks like CDK or Terraform but CloudFormation is best integrated into Amplify. All deployments and updates will still happen from the Amplify CLI and no other tool is needed. All code is in this repo.

How to add custom resources in general

Amplify has a docs page here on how to add custom AWS resources. We need to do basically two steps.

Add custom resource in backend config

In the first step, you need to add a custom category name and custom resource name within your backend config amplify/backend/backend-config.json.

{
  "<custom-category-name>": {
    "<custom-resource-name>": {
      "service": <custom-aws-service-name>,
      "providerPlugin": "awscloudformation"
    }
  }
}

This could look like that. Remember that Amplify always uses the category name as a prefix. Think about that when using the names to not have the same category all over the resource names.

Add custom folders & CloudFormation

After that, you need to add the folders you referenced your category and resource names to.

amplify
  \backend
    \<custom-category-name>
      \<custom-resource-name>
        parameters_<custom-resource-name>.json
        template_<custom-resource-name>.json

template_<custom-resource-name>.json: CloudFormation Templat
parameters_<custom-resource-name>.json: Additional parameters for CloudFormation

You need both the parameters and the template files. Create them directly with the suffix custom-resource-name to avoid any name clashes in the future.
Now you would need to add your CloudFormation code in the template file. We'll see that in a second.

Reference existing parameters

The good thing here now is that you can reference output parameters of other CloudFormation categories. For example if you need the lambda ARN which was added by Amplify you can reference it in a dependsOn block in the backend-config.json.

{
  "<custom-category-name>": {
    "<custom-resource-name>": {
      "service": <custom-aws-service-name>,
      "providerPlugin": "awscloudformation",
      "dependsOn": [
        {
          "category": "function",
          "resourceName": "lambda", // check `amplify status` to find resource name
          "attributes": [
              "arn" // Check Output Value of the resource specific cloudformation file to find available attributes
            ]
        }
      ]
    }
  }
}

Input parameters need to be defined in the created template_<custom-resource-name>.json file. The format is <category><resource-name><attribute-name .

{
"Parameters": {
  "functionlambdaarn": { 
    "Type": "String"
  }
}
}

The last step for referencing is to get the value into CloudFormation as well. For that you need to add an Fn:GetAtt to your attribute name.

{
  "functionlambdaarn": {
    "Fn::GetAtt": [
      "functionlambda",  
      "Outputs.ARN"
    ]
  }
}

Finally, you need to check out your environment again and push it to the cloud with amplify env checkout <ENV> and amplify push.

So that is the theory and seems a lot first. Let's add an SQS queue and see how it works in reality.

Add a custom SQS queue

We want an SQS queue that has access to a lambda function. At this point we don't care about what triggers the queue and which messages it will send, we simply want to create it.

In this setup we have an active Amplify application and have created one lambda function named service.

Backend Config

First of all, we add a custom SQS resource. I am in root of my project and show you the following steps you need to do to create the queue. We start by editing the amplify/backend/backend-config.json

{
  "function": {
    "service": {
      "build": true,
      "providerPlugin": "awscloudformation",
      "service": "Lambda"
    }
  },
  "queue": {
    "users": {
      "service": "SQS",
      "providerPlugin": "awscloudformation"
    }
  }
}

We added the queue parameter. Several things are defined here:

queue: That is the new category name. In lambda, it is function.
standardQueue: That is the resource name. That is how you name your queue. In lambda, this is service.
service: This is the custom AWS service we want to use. In this case, it is SQS.
providerPlugin: That is the provider we use for provisioning the resources. In our case and mostly with Amplify this is CloudFormation.

CloudFormation

Now we need to add the folders we need:

mkdir -p amplify/backend/queue/users
touch amplify/backend/queue/users/parameters_users.json
touch amplify/backend/queue/users/template_users.json

Now let's first fill the template_users.json. We need to add the CloudFormation code here.

{
    "AWSTemplateFormatVersion": "2010-09-09",
    "Description": "Queue resource stack creation using Amplify CLI",
    "Parameters": {
        "env": {
            "Type": "String"
        }
    },
    "Conditions": {
        "ShouldNotCreateEnvResources": {
            "Fn::Equals": [
                {
                    "Ref": "env"
                },
                "NONE"
            ]
        }
    },
    "Resources": {
        "SQS": {
            "Type": "AWS::SQS::Queue",
            "Properties": {
                "QueueName": {
                    "Fn::If": [
                        "ShouldNotCreateEnvResources",
                        "users",
                        {
                            "Fn::Join": [
                                "",
                                [
                                    "users",
                                    "-",
                                    {
                                        "Ref": "env"
                                    }
                                ]
                            ]
                        }
                    ]
                }
            }
        }
    },
    "Outputs": {
        "Name": {
            "Value": {
                "Ref": "SQS"
            }
        },
        "Arn": {
            "Value": {
                "Fn::GetAtt": [
                    "SQS",
                    "Arn"
                ]
            }
        },
        "Region": {
            "Value": {
                "Ref": "AWS::Region"
            }
        }
    }
}

At this point, we can push the SQS queue to the cloud and create it.

amplify env checkout <ENV> # e.g. dev
amplify push

The output of amplify push should now be:

❯ amplify push
✔ Successfully pulled backend environment dev from the cloud.

    Current Environment: dev

┌──────────┬───────────────┬───────────┬───────────────────┐
│ Category │ Resource name │ Operation │ Provider plugin   │
├──────────┼───────────────┼───────────┼───────────────────┤
│ Queue    │ users         │ Create    │ awscloudformation │
├──────────┼───────────────┼───────────┼───────────────────┤
│ Function │ service       │ No Change │ awscloudformation │
└──────────┴───────────────┴───────────┴───────────────────┘
? Are you sure you want to continue? (Y/n)

Let's hit Y and deploy our SQS queue.

Let's check the CloudFormation template. Go to your Amplify console → find your environment and click on View in CloudFormation.

If you look at one of the nested stacks you will see that your SQS queue was created:

Now open SQS and you see your queue being ready to process some messages.

Lambda Consumer

Now we want to connect our SQS queue with our lambda function. To accomplish that we need to connect the lambda with our SQS queue and then define that queue as the trigger for the lambda. We already added the URL and ARN of our SQS queue in the outputs section of the CloudFormation template. That means that the CloudFormation template will output these two parameters after it is finished and other resources have access to these parameters. Let's now use this as an input for lambda. For that, we need to edit the amplify/backend/backend-config.json file again.

"function": {
    "service": {
      "build": true,
      "providerPlugin": "awscloudformation",
      "service": "Lambda",
      "dependsOn": [
        {
          "category": "queue",
          "resourceName": "users",
          "attributes": ["Name", "Arn"]
        }
      ]
    }
  }

We change the function part of the file so, that we are depending on the category queue with our exact name users and the attributes we want to pass.

Now we just need to update the CloudFormation template of lambda to receive and handle these params.

Open amplify/backend/function/service/service-cloudformation-template.json and edit the following things:

You need to add your queue name and queue ARN as an input parameter in the section Parameters
You define the name of the SQS queue as an environment variable in the section Resources/LambdaFunction/Environment/Variables

{
  "AWSTemplateFormatVersion": "2010-09-09",
  "Description": "Lambda Function resource stack creation using Amplify CLI",
  "Parameters": {
# THIS IS ADDED in the format <CATEGORY RESOURCE>Name (without space)
    "queueusersName": {
      "Type": "String",
      "Default": "queueusersName"
    },
# THIS IS ADDED in the format <CATEGORY RESOURCE>Arn (without space)
    "queueusersArn": {
      "Type": "String",
      "Default": "queueusersArn"
    }

  },
  ...
  "Resources": {
    "LambdaFunction": {
      "Type": "AWS::Lambda::Function",
      "Metadata": {
        "aws:asset:path": "./src",
        "aws:asset:property": "Code"
      },
        ...

        "Environment": {
          "Variables": {
# THIS IS ADDED: Same name as above
            "SQS_QUEUE": {
              "Ref": "queueusersName"
            },
            "ENV": {
              "Ref": "env"
            },
            "REGION": {
              "Ref": "AWS::Region"
            }
          }
        },
        "Role": {
          "Fn::GetAtt": [
            "LambdaExecutionRole",
            "Arn"
          ]
        },
        "Runtime": "nodejs14.x",
        "Layers": [],
        "Timeout": 25
      }
    },
   ...
  }
}

After checking out the environment and pushing again with amplify env checkout dev && amplify push -y you see that the URL was added as an environment variable in lambda.

SQS as input trigger

Finally, let's call the lambda from the SQS queue. The connection of the environment variable should show how to pass arguments. Don't send a message from the SQS queue and call the lambda from the SQS queue because this ends up in a loop.

To add the queue as an input trigger we need to edit the CloudFormation template again. We will add the following things:

Add LambdaFunctionEventSourceMapping and add the SQS URL here
Add an IAM policy statement to give it access to the resource

See the full file here

{...
  "Resources": {
    "LambdaFunctionEventSourceMapping": {
      "DependsOn": [
        "lambdaexecutionpolicy"
      ],
      "Type": "AWS::Lambda::EventSourceMapping",
      "Properties": {
        "EventSourceArn": {
          "Ref": "queueusersArn"
        },
        "FunctionName": {
          "Ref": "LambdaFunction"
        }
      }
    },
# ADD THIS
    "lambdaexecutionpolicy": {
      "DependsOn": [
        "LambdaExecutionRole"
      ],
      "Type": "AWS::IAM::Policy",
      "Properties": {
        "PolicyName": "lambda-execution-policy",
        "Roles": [
          {
            "Ref": "LambdaExecutionRole"
          }
        ],
        "PolicyDocument": {
          "Version": "2012-10-17",
          "Statement": [
# ADD THIS
            {
              "Effect": "Allow",
              "Action": [
                "sqs:ReceiveMessage",
                "sqs:DeleteMessage",
                "sqs:GetQueueAttributes"
              ],
              "Resource": {
                "Ref": "queueusersArn"
              }
            },
            ...
                ]
    }
...

Let's push again amplify env checkout dev && amplify push -y and see that our lambda function now has SQS as a trigger

Finally just change the resource in the IAM policy of the CloudFormation template to:

"Statement": [
            {
              "Effect": "Allow",
              "Action": [
                "sqs:ReceiveMessage"
                "sqs:DeleteMessage",
                "sqs:GetQueueAttributes"
              ],
              "Resource": {
                "Ref": "queueusersArn"
              }
            },

With that, we follow the least privileged principle and don't give lambda access to all resources. This wasn't possible before because Amplify wouldn't recognize it.

Now we see that the IAM policy is restricted on this one resource.

Now we can go to SQS, send a message and see that the lambda was executed successfully.

The video shows it as well but can't be embedded right now 🤷🏽‍♂️

Summary

That's it. It seems a bit much at the beginning but with CloudFormation and Amplify you are very flexible in how to add and connect different resources.

Here is a short summary of the steps needed to do:

Custom resource in backend config
Folders & CloudFormation for custom resource
Pass parameter from SQS to lambda from backend config
Adjust CloudFormation template in lambda to use the parameter

If you want to know more about serverless, AWS, and building SaaS products on the cloud follow my Twitter 🙂

References: https://medium.com/@navvabian/how-to-add-an-sqs-queue-to-your-amplify-cli-bootstrapped-project-cb7781c636ed

Blog