Tathagata
Posted on March 31, 2019
Every now and then, you need to a script to run at regular interval and do something interesting. Before the advent of serverless computing, cron jobs were the defacto way of doing this. But now we have several options that are more cost effective. ECS scheduled tasks and lambdas are two serverless alternatives for task scheduling in AWS. Depending on your workload you can decide which option would be the right choice for you. Lambdas, with 15-minute runtime limit are ideal for small workloads. Let's explore how we can get this working.
Install SAM cli
AWS Sam cli is a tool with an adorable chipmunk as mascot. It simplifies building and deploying lambdas.
pip install --user aws-sam-cli
Create a project
sam init --name lambda-cron --runtime python3.7
[+] Initializing project structure...
Project generated: ./lambda-cron
Steps you can take next within the project folder
===================================================
[*] Invoke Function: sam local invoke HelloWorldFunction --event event.json
[*] Start API Gateway locally: sam local start-api
Read lambda-cron/README.md for further instructions
[*] Project initialization is now complete
Lets take a look at the generated code
tree lambda-cron/
lambda-cron/
|-- README.md
|-- event.json
|-- hello_world
| |-- __init__.py
| |-- __pycache__
| | |-- __init__.cpython-37.pyc
| | `-- app.cpython-37.pyc
| |-- app.py
| `-- requirements.txt
|-- template.yaml
`-- tests
`-- unit
|-- __init__.py
|-- __pycache__
| |-- __init__.cpython-37.pyc
| `-- test_handler.cpython-37.pyc
`-- test_handler.py
5 directories, 12 files
This might seem an overkill of artifacts compared to a line in your cron tab. But if you think you are getting rid of the entire server, this doesn't look that unreasonable. Let's take a look at the non-obvious files.
event.json
sam's scaffolding is targets the most popular lambda use case - a function servicing an API gateway endpoint. We can ignore this.
app.py
This file is of primary importance, as this is where your code goes. By default, lambda_handler is the name of the function which you populate. For now, we will just limit it to a printing a string.
def lambda_handler(event, context):
print("Do something interesting!")
requirements.txt
Standard stuff - your package dependencies go here.
template.yaml
This file is responsible for orchestrating the serverless-ness. It declaratively specifies what components make up your stack, where the stack is your entire software architecture deployed as one unit in AWS CloudFromation. If this is not magic, I don't know what is. I am a big fan of CloudFormation.
Note this file was generated to build a lambda function that gets triggered by an HTTP request to the API gateway endpoint. But our goal is to run it at scheduled intervals. Take a look at the section which describes the events.
Events:
HelloWorld:
Type: Api # More info about API Event Source: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#api
Properties:
Path: /hello
Method: get
We need to replace it with the schedule information. Sam template supports schedule expression in both cron syntax and a simpler rate expression.
Every minute using rate
Events:
HelloWorld:
Type: Schedule
Properties:
Schedule: rate(1 minute)
Every minute using cron
Events:
HelloWorld:
Type: Schedule
Properties:
Schedule: cron(1 * * * ? *)
test_handler.py
This is pre-generated unittest stubs for pytest. You can install pytest (pip install pytest) and run pytest like usual.
Build
With your tests passing, it is now time to do the builds
cd lambda-cron/
t lambda-cron $ sam build --use-container
2019-03-30 19:42:24 Starting Build inside a container
2019-03-30 19:42:24 Building resource 'HelloWorldFunction'
Fetching lambci/lambda:build-python3.7 Docker container image......
2019-03-30 19:42:25 Mounting /Users/t/projects/python/playground/lambda-deep-dive/lambda-cron/hello_world as /tmp/samcli/source:ro inside runtime container
Build Succeeded
Built Artifacts : .aws-sam/build
Built Template : .aws-sam/build/template.yaml
Commands you can use next
=========================
[*] Invoke Function: sam local invoke
[*] Package: sam package --s3-bucket <yourbucket>
Running PythonPipBuilder:ResolveDependencies
Running PythonPipBuilder:CopySource
Note the first time you run this, it will download the docker image it needs for running.
Local testing
One of the best functionality that sam offers is the ability to test your code locally in a docker container. This makes quick iteration possible without having to acutally deploy on AWS lambda environment.
sam local invoke --no-event -t template.yaml
2019-03-30 21:01:43 Found credentials in shared credentials file: ~/.aws/credentials
2019-03-30 21:01:43 Invoking app.lambda_handler (python3.7)
Fetching lambci/lambda:python3.7 Docker container image....................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
2019-03-30 21:02:53 Mounting /Users/t/projects/python/playground/lambda-deep-dive/lambda-cron/hello_world as /var/task:ro inside runtime container
START RequestId: 52fdfc07-2182-154f-163f-5f0f9a621d72 Version: $LATEST
Do something intersting!
END RequestId: 52fdfc07-2182-154f-163f-5f0f9a621d72
REPORT RequestId: 52fdfc07-2182-154f-163f-5f0f9a621d72 Duration: 16.52 ms Billed Duration: 100 ms Memory Size: 128 MB Max Memory Used: 22 MB
null
Package your code
Let's create a bucket and pass it to the packaging command. Note we are passing a file called packaged.yaml which is the generated cloud formation template that we will be using in the next command to deploy it.
aws s3 mb s3://lambda-cron-bucket
sam package --s3-bucket lambda-cron-bucket --output-template-file packaged.yaml
Deploy
Finally deploy your cron job. You'll need to provide a name for your stack and tell cloudformation explicitly that you are ok with the creation of IAM related resources by passing the CAPABILITY_IAM flag.
aws cloudformation deploy --template-file /absolute/path/packaged.yaml --stack-name lambda-cron --capabilities CAPABILITY_IAM
Validate
Lets take a look if our cron is executing every minute.
sam logs -n HelloWorldFunction --stack-name lambda-cron --tail
You should see something like the following:
2019-03-30 20:35:19 Found credentials in shared credentials file: ~/.aws/credentials
2019/03/31/[$LATEST]7cc1e71b6eb14c89881cb91d6c29b920 2019-03-31T01:29:12.613000 START RequestId: c2c8a898-dfab-450f-944b-d43d77741c35 Version: $LATEST
2019/03/31/[$LATEST]7cc1e71b6eb14c89881cb91d6c29b920 2019-03-31T01:29:12.618000 Do something intersting!
2019/03/31/[$LATEST]7cc1e71b6eb14c89881cb91d6c29b920 2019-03-31T01:29:12.638000 END RequestId: c2c8a898-dfab-450f-944b-d43d77741c35
2019/03/31/[$LATEST]7cc1e71b6eb14c89881cb91d6c29b920 2019-03-31T01:29:12.638000 REPORT RequestId: c2c8a898-dfab-450f-944b-d43d77741c35 Duration: 24.83 ms Billed Duration: 100 ms Memory Size: 128 MB Max Memory Used: 63 MB
2019/03/31/[$LATEST]7cc1e71b6eb14c89881cb91d6c29b920 2019-03-31T01:30:12.272000 START RequestId: 93645e34-850b-47b0-a4a3-00d10e502057 Version: $LATEST
2019/03/31/[$LATEST]7cc1e71b6eb14c89881cb91d6c29b920 2019-03-31T01:30:12.278000 Do something intersting!
2019/03/31/[$LATEST]7cc1e71b6eb14c89881cb91d6c29b920 2019-03-31T01:30:12.298000 END
...
Congratulations on your first serverless cron job!
What's with the cover picture?
Keeping with the theme of my other dev.to posts, it has nothing to do with the post. It's view of our beautiful city, Chicago, from my balcony.
Posted on March 31, 2019
Join Our Newsletter. No Spam, Only the good stuff.
Sign up to receive the latest update from our blog.
Related