Working on the fintech project backend I decided to use Django Q package for scheduling and executing different tasks in the background. Our CD-CI flow was based on Beanstalk new Amazon Linux 2 (AL2) machine. The trick is that Beanstalk configuration files have to include all instructions to run your code from scratch on a brand new virtual machine. Why? That's how the AWS Beanstalk works: it manages the server for you and requires precise instructions on what to install/configure in case it has to run an update or recreate/scale the virtual servers from scratch. I hoped to google some decent bash script for AL2, but there was none. So I had to dive into DevOps thing again and write my own.

Postdeploy hooks to get Django-Q cluster running on Beanstalk's AL2

Note #1: This guide is for Amazon Linux 2 EC2 instances which were rolled out for Beanstalk python platform in April 2020. If you're looking for a way to set up Django Q cluster on an older version (Amazon Linux), here is the working gist for it.

Note #2: If you're not familiar with Django Q cluster (celery based task manager) I strongly recommend reading the docs. Besides Beanstalk environment Django Q requires some message broker to schedule the tasks, so you'll need to set up a separate Redis server or Amazon SQS and connect it to your security group.

I assume you have your project configured to be deployed to Beanstalk. AWS Beanstalk allows you to place a custom number of bash scripts with .sh file extension into the following folder .platform/hooks/postdeploy in your root project directory. For convenience, I split it into 4 steps. Here's the full gist.

01_set_env.sh Grab the environmental variables and enable amazon-linux-extras

AL2 provides access to environmental variables we set in the Beanstalk dashboard through a specific file. Here we're grabbing them for further use. AL2 doesn't include supervisor. That's why we're also telling Beanstalk to enable extra extensions for us.

02_django_migrate.sh Just Django routine commands

This doesn't have much to do with Django Q cluster, but as part of Django app deployment process these steps are necessary.

03_django_q.sh Django Q supervisor configuration

Now we got Django up and running, but we still need to launch Django Q cluster and make sure it will be alive even if the process crashes. Here we're preparing a configuration block which tells the supervisor (Linux process control system) where to find and how to launch Django Q cluster.

04_supervisor_init.sh Supervisor daemon configuration

The last but not least: we need to take care about the supervisor itself and add config for its daemon. Thus, we make sure that supervisor will be loaded after EC2 instance reboot. This is the last bash script file, but pay attention to this line: sudo cp /var/app/current/supervisord.sample /etc/init.d/supervisord

/var/app/current/ - that's the default path to your project. In my root directory I placed a supervisord sample config file (by Dan MacKinlay). You can place it wherever you want, just don't forget to change the path in the bash script to it.

That's all, you're good to go. This guide doesn't cover full CI-CD flow with Django app and AWS, but I might be covering it in the next article. Stay tuned!

This article was originally published in my blog post