Robust queue job handling in Craft CMS

Craft CMS uses queue jobs for long-running tasks. Here’s how to avoid stalled or failed queue jobs by using an independent async queue runner

Queuing in line is something few people are fond of, but it allows each person to be serviced in order, in an efficient and fair manner.

Craft CMS also uses a queue, too. Any lengthy task is packaged off into a job , and jobs are put into a queue to be executed in order.

In computer science terms, this is a First In, First Out (FIFO) data structure that services things in what we’d consider to be a natural order.

Here are some lengthy tasks that end up as jobs in the Craft CMS queue:

Updating entry drafts and revisions
Deleting stale template caches
Generating pending image transforms
Re-saving Elements
Updating search indexes
Plugin created queue jobs

The queue exists to allow these often lengthy jobs to run in the background without blocking access to the Craft CMS Control Panel (CP) while they execute.

See How They Run

So now what we know what type of jobs will end up in the queue, let’s have a look at how these jobs are run. This ends up being a crucially important bit of information to understand about queue jobs.

Given that Craft CMS is built on top of the Yii2 framework, it makes sense that it leverages the existing Yii2-queue by layering a QueueInterface on top of it.

It does this so that it can provide a nice friendly status UI in the CP. The default Craft setup uses a database (Db) driver for queue jobs, storing them in the queue table.

Although there are other queue drivers available, you’re unlikely to see any gains from using anything other than the Craft CMS Db driver. And you’d lose the aforementioned status UI in the CP.

But how do the queue jobs run? Here’s how:

When someone access the CP
Via web request

Herein lies the rub with the default queue implementation in Craft CMS. No queue jobs will run at all unless someone logs into the CP, and even then, they are run via web request.

Here’s an excerpt from the Craft CMS CP.js JavaScript that kicks off the the web-based queue running:


runQueue: function() {
    if (!this.enableQueue) {
        return;
    }

    if (Craft.runQueueAutomatically) {
        Craft.queueActionRequest('queue/run', $.proxy(function(response, textStatus) {
            if (textStatus === 'success') {
                this.trackJobProgress(false, true);
            }
        }, this));
    }
    else {
        this.trackJobProgress(false, true);
    }
},

And here’s the queue/run controller method that ends up getting executed via AJAX:


/**
 * Runs any waiting jobs.
 *
 * @return Response
 */
public function actionRun(): Response
{
    // Prep the response
    $response = Craft::$app->getResponse();
    $response->content = '1';
    // Make sure Craft is configured to run queues over the web
    if (!Craft::$app->getConfig()->getGeneral()->runQueueAutomatically) {
        return $response;
    }
    // Make sure the queue isn't already running, and there are waiting jobs
    $queue = Craft::$app->getQueue();
    if ($queue->getHasReservedJobs() || !$queue->getHasWaitingJobs()) {
        return $response;
    }
    // Attempt to close the connection if this is an Ajax request
    if (Craft::$app->getRequest()->getIsAjax()) {
        $response->sendAndClose();
    }
    // Run the queue
    App::maxPowerCaptain();
    $queue->run();
    return $response;
}

They are doing what they can to make sure the web-based queue jobs will run smoothly, but there’s only so much that can be done. Brandon Kelly from Pixel & Tonic even noted “Craft 2, and Craft3 by default, basically have a poor-man’s queue”.

Web requests are really meant for relatively short, atomic operations like loading a web page. For this reason, both web servers and PHP processes have timeouts to prevent them from taking too long.

This can result in bad things happening:

Queue jobs stack up until someone logs into the CP
Queue jobs can fail or get stuck if they take too long and time out
Queue jobs can fail or get stuck if they run out of memory
Lengthy queue jobs can impact CP or frontend performance as they run

Fortunately, we can alleviate all of these problems, and it’s not hard to do!

If you do have tasks that have failed due to errors, check out the Zen and the Art of Craft CMS Log File Reading article to learn how to debug it.

Queue Running Solutions

Fortunately Yii2, and by extension Craft CMS, comes with a robust set of command line tools that I detail in the Exploring the Craft CMS 3 Console Command Line Interface (CLI) article.

The queue command gives us the ability to shift our queue jobs into high gear by running them directly. PHP code that is run via the console command line interface (CLI) uses a separate php.ini file than the web-oriented php-fpm. Here’s an example from my server:


/etc/php/7.1/fpm/php.ini
/etc/php/7.1/cli/php.ini

The reason this is important is that while the web-based php-fpm might have reasonable settings for memory_limit and max_execution_time, the CLI-based PHP will typically have no memory_limit and no max_execution_time.

This is because web processes are externally triggered and therefore untrusted, whereas things running via the CLI is internal and trusted.

So running our queue jobs via PHP CLI gives us the following benefits:

Can be run any time, not just when someone accesses the CP
Won’t be terminated due to lengthy execution time
Are unlikely to run out of memory

The queue console commands we use are:

queue/run — Runs all jobs in the queue, then exits
queue/listen — Perpetually runs a process that listens for new jobs added to the queue and runs them

There are several ways we can run these commands, discussed in the Yii2 Worker Starting Control article, and we’ll go into detail on them here.

Solution #1: Async Queue Plugin

If you want a no fuss, no muss solution that just involves installing a plugin, check out Oliver Stark’s Async Queue plugin. From the plugin’s description:

[Async Queue plugin] replaces Craft’s default queue handler and moves queue execution to a non-blocking background process. The command craft queue/run gets executed right after you push a Job to the queue.

If you want a simple solution that doesn’t require doing anything more than installing a plugin, go this route. You’ll end up with a more robust solution for running your queue jobs.

That’s it, you’re done!

Solution #2: Forge Daemons

If you use the wonderful Laravel Forge to provision your servers (discussed in detail in the How Agencies & Freelancers Should Do Web Hosting article), Forge provides a nice GUI interface to what it calls Daemons.

Go to your Server, then click on Daemons, and you’ll see a GUI that looks like this:

Here’s the command we’ve added for the devMode.fm website:


/usr/bin/nice -n 10 /usr/bin/php /home/forge/devmode.fm/craft queue/listen --verbose

Let’s break this down:

/usr/bin/nice -n 10 — this uses the Unix nice command to run the process using a lower priority, so it won’t interfere with the CP or frontend
/usr/bin/php /home/forge/devmode.fm/craft queue/listen --verbose — this starts a PHP CLI process to listen for any queue jobs, and run them

You should swap in your own path to the craft CLI executable; ours is /home/forge/devmode.fm/craft

We tell it that we want to run this as the user forge and we want 2 of these processes running. Click Start Daemon and you’ll be greeted with:

Then ensure that you’ve set the runQueueAutomatically Craft CMS general.php config setting to false for your live production environment:


    // Live (production) environment
    'live' => [
        // Craft defined config settings
        'runQueueAutomatically' => false,
    ],

This makes sure that Craft will no longer try to run its queue via web requests.

That’s it, you’re done!

The following is an entirely optional peek under the hood at how this works with Laravel Forge. Feel free to skip it if you’re not interested in the gory details.

What Forge calls Daemons are actually commands run via the Supervisor command. When we created the above daemon in the Forge GUI, it created a file in /etc/supervisor/conf.d:


forge@nys-production /etc/supervisor/conf.d $ ls -al
-rw-r--r-- 1 root root 293 Aug 3 14:55 daemon-157557.conf

The contents of the file look like this:


[program:daemon-157557]
command=/usr/bin/nice -n 10 /usr/bin/php /home/forge/devmode.fm/craft queue/listen --verbose

process_name=%(program_name)s_%(process_num)02d
autostart=true
autorestart=true
user=forge
numprocs=2
redirect_stderr=true
stdout_logfile=/home/forge/.forge/daemon-157557.log

And indeed, this is exactly one of the methods recommended in the in the Yii2 Worker Starting Control article. Huzzah!

Solution #3 Using Heroku

If you’re using Heroku for your hosting needs, you can use Worker Dynos to run your queue/listen command:


./craft queue/listen --verbose

Then ensure that you’ve set the runQueueAutomatically Craft CMS general.php config setting to false for your live production environment:


    // Live (production) environment
    'live' => [
        // Craft defined config settings
        'runQueueAutomatically' => false,
    ],

This makes sure that Craft will no longer try to run its queue via web requests.

That’s it, you’re done!

Solution #4: Utilizing systemd

If you’re using a Linux server, but you’re not using Forge or Heroku, or perhaps you just want to configure things yourself, we can accomplish the same thing using systemd.

systemd is a way to start, stop, and otherwise manage daemon processes that run on your server. You’re already using it whether you know it or not, for things like your web server, MySQL, and so on.

Here’s what I did for devMode.fm to use systemd to run our queue jobs. You’ll need sudo access to be able to do this, but fortunately most modern VPS hosting services give you this ability.

First I created a .service file in /etc/systemd/system/ named devmode-queue@.service via:


forge@nys-production ~ $ sudo nano /etc/systemd/system/devmode-queue@.service

Here’s what the contents of the file looks like:


[Unit]
Description=devMode.fm Queue Worker %I
After=network.target
After=mysql.service
Requires=mysql.service

[Service]
User=forge
Group=forge
ExecStart=/usr/bin/nice -n 10 /usr/bin/php /home/forge/devmode.fm/craft queue/listen --verbose
Restart=always

[Install]
WantedBy=multi-user.target

Let’s break this down:

/usr/bin/nice -n 10 — this uses the Unix nice command to run the process using a lower priority, so it won’t interfere with the CP or frontend
/usr/bin/php /home/forge/devmode.fm/craft queue/listen --verbose — this starts a PHP CLI process to listen for any queue jobs, and run them

You should swap in your own path to the craft CLI executable; ours is /home/forge/devmode.fm/craft

We tell it that we want to run this as the user forge , we want it to restart the process on failure and that it depends on both the network and mysql services being available.

The strange @ in the .service file name allows us to have multiple instances of this service running. We can then use the range syntax {1..2} to specify how many of these processes we want.

We’ve created the file, but we need to start the service via:


forge@nys-production ~ $ sudo systemctl start devmode-queue@{1..2}

And then just like our other services such as nginx and mysqld we want it to start up automatically when we restart our server:


forge@nys-production ~ $ sudo systemctl enable devmode-queue@{1..2}

Then ensure that you’ve set the runQueueAutomatically Craft CMS general.php config setting to false for your live production environment:


    // Live (production) environment
    'live' => [
        // Craft defined config settings
        'runQueueAutomatically' => false,
    ],

This makes sure that Craft will no longer try to run its queue via web requests.

That’s it, you’re done!

Now you can start, stop, enable, disable, etc. your Craft CMS queue runner just like you would any other system service. You can also monitor it via the journalctl command:


forge@nys-production ~ $ sudo journalctl -f -u devmode-queue@*.service
-- Logs begin at Sat 2019-08-03 13:23:17 EDT. --
Aug 03 14:06:04 nys-production nice[1364]: Processing element 42/47 - Neutrino: How I Learned to Stop Worrying and Love Webpack
Aug 03 14:06:04 nys-production nice[1364]: Processing element 43/47 - Google AMP: The Battle for the Internet's Soul?
Aug 03 14:06:04 nys-production nice[1364]: Processing element 44/47 - The Web of Future Past with John Allsopp
Aug 03 14:06:04 nys-production nice[1364]: Processing element 45/47 - Web Hosting with ArcusTech's Nevin Lyne
Aug 03 14:06:04 nys-production nice[1364]: Processing element 46/47 - Tailwind CSS utility-first CSS with Adam Wathan
Aug 03 14:06:04 nys-production nice[1364]: Processing element 47/47 - Talking Craft CMS 3 RC1!
Aug 03 14:06:04 nys-production nice[1364]: 2019-08-03 14:06:04 [1427] Generating episodes sitemap (attempt: 1) - Done (1.377 s)
Aug 04 09:25:25 nys-production nice[1364]: 2019-08-04 09:25:25 [1429] Generating calendar sitemap (attempt: 1) - Started
Aug 04 09:25:25 nys-production nice[1364]: Processing element 1/1 - Calendar of Upcoming Episodes
Aug 04 09:25:25 nys-production nice[1364]: 2019-08-04 09:25:25 [1429] Generating calendar sitemap (attempt: 1) - Done (0.174 s)

The above command will tail the systemd logs, only showing messages from our devmode-queue service.

You can learn more about using journalctl in the How To Use Journalctl to View and Manipulate Systemd Logs article.

Getting Dotenvious

This is an entirely optional section you can skip if you’re not interested in learning how to not use .env files in production, as recommended by the phpdotenv package authors:

phpdotenv is made for development environments, and generally should not be used in production. In production, the actual environment variables should be set so that there is no overhead of loading the .env file on each request. This can be achieved via an automated deployment process with tools like Vagrant, chef, or Puppet, or can be set manually with cloud hosts like Pagodabox and Heroku.

Additionally some setups have environmental variables introduced by alternate means. I made a Dotenvy package that makes it easy to generate .env key/value variable pairs as Apache, Nginx, and shell equivalents.

In this case, we’ll need to modified the command we run slightly, so that we’re defining our environmental variables in the shell before running Craft via the CLI.

All we need to do is change our command to this:


/bin/bash -c ". /home/forge/devmode.fm/.env_cli.txt && /usr/bin/nice -n 10 /usr/bin/php /home/forge/devmode.fm/craft queue/listen --verbose"

All we’re doing is executing the .env_cli.txt file (created by Dotenvy) that contains our environmental variables we export to the shell environment:


forge@nys-production ~/devmode.fm (master) $ cat .env_cli.txt
# CLI (bash) .env variables
# Paste these inside your .bashrc file in your $HOME directory:
export ENVIRONMENT="live"
export SECURITY_KEY="XXXXX"
export DB_DRIVER="pgsql"
export DB_SERVER="localhost"
export DB_USER="devmode"
export DB_PASSWORD="XXXXX"
export DB_DATABASE="devmode"
export DB_SCHEMA="public"
export DB_TABLE_PREFIX=""
export DB_PORT="5432"
export SITE_URL="https://devmode.fm/"
export BASE_PATH="/home/forge/devmode.fm/web/"
export BASE_URL="https://devmode.fm/"
export REDIS_HOSTNAME="localhost"
export REDIS_PORT="6379"
export REDIS_DEFAULT_DB="0"
export REDIS_CRAFT_DB="3"
export PUBLIC_PATH="/dist/"
export DEVSERVER_PUBLIC="http://192.168.10.10:8080"
export DEVSERVER_HOST="0.0.0.0"
export DEVSERVER_POLL="1"
export DEVSERVER_PORT="8080"
export DEVSERVER_HTTPS="0"

The rest is the same as the commands we’ve discussed above for running queue/listen, and then we wrap it all in /bin/sh -c so it appears as a single command that can be executed.

Deeply Queued

Here’s some queue job trivia that you may find interesting.

We mentioned earlier that the only way queue jobs are run normally is if you visit the CP. This is mostly true.

There’s actually a case where queue jobs can be run via frontend page load. If you have generateTransformsBeforePageLoad set to false (which is the default), and there are pending transforms that need to be generated.

What happens is the Assets::getAssetUrl() method pushes the image transform queue job, which calls Queue::push():


    /**
     * @inheritdoc
     */
    public function push($job)
    {
        // Capture the description so pushMessage() can access it
        if ($job instanceof JobInterface) {
            $this->_jobDescription = $job->getDescription();
        } else {
            $this->_jobDescription = null;
        }

        if (($id = parent::push($job)) === null) {
            return null;
        }

        // Have the response kick off a new queue runner if this is a site request
        if (Craft::$app->getConfig()->getGeneral()->runQueueAutomatically && !$this->_listeningForResponse) {
            $request = Craft::$app->getRequest();
            if ($request->getIsSiteRequest() && !$request->getIsAjax()) {
                Craft::$app->getResponse()->on(Response::EVENT_AFTER_PREPARE, [$this, 'handleResponse']);
                $this->_listeningForResponse = true;
            }
        }

        return $id;
    }

So if this is a site (frontend) request, and it’s not an AJAX request, Craft will add a call to Queue::handleResponse() that gets fired when the Response is about to be sent back to the client.

In this case, Craft will inject some JavaScript into the frontend request that pings the queue/run controller via AJAX:


    /**
     * Figure out how to initiate a new worker.
     */
    public function handleResponse()
    {
        // Prevent this from getting called twice
        $response = Craft::$app->getResponse();
        $response->off(Response::EVENT_AFTER_PREPARE, [$this, 'handleResponse']);

        // Ignore if any jobs are currently reserved
        if ($this->getHasReservedJobs()) {
            return;
        }

        // Ignore if this isn't an HTML/XHTML response
        if (!in_array($response->getContentType(), ['text/html', 'application/xhtml+xml'], true)) {
            return;
        }

        // Include JS that tells the browser to fire an Ajax request to kick off a new queue runner
        // (Ajax request code adapted from http://www.quirksmode.org/js/xmlhttp.html - thanks ppk!)
        $url = Json::encode(UrlHelper::actionUrl('queue/run'));
        $js = <<<EOD
<script type="text/javascript">
/*<![CDATA[*/
(function(){
    var XMLHttpFactories = [
        function () {return new XMLHttpRequest()},
        function () {return new ActiveXObject("Msxml2.XMLHTTP")},
        function () {return new ActiveXObject("Msxml3.XMLHTTP")},
        function () {return new ActiveXObject("Microsoft.XMLHTTP")}
    ];
    var req = false;
    for (var i = 0; i < XMLHttpFactories.length; i++) {
        try {
            req = XMLHttpFactories[i]();
        }
        catch (e) {
            continue;
        }
        break;
    }
    if (!req) return;
    req.open('GET', $url, true);
    if (req.readyState == 4) return;
    req.send();
})();
/*]]>*/
</script>
EOD;

        if ($response->content === null) {
            $response->content = $js;
        } else {
            $response->content .= $js;
        }
    }

While technically this could happen for any queue job that is Queue::pushed()‘d in the queue during a frontend request, asset transforms are the only case I’m aware of that this happens in practice.

For this reason, if you’re posting any XHR requests to Craft controllers or API endpoints, make sure you add the header:


'X-Requested-With': 'XMLHttpRequest'

Craft looks for this header to determine whether the request is AJAX or not, and will not attempt to attach its queue runner JavaScript to the request if the request is AJAX.

This header is set automatically by jQuery for $.ajax() requests, but it is not set automatically by popular libraries such as Axios.

Wrapping Up

So which method do I use? If I’m spinning up a VPS with Forge (which I normally am), then I’ll use Solution #2: Forge Daemons. If it’s not a Forge box, but I do have sudo access, then I’ll use Solution #3: Utilizing systemd.

I use Solution #1: Async Queue Plugin on client sites where neither of the above are the case. Oliver’s plugin is great, and much better than the default way that queue jobs are run, but I like leveraging existing systems if possible, and having nice control over the process priority is, well, nice.

Replacing the default web-based queue job system in Craft CMS with a more robust CLI-based system will pay dividends.

Your queue jobs will run more smoothly, fail less often, and have less of an impact on the CP/frontend.

It’s worth it. Do it.

Happy queuing!

Blog

Robust queue job handling in Craft CMS

Andrew Welch

Robust queue job handling in Craft CMS

Craft CMS uses queue jobs for long-running tasks. Here’s how to avoid stalled or failed queue jobs by using an independent async queue runner

See How They Run

Queue Running Solutions

Solution #1: Async Queue Plugin

Solution #2: Forge Daemons

Solution #3 Using Heroku

Solution #4: Utilizing systemd

Getting Dotenvious

Deeply Queued

Wrapping Up

Further Reading

Join Our Newsletter. No Spam, Only the good stuff.

Related

Robust queue job handling in Craft CMS

Andrew Welch

Robust queue job handling in Craft CMS

Craft CMS uses queue jobs for long-run­ning tasks. Here’s how to avoid stalled or failed queue jobs by using an inde­pen­dent async queue runner

See How They Run

Queue Run­ning Solutions

Solu­tion #1: Async Queue Plugin

Solu­tion #2: Forge Daemons

Solu­tion #3 Using Heroku

Solu­tion #4: Uti­liz­ing systemd

Get­ting Dotenvious

Deeply Queued

Wrap­ping Up

Further Reading

Join Our Newsletter. No Spam, Only the good stuff.

Related

Craft CMS uses queue jobs for long-running tasks. Here’s how to avoid stalled or failed queue jobs by using an independent async queue runner

Queue Running Solutions

Solution #1: Async Queue Plugin

Solution #2: Forge Daemons

Solution #3 Using Heroku

Solution #4: Utilizing systemd

Getting Dotenvious

Wrapping Up