Application hanged

I’ll start by describing a situation that prompted me to explore topic of timeouts. I worked with a typical PHP and MySQL application. The server receives a HTTP request, passes it to PHP using FastCGI interface, PHP establishes a connection to MySQL database, and application then executes queries over the connection.

Client
| (HTTPS:443)
Server
| (FastCGI:9000)
PHP-FPM
| (mysqli:3306)
MySQL

The problem I encountered was related to communication between PHP and MYSQL. The application was hanging until server timed out the request. An investigation revealed a hanging query and MySQL, which caused queries on a certain table to hang indefinitely.

Problems

Apart from the obvious problem that the application was hanging, the available processes can quickly be used up. A malicious actor can exploit this in a denial-of-service attack. Also, small issues, such as a single endpoint failing, can quickly snowball into all endpoints failing.

It may also be hard to log such issues, especially if the request indeed runs indefinitely. In such cases, the only indication of a problem would be excessive number of active processes in PHP-FPM.

Timeouts

Timeouts are a way to mitigate this problems and can be configured in several layers, and to achieve optimal results certainly should be done so. The layers we’re going to look at are server, application and database. Let’s look at what each level has to provide. I will be looking at Apache and Nginx web servers, PHP and MySQL.

Web server

Web servers are the first layer where the request arrives. I tested locally with official Apache PHP docker image, and official Nginx and PHP-FPM images.

The first curiosity was Apache timeouts, which didn’t appear to have any effect whatsoever. Even with Timeout and RequestReadTimeout set to low values, sleep(PHP_MAX_INT); would run obliviously for hours, until I finally closed the browser tab.

Nginx fared better – the request was terminated when fastcgi_read_timeout was exceeded. When there was no output at the time of termination, Nginx served a static 504 error page. I didn’t find a way to gracefully shut down the PHP application however, as there was no SIGTERM signal, and shutdown function wasn’t triggered. When I increased process_control_timeout setting in PHP-FPM, then the PHP application continued as normal, and connection_aborted function didn’t report anything unusual. Therefore, I would recommend to use web server timeout as a last resort.

PHP

An obvious solution to timing out the process is by setting the max_execution_time value in PHP configuration. What is not so obvious is that max_execution_time appears to ignore time spent on IO, such as running database queries. This is what a typical PHP application spends majority of it’s time on. Thus, it seems that it is only useful for catching problems like infinite loops. So something to think about, but not as useful as it appears.

The set_time_limit() function and the configuration directive max_execution_time only affect the execution time of the script itself. Any time spent on activity that happens outside the execution of the script such as system calls using system(), stream operations, database queries, etc. is not included when determining the maximum time that the script has been running. This is not true on Windows where the measured time is real.

Source: PHP: set_time_limit

A workaround involves using pcntl extension, which is not supported in many environments, such as mod_php module for Apache, because the PHP application shouldn’t be fiddling with processes in server environment.

The workaround seems fairly innocent, and is possible to use in PHP-FPM, so I will take a look at it anyway:

<?php

# https://stackoverflow.com/questions/7493676/detecting-a-timeout-for-a-block-of-code-in-php/7493838

# fork the process

$pid = pcntl_fork();

if ($pid === 0) {
    # application code

    # cleanup on SIGALRM
    pcntl_async_signals(true);
    pcntl_signal(SIGALRM, function() {
        sleep(60); # simulation of cleanup work
        exit;
    });

    # simulation of long running process
    sleep(60);
} else {
    # timeout process

    # timing variables 
    $softTimeout = 5; //s
    $hardTimeout = $softTimeout + 5; //s
    $interval = 16000; // μs
    $start = microtime(true);

    pcntl_async_signals(true);

    $handler = function($signo) use ($hardTimeout, $interval, $pid, $start) {
        posix_kill($pid, $signo);
        do {
            usleep($interval);
            $result = pcntl_wait($status, WNOHANG|WUNTRACED);
            if ((microtime(true) - $start) >= $hardTimeout) {
                # Immediate termination
                posix_kill($pid, SIGKILL);
                pcntl_wait($status);
                break;
            }
        } while($result === 0);
    };

    # Termination signals
    pcntl_signal(SIGTERM, $handler);
    pcntl_signal(SIGINT, $handler);
    pcntl_signal(SIGQUIT, $handler);
    pcntl_signal(SIGHUP, $handler);

    # Alarm signal
    pcntl_signal(SIGALRM, $handler);

    register_shutdown_function(function() {
        pcntl_alarm(0);
    });
    pcntl_alarm($softTimeout);

    pcntl_wait($pid);
}

EDIT: Reflecting back on this, a simpler approach without forking is also possible with pcntl_alarm, and is probably a better choice in most situations:

<?php
$timeout = 1;

# cleanup on SIGALRM
pcntl_async_signals(true);
pcntl_signal(SIGALRM, function() {
    echo "interupt";
    exit;
});

register_shutdown_function(function() {
    pcntl_alarm(0);
});
pcntl_alarm($timeout);

sleep(PHP_INT_MAX);

EDIT2: Preliminary testing with PHP-FPM shows a somewhat expected caveat with this approach, which is related to PHP-FPM reusing same processes for multiple requests. pcntl-alarm will send SIGALRM to the PHP-FPM worker, which may at time of triggering be doing nothing or processing another request. So at minimum, the script should do it's best to clean up the alarm in register_shutdown_function. I've update the previous example accordingly.

EDIT3: I'd like to see that done in PHP-FPM, and to some extent it is possible with request_terminate_timeout, but AFAIK this can only be set to same value for all workers in the pool, and it kills the worker (I'd prefer trying a soft termination first and not killing the worker unless necessary).

A safer approach could be to offload work using Gearman. With such approach PHP-FPM would only pass requests to Gearman. When designing such a solution, it is important to keep in mind that Gearman workers do not benefit from persistent database connections.

Database

Communication with database is where PHP applications tend to spend most of their time. It would be only natural to establish limits to database operations.

In my case, the application specified MYSQLI_OPT_CONNECT_TIMEOUT using mysqli_options. But why didn’t that stop the long running query? PHP documentation comment section provided a clue. Turns out that connection timeout only times out the phase where PHP establishes connection to MySQL server and it doesn’t affect the queries at all. In fact, with persistent database connections, the database connections are shared between requests, and can therefore be rather long lived. PHP has another option for limiting query execution times, MYSQLI_OPT_READ_TIMEOUT, but it does not appear in the documentation. Digging in PHP source, this is first mentioned in a non-released NEWS file for PHP 7.1 RC, but the constant doesn’t seem to make appearance before PHP 7.2 RC. And many users report success in earlier PHP versions by manually declaring the constant.

When testing this option, the behavior appears to be as follows:

• If the query execution interval is below timeout, nothing happens.
• When timeout is exceeded the connection is dropped (2006 MySQL server has gone away).

There seems to be more though. mysqlnd has an INI setting mysqlnd.net_read_timeout, which reads to do the same thing, but the documentation also notes MYSQL_OPT_READ_TIMEOUT only works for TCP/IP connections, and prior to MySQL 5.1.2, only for Windows.

Testing with mysqli revealed that both ways work identically, sockets included. However, the option can only be passed with mysqli_options, whereas the INI setting can be also used with PDO.

Summary

Control over timeouts in PHP appears rather complicated, especially when compared to asynchronous languages like Node JS, but can certainly be done right.

Given the issues with timeouts and lack of pcntl support in Apache, Nginx + PHP-FPM combo seems like a more solid option.

The timeout strategies in PHP need a careful thought. An architecture using Gearman certainly looks promising, and I have used it with success in the past. However, I will probably give process forking pcntl_alarm with PHP-FPM a try.

The read timeouts in MySQL is a must combined with connect timeout to avoid available worker pool exhaustion in case of a hanging query.