Greg Nokes
Posted on April 8, 2020
I was doing some performance testing on my blog, and while I was impressed, I felt like I could wring some more speed out of it. I decided to switch from Heroku’s Static Buildpack to a simple Nginx webserver using Heroku’s nice Nginx Buildpack. In theory this should be a little quicker and lighter.
The first step was to get Nginx set up on the Heroku App. I had the static buildpack already, so I simply had to remove it and add the Nginx Buildpack.
The next step is to modify the Procfile to run Ngnix in Solo mode:
web: bin/start-nginx-solo
Then I created a config directory and put my nginx.conf.erb file in it. The Nginx buildpack will grab that file, process it, and then use it to run Nginx on the dyno.
I found this config to work well:
daemon off;
# Heroku dynos have at least 4 cores
worker_processes <%= ENV['NGINX_WORKERS'] || 4 %>;
events {
use epoll;
accept_mutex on;
worker_connections <%= ENV['NGINX_WORKER_CONNECTIONS'] || 1024 %>;
}
http {
gzip on;
gzip_comp_level 2;
gzip_min_length 512;
server_tokens off;
log_format main '$time_iso8601 - $status $request - client IP: $http_x_forwarded_for - <%= ENV['DYNO'] %> to $upstream_addr - upstream status: $upstream_status, upstream_response_time $upstream_response_time, request_time $request_time';
access_log /dev/stdout main;
error_log /dev/stdout notice;
log_not_found on;
include mime.types;
default_type application/octet-stream;
sendfile on;
# Must read the body in 5 seconds.
client_body_timeout <%= ENV['NGINX_CLIENT_BODY_TIMEOUT'] || 5 %>;
server {
listen <%= ENV["PORT"] %>;
error_page 404 /404.html;
error_page 403 /403.html;
port_in_redirect off;
location / {
root _site/;
}
}
}
The important parts are the port_in_redirect off; and the root _site/; directives. The port_in_redirect tells Nginx to not embed it's port into any redirects, and root tells Nginx where the static files are.
Well, is it fast?
I ran ab on the site before and after, and there is a large improvement. Larger than I expected.
The old configuration had a perc99 under load of 1.7 seconds per request, and the new has a perc99 of 0.6 seconds. More than a second per request faster!
Old Config
Concurrency Level: 100
Time taken for tests: 7.284 seconds
Complete requests: 5000
Failed requests: 0
Total transferred: 6922000 bytes
HTML transferred: 6722000 bytes
Requests per second: 137.28 [#/sec] (mean)
Time per request: 728.426 [ms] (mean)
Time per request: 7.284 [ms] (mean, across all concurrent requests)
Transfer rate: 928.00 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 307 449 233.8 380 1612
Processing: 103 204 84.1 212 1026
Waiting: 103 201 83.5 211 1026
Total: 424 653 247.2 591 1939
Percentage of the requests served within a certain time (ms)
50% 591
66% 627
75% 658
80% 670
90% 822
95% 1084
98% 1738
99% 1769
100% 1939 (longest request)
New Config
Concurrency Level: 100
Time taken for tests: 26.680 seconds
Complete requests: 5000
Failed requests: 0
Total transferred: 34870000 bytes
HTML transferred: 33650000 bytes
Requests per second: 187.41 [#/sec] (mean)
Time per request: 533.592 [ms] (mean)
Time per request: 5.336 [ms] (mean, across all concurrent requests)
Transfer rate: 1276.36 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 297 385 42.1 378 620
Processing: 102 136 33.6 122 641
Waiting: 102 133 30.5 120 641
Total: 424 520 51.2 506 986
Percentage of the requests served within a certain time (ms)
50% 506
66% 524
75% 536
80% 544
90% 577
95% 643
98% 685
99% 696
100% 986 (longest request)
But what about SSL?
One of the things that I really liked about the Static Buildpack is that it is based on Rack, and I could use the rack-ssl-enforcer gem to make sure that all requests were directed to the encrypted version of the site. I did some investigation and found the following stanza in the server block would work with the Heroku router:
if ($http_x_forwarded_proto != "https") {
return 301 https://$host$request_uri;
}
A redirect would be preferable, however since we are behind the Heroku Router, a simple redirect seems to enter into an infinite redirect loop. The if loop slows down processing considerably.
With SSL Redirect
Concurrency Level: 100
Time taken for tests: 26.618 seconds
Complete requests: 5000
Failed requests: 0
Total transferred: 34870000 bytes
HTML transferred: 33650000 bytes
Requests per second: 187.85 [#/sec] (mean)
Time per request: 532.352 [ms] (mean)
Time per request: 5.324 [ms] (mean, across all concurrent requests)
Transfer rate: 1279.33 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 305 385 40.6 381 1618
Processing: 101 134 31.4 123 471
Waiting: 101 131 28.0 120 469
Total: 422 519 48.5 510 1740
Percentage of the requests served within a certain time (ms)
50% 510
66% 523
75% 533
80% 544
90% 569
95% 596
98% 630
99% 704
100% 1740 (longest request)
Still respectable, but you can see the impact of using the if statement in the Nginx config. The perc99 is only a few milliseconds slower with the SSL redirect, but milliseconds count!
I will continue to research and see if I can remove that if statement, and find a cleaner way to implement this.
One of the other uses for Nginx on Heroku is a on Dyno proxy and static file server. This experiment shows how efficient Nginx can be at hosting the static assets of a complex web application, like Ruby. Using Nginx as a front end to proxy requests, and serve static files offloads traffic from the language specific app server, and can improve overall performance of a web application.
Update 10/10/19
I quickly found out that I still needed the static buildpack in my development and staging environments. It’s used to build out the static assets that Nginx serves, after all.
If you refer back to How I post an article you will know that I use a Heroku Pipeline to manage posting. The nice thing is that I don’t need the Static buildpack or Ruby on my production site, as the slug is promoted intact between the Staging and Production apps. So I have Ruby and the Static buildpack installed and running in staging, but only the Nginx buildpack associated with the production app.
Posted on April 8, 2020
Join Our Newsletter. No Spam, Only the good stuff.
Sign up to receive the latest update from our blog.