TLDR

TLDR: Implementing mTLS in Ruby is a pretty straightforward process, and a dockerized local example of this, using self signed certificates, can be found at the following link: https://github.com/scoutapp/mtls_example.

Additionally, we didn’t need to make any changes to our infrastructure, except for adding the certificate and keys to the entities originating the requests from our private subnets.

See below for troubleshooting techniques which can be useful when setting up mTLS.

About mTLS

Here at Scout we recently released the ability to perform mTLS with our webhook alerting. Before jumping into how we implemented mTLS, a quick recap on the what and the why of mTLS would be beneficial.

Mutual Transport Layer Security (mTLS), unlike plain TLS, does a double ID check - both client and server need to show their certificates. Ultimately this helps validate where the traffic is originating from and can prevent man-in-the-middle attacks.

Since we are utilizing TLS, we also know that these messages are sent in an encrypted manner.

Most of our APM services are written in Ruby. As such, this article is more tailored to how we approached implementing this from a perspective of Ruby, as well as lightly going over how this played out in our systems architecture.

With that being said, most of this should be applicable to other languages and system designs.

Additionally, we will lay out some strategies for troubleshooting potential issues that may arise when setting up mTLS.

Let’s quickly take a look at what goes into performing an mTLS request, and to keep things simple we will be doing this with our own self signed certs.

Testing mTLS with self signed certificates:

This section gives a pretty high level overview of what goes into creating the root CA certificate as well as the leaf client and server certs. However, Mutual TLS Authentication De-Mystified by John Tucker gives a more in depth explanation on this subject:
https://codeburst.io/mutual-tls-authentication-mtls-de-mystified-11fa2a52e9cf. These commands can also be found in the create_certs.sh file in the repo above.

Before we create the client and server certs we will first need to create the root certificate. From this root CA certificate we can sign the leaf certificates, establishing the chain of trust.

When creating the root certificate, we are going to specify that we are creating a self signed x509 certificate (-x509), without the private key being encrypted (-nodes) valid for 365 days, with the common name (CN) / the identity of the entity being my-ca.

openssl req -new   -x509  -nodes  -days 365  -subj '/CN=my-ca'  -keyout ca.key  -out ca.crt -sha256

Before we create the server cert, we will need to create a private key as well as a Certificate Signing Request (CSR), and we are going to name the identity as ‘localhost’.

The CSR is used by the CA to confirm identity, such as the organization or domain that this certificate is being requested for. Since we are the CA in this case, I say it looks pretty pretty good/legit. Other servers that don’t contain our root CA will probably think they aren’t! More on that in a bit.

Now let’s generate the server certificate.

# Create the server private key
openssl genrsa -out server.key 2048

# Create the server certificate signing request (CSR)
openssl req -new -key server.key -subj '/CN=localhost' -out server.csr -sha256

# Use the CA key to sign the server CSR and get back the signed certificate. Localhost
# looks like a legit identity!
openssl x509 -req -in server.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out server.crt -days 365 -sha256

Now let’s generate the client cert. This is very similar to the creation of the server certs, first creating the private key then the CSR.

# Create the client private key
openssl genpkey -algorithm RSA -out client.key

# Create the client certificate signing request (CSR)
openssl req -new -key client.key -subj '/CN=client' -out client.csr -sha256

# Use the CA key to sign the client CSR and get back the signed certificate
openssl x509 -req -in client.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out client.crt -days 365 -sha256

There we have it. We have created all the certificates we need. Before moving on, there is one more thing we can do in regards to the generated client certificates and keys. We can also combine the client cert and key file into a P12 pem file, which has the benefit of only needing to pass/specify this file when making requests (instead of both the client cert and key).

# Create a combined PEM file which will contain both the client cert and key
cat client/client.crt client/client.key > client/combined.pem

Since we are using self signed certificates, we will also need to provide the CA certificate. In most cases, where the certificates are signed by a trusted CA such as Digicert, etc. we can most likely omit this as the certificate will already be preinstalled in the system’s/browser’s trust store.

Here are some examples of making this call from the client’s perspective:

curl https://mtls.example.com --cert ./client.crt --key ./client.key --cacert ./ca.crt

If we are utilizing the P12 pem file we can just do the following:

curl https://mtls.example.com --cert ./combined.pem  --cacert ./ca.crt

If the CA certificate is signed by a Trusted CA while using the P12 pem file:

curl https://mtls.example.com --cert ./combined.pem

In Ruby, making this call is pretty straightforward. On top of this, we can make this call utilizing libraries only found within the Ruby standard library, but is also supported by several Ruby HTTP libraries:

require 'net/http'
require 'net/https'
require 'uri'

client_cert_path = './client.crt'
client_key_path = './client.key'
server_ca_cert_path = './ca.crt'

server_url = URI.parse('https://localhost:443/')

client_cert = OpenSSL::X509::Certificate.new(File.read(client_cert_path))
client_key = OpenSSL::PKey::RSA.new(File.read(client_key_path))

http = Net::HTTP.new(server_url.host, server_url.port)
http.use_ssl = true
http.cert = client_cert
http.key = client_key
# In most cases, we can omit this as the signing CA cert is already in the system's trust store.
# However, since we are self-signing and it currently isn't in the trust store we need to provide it.
http.ca_file = server_ca_cert_path

request = Net::HTTP::Get.new(server_url.request_uri)

response = http.request(request)

puts "Response Code: #{response.code}"
puts "Response Body: #{response.body}"

From the server’s point of view, especially common in Rails land with the semi-exception of Passenger, most architectures utilize a reverse proxy (such as Nginx and Apache) in front of a web server (such as Unicorn and Puma). Here we can do the mTLS validation, and for this example we will use Nginx:

server {
    listen 443 ssl;

    ssl_certificate /etc/nginx/ssl/server_certs/server.crt;
    ssl_certificate_key /etc/nginx/ssl/server_certs/server.key;

    # Enable client certificate authentication
    ssl_client_certificate /etc/nginx/ssl/ca_certs/ca.crt;
    ssl_verify_client on;

    location / {
        if ($ssl_client_verify != SUCCESS) {
             return 403;
        }

        proxy_pass http://web_server:port;
         }
}

Let’s break down the parts of this which are responsible for mTLS.

    ssl_client_certificate /etc/nginx/ssl/ca_certs/ca.crt;
    ssl_verify_client on;

Which tells Nginx to enable client certificate validation as well as which certificate file Nginx will use to validate the cert:

http://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_verify_client
http://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_client_certificate

    if ($ssl_client_verify != SUCCESS) {
      return 403;
    }

This block will then return a 403 response if the client certificate is unable to be validated.

Infrastructure:

As mentioned in the TLDR, thankfully this section is pretty short. Ultimately, we did not need to make any changes to our infrastructure/networking/routing, except for adding the client cert and key to the various entities that could make outbound webhook requests.

Troubleshooting:

What can go wrong, will go wrong, and as such we need troubleshooting.

curl

The first tool in our arsenal (which is the easiest to use but quite powerful) is adding the -vvv flag to the curl command:

curl -vvv https://mtls.example.com --cert ./combined.pem

The important parts to look for are client (out) and server (in) hellos, both the certificate messages, the client and server key exchanges and finally, most importantly for mTLS, is the CERT verify from the server. Using this tool, it’s possible to figure out where in the handshake things are going errant.

s_client

The second tool in our arsenal (which gives more information) is s_client. S_client will go more low level and will give more insight into the SSL/TLS process. S_client is able to decrypt and view the values of the actual SSL handshake messages as opposed to just which handshake messages are occurring with curl.

Normally, these values/messages would be encrypted so if we were to do something like tcp dump all the traffic, and load it up in wireshark we wouldn’t be able to view the contents of the SSL handshake (without the session keys, which can be tricky to obtain).

A good combination is to use curl to get a high level overview of the handshake process, and dive into the individual parts of the handshake as needed with s_client.

openssl s_client -connect mtls.example.com:443 -key ./combined.pem -CAfile ./ca.crt

For example if we look at the screenshot below, in the certificate request message (Request Cert) sent by the server, it shows the “Acceptable client certificate CA names” that it is set up to handle. Note the CN = my-ca, this is the identity name we assigned when we created the CA certificate. Therefore, we know that the server is able to handle the cert we created.

Final thoughts

As we’ve seen, setting up mTLS is a pretty straightforward process in Ruby, and if problems arise we laid out some troubleshooting techniques that can make overcoming these hurdles easier.

That’s how we feel about our APM and Observability tools here at Scout. We help you uncover the issues quicker so you can get back to building. Talk to one of our team members today and understand how we can help you save time, energy and prevent future issues: support@scoutapm.com.