Chromium Spelunking: Connecting to Proxies

Having successfully fetched a URL with the Chromium network stack, it's time to return to the reason I began this journey: how does Chromium connect to proxies?

Proxy Review

First, a quick review of HTTP proxying.

In the beginning, there was a simple protocol to proxy an HTTP transaction: connect to a proxy and, instead of sending just a path in the request line, send the entire URL. For example:

GET http://httbin.org/uuid HTTP/1.1

The proxy server then makes an outbound connection to httpbin.org, sends GET /uuid HTTP/1.1 with the headers supplied by the client, and then relays the response back to the client.

This sort of proxy exposes all of the details of the transaction to the proxy (or to anyone, if not using TLS for the connection to the proxy). If the origin URL has scheme https, the proxy will use TLS to connect to the origin, but will still handle the transaction content in cleartext. This has some obvious downsides, but an advantage is that the proxy can cache responses, saving bandwidth. This was a popular use of proxies in the aughts, but is far less common now that bandwidth is cheap and privacy is important.

An alternative method, CONNECT, was defined about 25 years ago. It creates a "tunnel" through the proxy which can carry arbitrary data to and from another host. The request looks like

CONNECT httpbin.org:443 HTTP/1.1

The proxy server makes a TCP connection to the given host and port, sends back a response header, and then forwards bytes bidirectionally without further analyzing them. In fact, that protocol doesn't have to be HTTP (but it must use TCP, which will become important later).

To test proxying,I'm using tinyproxy, running a very simple config on port 8080. This supports SPDY (HTTP/2), which is a complication I don't really want to consider at this point, but the analysis ends up quite similar to HTTP/1.

Configuring a Proxy

So, how does the network service decide to use a proxy for a request? It's complicated.

Let's start at the bottom. net::ProxyServer defines an actual server to connect to. It has a scheme, host, and port. One of those schemes is "DIRECT" which means do not use a proxy. The others include HTTP, HTTPS, SOCKS, and QUIC, and just describe how to connect to the proxy server.

net::ProxyList represents a list of ProxyServer instances and handles fallback from one to the next. This allows, for example, an enterprise to configure traffic to go via a local proxy but fall back to DIRECT when that proxy is down, such as when using a caching proxy.

Scoping out another level, net::ProxyConfig represents a configuration for when to use which proxies. This can be manually configured or can refer to a PAC script that defines the configuration.

The ProxyConfig comes from a net::ProxyConfigService, which is used by a net::ProxyResolutionService to determine the ProxyList to use for a given URL. There's a lot of complexity here that we won't get into, including auto-configuration of proxies and downloading and executing PAC scripts, but the end result is a ProxyList.

For churl, we want to hard-code a proxy, so I updated the ProxyConfigService I defined earlier in this series to return a config pointing to a proxy server:

class ProxyConfigServiceHardCoded : public net::ProxyConfigService {
 public:
  // ProxyConfigService implementation:
  void AddObserver(Observer* observer) override {}
  void RemoveObserver(Observer* observer) override {}
  net::ProxyConfigService::ConfigAvailability GetLatestProxyConfig(
      net::ProxyConfigWithAnnotation* config) override {
    auto traffic_annotation = kHardCodedProxyTrafficAnnotation;
    auto proxy_config = net::ProxyConfig::CreateDirect();
    auto& proxy_rules = proxy_config.proxy_rules();
    proxy_rules.ParseFromString("localhost:8080");
    *config = net::ProxyConfigWithAnnotation(proxy_config, traffic_annotation);
    return CONFIG_VALID;
  }
};

Tracing a Simple Request

OK, let's see how this works. My strategy for this sort of investigation is to add lots of debugging output -- at the beginning of each relevant function, and sometimes at key points in longer functions. Then I can follow execution within the source, comparing to the debugging output. I prefer this approach over using a debugger like GDB because I find it more efficient. Use the tools you prefer!

I'll be making a request to https://ip.cow.org using tinyproxy running on http://localhost:8080. Because the origin URL is https, this should use CONNECT.

HTTP Cache

We saw in previous posts that the URLRequest ends up using an HttpTransactionFactory::CreateTransaction to create an HttpCache::Transaction, and starting it. That class has a rather large number of states, but adding some logging in DoLoop shows the sequence of states (formatted to fit your screen):

HttpCache::Transaction -> STATE_GET_BACKEND
HttpCache::Transaction -> STATE_GET_BACKEND_COMPLETE
HttpCache::Transaction -> STATE_INIT_ENTRY
HttpCache::Transaction -> STATE_OPEN_OR_CREATE_ENTRY
HttpCache::Transaction -> STATE_OPEN_OR_CREATE_ENTRY_COMPLETE
HttpCache::Transaction -> STATE_ADD_TO_ENTRY
HttpCache::Transaction -> STATE_ADD_TO_ENTRY_COMPLETE
HttpCache::Transaction -> STATE_SEND_REQUEST
[0925/201309.849178:ERROR:churl_bin.cc(89)] OnConnected
HttpCache::Transaction -> STATE_SEND_REQUEST_COMPLETE
HttpCache::Transaction -> STATE_FINISH_HEADERS
HttpCache::Transaction -> STATE_SUCCESSFUL_SEND_REQUEST
HttpCache::Transaction -> STATE_OVERWRITE_CACHED_RESPONSE
HttpCache::Transaction -> STATE_CACHE_WRITE_RESPONSE
HttpCache::Transaction -> STATE_CACHE_WRITE_RESPONSE_COMPLETE
HttpCache::Transaction -> STATE_TRUNCATE_CACHED_DATA
HttpCache::Transaction -> STATE_TRUNCATE_CACHED_DATA_COMPLETE
HttpCache::Transaction -> STATE_PARTIAL_HEADERS_RECEIVED
HttpCache::Transaction -> STATE_FINISH_HEADERS
HttpCache::Transaction -> STATE_FINISH_HEADERS_COMPLETE
[0925/201310.106633:ERROR:churl_bin.cc(165)] OnResponseStarted
[0925/201310.106669:ERROR:churl_bin.cc(171)] Got HTTP response code 200
HttpCache::Transaction -> STATE_NETWORK_READ_CACHE_WRITE
HttpCache::Transaction -> STATE_NETWORK_READ_CACHE_WRITE_COMPLETE
HttpCache::Transaction -> STATE_NETWORK_READ_CACHE_WRITE
HttpCache::Transaction -> STATE_NETWORK_READ_CACHE_WRITE_COMPLETE

So this provides a map to focus on what's going on in this particular request. Looking at the code, the backend- and entry-related items are just looking for values in the cache, of which there are none. The OnConnected callback in the URLRequest delegate occurs during the STATE_SEND_REQUEST segment. So let's dig in there.

HttpCache::Transaction::DoStartRequest calls cache_->network_layer_->CreateTransaction(..). That network_layer_ is another HttpTransactionFactory. The code-search for that class shows a few classes that extend it, and a little debug printing reveals that this layer is an HttpNetworkLayer, and CreateTransaction returns an HttpNetworkTransaction.

HTTP Network Layer

This class also has a large collection of states, but the same trick helps us find the right one:

HttpNetworkTransaction -> STATE_NOTIFY_BEFORE_CREATE_STREAM
HttpNetworkTransaction -> STATE_CREATE_STREAM        
HttpNetworkTransaction -> STATE_CREATE_STREAM_COMPLETE
HttpNetworkTransaction -> STATE_CONNECTED_CALLBACK     
[0925/205926.726872:ERROR:churl_bin.cc(89)] OnConnected
HttpNetworkTransaction -> STATE_CONNECTED_CALLBACK_COMPLETE
HttpNetworkTransaction -> STATE_INIT_STREAM
HttpNetworkTransaction -> STATE_INIT_STREAM_COMPLETE
HttpNetworkTransaction -> STATE_GENERATE_PROXY_AUTH_TOKEN
HttpNetworkTransaction -> STATE_GENERATE_PROXY_AUTH_TOKEN_COMPLETE
HttpNetworkTransaction -> STATE_GENERATE_SERVER_AUTH_TOKEN
HttpNetworkTransaction -> STATE_GENERATE_SERVER_AUTH_TOKEN_COMPLETE
HttpNetworkTransaction -> STATE_INIT_REQUEST_BODY
HttpNetworkTransaction -> STATE_INIT_REQUEST_BODY_COMPLETE
HttpNetworkTransaction -> STATE_BUILD_REQUEST 
HttpNetworkTransaction -> STATE_BUILD_REQUEST_COMPLETE
HttpNetworkTransaction -> STATE_SEND_REQUEST
HttpNetworkTransaction -> STATE_SEND_REQUEST_COMPLETE
HttpNetworkTransaction -> STATE_READ_HEADERS
HttpNetworkTransaction -> STATE_READ_HEADERS_COMPLETE
[0925/225438.616076:ERROR:churl_bin.cc(165)] OnResponseStarted
[0925/225438.616113:ERROR:churl_bin.cc(171)] Got HTTP response code 200
HttpNetworkTransaction -> STATE_READ_BODY
HttpNetworkTransaction -> STATE_READ_BODY_COMPLETE
HttpNetworkTransaction -> STATE_READ_BODY
HttpNetworkTransaction -> STATE_READ_BODY_COMPLETE

The STATE_CONNECTED_CALLBACK is just calling the OnConnected callback. The more interesting bit is in the states before that, STATE_CREATE_STREAM(_COMPLETE). The important bit of this state seems to be calling HttpStreamFactory::RequestStream. This calls back through a delegate method named OnStreamReady, and in this case HttpStreamFactory itself is the delegate. Adding a debug print in its OnStreamReady method shows that used_proxy_info includes localhost:8080, so that stream involves the proxy.

HTTP Stream

The HttpStreamFactory class has three nested helper classes: JobFactory, Job, and JobController. The HttpStreamFactory::JobFactory class is trivial: it has a CreateJob method that calls the Job constructor. I suspect this was done as a kind of dependency injection to support testing the JobController.

HttpStreamFactory::JobController is a bit more interesting: it has a small state machine that simply resolves the proxy and then creates some jobs. The proxy resolution simply calls the ProxyResolutionService described above. Some debug prints confirm that this returns a ProxyInfo containing localhost:8080.

The job controller manages several jobs that run in parallel, implementing the "happy eyeballs" that I mentioned in the Life and Times post. All of these jobs are implemented with the same class. I would have expected different job subclasses per job type. Anyway, since we're not using QUIC or pre-connecting or any of that stuff, we'll just focus on the "MAIN" job.

Among many parameters, the HttpStreamFactory::Job constructor takes a ProxyInfo, so we can look for where that is used.

HttpStreamFactory::Job -> STATE_START                                                                                                  
HttpStreamFactory::Job -> STATE_WAIT
HttpStreamFactory::Job -> STATE_WAIT_COMPLETE
HttpStreamFactory::Job -> STATE_INIT_CONNECTION
HttpStreamFactory::Job -> STATE_INIT_CONNECTION_COMPLETE
HttpStreamFactory::Job -> STATE_CREATE_STREAM
HttpStreamFactory::Job -> STATE_CREATE_STREAM_COMPLETE

The STATE_WAIT(_COMPLETE) states are related to the job controller's coordination of multiple parallel jobs. The interesting bit is STATE_INIT_CONNECTION, in the HttpStreamFactory::Job::DoInitConnectionImpl method. This method embodies dozens of concerns -- in my opinion this a perfect example of how not to implement something like this. But, ignoring QUIC, SPDY, WebSockets, TLS, PRECONNECT, and all the rest, it comes down to a call to InitSocketHandleForHttpRequest, passing along the ProxyInfo and a plethora of additional arguments.

Let's take a moment here to notice the shift from deeply nested Java-style factories and controllers to a plain old C-style function. There's probably some interesting history here, perhaps in who wrote which bits of this code, or when they were written.

HTTP Connection Pools

InitSocketHandleForHttpRequest, or more accurately InitSocketPoolHelper, gets a pool from the current HttpNetworkSession with session->GetSocketPool(socket_pool_type, proxy_info.proxy_server()). In this case the socket_pool_type is NORMAL_SOCKET_POOL, so this amounts to a call to ClientSocketPoolManagerImpl::GetSocketPool passing the proxy server through which the connection should be made (which might be ProxyServer::Direct() when not using a proxy). The function also creates a ClientSocketPool::GroupId built from the endpoint URL (ip.cow.org in this case) and a few partitioning parameters.

Summarizing, then, the HttpNetworkSession stores a connection pool for each proxy server (including direct), and within each pool indexes connections by group ID.

When a socket in a socket pool is claimed, that claim is represented by a ClientSocketHandle, an empty instance of which is among the parameters to InitSocketHandleForHttpRequest, which calls ClientSocketHandle::Init(..).

This Init method calls the pool's RequestSocket method. There are two implementations of this method, one of which is for WebSockets, so in this case we're calling TransportClientSocketPool::RequestSocket and on to TransportClientSocketPool::RequestSocketInternal. Assuming that there are no existing connections in the pool, and there are free slots to create new connections, this makes a new connection.

Creating a Connection

This occurs with another set of jobs and job factories, this time with subclasses.

ClientSocketPool::CreateConnectJob uses a ConnectJobFactory to create a ConnectJob, passing along the origin URL (endpoint) and proxy server.

ConnectJobFactory::CreateConnectJob examines the proxy server and, in the case that it's not direct (and HTTP-like, not SOCKS) defers to an HttpProxyConnectJob::Factory, which simply creates an HttpProxyConnectJob, a subclass of ConnectJob. This, too, has a large set of states, although only a few are used in this situation:

HttpProxyConnectJob -> STATE_BEGIN_CONNECT
HttpProxyConnectJob -> STATE_TRANSPORT_CONNECT
HttpProxyConnectJob -> STATE_TRANSPORT_CONNECT_COMPLETE                                                                                                                                                             
HttpProxyConnectJob -> STATE_HTTP_PROXY_CONNECT
HttpProxyConnectJob -> STATE_HTTP_PROXY_CONNECT_COMPLETE

Checking the implementation of those states, STATE_TRANSPORT_CONNECT involves creating a TransportConnectJob (since the connection to localhost:8080 is not using HTTPS). But at this point my head is starting to spin at the number of nested "jobs", so I'll stop here and assume that TransportConnectJob::Connect does what it says on the tin: connects to the host (the proxy server) specified in the HttpProxySocketParams.

Initializing the Connection

The next state, STATE_HTTP_PROXY_CONNECT, wraps the socket returned from the TransportConnectJob in an HttpProxyClientSocket and calls its Connect method. And no surprise, there's another state machine here:

HttpProxyClientSocket -> STATE_GENERATE_AUTH_TOKEN
HttpProxyClientSocket -> STATE_GENERATE_AUTH_TOKEN_COMPLETE
HttpProxyClientSocket -> STATE_SEND_REQUEST
HttpProxyClientSocket -> STATE_SEND_REQUEST_COMPLETE
HttpProxyClientSocket -> STATE_READ_HEADERS
HttpProxyClientSocket -> STATE_READ_HEADERS_COMPLETE

We're not using proxy authentication (which is an additional complication sprinkled evenly over this entire stack!), so the interesting state here is STATE_SEND_REQUEST. This calls out to the ProxyDelegate, if one is configured, and then calls ProxyClientSocket::BuildTunnelRequest which finally does something recognizable: creates a "CONNECT" request line, with the host and port for the endpoint (so, CONNECT ip.cow.org:443 HTTP/1.1 in this example).

The next state is STATE_READ_HEADERS, which reads the response from the proxy. If that's a 200 OK, then the socket is connected through the proxy and to the endpoint, and from here on out can be treated just like a socket connected directly to the endpoint.

Re-Surfacing

So, let's trace the result back up through the stack. The interleaving of the logging added above helps quite a bit here:

HttpProxyConnectJob -> STATE_HTTP_PROXY_CONNECT_COMPLETE
HttpStreamFactory::Job -> STATE_INIT_CONNECTION_COMPLETE
HttpStreamFactory::Job -> STATE_CREATE_STREAM
HttpStreamFactory::Job -> STATE_CREATE_STREAM_COMPLETE
HttpNetworkTransaction -> STATE_CREATE_STREAM_COMPLETE
HttpNetworkTransaction -> STATE_CONNECTED_CALLBACK
[0926/173834.596927:ERROR:churl_bin.cc(89)] OnConnected

STATE_HTTP_PROXY_CONNNECT_COMPLETE calls the parent class's SetSocket to use the socket prepared earlier.

HttpStreamFactory::Job gets the result wrapped in a ClientSocketHandle. As always, it handles a half-dozen concerns in STATE_INIT_CONNECTION_COMPLETE, then wraps that in an HttpBasicStream in STATE_CREATE_STREAM.

The HttpNetworkTransaction STATE_CREATE_STREAM_COMPLETE then calls the OnConnected callback, which results in a debug log message in churl.

From that point, there's no further special handling of proxies -- this is a socket carrying an HTTP stream, like any other.

What's Next

This will be the last post on this topic -- I've learned the things I wanted to learn already. However, there are certainly more things to explore:

What happens when making a simple proxy request, rather than tunneled?
What happens when a proxy tunnel fails?
What happens when a proxy requires authentication and the browser must prompt the user?
What happens when a proxy implements QUIC?

All of these are handled somewhere in the stack, but I've skipped over them to try to reduce the breadth of knowledge I had to understand. And it was still quite broad!

Blog