Gathering HAR and deciphered PCAP from you webdriver tests

To gather network data without relying solely on the browser, we need an intermediary that can monitor the traffic between the browser and the remote server. For the following tutorial, we will be using Fluxzy, an open-source Man-In-The-Middle (MITM) intermediary that can be used as a .NET library. We will use it along with Selenium to extract HTTP Archive (HAR) Files and PCAPNG from any browser or remote devices.

The code

Let's start by creating a new console project. For all the following, I'm using .NET 8.

dotnet new console -o webdriver_with_har
cd webdriver_with_har

Install fluxzy packages:

dotnet add package Fluxzy.Core.Pcap

With a quick check on the main docs, we can run the proxy with a few lines of code


using Fluxzy;

var fluxzySetting = FluxzySetting.CreateDefault()
    .SetOutDirectory("temp_directory");

await using (var proxy = new Proxy(fluxzySetting))
{
    _ = proxy.Run(); 
    // The proxy now runs until this scope is disposed
}

Packager.ExportAsHttpArchive("temp_directory", "export.har");

The code above is quiet straightforward. We create a new proxy instance with the default settings.
Then, we start the proxy and wait for the proxy to be disposed. Once the proxy is disposed, we export the network dump into an HTTP Archive file.

For this example above will generate an empty HAR as we did not redirect actual traffic to the proxy.

Integrate with Selenium

Let's add Selenium to the project:

dotnet add package Selenium.WebDriver

Proxy.Run() method returns all endpoints on which the proxy is listening. We can use any of them to configure the webdriver.

var endpoint = proxy.Run().First(); 

var options = new FirefoxOptions
{
    Proxy = new OpenQA.Selenium.Proxy
    {
        HttpProxy = $"{endpoint.Address}:{endpoint.Port}",
        SslProxy = $"{endpoint.Address}:{endpoint.Port}"
    },
    // Necessary if you did not install the root certificate computer-wide
    AcceptInsecureCertificates = true 
};

Note that, if the root certificate is not installed computer-wide, you need to set AcceptInsecureCertificates to true. Otherwise the browser will trigger errors when connecting to the proxy.

Now, a full working example making a request to google.com and exporting the HAR file would look like this:

using Fluxzy;
using OpenQA.Selenium;
using OpenQA.Selenium.Firefox;

var fluxzySetting = FluxzySetting.CreateDefault()
    .SetOutDirectory("temp_directory");

await using (var proxy = new Fluxzy.Proxy(fluxzySetting))
{
    var endpoint = proxy.Run().First(); 
    var options = new FirefoxOptions
    {
        Proxy = new OpenQA.Selenium.Proxy
        {
            HttpProxy = $"{endpoint.Address}:{endpoint.Port}",
            SslProxy = $"{endpoint.Address}:{endpoint.Port}"
        },
        AcceptInsecureCertificates = true
    };

    using var driver = new FirefoxDriver(options); 

    driver.Navigate().GoToUrl("https://www.google.com");
    driver.FindElement(By.TagName("textarea")); 
    driver.Quit();
}

Packager.ExportAsHttpArchive("temp_directory", "export.har");
Packager.Export("temp_directory", "export.fxzy");

Going further with PCAPNG data extraction

PCAPNG (stands for PCAP new generation) is a newer version of the PCAP format. It fullfills basically the same purpose as regular PCAP and can be open with a recent version of Wireshark. What's interesting with the fluxzy integration is its ability to hold, out of the box, NSS Key log datas which can be used to view the decrypted TLS traffic in Wireshark.

To enable NSS Key log datas, we instruct fluxzy to use the Bouncy Castle library during the proxy setup:

fluxzySetting.UseBouncyCastleSslEngine();

Then we can configure the proxy to enable PCAPNG extraction.

await using (var tcpConnectionProvider = await CapturedTcpConnectionProvider.CreateInProcessCapture())
await using (var proxy = new Fluxzy.Proxy(fluxzySetting, tcpConnectionProvider: tcpConnectionProvider)){
    // Use proxy
}

The code above tells fluxzy to use a tcpConnectionProvider that will capture any created TCP connections into a PCAPNG file.

Full code with HAR and pcapng

Here is a full working example with HAR and PCAPNG extraction:

using System.Net;
using Fluxzy;
using Fluxzy.Core.Pcap;
using OpenQA.Selenium;
using OpenQA.Selenium.Firefox;

var fluxzySetting = FluxzySetting.CreateDefault(IPAddress.Loopback, 0)
    .SetOutDirectory("temp_directory");

fluxzySetting.UseBouncyCastleSslEngine();

await using (var tcpConnectionProvider = await CapturedTcpConnectionProvider.CreateInProcessCapture())
await using (var proxy = new Fluxzy.Proxy(fluxzySetting, tcpConnectionProvider: tcpConnectionProvider))
{
    var endpoint = proxy.Run().First(); 
    var options = new FirefoxOptions
    {
        Proxy = new OpenQA.Selenium.Proxy
        {
            HttpProxy = $"{endpoint.Address}:{endpoint.Port}",
            SslProxy = $"{endpoint.Address}:{endpoint.Port}"
        },
        AcceptInsecureCertificates = true
    };

    using var driver = new FirefoxDriver(options); 

    driver.Navigate().GoToUrl("https://www.google.com");
    driver.FindElement(By.TagName("textarea")); 
    driver.Quit();
}

Packager.ExportAsHttpArchive("temp_directory", "export.har");
Packager.Export("temp_directory", "export.fxzy");

Getting the PCAPNG file

The PCAPNG file is not exported directly because it's sliced individually for each TCP connection made by the browser.

There are, two non programmatic ways to get the PCAPNG file for a specific HTTP exchange or connection.

Using Fluxzy CLI

Fluxzy CLI is a command line tool that can be used to access most of fluxzy features without writing a single line of code.

The command that we are interested in is fluxzy dissect. It allows to make various read operations on a fluxzy dump.

First we list the content of the archive:

fluxzy dissect export.fxzy

This should output something like this:

...
22 - https://www.google.com/ - 200
23 - https://www.google.com/images/branding/googlelogo/1x/googlelogo_color_272x92dp.png - 200
24 - https://shavar.services.mozilla.com/downloads?client=navclient-auto-ffox&appver=121.0&pver=2.2 - 200
25 - https://www.google.com/gen_204?atyp=i&ct=bxjs&cad=&b=0&ei=2GulZf2oCbuikdUP6LCs8As&zx=1705339867346&opi=89978449 - 204
26 - https://www.google.com/ - 200
27 - https://www.google.com/gen_204?ei=smulZYG4GLqrkdUPyI6RGA&vet=10ahUKEwiB2cv09d-DAxW6VaQEHUhHBAMQhJAHCB8..s&bl=xaqk&s=webhp&gl=fr&pc=SEARCH_HOMEPAGE&isMobile=false - 204
...

Each line of the output is a specific HTTP exchange. The first column is the ID of the exchange. The second column is the URL of the exchange. The third column is the HTTP status code of the exchange. This output is customizable with the --format option but now, what we want is to get the id of the exchange that we are interested in.

To extract the PCAPNG matching the exchange with id 22, we can run the following command:

fluxzy dissect export.fxzy -i 22 -f "{pcap}" -o exchange22.pcapng

The -i option is used to specify the id of the exchange. The -f option is used to specify the format of the output. The {pcap} placeholder is used to specify that we want the PCAPNG file. The -o option is used to specify the output file which is exchange22.pcapng in this case.

Using Fluxzy Desktop

Fluxzy Desktop is a free HTTP debuggger that provides a graphical way to access easily a fluxzy archive. To extract the PCAPNG file of an exchange, you can simply select the matching exchange and choose "Connection -> Pcapng -> Download". And that's all.

The downloaded PCAPNG file should be opened deciphered in wireshark without any extra handling.

Conclusion

That's it! We've demonstrated how to integrate Fluxzy with Selenium for effective network data extraction in web testing. This approach not only simplifies the process of capturing HTTP and network traffic data but also opens up new possibilities for in-depth analysis and debugging.

Blog