Three ways to configure HTTP Proxy in Playwright

rg9

Rafa艂 Garbowski

Posted on March 23, 2023

Three ways to configure HTTP Proxy in Playwright

TL;DR

In fact, in Playwright v.1.31.1 there are three ways to configure an HTTP proxy:

  • chromium.launch
  • browser.newContext
  • page.setExtraHTTPHeaders, but works only on Firefox

Today I was struggling with proxy setup for Browserless ...

Official docs says:

Both browserless, and Chrome itself, support the usage of external proxies. In order to fully utilize a 3rd-party proxy you'll need to do two things:

-   Specify the address of where the proxy is with the `--proxy-server` switch.
-   Optionally, you'll also need to send in your username and password if the proxy is authenticated.
Enter fullscreen mode Exit fullscreen mode

Ok, so I need to set --proxy-server. I was using a Docker image to run Browserless locally. I discovered that there is an environment variable called PROXY_URL:

podman run --rm -p 3000:3000 -e "PROXY_URL=my-proxy-url" browserless/chrome:latest
Enter fullscreen mode Exit fullscreen mode

Fine, but how do I pass on the username and password?
Docs say there are two methods:

## Using username and password
### Method 1: page.authenticate
### Method 2: page.setExtraHTTPHeaders
Enter fullscreen mode Exit fullscreen mode

I'm using playwright-java and there is no method page.authenticate, so page.setExtraHTTPHeaders left, I thought.

page.setExtraHTTPHeaders(Map.of("Proxy-Authorization", "Basic "  
   + new String(Base64.getEncoder().encode("username:password".getBytes()))));
Enter fullscreen mode Exit fullscreen mode

Unfortunately, it does not work. I received the following strange error: net::ERR_INVALID_ARGUMENT. Besides there was no errors in container's logs. What's the hack?

Google's top two results for playwright Proxy-Authorization:

I quickly checked the first link. The answer says that there are two ways. I didn't read further, because I thought that I know these ways - one is method I'm struggling with and second chromium.launch, right? :) Later it turned out that I was wrong!

The second link looks like what I need. However issue was closed without any conclusion. The alternative shown, was one that I already knew:

browser = await playwright.chromium.launch(
    proxy={"server": "localhost:8080", "username": "user", "password": ""},
)
Enter fullscreen mode Exit fullscreen mode

The case was that I didn't want to lanuch browser, but rather connect to one created inside container.

I dug even deeper and found another similar issue in Playwright project:
# [Feature] Directly set Proxy-Authorization in Chrome without configuring any other proxy settings

The problem (in Chrome/Chromium anyway) is that Proxy-Authorization may not be set explicitly via extraHTTPHeaders. If you do this and then try to navigate to any page Chrome will give you the following error: net::ERR_INVALID_ARGUMENT.

I was lost, considering changing my approach...

Fortunately, my teammate assisted me by showing some examples of where I found a missing piece:

Browser.NewContextOptions options = new Browser.NewContextOptions()  
   .setProxy(new Proxy("my-proxy-url")  
      .setUsername("username")  
      .setPassword("password"));  
BrowserContext context = browser.newContext(options);
Enter fullscreen mode Exit fullscreen mode

Ta da! As simple as that! 馃槂

I checked the playwright docs and it appears to be possible:

Proxy can be either set globally for the entire browser, or for each browser context individually.
Enter fullscreen mode Exit fullscreen mode

Unluckily there is no example, so it's hard to spot.

I also double-checked node.js - How do I authenticate a proxy in playwright - Stack Overflow and configuring proxy via browser.newContext was actually there, but I missed it (confused by Browserless docs?):

const browser = await chromium.launch({
    proxy: { server: 'per-context' }
});
const context = await browser.newContext({
    proxy: { server: 'http://myproxy.com:3128' }
})
Enter fullscreen mode Exit fullscreen mode

Getting to the point:
Festina lente ("hurry slowly") and read comprehension!

馃挅 馃挭 馃檯 馃毄
rg9
Rafa艂 Garbowski

Posted on March 23, 2023

Join Our Newsletter. No Spam, Only the good stuff.

Sign up to receive the latest update from our blog.

Related