Demo code here

Disclaimer: While this is not a sponsored post, the links contained here are affiliate links. Luminati is a product, however, that I use currently in my production code.

Sometimes using a proxy service is worth the cost. Using Luminati is a great example of when that is the case. I don’t often use proxies. I like to investigate and reverse engineer. Learn more about Luminati here.

The fun in web scraping is understanding what is stopping you from automated access to the website and then creating a tool that is able to emulate perfectly the expected request. When I need to have code that is running regularly, however, I need to run it from the cloud. And sometimes, cloud ip addresses are blocked at the ip level. This is a time when using a proxy makes sense.

Enter Luminati

Luminati makes it very simple to sign up. Go here and hit “Start Now”. They are very thorough with verification and before I could use it fully they had to verify that I had a valid business with the Secretary of State. I’m not sure what would happen if I didn’t have an actual business that was registered with the Secretary of State.

Once within Luminati, you’ll see a dashboard and a place to configure your “Zones” or different proxies.

The one in which I was most interested was the rotating residential proxy. I have an example of a target website from which I gather information that when scraping from my home, it works without any problem. When scraping from the cloud, the request just times out. At one point I even set up a VNC connection and tried to access the site from a chrome browser from the GUI and it still timed out. This is an example of when a proxy makes sense.

Hitting the “Edit” icon near the residential settings provides the username and password that you will need to use. If you are using the sample code that I am providing, just rename .sample.env to .env and replace the sample username and password with the ones found in the above section.

When you want to use the proxy, just add the prod=true property to the .env file.

And that’s really all you need to do within Luminati to get setup.

The code

I have examples using axios, request (well, request-promise), and Puppeteer.

Axios is the most strange. It seems that using the proxy option doesn’t work when calling to https sites. Luminati has a test site that gives the ip address from which we are calling.

async function withAxios() {
    const url = 'https://lumtest.com/myip.json';
    const options: AxiosRequestConfig = { };

    if (process.env.prod) {
        console.log('Production enabled, using Luminati');
        options.proxy = false;
        options.httpsAgent = new HttpsProxyAgent(`https://${process.env.luminatiUsername}:${process.env.luminatiPassword}@zproxy.lum-superproxy.io:22225`)
    }

    const axiosResponse = await axios.get(url, options);

    console.log('Response from axios', axiosResponse.data);

}

And here is the difference with and without the proxy:

Here’s the code when using request. The proxy works as expected here.

async function withRequest() {
    const url = 'http://lumtest.com/myip.json';

    const options: Request.Options = {
        url: url,
        method: 'GET'
    };

    if (process.env.prod) {
        console.log('Production enabled, using Luminati');
        options.proxy = `http://${process.env.luminatiUsername}:${process.env.luminatiPassword}@zproxy.lum-superproxy.io:22225`;
    }

    const response = await requestPromise(options);

    console.log('Response from request', JSON.parse(response));

}

And the response with and without the proxy:

And finally, the code with Puppeteer. It’s a simple thing; just add in the proxy as an args option and then authenticate with the page object.

async function withPuppeteer() {    
    const url = 'https://lumtest.com/myip.json';

    const options = {
        args: []
    };

    if (process.env.prod) {
        options.args.push('--proxy-server=zproxy.lum-superproxy.io:22225');
    }

    const browser = await puppeteer.launch(options);
    const page = await browser.newPage();

    if (process.env.prod) {
        console.log('Production enabled, using Luminati');
        await page.authenticate({
            username: process.env.luminatiUsername,
            password: process.env.luminatiPassword
        });
    }

    await page.goto(url);

    const response = await page.$eval('pre', elem => elem.textContent);

    console.log('Response from puppeteer', JSON.parse(response));

    await browser.close();

}

And the response with and without the proxy:

Customer service

I would be very remiss if I didn’t speak of Luminati’s customer service. As soon as I created my account, I was assigned a case manager and it included his Skype information. I had a few questions and so I reached out. He responded quickly and got me in contact with a technical rep.

Both him and the technical rep answered my questions always quickly and extremely helpfully. I just recently had a call with the technical rep to discuss a problem I was having. He spent a good amount of time helping me find a solution to a problem that was mine and not related to Luminati.

Luminati’s customer service is GREAT. I cannot recommend it enough. Luminati is truly the Rolls-Royce of proxies and I use them now and I intend to keep using them. Check out more about Luminati here.

Demo code here

Looking for business leads?

Using the techniques talked about here at javascriptwebscrapingguy.com, we’ve been able to launch a way to access awesome web data. Learn more at Cobalt Intelligence!

The post Jordan Uses Luminati appeared first on JavaScript Web Scraping Guy.