Scrape Google Autocomplete Suggestions

In this post, we will learn to scrape Google Autocomplete Suggestions.

Requirements:

Before we begin, we should install everything we may need in this tutorial to move forward.

So before starting, set up your Node JS project and after that, install both the packages - Unirest JS and Cheerio JS. You can install both packages from the above stated link.

Target:

We will target to scrape the autocomplete suggestions of coffee.

Process:

Now, all the things are installed that we will need to prepare our scraper. We will use an npm library Unirest JS to make a get request to our target URL so we can get our raw HTML data. Then we will use Cheerio JS for parsing the extracted HTML data.

We will target this URL:

https://www.google.com/complete/search?&hl=en&q=coffee&gl=us&client=chrome

Copy this URL in your browser and press enter. You will see a text file downloading in your browser by entering this URL in your browser. Open this file in your respective code editor.

We will now first convert this JSON string into object and get the respective data we need.

You can copy this code from the following link: https://github.com/Darshan972/GoogleScrapingBlogs/blob/main/GoogleAutocompleteScraper.js

Result:

[
  { value: 'coffee near me', relevance: 1250, type: 'QUERY' },
  { value: 'coffee shops near me', relevance: 650, type: 'QUERY' },
  { value: 'coffee shop', relevance: 601, type: 'QUERY' },
  { value: 'coffee table', relevance: 600, type: 'QUERY' },
  { value: 'coffee maker', relevance: 553, type: 'QUERY' },
  { value: 'coffee bean', relevance: 552, type: 'QUERY' },
  { value: 'coffee grinder', relevance: 551, type: 'QUERY' },
  { value: 'coffee meets bagel', relevance: 550, type: 'QUERY' }
]
1300

With Google Autocomplete API

If you want to scrape Google Autocomplete Results easily, without making a scraper, as scraping can take a lot of time sometimes, you can try this API.

Serpdog | Google Search API also provides 100 free searches per month, and if you want to scale your requests quota, you can buy paid plans also.

Example request code:

const axios = require('axios');

axios.get('https://api.serpdog.io/autocomplete?api_key=APIKEY&q=football&gl=us')
  .then(response => {
    console.log(response.data);
  })
  .catch(error => {
    console.log(error);
  });

Results:

{
  "meta": {
    "api_key": "APIKEY",
    "q": "football",
    "gl": "us"
  },
  "suggestions": [
    {
      "value": "football cleats",
      "relevance": 601,
      "type": "QUERY"
    },
    {
      "value": "football games",
      "relevance": 600,
      "type": "QUERY"
    },
    {
      "value": "football wordle",
      "relevance": 555,
      "type": "QUERY"
    },
    {
      "value": "football today",
      "relevance": 554,
      "type": "QUERY"
    },
    {
      "value": "football gloves",
      "relevance": 553,
      "type": "QUERY"
    },
    {
      "value": "football field",
      "relevance": 552,
      "type": "QUERY"
    },
    {
      "value": "football movies",
      "relevance": 551,
      "type": "QUERY"
    },
    {
      "value": "football positions",
      "relevance": 550,
      "type": "QUERY"
    }
  ],
  "verbatim_relevance": 1300
}

Conclusion

In this tutorial we learned to scrape Google Autocomplete Suggestions Results. If you have any questions, feel free to ask me in the comments. Follow me on Twitter. Thanks for reading!