Web scraping Google Jobs Listing with Nodejs

Intro

In this blog post I'll show you how to use Google Jobs Listing Results API from SerpApi.

The main advantages of our API is that you don't need to use browser automation to scrape results, create the parser from scratch and maintain it.

There's also a chance that the request might be blocked at some point from Google, we handle it on our backend so there's no need to figure out how to do it yourself or figure out which CAPTCHA, proxy provider to use.

What will be scraped

Full code

If you don't need an explanation, have a look at the full code example in the online IDE

const SerpApi = require("google-search-results-nodejs");
const search = new SerpApi.GoogleSearch(process.env.API_KEY);

const jobId =
  "eyJqb2JfdGl0bGUiOiJCYXJpc3RhIiwiY29tcGFueV9uYW1lIjoiQmx1ZSBCb3R0bGUgQ29mZmVlIiwiY29tcGFueV9taWQiOiIvbS8wM3AxMnFkIiwiYWRkcmVzc19jaXR5IjoiU2FuIEZyYW5jaXNjbyIsImFkZHJlc3Nfc3RhdGUiOiJDYWxpZm9ybmlhIiwiaHRpZG9jaWQiOiJBNnFRZGw1VjZvU1ZxbnI0QUFBQUFBPT0iLCJ1dWxlIjoidytDQUlRSUNJa1UyRnVJRVp5WVc1amFYTmpieUJDWVhrZ1FYSmxZU3hWYm1sMFpXUWdVM1JoZEdWeiIsImZjIjoiRW80QkNtY3dZVFZSY0RkWE9VTkJSek42WVdocVVHRkZXV2hqTkdGamJGSm5lRUpmWVhOc2NVRmlhRXhKWHpaek1Hb3hWamx2UWxKSlpGVnBVRUpyV1ZkeU1sQXpXRmhVVDB3MVRHbFNjamh5T0VzelRVcGhhM2x6WjFsQ1NuZFBWMjVJTlVKQlJWTjJOREZCRWhaME5rUXdXSE5pYjB4TmRWVXRaMVEwTkc5SFNVTkJHZ3RQTFVKSGRqaFNVWEp4TkEiLCJmY2VudiI6IjEiLCJmY3YiOiIyIiwiZmNfaWQiOiJmY183In0";

const params = {
  engine: "google_jobs_listing", // search engine
  q: jobId, // job id from https://serpapi.com/playground?engine=google_jobs&q=Barista
};

const getJson = () => {
  return new Promise((resolve) => {
    search.json(params, resolve);
  });
};

const getResults = async () => {
  const json = await getJson();
  const { apply_options = "No apply options for this job", salaries = "No salaries for this job", ratings = "No ratings for this job" } = json;
  return { applyOptions: apply_options, salaries, ratings };
};

getResults().then(console.log);

Preparation

First, we need to create a Node.js* project and add npm package google-search-results-nodejs to scrape and parse what you need using SerpApi.

To do this, in the directory with our project, open the command line and enter:

$ npm init -y

And then:

$ npm i google-search-results-nodejs

*If you don't have Node.js installed, you can download it from nodejs.org and follow the installation documentation.

Code explanation

First, we need to declare SerpApi from google-search-results-nodejs library and define new search instance with your API key from SerpApi:

const SerpApi = require("google-search-results-nodejs");
const search = new SerpApi.GoogleSearch(API_KEY);

Next, we write the job ID and the necessary parameters for making a request. You can get the job Id parameter using Web scraping Google Jobs organic results with Nodejs blog post:

const jobId =
  "eyJqb2JfdGl0bGUiOiJCYXJpc3RhIiwiY29tcGFueV9uYW1lIjoiQmx1ZSBCb3R0bGUgQ29mZmVlIiwiY29tcGFueV9taWQiOiIvbS8wM3AxMnFkIiwiYWRkcmVzc19jaXR5IjoiU2FuIEZyYW5jaXNjbyIsImFkZHJlc3Nfc3RhdGUiOiJDYWxpZm9ybmlhIiwiaHRpZG9jaWQiOiJBNnFRZGw1VjZvU1ZxbnI0QUFBQUFBPT0iLCJ1dWxlIjoidytDQUlRSUNJa1UyRnVJRVp5WVc1amFYTmpieUJDWVhrZ1FYSmxZU3hWYm1sMFpXUWdVM1JoZEdWeiIsImZjIjoiRW80QkNtY3dZVFZSY0RkWE9VTkJSek42WVdocVVHRkZXV2hqTkdGamJGSm5lRUpmWVhOc2NVRmlhRXhKWHpaek1Hb3hWamx2UWxKSlpGVnBVRUpyV1ZkeU1sQXpXRmhVVDB3MVRHbFNjamh5T0VzelRVcGhhM2x6WjFsQ1NuZFBWMjVJTlVKQlJWTjJOREZCRWhaME5rUXdXSE5pYjB4TmRWVXRaMVEwTkc5SFNVTkJHZ3RQTFVKSGRqaFNVWEp4TkEiLCJmY2VudiI6IjEiLCJmY3YiOiIyIiwiZmNfaWQiOiJmY183In0";

const params = {
  engine: "google_jobs_listing", // search engine
  q: jobId, // job id from https://serpapi.com/playground?engine=google_jobs&q=Barista
};

Next, we wrap the search method from the SerpApi library in a promise to further work with the search results:

const getJson = () => {
  return new Promise((resolve) => {
    search.json(params, resolve);
  });
};

And finally, we declare the function getResult that gets data from the page and return it:

const getResults = async () => {
  ...
};

In this function, we get json with results from the page (getJson function). Then we destructure received json, set default values to each destructured property and return object with results:

const json = await getJson();
const { 
    apply_options = "No apply options for this job",
    salaries = "No salaries for this job",
    ratings = "No ratings for this job"
    } = json;
return { applyOptions: apply_options, salaries, ratings };

After, we run the getResults function and print all the received information in the console:

getResults().then(console.log);

Output

{
  "applyOptions": "No apply options for this job",
  "salaries": [
    {
      "job_title": "Barista",
      "link": "https://www.ziprecruiter.com/Salaries/Barista-Salary-in-San-Francisco,CA?utm_campaign=google_jobs_salary&utm_source=google_jobs_salary&utm_medium=organic",
      "source": "ZipRecruiter",
      "salary_from": 21000,
      "salary_to": 40000,
      "salary_currency": "$",
      "salary_periodicity": "year",
      "thumbnail": "https://serpapi.com/searches/6363e399a3f4ef79bb0ff373/images/f1d87c26c4968270a8e938f7cf10b5cdc1a6c6457fd1184e.png",
      "based_on": "Based on local employers"
    },
    {
      "job_title": "Coffee Barista",
      "link": "https://www.salary.com/research/salary/alternate/coffee-barista-salary/san-francisco-ca?utm_campaign=google_jobs_salary&utm_source=google_jobs_salary&utm_medium=organic",
      "source": "Salary.com",
      "salary_from": 27000,
      "salary_to": 36000,
      "salary_currency": "$",
      "salary_periodicity": "year",
      "thumbnail": "https://serpapi.com/searches/6363e399a3f4ef79bb0ff373/images/f1d87c26c496827075cfe00b6aeb887d3533b03d6f8540fc.png",
      "based_on": "Based on local employers"
    },
    {
      "job_title": "Barista",
      "link": "https://www.payscale.com/research/US/Job=Barista/Hourly_Rate/ae670208/San-Francisco-CA?utm_campaign=google_jobs_salary&utm_source=google_jobs_salary&utm_medium=organic",
      "source": "Payscale",
      "salary_from": 14,
      "salary_to": 20,
      "salary_currency": "$",
      "salary_periodicity": "hour",
      "thumbnail": "https://serpapi.com/searches/6363e399a3f4ef79bb0ff373/images/f1d87c26c4968270a20489154adfa8fce64f52ba237d9ed1.png",
      "based_on": "Based on local employers"
    }
  ],
  "ratings": [
    {
      "link": "https://www.glassdoor.com/Reviews/Blue-Bottle-Coffee-Reviews-E803867.htm?utm_campaign=google_jobs_reviews&utm_source=google_jobs_reviews&utm_medium=organic",
      "source": "Glassdoor",
      "rating": 3.6,
      "reviews": 303
    },
    {
      "link": "https://www.indeed.com/cmp/Blue-Bottle-Coffee-Company/reviews?utm_campaign=google_jobs_reviews&utm_source=google_jobs_reviews&utm_medium=organic",
      "source": "Indeed",
      "rating": 3.3,
      "reviews": 70
    },
    {
      "link": "https://www.comparably.com/companies/blue-bottle-coffee?utm_campaign=google_jobs_reviews&utm_source=google_jobs_reviews&utm_medium=organic",
      "source": "Comparably",
      "rating": 2.5,
      "reviews": 40
    }
  ]
}

Links

If you want to see some projects made with SerpApi, write me a message.

Join us on Twitter | YouTube

Add a Feature Request💫 or a Bug🐞

Blog