The Easiest Way to Scrape Amazon Products and Reviews

Introduction

Amazon is a vast Internet-based enterprise that sells all kinds of goods either directly or as the middleman between other retailers and Amazon's millions of customers.

You can scrape their data to run different types of tasks such as:

competitor analysis
improving your products and value proposition
identifying market trends and what influences them
price monitoring

Luckily, Amazon is a website that is pretty easy to scrape if you have the right tools.

For this purpose, we will use Page2API - a powerful and delightful API that makes web scraping easy and fun.

In this case, we will scrape 2 types of amazon pages:

Search results page
Product reviews page

Prerequisites

To perform this simple task, you will need the following things:

A Page2API account

The free trial offers a credit that covers up to 1000 web pages to scrape, and it takes under 10 seconds to create the account if you sign up with Google.

A product or a category of products that we are about to scrape.

In our case, we will search for 'luminox watches'
and then scrape the reviews for a random product.

Scraping all products from the search page

First what we need is to type 'luminox watches' into the search input from amazon's search page.

This will change the browser URL to something similar to:

https://www.amazon.com/s?k=luminox+watches

The URL is the first parameter we need to perform the scraping.

The page that you see must look like the following one:

If you inspect the page HTML, you will find out that a single result is wrapped into a div that looks like the following:

The HTML for a single result element will look like this:

The last part is the pagination handling.

In our case, we must click on the Next → button while the list item's class will be active:

And stop our scraping request when the Next → button became disabled.

In our case, a new class (a-disabled) is assigned to the list element where the button is located:

Now it's time to prepare the request that will scrape all products that the search page returned.

Setting the api_key as an environment variable

export API_KEY=YOUR_PAGE2API_KEY

Running the scraping request with cURL

    curl -v -XPOST -H "Content-type: application/json" -d '{
      "api_key": "'"$API_KEY"'",
      "url": "https://www.amazon.com/s?k=luminox+watches",
      "real_browser": true,
      "merge_loops": true,
      "scenario": [
        {
          "loop" : [
            { "wait_for": ".a-pagination li.a-last" },
            { "execute": "parse" },
            { "execute_js": "document.querySelector(\".a-pagination li.a-last a\").click()" }
          ],
          "stop_condition": "document.querySelector(\".a-last.a-disabled\") !== null"
        }
      ],
      "parse": {
        "watches": [
          {
            "_parent": "[data-component-type='s-search-result']",
            "title": "h2 >> text",
            "link": ".a-link-normal >> href",
            "price": ".a-price-whole >> text",
            "stars": ".a-icon-alt >> text"
          }
        ]
      }
    }' 'https://www.page2api.com/api/v1/scrape' | python -mjson.tool

The result

  {
    "result": {
      "watches": [
        {
          "title": "Men's Luminox Leatherback Sea Turtle 44mm Watch",
          "link": "https://www.amazon.com/Luminox-Leatherback-Turtle-Giant-Black/dp/B07CVFWXMR/ref=sr_1_2?dchild=1&keywords=luminox+watches&qid=1634327863&sr=8-2",
          "price": "$223.47",
          "stars": "4.5 out of 5 stars"
        },
        {
          "title": "The Original Navy Seal Mens Watch Black Display (XS.3001.F/Navy Seal Series): 200 Meter Water Resistant + Light Weight Case + Constant Night Visibility",
          "link": "https://www.amazon.com/Luminox-Wrist-Watch-Navy-Original/dp/B07NYXV77C/ref=sr_1_3?dchild=1&keywords=luminox+watches&qid=1634327863&sr=8-3",
          "price": "$254.12",
          "stars": "4.3 out of 5 stars"
        },
        {
          "title": "Leatherback SEA Turtle Giant - 0323",
          "link": "https://www.amazon.com/Luminox-Leatherback-SEA-Turtle-Giant/dp/B07PBC31N8/ref=sr_1_4?dchild=1&keywords=luminox+watches&qid=1634327863&sr=8-4",
          "price": "$179.00",
          "stars": "4.3 out of 5 stars"
        },
        ...
      ]
    },
    ...
  }

Scraping product reviews

First what we need is to click on the See all reviews link from the Product page.

This will change the browser URL to something similar to:

https://www.amazon.com/product-reviews/B072FNJLBC

The URL is the first parameter we need to perform the reviews scraping.

The HTML from a single review will look like this:

Luckily, the pagination handling is similar to the one described above, so we will use the same flow.

Now it's time to prepare the request that will scrape all reviews.

Setting the api_key as an environment variable (if needed)

export API_KEY=YOUR_PAGE2API_KEY

Running the scraping request with cURL

    curl -v -XPOST -H "Content-type: application/json" -d '{
      "api_key": "'"$API_KEY"'",
      "url": "https://www.amazon.com/product-reviews/B072FNJLBC",
      "real_browser": true,
      "merge_loops": true,
      "scenario": [
        {
          "loop" : [
            { "wait_for": ".a-pagination li.a-last" },
            { "execute": "parse" },
            { "execute_js": "document.querySelector(\".a-pagination li.a-last a\").click()" }
          ],
          "stop_condition": "document.querySelector(\".a-last.a-disabled\") !== null"
        }
      ],
      "parse": {
        "reviews": [
          {
            "_parent": "[data-hook='review']",
            "title": ".review-title >> text",
            "author": ".a-profile-name >> text",
            "stars": ".review-rating >> text",
            "content": ".review-text >> text"
          }
        ]
      }
    }' 'https://www.page2api.com/api/v1/scrape' | python -mjson.tool

The result

  {
    "result": {
      "reviews": [
          {
            "title": "Great watch & easy to read in low light conditions",
            "author": "Paul E. Papas",
            "stars": "5.0 out of 5 stars",
            "content": "I'm a 60+ year old equestrian and outdoorsman. I was looking for a watch that could take the shock of firearm discharge ..."
          },
          {
            "title": "Not Water Resistant, impossible to get amazon help",
            "author": "Benjamin H. Curry",
            "stars": "2.0 out of 5 stars",
            "content": "This watch has a 2 year warranty from date of purchase however after not even on full year my adult son went swimming with it ..."
          },
        ...
      ]
    },
    ...
  }

Conclusion

That's pretty much of it!

As mentioned, Amazon is a website that is pretty easy to scrape if you have the right tools.
This is the place where Page2API shines, making web scraping super easy and fun.

The original article can be found here:

https://www.page2api.com/blog/scrape-amazon-product-data

Blog

The Easiest Way to Scrape Amazon Products and Reviews

Nicolae Rotaru

Introduction

Prerequisites

Scraping all products from the search page

Scraping product reviews

Conclusion

Join Our Newsletter. No Spam, Only the good stuff.

Related