--- title: How to Scrape Tokopedia Data published: true description: This article explores how to scrape Tokopedia using Python and Crawlbase. You will learn how to extract search listings and product pages for valuable insights. Dive in! tags: howtoscrapetokopedia, tokopediascraping cover_image: https://crawlbase.com/blog/how-to-scrape-tokopedia-data/how-to-scrape-tokopedia-data-og.jpg canonical_url: https://crawlbase.com/blog/how-to-scrape-tokopedia-data/ # Use a ratio of 100:42 for best results. # published_at: 2024-11-01 11:23 +0000 --- This blog was originally posted to [Crawlbase Blog](https://crawlbase.com/blog/how-to-scrape-tokopedia-data/?utm_source=dev.to&utm_medium=referral&utm_campaign=content_distribution) Tokopedia, one of Indonesia’s biggest e-commerce platforms, has 90+ million active users and 350 million monthly visits. The platform has a wide range of products, from electronics, fashion, groceries to personal care. For businesses and developers, scraping Tokopedia data can give you insights into product trends, pricing strategy, and customer preference. Tokopedia uses JavaScript to render its content; the traditional scraping method doesn’t work. [Crawlbase Crawling API](https://crawlbase.com/crawling-api-avoid-captchas-blocks 'Crawlbase Crawling API') helps by handling JavaScript-rendered content seamlessly. In this tutorial, you’ll learn how to use Python and Crawlbase to scrape Tokopedia search listings and product pages for product names, prices, and ratings. Let’s get started! ## Why Scrape Tokopedia Data? Scraping Tokopedia data can be beneficial for businesses and developers. As one of Indonesia’s biggest e-commerce platform, Tokopedia has a lot of information about products, prices and customer behavior. By extracting this data, you can get ahead in the online market. There are many reasons why one would choose to scrape data from Tokopedia: ![An image that shows reasons to scrape Tokopedia data](https://crawlbase.com/blog/how-to-scrape-tokopedia-data/why-scrape-tokopedia-data.jpg) 1. **Market Research**: Knowing the current demand will help you with inventory and marketing planning. Opportunities can always be found by looking at the general trends. 2. **Price Comparison**: One may be able to scrape Tokopedia and get several prices on products from various categories. This would allow one to make price adjustments in order to remain competitive. 3. **Competitor Analysis**: Compiling the data about the products of the competitors will help you understand how they position themselves and where are their weak points. 4. **Customer Insights**: Looking into product reviews and ratings will help understand major pros and cons of various goods from customers’ point of view. 5. **Product Availability**: Monitor products so that you know when the hot ones are getting low, bob up the stocks to appease customers. In the next section we will see what we can scrape from Tokopedia. ## Key Data Points to Extract from Tokopedia When scraping Tokopedia, focus on the important data points and you’ll get actionable insights for your business or research. Here are the data points to grab: 1. **Product Name**: Identifies the product. 2. **Price**: For price monitoring and competition analysis. 3. **Ratings and Reviews**: For User experience and products usability. 4. **Availability**: For stock level and product availability. 5. **Seller Information**: Details on third-party vendors, seller ratings and location. 6. **Product Images**: Images for visual representation and understanding of the product. 7. **Product Description**: For the details of the product. 8. **Category and Tags**: For arrangement of products and categorized analysis. Concentrating on these aspects of data, allows one to collect useful insights from Tokopedia that can aid one in refining or making better decisions. Next, we will see how to set up your Python environment for scraping. ## Crawlbase Crawling API for Tokopedia Scraping The [Crawlbase Crawling API](https://crawlbase.com/crawling-api-avoid-captchas-blocks 'Crawlbase Crawling API') makes scraping Tokopedia fast and straightforward. Since Tokopedia's website uses dynamic content, much of the data is loaded via JavaScript, making it challenging to scrape with traditional methods. But Crawlbase Crawling API renders the pages like a real browser so you can access the data. Here’s why Crawlbase Crawling API is good for scraping Tokopedia: - **Handles Dynamic Content**: Crawlbase handles JavaScript heavy pages so all product data is fully loaded and ready to scrape. - **IP Rotation**: To prevent getting blocked by Tokopedia’s security systems, Crawlbase automatically rotates IPs, letting you scrape without worrying about rate limits or bans. - **Fast Performance**: Crawlbase allows you to efficiently scrape massive amounts of data while saving time and resources. - **Customizable Request**s: You can change the headers, cookies and control requests to fit your needs. With these features, Crawlbase Crawling API makes scraping Tokopedia easier and more efficient. ### Crawlbase Python Library Crawlbase also provides a Python library to make web scraping even easier. To use this library you will need an access token that you can get by [signing up](https://crawlbase.com/signup 'Crawlbase Signup') to Crawlbase. Here’s an example function to send a request to Crawlbase Crawling API: ```python from crawlbase import CrawlingAPI # Initialize Crawlbase API with your access token crawling_api = CrawlingAPI({ 'token': 'YOUR_CRAWLBASE_TOKEN' }) def make_crawlbase_request(url): response = crawling_api.get(url) if response['headers']['pc_status'] == '200': html_content = response['body'].decode('utf-8') return html_content else: print(f"Failed to fetch the page. Crawlbase status code: {response['headers']['pc_status']}") return None ``` **Note**: Crawlbase provides two types of tokens. **Normal Token** for static sites. **JavaScript (JS) Token** for dynamic or browser-rendered content, which is required for scraping Tokopedia. Crawlbase offers 1,000 free requests to help you get started, and you can sign up without a credit card. For more details, refer to the [Crawlbase Crawling API documentation](https://crawlbase.com/docs/crawling-api/ 'Crawlbase Crawling API documentation'). In the next section, we’ll learn how to setup Python environment for Tokopedia scraping. ## Setting Up Your Python Environment To start scraping Tokopedia, you need to setup your Python environment. Follow these steps to get started: ### Installing Python and Required Libraries Make sure Python is installed on your machine. You can download it [here](https://www.python.org/downloads/ 'Download Python'). After installation, run the following command to install the necessary libraries: ```bash pip install crawlbase beautifulsoup4 ``` - **Crawlbase**: For interacting with the Crawlbase Crawling API to handle dynamic content. - **BeautifulSoup**: For parsing and extracting data from HTML. These tools are essential for scraping Tokopedia’s data efficiently. ### Selecting an IDE Choose an IDE for seamless development: - [**Visual Studio Code**](https://code.visualstudio.com/ 'VS Code'): Lightweight and frequently used. - [**PyCharm**](https://www.jetbrains.com/pycharm/download/ 'PyCharm'): A full-featured IDE with powerful Python capabilities. - [**Jupyter Notebook**](https://jupyter.org/install 'Jupyter Notebook'): Ideal for interactive coding and testing. Once your environment is set up, you can begin scraping Tokopedia. Next, we’ll cover how to create Tokopedia SERP Scraper. ## Scraping Tokopedia Search Listings Now that you have your Python environment ready, we can start scraping Tokopedia’s search listings. In this section, we’ll guide you through inspecting the HTML, writing the scraper, handling pagination and storing the data in a JSON file. ### Inspecting the HTML Structure First, you need to inspect the HTML of the Tokopedia search results page from which you want to scrape product listings. For this example, we’ll be scraping headset listings from the following URL: ``` https://www.tokopedia.com/search?q=headset ``` Open the developer tools in your browser and navigate to this URL. ![A screenshot of Tokopedia search listings HTML structure](https://crawlbase.com/blog/how-to-scrape-tokopedia-data/tokopedia-serp-inspect.jpg) Here are some key selectors to focus on: - **Product Title**: Found in a `` tag with class `OWkG6oHwAppMn1hIBsC3pQ==` which contains the name of the product. - **Price**: In a `

` tag with class `ELhJqP-Bfiud3i5eBR8NWg==` that displays the product price. - **Store Name**: Found in a `` tag with class `X6c-fdwuofj6zGvLKVUaNQ==`. - **Product Link**: Product page Link found in an `` tag with class `Nq8NlC5Hk9KgVBJzMYBUsg==`, accessible via the `href` attribute. ### Writing the Search Listings Scraper We’ll write a function that makes a request to the Crawlbase Crawling API, retrieves the HTML, and then parses the data using BeautifulSoup. Here’s the code to scrape the search listings: ```python from crawlbase import CrawlingAPI from bs4 import BeautifulSoup import json # Initialize Crawlbase API with your access token crawling_api = CrawlingAPI({ 'token': 'YOUR_CRAWLBASE_TOKEN' }) # Function to get HTML content from Crawlbase def fetch_html(url): options = { 'ajax_wait': 'true', 'page_wait': '5000' } response = crawling_api.get(url, options) if response['headers']['pc_status'] == '200': return response['body'].decode('utf-8') else: print(f"Failed to fetch page. Status code: {response['headers']['pc_status']}") return None # Function to parse and extract product data def parse_search_listings(html): soup = BeautifulSoup(html, 'html.parser') products = [] for product in soup.select('div[data-testid="divSRPContentProducts"] div.css-5wh65g'): name = product.select_one('span.OWkG6oHwAppMn1hIBsC3pQ\\=\\=').text.strip() if product.select_one('span.OWkG6oHwAppMn1hIBsC3pQ\\=\\=') else 'N/A' price = product.select_one('div.ELhJqP-Bfiud3i5eBR8NWg\\=\\=').text.strip() if product.select_one('div.ELhJqP-Bfiud3i5eBR8NWg\\=\\=') else 'N/A' store = product.select_one('span.X6c-fdwuofj6zGvLKVUaNQ\\=\\=').text.strip() if product.select_one('span.X6c-fdwuofj6zGvLKVUaNQ\\=\\=') else 'N/A' product_url = product.select_one('a.Nq8NlC5Hk9KgVBJzMYBUsg\\=\\=')['href'] if product.select_one('a.Nq8NlC5Hk9KgVBJzMYBUsg\\=\\=') else 'N/A' products.append({ 'name': name, 'price': price, 'store': store, 'product_url': product_url }) return products ``` This function first fetches the HTML using the Crawlbase Crawling API and then parses the data using BeautifulSoup to extract the product information. ### Handling Pagination in Tokopedia Tokopedia’s search results are spread across multiple pages. To scrape all listings, we need to handle pagination. Each subsequent `page` can be accessed by appending a page parameter to the URL, such as `?page=2`. Here’s how to handle pagination: ```python # Function to scrape multiple pages of search listings def scrape_multiple_pages(base_url, max_pages): all_products = [] for page in range(1, max_pages + 1): paginated_url = f"{base_url}&page={page}" html_content = fetch_html(paginated_url) if html_content: products = parse_search_listings(html_content) all_products.extend(products) else: break return all_products ``` This function loops through the search result pages, scrapes the product listings from each page, and aggregates the results. ### Storing Data in a JSON File After scraping the data, you can store it in a JSON file for easy access and future use. Here’s how you can do it: ```python # Function to save data to a JSON file def save_to_json(data, filename='tokopedia_search_results.json'): with open(filename, 'w') as json_file: json.dump(data, json_file, indent=4) print(f"Data saved to {filename}") ``` ### Complete Code Example Below is the complete code to scrape Tokopedia search listings for headsets, including pagination and saving the data to a JSON file: ```python from crawlbase import CrawlingAPI from bs4 import BeautifulSoup import json # Initialize Crawlbase API with your access token crawling_api = CrawlingAPI({ 'token': 'YOUR_CRAWLBASE_TOKEN' }) def fetch_html(url): options = { 'ajax_wait': 'true', 'page_wait': '5000' } response = crawling_api.get(url, options) if response['headers']['pc_status'] == '200': return response['body'].decode('utf-8') else: print(f"Failed to fetch page. Status code: {response['headers']['pc_status']}") return None def parse_search_listings(html): soup = BeautifulSoup(html, 'html.parser') products = [] for product in soup.select('div[data-testid="divSRPContentProducts"] div.css-5wh65g'): name = product.select_one('span.OWkG6oHwAppMn1hIBsC3pQ\\=\\=').text.strip() if product.select_one('span.OWkG6oHwAppMn1hIBsC3pQ\\=\\=') else 'N/A' price = product.select_one('div.ELhJqP-Bfiud3i5eBR8NWg\\=\\=').text.strip() if product.select_one('div.ELhJqP-Bfiud3i5eBR8NWg\\=\\=') else 'N/A' store = product.select_one('span.X6c-fdwuofj6zGvLKVUaNQ\\=\\=').text.strip() if product.select_one('span.X6c-fdwuofj6zGvLKVUaNQ\\=\\=') else 'N/A' product_url = product.select_one('a.Nq8NlC5Hk9KgVBJzMYBUsg\\=\\=')['href'] if product.select_one('a.Nq8NlC5Hk9KgVBJzMYBUsg\\=\\=') else 'N/A' products.append({ 'name': name, 'price': price, 'store': store, 'product_url': product_url }) return products def scrape_multiple_pages(base_url, max_pages): all_products = [] for page in range(1, max_pages + 1): paginated_url = f"{base_url}&page={page}" html_content = fetch_html(paginated_url) if html_content: products = parse_search_listings(html_content) all_products.extend(products) else: break return all_products def save_to_json(data, filename='tokopedia_search_results.json'): with open(filename, 'w') as json_file: json.dump(data, json_file, indent=4) print(f"Data saved to {filename}") # Scraping data from Tokopedia search listings base_url = 'https://www.tokopedia.com/search?q=headset' max_pages = 5 # Adjust the number of pages you want to scrape search_results = scrape_multiple_pages(base_url, max_pages) # Save results to a JSON file save_to_json(search_results) ``` Example Output: ```json [ { "name": "Ipega PG-R008 Gaming Headset for P4 /X1 series/N-Switch Lite/Mobile/ta", "price": "Rp178.000", "store": "ipegaofficial", "product_url": "https://www.tokopedia.com/ipegaofficial/ipega-pg-r008-gaming-headset-for-p4-x1-series-n-switch-lite-mobile-ta?extParam=ivf%3Dfalse&src=topads" }, { "name": "Hippo Toraz Handsfree Earphone Stereo Sound - headset, Putih", "price": "Rp13.000", "store": "HippoCenter", "product_url": "https://www.tokopedia.com/hippocenter88/hippo-toraz-handsfree-earphone-stereo-sound-headset-putih?extParam=ivf%3Dfalse&src=topads" }, { "name": "HEADSET ORIGINAL COPOTAN VIVO OPPO XIAOMI REALMI JACK 3.5MM SUPERBASS - OPPO", "price": "Rp5.250", "store": "BENUAACELL", "product_url": "https://www.tokopedia.com/bcbenuacell/headset-original-copotan-vivo-oppo-xiaomi-realmi-jack-3-5mm-superbass-oppo?extParam=ivf%3Dfalse&src=topads" }, { "name": "earphone bluetooth wireless headset gaming full bass", "price": "Rp225.000", "store": "Kopi 7 Huruf", "product_url": "https://www.tokopedia.com/kopi7huruf/earphone-bluetooth-wireless-headset-gaming-full-bass?extParam=ivf%3Dfalse&src=topads" }, { "name": "Earphone In-Ear 4D Stereo Super Bass dengan Mic with Kabel Jack 3.5mm Headset Crystal-Clear Sound - Putih", "price": "Rp15.000Rp188.000", "store": "MOCUTE STORE", "product_url": "https://www.tokopedia.com/mocutestore/earphone-in-ear-4d-stereo-super-bass-dengan-mic-with-kabel-jack-3-5mm-headset-crystal-clear-sound-putih-97573?extParam=ivf%3Dtrue&src=topads" }, .... more ] ``` In the next section, we’ll cover scraping individual product pages on Tokopedia to get detailed information. ## Scraping Tokopedia Product Pages Now that we have scraped search listings, let’s move on to scraping product details from individual product pages. In this section, we will scrape product name, price, store name, description and image URL from a Tokopedia product page. ### Inspecting the HTML for CSS Selectors Before we write the scraper, we need to inspect the HTML structure of the product page to find the correct CSS selectors for the data we want to scrape. For this example, we’ll scrape the product page from the following URL: ``` https://www.tokopedia.com/thebigboss/headset-bluetooth-tws-earphone-bluetooth-stereo-bass-tbb250-beige-8d839 ``` Open the developer tools in your browser and navigate to this URL. ![A screenshot of Tokopedia product page HTML structure](https://crawlbase.com/blog/how-to-scrape-tokopedia-data/tokopedia-product-page-inspect.jpg) Here's what we need to focus on: - **Product** Name: Found in an `

` tag with the attribute `data-testid="lblPDPDetailProductName"`. - Price: The price is located in a `
` tag with the attribute `data-testid="lblPDPDetailProductPrice"`. - Store Name: The store name is inside an `` tag with the attribute `data-testid="llbPDPFooterShopName"`. - Product Description: Located in a `
` tag with the attribute `data-testid="lblPDPDescriptionProduk"` which contains detailed information about the product. - Images URL: The main product image is found within a `

tokopediascraping

How to Scrape Tokopedia Data

Join Our Newsletter. No Spam, Only the good stuff.