Scrape but Validate: Data scraping with Pydantic Validation

Note: Not an output of chatGPT/ LLM

Data scraping is process of collecting data from public web sources and it is mostly done using script in a automated way. Due to automation, often collected data have errors and need to filter out and clean for use. However, it will be better if scraped data can be validate during scraping.

Considering the data validation requirement, most of scraping framework like Scrapy have inbuilt pattern that can be used for data validation. However, many a time, during the data scraping process, we often just use general purpose modules like requests and beautifulsoup for scraping. In such case, it is hard to validate the collected data, so this blog post explain a simple approach for data scraping with validation using Pydantic.
https://docs.pydantic.dev/latest/
Pydantic is a data validation python module. It is the backbone of popular api module FastAPI too, like Pydantic there are other python modules too, that can be used for validation during data scraping. However, this blog explore pydantic and here are link of alternatives packages (you can try changing pydantic with any other module as a learning exercise )

Cerberus is a lightweight and extensible data validation library for Python. https://pypi.org/project/Cerberus/

Plan of scraping :

In this blog, we will scrap quotes from the quotes site.
We will use requests and beautifulsoup to get the data Will create a pydantic data class to validate each scraped data Save the filtered and validated data in a json file.

For better arrangement and understanding, each step is implemented as a python method that can be used under main section.

Basic import

import requests # for web request
from bs4 import BeautifulSoup # cleaning html content

# pydantic for validation

from pydantic import BaseModel, field_validator, ValidationError

import json

1. Target site and getting quotes

We are using (http://quotes.toscrape.com/) to scrape the quotes. Each quote will have three fields: quote_text, author, and tags. For example:

Below method is a general script to get html content for a given url.


def get_html_content(page_url: str) -> str:
    page_content =""
    # Send a GET request to the website
    response = requests.get(url)
    # Check if the request was successful (status code 200)
    if response.status_code == 200:
        page_content = response.content
    else:
        page_content = f'Failed to retrieve the webpage. Status code: {response.status_code}'
    return page_content

2. Get the quote data from scraping

We will use requests and beautifulsoup to scraped the data from given urls. The process is broken into three parts: 1) Get the html content from the web 2) Extract the desired html tags for each targeted fields 3) Get the values from each tags


def get_tags(tags):
    tags =[tag.get_text() for tag in tags.find_all('a')]
    return tags


def get_quotes_div(html_content:str) -> str :    
    # Parse the page content with BeautifulSoup
    soup = BeautifulSoup(html_content, 'html.parser')

    # Find all the quotes on the page
    quotes = soup.find_all('div', class_='quote')

    return quotes

Below script get the data point from each quote's div.

    # Loop through each quote and extract the text and author
    for quote in quotes_div:
        quote_text = quote.find('span', class_='text').get_text()
        author = quote.find('small', class_='author').get_text()
        tags = get_tags(quote.find('div', class_='tags'))

        # yied data to a dictonary 
        quote_temp ={'quote_text': quote_text,
                'author': author,
                'tags':tags
        }

3. Create Pydantic dataclass and Validate the data for each quote

As per each fields of the quote, create a pydantic class and use same class for data validation during data scraping.

The pydantic model Quote

Below is the Quote class that is extended from BaseModel having three fields like quote_text, author, and tags. Out of these three, quote_text and author are type of string (str) and tags is a list type.

We have two validator methods (with decorators):

1) tags_more_than_two () : Will check that it must have more than two tags. (it is just for example, you can have any rule here)

2.) check_quote_text(): This method will remove "" from quote and test for text.


class Quote(BaseModel):
    quote_text:str
    author:str
    tags: list

    @field_validator('tags')
    @classmethod
    def tags_more_than_two(cls, tags_list:list) -> list:
        if len(tags_list) <=2:
            raise ValueError("There should be more than two tags.")
        return tags_list

    @field_validator('quote_text')
    @classmethod    
    def check_quote_text(cls, quote_text:str) -> str:
        return quote_text.removeprefix('“').removesuffix('”')

Getting and validating data

Data validation is very easy with pydantic, for example, below code, pass scraped data to pydantic class Quote.

quote_data = Quote(**quote_temp)


def get_quotes_data(quotes_div: list) -> list:
    quotes_data = []

    # Loop through each quote and extract the text and author
    for quote in quotes_div:
        quote_text = quote.find('span', class_='text').get_text()
        author = quote.find('small', class_='author').get_text()
        tags = get_tags(quote.find('div', class_='tags'))

        # yied data to a dictonary 
        quote_temp ={'quote_text': quote_text,
                'author': author,
                'tags':tags
        }

        # validate data with Pydantic model
        try:
            quote_data = Quote(**quote_temp)            
            quotes_data.append(quote_data.model_dump())            
        except  ValidationError as e:
            print(e.json())
    return quotes_data

4. Store the data

Once data is validated that will be save to a json file. (A general purpose method is written that will convert Python dictionary to json file)


def list_of_dict_to_json(list_data_dict:list, filename:str)-> None:
    """This is a utiltiy method to write python dictonary data
    to a json file.

    Args:
        list_data_dict (list): Python list of dict having the data
        filename (str): Filename for the json file
    example_data = [
    {'name':'ajit', 'age':30},
    {'name':'Raushan', 'age':30}
    ]

    list_of_dict_to_json(example_data, 'names') # output: names.json
    """
    # check given filename, if ends with .json or not
    if not filename.endswith('.json'):
        filename = filename+'.json'

    # Write data to a JSON file
    with open(filename, 'w') as json_file:
        json.dump(list_data_dict, json_file, indent=4)

Putting all together

After understanding each piece of scraping, now , you can put all together and run the scraping for data collection.


if __name__ == '__main__':        
    # URL of the Quotes to Scrape website    
    url = 'http://quotes.toscrape.com/'
    # 1. get the content
    page_html = get_html_content(url)
    # 2. get the div for each quote
    quotes_div = get_quotes_div(page_html)
    # 3. validate and get the data point
    quotes_data = get_quotes_data(quotes_div)
    # 4. Store the data to json
    list_of_dict_to_json(quotes_data, 'quotes')

Note: A revision is planned, let me know your idea or suggestion to include in the revised version.

Links and resources:

Blog