TheLazy.Dev
Portfolio GitHub

Blog

How to create a web crawler with Beautiful Soup

kaveku

AngelaMunyao

Posted on April 29, 2023

How to create a web crawler with Beautiful Soup

Web crawling is an essential technique used in data extraction, indexing, and web scraping. Web crawlers are automated programs that navigate through websites, extract data, and save it for further processing. Python is an excellent language for building web crawlers, and the Beautiful Soup library is one of the most popular tools for web scraping.

In this article, we will show you how to create a web crawler using Python and Beautiful Soup. We will extract text data from a website and save it to a CSV file.

First, we need to import the required libraries. We will use the requests library to send GET requests to the website and the Beautiful Soup library to parse the HTML.

import requests
import csv
from bs4 import BeautifulSoup
Enter fullscreen mode Exit fullscreen mode 
url = "https://www.businessdailyafrica.com/"
Enter fullscreen mode Exit fullscreen mode

We will use the requests library to send a GET request to the website and store the response

response = requests.get(url)
Enter fullscreen mode Exit fullscreen mode

Now, we can create a Beautiful Soup object from the response HTML.

soup = BeautifulSoup(response.text, "html.parser")
Enter fullscreen mode Exit fullscreen mode

We will use the find_all method to extract all anchor tags from the HTML.

links = soup.find_all("a")
Enter fullscreen mode Exit fullscreen mode

We need to filter out links that do not belong to the website. We will create an empty list to store the internal links.

internal_links = []
Enter fullscreen mode Exit fullscreen mode

We will loop through all the links and append the internal links to the internal_links list.

for link in links:
    href = link.get("href")
    if href and href.startswith("/bd"):
        href1 = 'https://www.businessdailyafrica.com'+href
        internal_links.append(href1)
Enter fullscreen mode Exit fullscreen mode

Next, we will visit each internal link and repeat the process. We will create a set to store the visited links and a list to store the text data.

visited_links = set()
p_array = []
Enter fullscreen mode Exit fullscreen mode

We will open a CSV file to store the text data.

with open('bdailyps.csv', mode='w') as csv_file:
    fieldnames = ['words']
    writer = csv.DictWriter(csv_file, fieldnames=fieldnames)
    writer.writeheader()
Enter fullscreen mode Exit fullscreen mode

We will loop through each internal link and send a GET request to the website. We will create a Beautiful Soup object from the response HTML and extract all the paragraphs.

    for link in internal_links:
        if (len(visited_links) >= 100000 & len(p_array)>=10000001):
            break
        if link not in visited_links:
            visited_links.add(link)
            response = requests.get(link)
            soup = BeautifulSoup(response.text, "html.parser")
            h2s = soup.find_all("h2")
            ps = soup.find_all("p")
Enter fullscreen mode Exit fullscreen mode

We will extract all the links from the HTML and append the internal links to the internal_links list.

links = soup.find_all("a")
for sublink in links:
    href = sublink.get("href")
    if href and href.startswith("/bd"):
        href1 = 'https://www.businessdailyafrica.com'+href
        internal_links.append(href1)
Enter fullscreen mode Exit fullscreen mode

We will loop through all the paragraphs and extract the text. We will split the text into words and save each word to the p_array list and the CSV file.

for p in ps:
    paragraph =  p.text.strip()
    for word in paragraph.split():
        p_array.append(word)
        writer.writerow({'words': word})
Enter fullscreen mode Exit fullscreen mode

Finally, we will print the number of visited links.

print(f'{len(visited_links)} links visited.')
Enter fullscreen mode Exit fullscreen mode

A summary of the code:

import requests
import csv
from bs4 import BeautifulSoup

# Set the URL of the website to crawl
url = "https://www.businessdailyafrica.com/"

# Send a GET request to the URL and store the response
response = requests.get(url)

# Create a BeautifulSoup object from the response HTML
soup = BeautifulSoup(response.text, "html.parser")

# Find all anchor tags in the HTML
links = soup.find_all("a")

# Filter out links that do not belong to the website
internal_links = []
p_array = []
for link in links:
    href = link.get("href")
    if href and href.startswith("/bd"):
        href1 = 'https://www.businessdailyafrica.com'+href
        internal_links.append(href1)

# Visit each internal link and repeat the process
visited_links = set()
with open('bdailyps.csv', mode='w') as csv_file:
    fieldnames = ['words']
    writer = csv.DictWriter(csv_file, fieldnames=fieldnames)
    writer.writeheader()

    for link in internal_links:
        if (len(visited_links) >= 100000 & len(p_array)>=10000001):
            break
        if link not in visited_links:
            visited_links.add(link)
            response = requests.get(link)
            soup = BeautifulSoup(response.text, "html.parser")
            h2s = soup.find_all("h2")
            ps = soup.find_all("p")
            links = soup.find_all("a")
            for sublink in links:
                href = sublink.get("href")
                if href and href.startswith("/bd"):
                    href1 = 'https://www.businessdailyafrica.com'+href
                    internal_links.append(href1)

            # Print each h1 tag and the element directly below it
            for p in ps:
                print(link, p)
                paragraph =  p.text.strip()
                for word in paragraph.split():
                    p_array.append(word)
                    writer.writerow({'words': word})

# Print the number of visited links
print(f'{len(visited_links)} links visited.')
Enter fullscreen mode Exit fullscreen mode

That's it! We have created a web crawler using Beautiful Soup

💖 💪 🙅 🚩
kaveku
AngelaMunyao

Posted on April 29, 2023

Join Our Newsletter. No Spam, Only the good stuff.

Sign up to receive the latest update from our blog.

Related

I am purposefully super nice to AI because I imagine they are trained on data where nice people get better help. Doing this, though, has an existential uneasiness to it.
ai I am purposefully super nice to AI because I imagine they are trained on data where nice people get better help. Doing this, though, has an existential uneasiness to it.

November 28, 2024

If you're a beginner, definitely check this open source guide. I've explained almost everything you need to know.
undefined If you're a beginner, definitely check this open source guide. I've explained almost everything you need to know.

November 28, 2024

The best way to get better at writing code is...
development The best way to get better at writing code is...

November 28, 2024

AI Innovations at Microsoft Ignite 2024 What You Need to Know (Part 2)
githubcopilot AI Innovations at Microsoft Ignite 2024 What You Need to Know (Part 2)

November 29, 2024

Buying nothing, or how digital art is disappearing
discuss Buying nothing, or how digital art is disappearing

November 28, 2024