Introducing web scraping

Simply put, web scraping is one of the tools developers use to gather and analyze information from the internet. Some websites and platforms offer application programming interfaces (APIs) which we can use to access information in a structured way, but others might not. While APIs are certainly becoming the standard way of interacting with today’s popular platforms, we don’t always have this luxury when interacting with most of the websites on the internet. Rather than reading data from standard API responses, we’ll need to find the data ourselves by reading the website’s pages and feeds.

Some uses

The World Wide Web was born in 1989 and web scraping and crawling entered the conversation not long after in 1993. Before scraping, search engines were compiled lists of links collected by the website administrator, and arranged into a long list of links somewhere on their website. The first web scraper and crawler, the World Wide Web Wanderer, were created to follow all these indexes and links to try and determine how big the internet was. It wasn’t long after this that developers started using crawlers and scrapers to create crawler-based search engines that didn’t require human assistance. These crawlers would simply follow links that would come across each page and save information about the page. Since the web is a collaborative effort, the crawler could easily and infinitely follow embedded links on websites to other platforms, and the process would continue forever.

Nowadays, web scraping has its place in nearly every industry. In newsrooms, web scrapers are used to pull in information and trends from thousands of different internet platforms in real time. Spending a little too much on Amazon this month? Websites exist that will let you know, and, in most cases, will do so by using web scraping to access that specific information on your behalf. Machine learning and artificial intelligence companies are scraping billions of social media posts to better learn how we communicate online.

So how does it work?

The process a developer builds for web scraping looks a lot like the process a user takes with a browser:

Read more here!