Simple Web Scraper

Python Projects — Simple Web Scraper
Python Projects (Beginner)
Course 4 · Chapter 7 · Simple Web Scraper

🕸️ Simple Web Scraper

Every project so far has used only the standard library. This one introduces two genuinely popular third-party packages — requests for fetching a web page, and BeautifulSoup for parsing its HTML — putting Course 1, Chapter 9's pip and venv coverage to real use for the first time in this course.

🎮 What We're Building

A scraper for quotes.toscrape.com — a site built specifically as safe, legal scraping practice — that fetches the page, pulls out every quote's text and author, and saves the results to a JSON file.

Step 1: Installing the Libraries

# with a venv active (Course 1, Chapter 9): pip install requests beautifulsoup4

Two packages installed in one pip install call — requests handles the network request, beautifulsoup4 (imported as bs4) handles parsing the returned HTML into something searchable.

Step 2: Fetching a Page with requests

import requests response = requests.get("https://quotes.toscrape.com") print(response.status_code) # 200 — success print(response.text[:200]) # the raw HTML, as a string

requests.get(url) sends an HTTP request and returns a Response object. status_code follows the same HTTP status conventions from the site's own API-focused courses (200 = success, 404 = not found); .text is the full HTML the server sent back, as a plain string.

Step 3: Parsing HTML with BeautifulSoup

from bs4 import BeautifulSoup soup = BeautifulSoup(response.text, "html.parser") quote_divs = soup.find_all("div", class_="quote") print(len(quote_divs)) # 10 — one per quote on the page

BeautifulSoup(html, "html.parser") turns the raw HTML string into a navigable object. .find_all("div", class_="quote") searches for every <div class="quote"> element — the trailing underscore in class_ avoids colliding with Python's own class keyword (Course 2, Chapter 1).

Step 4: Extracting the Data

quotes = [] for div in quote_divs: text = div.find("span", class_="text").get_text() author = div.find("small", class_="author").get_text() quotes.append({"text": text, "author": author}) print(quotes[0]) # {'text': '"The world as we have created it...', 'author': 'Albert Einstein'}

Within each quote's div, .find() (singular — one match) locates the specific <span>/<small> tags holding the text and author; .get_text() extracts just the readable text, stripping the surrounding HTML tags. quotes.append({...}) reuses the exact list-of-dicts pattern from Chapters 4-6's to-do app and contact book.

Step 5: Saving the Results

import json from pathlib import Path Path("quotes.json").write_text(json.dumps(quotes, indent=2))

The same pathlib + json save pattern from every earlier project in this course — scraped data is just data, and it persists the same way hand-entered data does.

🏁 The Complete Web Scraper

import json import requests from bs4 import BeautifulSoup from pathlib import Path response = requests.get("https://quotes.toscrape.com") soup = BeautifulSoup(response.text, "html.parser") quote_divs = soup.find_all("div", class_="quote") quotes = [] for div in quote_divs: text = div.find("span", class_="text").get_text() author = div.find("small", class_="author").get_text() quotes.append({"text": text, "author": author}) Path("quotes.json").write_text(json.dumps(quotes, indent=2)) print(f"Saved {len(quotes)} quotes.")
⚠ Scraping Has Real Legal and Ethical Limits

Not every site allows scraping — many explicitly forbid it in their Terms of Service, and even where it's technically allowed, hammering a server with rapid repeated requests can degrade it for real users or get your IP address blocked. Before scraping any real site: check its /robots.txt file (a standard, machine-readable statement of what's allowed), read its Terms of Service, and add a delay between requests (time.sleep(), Course 3, Chapter 3) if fetching multiple pages. quotes.toscrape.com is used here specifically because it's built and maintained for scraping practice — that permission doesn't transfer to other sites.

requests.get()

Fetches a page — .status_code and .text on the response.

BeautifulSoup

Parses HTML into a searchable structure.

find_all() / find()

Locate matching elements by tag name and class.

Scraping ethics

Check robots.txt and Terms of Service; don't hammer a server.

🚀 Extend This Project

Try these on your own:

  • Follow the "Next →" link on each page to scrape all 10 pages of quotes, not just the first — adding a time.sleep(1) between requests, per this chapter's warn-box.
  • Wrap requests.get() in try/except (Course 2, Chapter 3) to handle a network failure gracefully instead of crashing.
  • Also scrape each quote's tags (class_="tags"), adding a "tags" key to each quote's dict.
  • Count and print how many quotes each unique author has, using a dict as a counter (Course 1, Chapter 7's word-frequency pattern).

🎯 What's Next

Final chapter: Capstone: Expense Tracker — a small CLI app tying together file I/O, functions, and error handling into one complete, slightly larger project.