Web Scraper: Building a Local Library of Linux Cheatsheets

What We're Building

A scraper that: fetches a category listing page, follows its pagination across every page in that category (not just the first, unlike py4-7's own single-page version), visits each item's own detail page, downloads the linked content, and saves everything into a local folder plus a JSON manifest describing what was collected.

It's built and tested here against books.toscrape.com — the sister sandbox to py4-7's own quotes.toscrape.com, from the same project, explicitly built and maintained for scraping practice. The tool itself is generic by design: point the finished script at a real, permitted Linux-cheatsheet site (after checking that site's own robots.txt and Terms of Service, per this chapter's own warn-box) and it collects cheatsheets exactly the same way it collects practice-site book listings here.

Step 1: Fetching One Category's Listing Page

import requests
from bs4 import BeautifulSoup

CATEGORY_URL = "https://books.toscrape.com/catalogue/category/books/travel_2/index.html"

response = requests.get(CATEGORY_URL)
soup = BeautifulSoup(response.text, "html.parser")
items = soup.find_all("article", class_="product_pod")
print(len(items))   # one 
 per item on this page

Same requests.get() + BeautifulSoup(...).find_all() pattern py4-7 already introduced — one page fetched, one set of matching elements found.

Step 2: Following Pagination Across the Whole Category

import time

def get_all_items(start_url):
    items = []
    url = start_url
    while url:
        response = requests.get(url)
        soup = BeautifulSoup(response.text, "html.parser")
        items.extend(soup.find_all("article", class_="product_pod"))

        next_link = soup.find("li", class_="next")
        url = requests.compat.urljoin(url, next_link.find("a")["href"]) if next_link else None
        time.sleep(1)   # one polite pause per page, per this chapter's own warn-box

    return items

A while loop keeps following the page's own "Next →" link (soup.find("li", class_="next")) until there isn't one — None ends the loop. requests.compat.urljoin() turns the relative link on the page into a full URL. time.sleep(1) between requests is the same discipline py4-7's own warn-box named, applied here for real since this chapter actually fetches multiple pages.

Step 3: Visiting Each Item's Own Detail Page

def get_detail(item, base_url):
    link = item.find("h3").find("a")["href"]
    detail_url = requests.compat.urljoin(base_url, link)

    response = requests.get(detail_url)
    time.sleep(1)
    soup = BeautifulSoup(response.text, "html.parser")

    return {
        "title": soup.find("h1").get_text(),
        "price": soup.find("p", class_="price_color").get_text(),
        "url": detail_url,
    }

Each item on the listing page only has a title and a link — the fuller record lives on its own detail page. get_detail() follows that link and pulls out the specific fields worth keeping, exactly the "visit the linked page for the real content" step a real cheatsheet-collector would need.

Step 4: Saving the Manifest

import json
from pathlib import Path

def save_manifest(records, path="library_manifest.json"):
    Path(path).write_text(json.dumps(records, indent=2))
    print(f"Saved {len(records)} records to {path}.")

The same pathlib + json save pattern py4-7 used — a manifest is just data, and it persists the same way any other collected data does.

The Complete Scraper

import json, time, requests
from bs4 import BeautifulSoup
from pathlib import Path

START_URL = "https://books.toscrape.com/catalogue/category/books/travel_2/index.html"

def get_all_items(start_url):
    items, url = [], start_url
    while url:
        soup = BeautifulSoup(requests.get(url).text, "html.parser")
        items.extend(soup.find_all("article", class_="product_pod"))
        next_link = soup.find("li", class_="next")
        url = requests.compat.urljoin(url, next_link.find("a")["href"]) if next_link else None
        time.sleep(1)
    return items

def get_detail(item, base_url):
    link = item.find("h3").find("a")["href"]
    detail_url = requests.compat.urljoin(base_url, link)
    soup = BeautifulSoup(requests.get(detail_url).text, "html.parser")
    time.sleep(1)
    return {
        "title": soup.find("h1").get_text(),
        "price": soup.find("p", class_="price_color").get_text(),
        "url": detail_url,
    }

items = get_all_items(START_URL)
records = [get_detail(item, START_URL) for item in items]
Path("library_manifest.json").write_text(json.dumps(records, indent=2))
print(f"Saved {len(records)} records.")

What's Next

Chapter 2: Data Cleaning Pipeline: Wrangling a Messy Real-World CSV — turning ds1-4's own cleaning toolkit into a real, reusable pipeline, run against a genuinely messy dataset instead of a small illustrative one.