Web scraping is the automated process of extracting data from websites. You send HTTP requests, receive HTML responses, and parse the content to pull out exactly what you need. Teams use it for price monitoring, lead generation, academic research, training ML models, and competitive analysis.

The three building blocks are always the same regardless of which library you choose:

1. Fetch the page (HTTP GET request)
2. Parse the HTML (DOM traversal or CSS/XPath selectors)
3. Store the data (CSV, JSON, database)

The difference between beginner scraping and production scraping is mostly what happens between steps 1 and 2: dealing with login walls, JavaScript rendering, CAPTCHAs, and IP rate limits.

Four libraries cover the full range of scraping tasks in 2026. You don't need all four for every project. Start with requests and BeautifulSoup for simple static sites, then add the others when you hit their limits.

# Install all four in one command
pip install requests beautifulsoup4 lxml httpx parsel playwright

# Then install Playwright's browser binaries (Chromium, Firefox, WebKit)
playwright install chromium

The lxml package is the fastest HTML parser for BeautifulSoup. It's optional but worth including. The playwright install step downloads a bundled Chromium browser (about 130 MB), which Playwright controls directly.

Confirm everything works:

python -c "import requests, bs4, httpx, parsel, playwright; print('All libraries loaded')"

If you see All libraries loaded, you're ready. Any ModuleNotFoundError means a package didn't install. Re-run the pip install for that specific package.

Static pages are the easiest case. The full HTML is in the initial server response, so you just fetch it and parse it. requests handles the HTTP part; BeautifulSoup handles the parsing.

Here's a complete working scraper that pulls book titles and prices from books.toscrape.com, a public sandbox built specifically for scraping practice:

# scrape_books.py

import requests
from bs4 import BeautifulSoup
import csv

BASE_URL = "https://books.toscrape.com/catalogue/"

def get_books(page_url):
    headers = {
        "User-Agent": (
            "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
            "AppleWebKit/537.36 (KHTML, like Gecko) "
            "Chrome/124.0.0.0 Safari/537.36"
        )
    }
    response = requests.get(page_url, headers=headers, timeout=10)
    response.raise_for_status()  # raise an error for 4xx/5xx status codes

    soup = BeautifulSoup(response.text, "lxml")

    books = []
    for article in soup.select("article.product_pod"):
        title = article.h3.a["title"]
        price = article.select_one("p.price_color").text.strip()
        rating = article.p["class"][1]  # e.g., "Three"
        books.append({"title": title, "price": price, "rating": rating})

    return books

def get_next_page(soup):
    next_btn = soup.select_one("li.next a")
    return BASE_URL + next_btn["href"] if next_btn else None

def scrape_all_books():
    url = "https://books.toscrape.com/catalogue/page-1.html"
    all_books = []

    while url:
        print(f"Scraping: {url}")
        response = requests.get(url, timeout=10)
        soup = BeautifulSoup(response.text, "lxml")

        all_books.extend(get_books(url))
        url = get_next_page(soup)

    return all_books

if __name__ == "__main__":
    books = scrape_all_books()
    print(f"Scraped {len(books)} books")

    with open("books.csv", "w", newline="", encoding="utf-8") as f:
        writer = csv.DictWriter(f, fieldnames=["title", "price", "rating"])
        writer.writeheader()
        writer.writerows(books)

    print("Saved to books.csv")

Run it with python scrape_books.py. It'll page through all 50 pages and write a CSV with 1,000 book records.

Most modern websites load content after the initial HTML through JavaScript. React, Vue, Angular, and similar frameworks render the actual data client-side. When you fetch the raw HTML with requests, you get an empty shell. Playwright solves this by running a real browser and waiting for the page to fully render before you extract anything.

How do you know if a page needs Playwright? Open the page in Chrome, right-click, select "View Page Source" (Ctrl+U). If the content you want isn't visible in the raw source, the page is rendering it with JavaScript and you'll need a headless browser.

# scrape_js_page.py

import asyncio
from playwright.async_api import async_playwright

async def scrape_quotes():
    async with async_playwright() as p:
        # launch=False means headless (no visible browser window)
        browser = await p.chromium.launch(headless=True)
        page = await browser.new_page()

        # Set a realistic user agent
        await page.set_extra_http_headers({
            "User-Agent": (
                "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
                "AppleWebKit/537.36 (KHTML, like Gecko) "
                "Chrome/124.0.0.0 Safari/537.36"
            )
        })

        await page.goto("https://quotes.toscrape.com/js/", wait_until="networkidle")

        # Wait for the dynamic content to appear in the DOM
        await page.wait_for_selector("div.quote")

        quotes = await page.eval_on_selector_all(
            "div.quote",
            """elements => elements.map(el => ({
                text: el.querySelector("span.text").innerText,
                author: el.querySelector("small.author").innerText,
                tags: [...el.querySelectorAll("a.tag")].map(t => t.innerText)
            }))"""
        )

        await browser.close()
        return quotes

if __name__ == "__main__":
    results = asyncio.run(scrape_quotes())
    for q in results:
        print(f'{q["author"]}: {q["text"][:60]}...')
    print(f"\nTotal: {len(results)} quotes")

The key line is wait_until="networkidle". Playwright waits until there are no more than 2 network connections for at least 500ms, which means the page has finished loading its dynamic content.

For pages that load content on scroll (infinite scroll), replace networkidle with a manual scroll loop:

# Scroll to the bottom of the page three times
for _ in range(3):
    await page.keyboard.press("End")
    await page.wait_for_timeout(1500)  # wait 1.5s for new content to load

Personal Experience We've found that wait_for_selector is more reliable than networkidle for SPAs that keep background connections alive. Waiting for the specific element you want guarantees it's present in the DOM before you try to read it.

When you need to scrape hundreds or thousands of static pages, requests becomes the bottleneck. It's synchronous: one request runs, finishes, then the next starts. httpx supports async HTTP, so you can fire dozens of requests concurrently without spinning up browser instances.

# scrape_async.py

import asyncio
import httpx
from parsel import Selector
import json

URLS = [
    "https://books.toscrape.com/catalogue/page-1.html",
    "https://books.toscrape.com/catalogue/page-2.html",
    "https://books.toscrape.com/catalogue/page-3.html",
    "https://books.toscrape.com/catalogue/page-4.html",
    "https://books.toscrape.com/catalogue/page-5.html",
]

HEADERS = {
    "User-Agent": (
        "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) "
        "AppleWebKit/537.36 (KHTML, like Gecko) "
        "Chrome/124.0.0.0 Safari/537.36"
    )
}

async def fetch_page(client, url):
    response = await client.get(url, headers=HEADERS, timeout=15.0)
    response.raise_for_status()
    return response.text

def parse_books(html):
    sel = Selector(text=html)
    books = []
    for article in sel.css("article.product_pod"):
        books.append({
            "title": article.css("h3 a::attr(title)").get(),
            "price": article.css("p.price_color::text").get().strip(),
        })
    return books

async def scrape_pages(urls):
    all_books = []
    async with httpx.AsyncClient() as client:
        # Fetch all pages concurrently
        tasks = [fetch_page(client, url) for url in urls]
        pages = await asyncio.gather(*tasks)

    for html in pages:
        all_books.extend(parse_books(html))

    return all_books

if __name__ == "__main__":
    books = asyncio.run(scrape_pages(URLS))
    print(f"Scraped {len(books)} books from {len(URLS)} pages concurrently")

    with open("books_async.json", "w") as f:
        json.dump(books, f, indent=2)
    print("Saved to books_async.json")

The Parsel library uses the same CSS selector syntax as BeautifulSoup but adds full XPath support. The ::text and ::attr(name) pseudo-elements make extracting text and attributes much cleaner than BeautifulSoup's approach.

Compare the two styles:

# BeautifulSoup
title = article.css("h3 a")["title"]       # AttributeError if missing
title = article.find("h3").find("a")["title"]

# Parsel (safer)
title = article.css("h3 a::attr(title)").get()   # returns None if missing, not an error
title = article.xpath('.//h3/a/@title').get()     # same result via XPath

Parsel's .get() returns None on a miss instead of raising an exception. That matters for production scrapers where missing elements are expected.

Over 40% of all internet traffic is automated bot activity (Imperva Bad Bot Report, 2025). Sites know this, and most have countermeasures in place. A 403 Forbidden or CAPTCHA page doesn't mean you're blocked permanently; it means your request looks automated. The goal is to look like a real browser.

Four techniques cover the majority of anti-bot systems you'll encounter:

Every scraper hits the same wall of errors at some point. Here are the most frequent ones and how to get past them fast:

The most reliable approach for any unexplained block: open the browser DevTools Network tab, observe what a real browser sends in its request headers, and replicate those headers exactly in your script.

You now have four working patterns: requests + BeautifulSoup for static pages, Playwright for JavaScript-rendered content, httpx + Parsel for concurrent high-volume scraping, and proxy rotation for sites with IP rate limiting.

Start with the books.toscrape.com example, get comfortable with CSS selectors, then move to a real target you care about. The jump from tutorial to production mostly comes down to error handling, retry logic, and deciding how much stealth you need for your specific target.

Python scraping project ideas