What Is Web Scraping? How It Works, Tools & Legal Guide (2026)

The scraper sends an HTTP GET (or POST) request to the target URL. At minimum this includes a User-Agent header. Realistic scrapers also send Accept-Language, Accept-Encoding, Referer, and browser-matching sec-ch-ua client hint headers — because anti-bot systems fingerprint the header set as reliably as the IP address.

The server returns HTML (and for dynamic sites, JavaScript that must execute to render the final DOM). Static scrapers parse the raw HTML directly. Dynamic scrapers hand the response to a headless browser that executes the JavaScript and waits for the page to reach a stable state before extracting data.

The HTML is parsed into a tree structure (the DOM). The scraper then uses CSS selectors (soup.select("div.price")) or XPath expressions (//div[@class="price"]) to locate the specific nodes containing target data. This is the most maintenance-intensive step — when a site redesigns its HTML structure, selectors break and need updating.

Extracted data is written to a destination: CSV, JSON, PostgreSQL, BigQuery, or a real-time pipeline like Kafka. A cleaning step normalises formats — converting price strings to floats, parsing dates, deduplicating records. Production pipelines add schema validation here to catch extraction failures before corrupt data reaches downstream systems.

# Minimal working scraper — Python + requests + BeautifulSoup
import requests
from bs4 import BeautifulSoup

url = "https://example.com/products"
headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
                  "AppleWebKit/537.36 (KHTML, like Gecko) "
                  "Chrome/124.0.0.0 Safari/537.36"
}

response = requests.get(url, headers=headers, timeout=15)
soup = BeautifulSoup(response.text, "html.parser")

# Extract all product names and prices
for item in soup.select("div.product-card"):
    name  = item.select_one("h2.product-title").text.strip()
    price = item.select_one("span.price").text.strip()
    print(f"{name}: {price}")

Works when the server sends fully-rendered HTML — no JavaScript execution needed. Use requests (Python HTTP client) plus BeautifulSoup or lxml for parsing. This is the simplest, fastest, and cheapest technique. It's still effective on news sites, blogs, government data portals, and many e-commerce product pages. Doesn't work on React/Vue/Angular SPAs where content loads via client-side JavaScript after the initial HTML.

Playwright and Selenium launch a real browser engine (Chromium or Firefox) in headless mode, execute all JavaScript on the page, and expose the rendered DOM. Necessary for SPAs, infinite-scroll feeds, and any site where critical data loads via client-side fetch calls. Slower (3–10× vs. static scraping) and more expensive to run at scale, but the only reliable option for JavaScript-gated content.

Many modern sites load data via internal REST or GraphQL APIs. By inspecting the browser's Network tab, you can identify the API endpoint (often /api/v1/products?page=1) and query it directly — getting clean JSON without HTML parsing. This produces more stable scrapers because API schemas change less often than HTML layouts. Works on roughly 40% of modern web apps.

For scenarios requiring authenticated sessions (scraping your own CRM, exporting data from SaaS tools with no export feature), injecting JavaScript into a page via a browser extension or via page.evaluate() in Playwright can extract data already rendered in the DOM without making additional network requests. Useful for single-user automation, not scalable for bulk collection.

The average e-commerce site is scraped 8–12 times per day by competitor intelligence tools (Oxylabs Research, 2024). Retailers use this data to feed dynamic repricing algorithms — Amazon alone runs millions of price adjustments per day, largely informed by scraped competitor data.

Hedge funds scrape earnings call transcripts, SEC filings, job postings (as a leading economic indicator), and news sentiment at scale. Quant firms treat scraped data as an alternative data source — often giving them a 6–24 hour edge over competitors relying on traditional data vendors.

Large language models are trained on scraped datasets — CommonCrawl, the Pile, and C4 are all products of web-scale scraping. Academic researchers scrape social media, news archives, and government portals to build datasets for social science, computational linguistics, and public health studies.

• Scraping data publicly visible without logging in (prices, listings, public posts, open government data)
• Scraping for research, journalism, analysis, or competitive intelligence
• Scraping data you have a legitimate interest in collecting (your own product reviews, public mentions of your brand)

• Authenticated pages — scraping behind a login wall after accepting a ToS is a CFAA violation risk (the authorization analysis changes)
• ToS violations — even for public data, violating Terms of Service can lead to civil breach-of-contract claims (lower bar than CFAA)
• Personal data in the EU — collecting EU residents' personal data triggers GDPR obligations: legal basis, data minimization, retention limits
• Copyright — raw facts aren't copyrightable, but creative compilations can be; scraping a database's unique selection and arrangement may infringe

Best practice: Check robots.txt before scraping (disrespecting it won't incur criminal liability, but courts have cited it in ToS claims). Rate-limit requests to avoid overloading servers (DDoS-adjacent behaviour creates liability regardless of data legality). Never scrape personal data without a GDPR-compliant legal basis.

→ See also: What Is a Web Scraping Proxy? — Rotating IPs to scrape without IP bans