Incremental Web Scraping and Change Detection

Picture a price monitor watching 50,000 product pages every day. On a typical day maybe 2 to 5 percent of them changed a price, dropped out of stock, or edited a title. The other 95-plus percent returned byte-for-byte the content you already had. A naive scraper renders all 50,000 anyway. On a metered API at 5 credits per render, that's 250,000 credits a day to learn that roughly 47,500 pages didn't move.

Incremental scraping flips the default. You assume a page is unchanged until something proves otherwise, and you spend the expensive full fetch only on the pages that clear that bar. Three signals prove change, from cheapest to most reliable:

• Discovery signals tell you which URLs might have changed without fetching them one by one. A sitemap's and an RSS feed's pubDate are the main two.
• Validators let the server confirm "not changed" in a single round trip with no body. Those are ETag and Last-Modified, checked with conditional requests.
• Fingerprints catch changes on servers that send no validators at all. You hash the content you fetched and compare it to the hash you stored.

Layer them and each request does the least work that still answers one question: did this page change? The rest of this guide wires all three into a single loop, backed by a state store that remembers the last run.

Read the table top to bottom and you have the request's decision tree. Discovery narrows the candidate set. A conditional request tries to settle it for free. If the server sends no validators, the fingerprint settles it. If something did change, the field diff records what. The state store threads through every layer, because none of these checks mean anything without a memory of the previous run.

The cheapest change signal is one you can read for an entire site in a single request. Most sites publish an XML sitemap, and the sitemap protocol defines an optional element holding the date a URL was last modified (sitemaps.org). Pull the sitemap, compare each against the last_seen timestamp in your state store, and you have a candidate list of changed URLs before touching a single product page.

import requests, xml.etree.ElementTree as ET
from datetime import datetime

NS = {"sm": "http://www.sitemaps.org/schemas/sitemap/0.9"}
API = "https://scrape.sparkproxy.io/api/v1"
HEADERS = {"X-API-Key": "YOUR_API_KEY"}

def changed_urls_from_sitemap(sitemap_url: str, seen: dict) -> list[str]:
    # render_js=false: a sitemap is plain XML, so this is a 1-credit fetch.
    r = requests.get(API, headers=HEADERS,
                     params={"url": sitemap_url, "render_js": "false"}, timeout=30)
    r.raise_for_status()
    root = ET.fromstring(r.text)

    candidates = []
    for url_el in root.findall("sm:url", NS):
        loc = url_el.findtext("sm:loc", namespaces=NS)
        lastmod_txt = url_el.findtext("sm:lastmod", namespaces=NS)
        if not loc:
            continue
        if lastmod_txt is None:
            candidates.append(loc)          # no hint, check it the normal way
            continue
        lastmod = datetime.fromisoformat(lastmod_txt.replace("Z", "+00:00"))
        prev = seen.get(loc, {}).get("last_seen")
        if prev is None or lastmod > prev:
            candidates.append(loc)          # new or moved since we last looked
    return candidates

Two cautions keep this honest. First, is self-reported by the site, and plenty of content management systems stamp it with the build time rather than a real content edit, so treat it as a hint that narrows the set, not proof. Google says it only trusts lastmod when a site keeps it consistently accurate (Google Search Central). Second, larger sites split the sitemap into an index of child sitemaps, so walk the index first, then each child.

For sources that publish new items rather than edit old ones (news, forums, job boards), an RSS or Atom feed is the better discovery source. Each entry carries a pubDate, and you fetch only entries newer than the last one you stored.

import feedparser  # pip install feedparser
from datetime import datetime, timezone

def new_items_from_feed(feed_url: str, watermark: datetime) -> list[str]:
    feed = feedparser.parse(feed_url)
    fresh = []
    for entry in feed.entries:
        published = datetime(*entry.published_parsed[:6], tzinfo=timezone.utc)
        if published > watermark:
            fresh.append(entry.link)
    return fresh

When a discovery signal flags a URL, don't download it blind. Ask the server whether it actually changed. HTTP has done this for decades through conditional requests (RFC 9110, Section 13). Two validators drive it:

• ETag is an opaque version token the server assigns to a response. Send it back in an If-None-Match header and the server compares: same token means 304 Not Modified with no body, a different token means 200 with the new content.
• Last-Modified is a timestamp. Send it back in If-Modified-Since and the server returns 304 if the page hasn't changed since that moment.

A 304 is the outcome you want. It carries no response body, so on self-managed proxies you pay almost no bandwidth, and the round trip confirms "nothing changed" from the source itself.

With the SparkProxy Scraping API you pass the stored validators through forward_headers and set transparent_status_code=true so a 304 from the target reaches you as a 304 instead of being normalized to 200. Keep render_js=false, because a validator check needs no browser and stays a 1-credit request.

import json, requests

def conditional_fetch(url: str, state: dict) -> requests.Response:
    cond = {}
    if state.get("etag"):
        cond["If-None-Match"] = state["etag"]
    if state.get("last_modified"):
        cond["If-Modified-Since"] = state["last_modified"]

    params = {
        "url": url,
        "render_js": "false",              # 1 credit: no render for a validator check
        "transparent_status_code": "true", # surface the target's real 304
    }
    if cond:
        params["forward_headers"] = json.dumps(cond)

    return requests.get(API, headers=HEADERS, params=params, timeout=30)

Then act on the outcome:

resp = conditional_fetch(url, state)

if resp.status_code == 304:
    pass                                   # unchanged, spend nothing more on it
elif resp.status_code == 200:
    etag = resp.headers.get("ETag")
    last_mod = resp.headers.get("Last-Modified")
    # A 200 does not prove a real change yet. Verify with a hash, then act.

One catch is worth internalizing: a 200 does not prove the content changed. Some servers ignore conditional headers outright, and others regenerate the page and its ETag on every request even when the visible content is identical. So 304 is trustworthy, and 200 is only a "maybe." That gap is what content hashing closes.

Many pages send no ETag, no Last-Modified, or hand you a fresh ETag on every hit. For those you fall back to the signal that always works: fingerprint the content and compare it to last time. Fetch the page, reduce it to a stable string, hash that string with SHA-256, and compare the hex digest against the one in your state store. Same digest, no change. Different digest, something moved.

The trap is hashing the raw response. Do that and nearly every page looks "changed" on every run, because real pages are full of per-request noise: CSRF tokens, session ids, cache-buster query strings, rotating ad slots, a "generated at" timestamp in the footer. Hash raw HTML and you defeat the entire point. Normalize first, then hash.

import hashlib

def content_hash(text: str) -> str:
    normalized = canonicalize(text)                 # strip noise (next section)
    return hashlib.sha256(normalized.encode("utf-8")).hexdigest()

def has_changed(text: str, state: dict) -> bool:
    new_hash = content_hash(text)
    old_hash = state.get("content_hash")
    return new_hash != old_hash                      # True on first sight (old is None)

A useful shortcut with a scraping API: request format=md instead of raw HTML for the hash input. SparkProxy's markdown conversion drops scripts, styles, and most attribute-level markup, so the response is already close to a stable fingerprint before you normalize it. Hashing the markdown of a page is far less jumpy than hashing its HTML.

Canonicalization is the step that decides what "changed" means for your use case. The goal is simple to state and fiddly to get right: two fetches a human would call identical must produce the same string. Strip or neutralize everything that varies without carrying meaning.

A practical canonicalizer for HTML does at least this much:

import re
from bs4 import BeautifulSoup

NOISE_SELECTORS = ["script", "style", "noscript", "svg",
                   ".ad", ".ads", "[data-ad]", ".timestamp", "#csrf-token"]

def canonicalize(html: str) -> str:
    soup = BeautifulSoup(html, "html.parser")
    for sel in NOISE_SELECTORS:
        for node in soup.select(sel):
            node.decompose()                          # remove known-volatile regions
    text = soup.get_text(separator=" ")
    text = re.sub(r"\s+", " ", text).strip()          # collapse whitespace
    text = re.sub(r"\b[0-9a-f]{32,}\b", "", text)     # drop long hex tokens/nonces
    return text.lower()

Tune the selector list per target. If a page shows a live "3 items in your cart" widget or a rotating testimonial, add it to the noise list or your diff will scream change on every run. The opposite mistake hurts more: if you canonicalize away the price because it sits inside a .badge you nuked, you'll never detect the one change you actually care about. Validate any canonicalization rule by running it twice against a page you know hasn't changed and confirming the hash holds steady.

A hash answers a yes or no question: did anything change? Often you need more than that. You need what changed. For a product page, "the price dropped from 49.99 to 39.99 and stock flipped to in-stock" is the real deliverable, not "this page is different." That is field-level diffing.

Extract the fields you care about into a small dict, store that dict, and compare it field by field on the next run. With SparkProxy you can pull the fields server-side using extract_rules, so you never parse HTML in your own code.

import requests, json

def fetch_fields(url: str) -> dict:
    rules = {
        "title": "h1",
        "price": "span.price",
        "in_stock": "div.availability",
    }
    r = requests.get(API, headers=HEADERS, params={
        "url": url,
        "render_js": "true",                 # render only once we know it changed
        "extract_rules": json.dumps(rules),
        "json_response": "true",
    }, timeout=60)
    r.raise_for_status()
    return r.json().get("data", {})

def diff_fields(old: dict, new: dict) -> dict:
    changes = {}
    for key in set(old) | set(new):
        before, after = old.get(key), new.get(key)
        if before != after:
            changes[key] = {"from": before, "to": after}
    return changes

Now a run emits a changeset you can act on: write a price-history row, fire an alert, or trigger a downstream job. Store the field dict next to the hash so the following diff has a baseline to compare against. The storage side of that (schema design and upserts that never double-insert) is covered in how to store scraped data; this guide focuses on deciding what is worth storing in the first place.

Every layer above needs a memory of the previous run, and that memory is the seen-store: one row per URL holding the validators, the content hash, and a couple of timestamps. SQLite is the right default for a single-machine job, and the schema is tiny.

CREATE TABLE IF NOT EXISTS seen (
  url            TEXT PRIMARY KEY,
  etag           TEXT,
  last_modified  TEXT,
  content_hash   TEXT,
  fields_json    TEXT,
  last_status    INTEGER,
  last_seen      TIMESTAMP NOT NULL,
  last_changed   TIMESTAMP
);

import sqlite3, json
from datetime import datetime, timezone

db = sqlite3.connect("seen.db")

def get_state(url: str) -> dict:
    row = db.execute(
        "SELECT etag, last_modified, content_hash, fields_json, last_seen "
        "FROM seen WHERE url = ?", (url,)).fetchone()
    if not row:
        return {}
    return {"etag": row[0], "last_modified": row[1], "content_hash": row[2],
            "fields": json.loads(row[3]) if row[3] else {},
            "last_seen": datetime.fromisoformat(row[4]) if row[4] else None}

def save_state(url, etag, last_modified, content_hash, fields, changed):
    now = datetime.now(timezone.utc).isoformat()
    db.execute("""
        INSERT INTO seen (url, etag, last_modified, content_hash, fields_json,
                          last_status, last_seen, last_changed)
        VALUES (?,?,?,?,?,?,?,?)
        ON CONFLICT(url) DO UPDATE SET
          etag=excluded.etag, last_modified=excluded.last_modified,
          content_hash=excluded.content_hash, fields_json=excluded.fields_json,
          last_status=excluded.last_status, last_seen=excluded.last_seen,
          last_changed=COALESCE(excluded.last_changed, seen.last_changed)
    """, (url, etag, last_modified, content_hash, json.dumps(fields),
          200, now, now if changed else None))
    db.commit()

The url primary key gives you the only lookup you need. On Postgres for a multi-writer job the same schema works with ON CONFLICT (url) DO UPDATE. Keep last_seen (touched every run) separate from last_changed (touched only on a real diff): the gap between them tells you how stale each page's data is, which is exactly the number you want when deciding how often to re-check a URL.

Some sources never edit old records, they only append new ones: a "newest first" paginated listing, an audit log, an API with a monotonically increasing id or an updated_at cursor. For those, per-URL hashing is overkill. Track a single high-water mark, the newest id or timestamp you've already ingested, and fetch only what sits beyond it.

def sync_append_only(list_url: str, watermark: str) -> tuple[list[dict], str]:
    """Fetch items newer than the stored high-water mark; return them plus the new mark."""
    r = requests.get(API, headers=HEADERS, params={
        "url": f"{list_url}?since={watermark}",
        "render_js": "false",
        "json_response": "true",
    }, timeout=60)
    r.raise_for_status()
    items = r.json().get("data", {}).get("items", [])

    fresh = [it for it in items if it["id"] > watermark]
    new_mark = max((it["id"] for it in fresh), default=watermark)
    return fresh, new_mark

Store the watermark next to your seen-store, in a one-row cursors table keyed by source. On the next run you resume exactly where you stopped, so the work per run stays constant no matter how large the source grows. Schedulers use this same idea to run jobs incrementally; the orchestration side of it is covered in how to schedule and automate web scrapers.

Here is where incremental scraping pays for itself, and where a scraping API changes the arithmetic. Rendering a page with render_js=true costs 5 credits on the standard tier, and up to 25 with a premium residential proxy and rendering combined, per the pricing in the SparkProxy Scraping API docs. A validator check with render_js=false costs 1. So the incremental pattern is a cost cascade:

1. Discovery (one fetch per sitemap) narrows 50,000 URLs to a few hundred candidates.
2. Conditional probe (1 credit each) turns most candidates into 304s that cost nothing more.
3. Hash check on the 200s that remain filters out the "changed ETag, identical content" false alarms for the price of one cheap fetch.
4. Full render (5 to 25 credits) runs only on the pages that genuinely changed.

Put numbers on it. A daily run over 50,000 pages where 3 percent truly change:

*1,500 renders at 5 credits (7,500) plus 50,000 conditional or hash probes at 1 credit (50,000) is 57,500, about 4.3x less than re-rendering everything. Sites that honor 304 push the probe cost lower still, since a confirmed 304 never escalates to a render.

On self-managed proxies the same win shows up as bandwidth. A 304 transfers headers only, not the page body, so you stop paying to download megabytes you already hold. Either way you also cut block rate, because you send a fraction of the requests and the ones you skip were the most pointless. For the broader set of tactics that keep a high-volume crawl under a target's radar, see scraping high-volume data without rate limiting.

Now assemble the layers into the function your scheduler calls per URL. It reads prior state, tries a conditional request, escalates to a hash check only on a 200, renders only on a real change, diffs the fields, and writes the new state back.

def incremental_fetch(url: str) -> dict:
    state = get_state(url)

    # 1. Cheap conditional probe (1 credit, no render).
    resp = conditional_fetch(url, state)

    if resp.status_code == 304:
        save_state(url, state.get("etag"), state.get("last_modified"),
                   state.get("content_hash"), state.get("fields", {}), changed=False)
        return {"url": url, "status": "unchanged", "via": "304"}

    if resp.status_code != 200:
        return {"url": url, "status": "error", "code": resp.status_code}

    # 2. A 200 doesn't prove change. Hash the normalized body to be sure.
    if not has_changed(resp.text, state):
        save_state(url, resp.headers.get("ETag"), resp.headers.get("Last-Modified"),
                   state.get("content_hash"), state.get("fields", {}), changed=False)
        return {"url": url, "status": "unchanged", "via": "hash"}

    # 3. Real change. Now spend the expensive render plus field extract.
    fields = fetch_fields(url)
    changes = diff_fields(state.get("fields", {}), fields)
    save_state(url, resp.headers.get("ETag"), resp.headers.get("Last-Modified"),
               content_hash(resp.text), fields, changed=True)
    return {"url": url, "status": "changed", "changes": changes}

The control flow is the whole thesis on one screen. A 304 costs one cheap credit and exits. A 200 with an unchanged hash costs one cheap credit and exits. Only a genuine change pays for a render and a field extract. Run this behind a scheduler with retry and backoff on the transient failures (see retry and backoff strategies for web scraping), and a job that used to re-crawl everything now touches only what moved.