Website Returning Fake Data to Scraper? Catch Soft Blocks

Nobody gets paged for a soft block. The scrape finishes, status codes are all 200, row counts look normal, and the first person to notice is an analyst asking why a competitor cut prices by 30% on a Tuesday. These are the patterns we see most when a target is feeding a scraper something other than the real page:

• Prices that are close but not right. Off by a fixed percentage, rounded to a different ending (.00 instead of .99), or frozen at last month's value while the live site moved.
• Availability that flips in bulk. Dozens of SKUs go to Currently unavailable or Out of stock in one run and come back the next, with no matching change on the site.
• Result lists that stop short. A category that normally yields 40 pages yields 2. Search for a common term returns 0 results found or "We couldn't find any matches".
• Content that is fluent and off-topic. Pages that parse, contain paragraphs, and have nothing to do with the URL you asked for.

The common thread: your parser succeeded. The response is well-formed, so every structural alarm stays quiet. If you have not yet built structural detection for interstitials and empty containers, start with how to detect when your scraper is blocked, which covers baselines, selector contracts and skeleton hashes. This post picks up where those stop: the response has the right shape and the wrong values.

A soft block is any defensive response that returns success instead of refusal. Defenders do this deliberately, because a scraper that gets a 403 adapts in minutes and a scraper that gets plausible data may never adapt at all. It is a documented product feature, not folklore. Akamai's Bot Manager has a configurable "serve alternate" action (it appears as the akamai_botman_serve_alternate_action resource in Akamai's own Terraform provider docs). Cloudflare's AI Labyrinth announcement (published 19 March 2025) says that "rather than blocking the request, we will link to a series of AI-generated pages", and that the feature is an opt-in toggle open to all customers, including the Free plan.

None of these is guaranteed to be malicious. A price can legitimately differ by region, stock can really run out, and a search index can genuinely be thin. Every check below is built to separate "the site showed everyone this" from "the site showed you this".

Run the checks in this order. Each one is cheaper than the next, and the first three need no extra requests at all.

The cost column matters for budget planning. On SparkProxy proxy plans the billing unit is the plan and its concurrency, with unlimited bandwidth included, so a cross-IP vote costs you concurrency slots rather than gigabytes. On the Scraping API every extra fetch is credits, so the vote runs on a sample, never the full crawl.

A modern product page states the same facts in several places: the visible DOM, a JSON-LD Product block for search engines, and often a hydration blob such as __NEXT_DATA__ that the front end renders from. A defender that rewrites the visible price does not always rewrite all three. When they disagree, you don't need to know which one lied. You already know the row cannot ship.

import json
import re
from bs4 import BeautifulSoup

PRICE_RE = re.compile(r"(\d[\d,]*\.?\d*)")

def to_float(text):
    m = PRICE_RE.search(text or "")
    return float(m.group(1).replace(",", "")) if m else None

def price_layers(html):
    """Return the price and availability as stated by each layer of one page."""
    soup = BeautifulSoup(html, "lxml")
    layers = {}

    node = soup.select_one("[data-testid='price'], .price")
    if node:
        layers["dom_price"] = to_float(node.get_text())

    for tag in soup.select("script[type='application/ld+json']"):
        try:
            data = json.loads(tag.string or "")
        except json.JSONDecodeError:
            continue
        items = data if isinstance(data, list) else [data]
        for item in items:
            if item.get("@type") == "Product":
                offer = item.get("offers") or {}
                if isinstance(offer, list):
                    offer = offer[0] if offer else {}
                layers["ld_price"] = to_float(str(offer.get("price", "")))
                layers["ld_availability"] = str(offer.get("availability", "")).rsplit("/", 1)[-1]

    blob = soup.select_one("script#__NEXT_DATA__")
    if blob and blob.string:
        m = re.search(r'"price"\s*:\s*"?(\d[\d.]*)', blob.string)
        if m:
            layers["state_price"] = float(m.group(1))
    return layers

Then assert the invariants. Prices must agree across layers within a cent, a sale price must sit below the list price, and the stock badge must agree with whether a buy button exists:

def consistency_failures(layers, has_add_to_cart, list_price=None):
    fails = []
    prices = {k: v for k, v in layers.items() if k.endswith("_price") and v is not None}
    if len(set(round(v, 2) for v in prices.values())) > 1:
        fails.append(f"price layers disagree: {prices}")

    sale = layers.get("dom_price")
    if list_price is not None and sale is not None and sale > list_price:
        fails.append(f"sale price {sale} above list price {list_price}")

    ld_stock = layers.get("ld_availability")
    if ld_stock in ("OutOfStock", "SoldOut", "Discontinued") and has_add_to_cart:
        fails.append(f"structured data says {ld_stock} but an add-to-cart button is present")
    if ld_stock == "InStock" and not has_add_to_cart:
        fails.append("structured data says InStock but there is no add-to-cart button")
    return fails

The availability names come from schema.org's ItemAvailability enumeration, which lists twelve members including InStock, OutOfStock, SoldOut, BackOrder and PreOrder. Treat BackOrder and PreOrder as "purchasable" if the site still shows a button for them, or you'll raise false alarms on every preorder launch.

A self-consistency failure is a strong signal and a cheap one. A self-consistency pass is weak, since a competent defender can fake all layers at once. That's why the next checks exist.

Listing pages usually tell you how big they are. "Showing 1, 24 of 1,312 results", a totalResults field in the JSON the page fetches, a page count in the pagination bar. Truncation shows up as arithmetic that doesn't close:

import math
import re

TOTAL_RE = re.compile(r"of\s+([\d,]+)\s+results", re.I)

def truncation_check(first_page_html, pages_fetched, items_seen, per_page):
    m = TOTAL_RE.search(first_page_html)
    if not m:
        return {"verdict": "no_total", "detail": "page states no total; compare against history"}
    stated = int(m.group(1).replace(",", ""))
    expected_pages = math.ceil(stated / per_page)
    # Allow a little drift: items sell out between page 1 and page N.
    coverage = items_seen / stated if stated else 1.0
    if pages_fetched < expected_pages and coverage < 0.95:
        return {"verdict": "truncated", "stated": stated, "seen": items_seen,
                "pages": f"{pages_fetched}/{expected_pages}"}
    return {"verdict": "ok", "stated": stated, "seen": items_seen}

Three variants of this check are worth adding once the basic one runs:

• The total itself shrank. Some targets lower the stated total for suspected bots so the arithmetic closes. Keep the stated total per category per day and alert when it drops by more than a few percent overnight while the same query in a browser does not.
• Page N repeats page 1. Hash the item IDs on each page. If page 7 has the same ID set as page 1, the site is cycling you rather than ending the list.
• Zero results on a query that never returns zero. Keep a short list of queries that always have results (your own brand name on a marketplace, a common category term) and treat an empty answer as a block, not a fact.

The blocked-scraper guide makes the case for canary URLs with a fixed known value, such as a static title. For fake-data problems you need the harder version: a canary on the field that gets poisoned, which is usually the price. A price changes, so the canary needs a tolerance and a fresh truth source.

Pick 10 to 30 SKUs per target and refresh their correct values out of band, either by a person checking in a normal browser at a set time, or by your own purchase records, or from a partner feed you trust. Then check every scraped run against them:

from dataclasses import dataclass
from datetime import datetime, timedelta, timezone

@dataclass
class Truth:
    sku: str
    price: float
    in_stock: bool
    checked_at: datetime

def canary_report(truths, scraped, tolerance=0.01, max_age_hours=24):
    """scraped: {sku: {"price": float, "in_stock": bool}} from this run."""
    now = datetime.now(timezone.utc)
    stale, wrong = [], []
    for t in truths:
        if now - t.checked_at > timedelta(hours=max_age_hours):
            stale.append(t.sku)
            continue
        got = scraped.get(t.sku)
        if got is None:
            wrong.append((t.sku, "missing from run"))
        elif abs(got["price"] - t.price) > max(tolerance * t.price, 0.01):
            wrong.append((t.sku, f"price {got['price']} vs truth {t.price}"))
        elif got["in_stock"] != t.in_stock:
            wrong.append((t.sku, f"stock {got['in_stock']} vs truth {t.in_stock}"))
    checked = len(truths) - len(stale)
    return {"checked": checked, "wrong": wrong, "stale": stale,
            "fail": checked > 0 and len(wrong) / checked > 0.1}

Two design choices carry most of the value. The canaries must be scraped in the same job, through the same exits, mixed into the normal URL list, so the target cannot tell them apart from the rest. And stale truth is reported as stale rather than silently passing, because a canary whose truth is three days old will flag every legitimate price change as poisoning and get muted within a week.

Canaries only cover the SKUs you track. To catch poisoning anywhere in the catalogue, sample a few percent of URLs and fetch each one through two more exits. If the target is poisoning per IP, the poisoned answer is the odd one out. With a rotating gateway this is cheap, but there's a catch that trips most first attempts: an HTTPS request travels inside a CONNECT tunnel that the gateway cannot see into, so Random Rotation on gateway.sparkproxy.io:11000 can only pick an exit when a new tunnel opens. Reuse a pooled connection and all three fetches leave from the same IP. Open a fresh connection for each vote.

import requests
from collections import Counter

PROXY = "http://USER:PASS@gateway.sparkproxy.io:11000"   # Random Rotation

def fetch_fresh(url):
    """New session plus Connection: close, so the gateway picks a new exit."""
    with requests.Session() as s:
        s.headers["Connection"] = "close"
        r = s.get(url, proxies={"http": PROXY, "https": PROXY}, timeout=30)
        r.raise_for_status()
        return r.text

def vote(url, extract, n=3):
    """extract(html) -> hashable value, e.g. (price, in_stock)."""
    answers = []
    for _ in range(n):
        try:
            answers.append(extract(fetch_fresh(url)))
        except requests.RequestException as exc:
            answers.append(("error", type(exc).__name__))
    counts = Counter(answers)
    value, votes = counts.most_common(1)[0]
    return {"url": url, "answers": answers,
            "consensus": value if votes >= 2 else None,
            "dissent": votes < n}

Read the result carefully. Three agreeing answers do not prove the value is true; they prove the target gave the same answer to three exits. Two agreeing and one dissenting points at a flagged exit, and the dissenting row should be dropped and its IP logged. Three different answers on one URL is the pattern that deserves a human: either the page is genuinely volatile (flash pricing, A/B tests) or poisoning is widespread.

Two refinements stop the vote from lying to you. First, pin the country. If your three exits land in three countries you'll measure geo pricing, not poisoning, which is a separate job covered in proxies for geo-pricing audits. Second, make the exits genuinely independent. Three IPs from one /24 share reputation, so a range-level flag poisons all three identically and the vote passes. Log the exit IP for each answer and require the voters to come from different subnets.

The same vote through the Scraping API looks like this. json_response=true returns the envelope with status_code, duration_ms, credits_used and a body field, and that body is base64, so decode it before you extract anything:

import base64
import requests

API = "https://scrape.sparkproxy.io/api/v1"
HEADERS = {"X-API-Key": "YOUR_API_KEY"}

def api_fetch(url, premium=False):
    params = {
        "url": url,
        "render_js": "false",        # plain fetch, 1 credit on the rotating pool
        "country_code": "US",        # +5 credits, keeps all voters in one market
        "json_response": "true",
    }
    if premium:
        params["proxy_type"] = "premium"
    r = requests.get(API, headers=HEADERS, params=params, timeout=90)
    r.raise_for_status()
    env = r.json()
    html = base64.b64decode(env["body"]).decode("utf-8", "replace")
    return {"html": html, "status": env.get("status_code"),
            "credits": env.get("credits_used"), "ms": env.get("duration_ms")}

Making one of the three voters a proxy_type=premium fetch gives you a voter from a different pool, which addresses the "whole pool flagged together" blind spot. Per the Scraping API docs read on 30 October 2026, the premium pool costs 10 credits without JS rendering and 25 with it, against 1 and 5 on the default rotating pool, so keep premium for the tie-breaker. Note that session_id on the API labels logs only and does not pin an exit, so it won't help you hold one IP across a vote.

If the target only renders prices client-side, switch to render_js=true (the default, 5 credits). Don't add extract_rules to the call above: it replaces the response with an extracted object, so it doesn't combine with json_response.

Decoy content is the soft block that pollutes crawls rather than price feeds. Cloudflare's AI Labyrinth post describes the mechanism plainly: links added to real pages that "remain invisible to human visitors", leading to generated pages that are not the site's real content, with the stated goal of identifying crawlers that follow them. A crawler that follows every href it parses walks straight in, and its link graph fills with junk.

Two filters handle most of it. Drop links a human could not see or click, and drop pages whose text has nothing in common with what the URL promised:

from bs4 import BeautifulSoup

HIDDEN_STYLE = ("display:none", "visibility:hidden", "opacity:0", "left:-9999px")

def visible_links(html):
    soup = BeautifulSoup(html, "lxml")
    keep = []
    for a in soup.find_all("a", href=True):
        chain = [a] + list(a.parents)
        hidden = False
        for el in chain:
            if not hasattr(el, "get"):
                continue
            style = (el.get("style") or "").replace(" ", "").lower()
            if (el.get("hidden") is not None or el.get("aria-hidden") == "true"
                    or any(h in style for h in HIDDEN_STYLE)):
                hidden = True
                break
        rel = a.get("rel") or []
        if not hidden and "nofollow" not in rel:
            keep.append(a["href"])
    return keep

def on_topic(page_text, expected_terms, min_hits=2):
    text = page_text.lower()
    return sum(1 for t in expected_terms if t.lower() in text) >= min_hits

Inline-style checks miss links hidden by an external stylesheet. If decoys get past this filter, render the page with render_js=true and test real visibility in the browser instead of guessing from attributes. Restricting the crawl to URL patterns you already know (product, category, search) is the bluntest and most effective defence of all: a decoy link outside those patterns is never queued.

Fresh exits help with exactly one class of soft block: the kind keyed to your IP's reputation. Everything else needs a different fix, and rotating harder only hides the evidence.

If a cross-IP vote shows all exits agreeing on the fake value, stop rotating. Either the poisoning keys on something other than IP, or the whole pool shares a reputation, and the premium tie-breaker plus a manual browser check will tell you which.

Detection is only useful if it controls the write. Every check above returns a verdict, and the gate turns verdicts into one decision per batch: publish, publish with rows dropped, or hold.

def release_decision(batch):
    """batch: dict of check results for one run of one target."""
    canary = batch["canary"]
    votes = batch["votes"]
    inconsistent = batch["inconsistent_rows"]
    total = max(batch["rows"], 1)

    if canary["fail"] or batch["truncation"]["verdict"] == "truncated":
        return "HOLD", "canary or truncation failed; keep the run in staging"

    dissent_rate = sum(v["dissent"] for v in votes) / max(len(votes), 1)
    if dissent_rate > 0.2:
        return "HOLD", f"{dissent_rate:.0%} of sampled URLs disagree across exits"

    if inconsistent / total > 0.05:
        return "HOLD", f"{inconsistent} rows fail self-consistency"

    drop = [v["url"] for v in votes if v["consensus"] is None]
    return "PUBLISH", f"drop {len(drop)} unresolved rows plus {inconsistent} inconsistent rows"

The thresholds are starting points, not findings. Run the gate in report-only mode for two weeks, look at what it would have held, and tune from your own data. Held runs go to a staging table, never the bin: if the hold turns out to be a false alarm, a single INSERT ... SELECT moves it to production, and you never have to re-scrape a window that no longer exists.

The gate sits after cleaning, not instead of it. Normalising currency symbols and decimal separators first, as in how to clean scraped data, keeps format noise from looking like disagreement. And when you report pipeline health upward, count a held or poisoned row as a failure: a run that returned 200 on every request but shipped wrong prices did not succeed, which is the "soft 200" trap described in what proxy success rate means and how to measure it.