Scraper Stopped Working Suddenly? A 10-Minute Triage

Start with the text you actually have. Every string below was either captured by us on 2026-10-30 (Python 3.14.2, requests 2.32.5, urllib3 2.6.3, against the SparkProxy gateway) or is quoted from the documentation named in the row.

The last row is the trap. An endpoint that used to return JSON and now returns HTML is either a moved API (branch 3) or a challenge page served in place of the data (branch 2). The saved body settles it in seconds, which is why the first step is saving it.

Don't rotate keys, bump retries or edit selectors yet. Each of those destroys evidence. Reproduce one failing URL with the same proxy settings your job uses, and write everything about the outcome to disk: the exception string if there was one, otherwise status, final URL, headers, byte count and page title.

import hashlib
import json
import re
import time
from pathlib import Path

import requests

PROXY = "http://USERNAME:PASSWORD@gateway.sparkproxy.io:11000"  # Random Rotation port
PROXIES = {"http": PROXY, "https": PROXY}
URL = "https://shop.sparkproxy.io/products/widget-42"


def freeze(url, proxies=PROXIES, out="evidence"):
    """Save one request's full outcome so the diagnosis works from facts."""
    Path(out).mkdir(exist_ok=True)
    stamp = time.strftime("%Y%m%d-%H%M%S")
    record = {"url": url, "at": stamp}
    try:
        r = requests.get(url, proxies=proxies, timeout=30)
    except requests.exceptions.RequestException as exc:
        record["exception"] = f"{type(exc).__name__}: {exc}"
    else:
        body = r.content
        m = re.search(rb"<title[^>]*>(.*?)</title>", body, re.I | re.S)
        record.update(
            status=r.status_code,
            final_url=r.url,
            headers=dict(r.headers),
            bytes=len(body),
            sha1=hashlib.sha1(body).hexdigest(),
            title=m.group(1).strip()[:120].decode("utf-8", "replace") if m else None,
        )
        Path(out, f"{stamp}.html").write_bytes(body)
    Path(out, f"{stamp}.json").write_text(json.dumps(record, indent=2))
    return record


if __name__ == "__main__":
    print(json.dumps(freeze(URL), indent=2))

Run it twice, a minute apart. If both runs give the same outcome, you have a stable failure and the tree below will classify it. If one passes and one fails, you are almost certainly in branch 2, because selectors and credentials don't flicker.

Also open the URL in a normal browser on your own connection. If the data you want is visibly there, write down one exact value (a price, a product name). You'll grep for it later.

The tree asks the cheapest question first. Reading an exception costs nothing. Reading a status code and headers costs nothing. Only the last question needs you to look inside the page.

Q1. Did the request raise, or come back 407?
 ├─ yes, and it names the proxy (ProxyError, 407, the gateway host or port)
 │     → BRANCH 1: your proxy changed
 └─ no, the target answered
       Q2. Is status 403 / 429 / 503, is cf-mitigated: challenge set,
           or is the <title> a challenge or "access denied" page?
       ├─ yes → BRANCH 2: the site blocked you
       └─ no, it is a 200 that looks like the real site
             Q3. Is the value you saw in the browser inside the saved bytes?
             ├─ yes, but your selector finds nothing → BRANCH 3: the site changed
             └─ no → run the rotation probe in Branch 2 before touching selectors

The same logic as a function you can drop next to freeze():

BLOCK_TITLES = (
    "just a moment", "attention required", "access denied",
    "verify you are human", "are you a robot", "pardon our interruption",
)


def classify(record, html=b"", known_value=None):
    """Return 'proxy', 'blocked', 'site-changed', 'network' or 'unclear'."""
    exc = record.get("exception", "")
    if exc:
        if "407" in exc or "ProxyError" in exc:
            return "proxy"
        return "network"  # DNS, TLS or timeout talking to the target itself
    if record["status"] == 407:
        return "proxy"
    headers = {k.lower(): v for k, v in record["headers"].items()}
    title = (record.get("title") or "").lower()
    if (record["status"] in (403, 429, 503)
            or headers.get("cf-mitigated") == "challenge"
            or any(t in title for t in BLOCK_TITLES)):
        return "blocked"
    if known_value and known_value.encode() not in html:
        return "unclear"  # real data missing from a 200: probe before editing selectors
    return "site-changed"

The cf-mitigated check is the one deterministic marker in that list. Cloudflare's challenge detection docs say every challenge page carries the header and that "challenge is the only valid value". The title strings are a starter list we have seen on challenge pages, not documented by any vendor, so extend them from your own saved bodies. Building a full block detector is its own job, and it is covered in how to detect when your scraper is blocked.

This branch is the cheapest to confirm and the most common reason for a clean break after "weeks of working". Three things change on the proxy side without you touching code: the plan, the credentials, or the IP your server presents.

Confirm it from the shell. These three commands separate them in under a minute:

# 1. Does the gateway accept these credentials? A healthy run prints connect=200.
curl -s -o /dev/null -w "connect=%{http_connect} code=%{http_code}\n" \
  -x http://USERNAME:PASSWORD@gateway.sparkproxy.io:11000 https://www.sparkproxy.io/

# 2. Which public IP is this server using today? (matters for IP-whitelist auth)
curl -s https://checkip.amazonaws.com

# 3. Does the proxy hostname in your config still resolve?
nslookup gateway.sparkproxy.io

With bad credentials, command 1 printed connect=407 code=000 and curl exited with code 56 in our test. The gateway's refusal carries Proxy-Authenticate: Basic realm="SparkProxy" and X-Squid-Error: ERR_CACHE_ACCESS_DENIED 0.

One measured quirk is worth knowing before you grep logs. With requests, a refused proxy shows up two different ways depending on the target's scheme. An https:// target raises ProxyError with "Tunnel connection failed: 407", because the refusal happens during CONNECT. A plain http:// target raises nothing at all: you get a normal response object whose status_code is 407. A scraper that only logs status codes will file that under "blocked", and you'll go hunting in the wrong branch. This check reports either form:

def proxy_state(proxies):
    """Ask the gateway directly; plain HTTP so a 407 arrives as a status."""
    try:
        r = requests.get("http://www.sparkproxy.io/", proxies=proxies,
                         timeout=20, allow_redirects=False)
    except requests.exceptions.ProxyError as exc:
        return f"unreachable: {exc}"
    if r.status_code == 407:
        return "rejected: " + r.headers.get("X-Squid-Error", "407")
    return f"accepted (target answered {r.status_code})"

Against our deliberately wrong credentials it returned rejected: ERR_CACHE_ACCESS_DENIED 0.

The fixes, by cause.

• Plan lapsed or trial ended. The 24-hour free trial ends on its own, and on 2026-10-30 the SparkProxy pricing card listed the $75 Starter pack as "Billed every 30 days". A job that ran for roughly a month and then started returning 407 fits a renewal that didn't go through. From the client, a lapsed plan and a wrong password look identical, so check the dashboard before anything else. Plans are packs with unlimited bandwidth, so you can't have "run out of traffic" mid-cycle; the billing unit is the plan and its concurrency.
• Credentials rotated. Someone regenerated the password or removed a sub-user. Copy them fresh from the dashboard. If the new password contains @, : or /, percent-encode it in the URL.
• Your server's IP changed. If you authenticate by IP whitelist rather than user:pass, a redeploy, a new NAT gateway or a cloud instance restart can hand your job a new public IP. Compare command 2's output with the whitelist. The same pricing card listed 5 whitelisted IPs on Starter, so an autoscaling fleet can quietly exceed it. Switching that job to user:pass auth removes the problem.
• Stale hostname or port. Use gateway.sparkproxy.io with port 11000 for HTTP/HTTPS (Random Rotation), 11002 for Sticky Sessions, and 13000 for SOCKS5. A snippet copied from an old config that names a hostname which no longer resolves fails with the NameResolutionError from the table.

On the Scraping API, the account answers for itself. The Scraping API docs separate account errors from target errors by status code, so read them before suspecting the site:

API = "https://scrape.sparkproxy.io/api/v1"
HEADERS = {"X-API-Key": "YOUR_API_KEY"}

r = requests.get(API, headers=HEADERS,
                 params={"url": URL, "render_js": "false"}, timeout=120)
if r.status_code in (401, 402, 429):
    # 401 bad key, 402 out of credits, 429 rate or concurrency limit.
    print("account side:", r.status_code, r.json())

A 402 comes back as {"error": "Insufficient credits", "credits_required": 10, "credits_remaining": 3}. Remember that render_js defaults to true at 5 credits, so the 1,000 free credits cover 200 default requests, or 1,000 with render_js=false. A prototype that "stopped" on day four may simply have spent them. A 429 with "Concurrency limit reached" means you raised parallelism on your side.

If the proxy is accepted but the target still fails, you are done with this branch. For a full health check of the proxy itself (exit IP, HTTPS, latency), use how to test if your proxy is working.

The target answered, and the answer was a refusal. Before choosing a fix you need one more fact: is the block tied to specific IPs, to the whole class of IP you use, or to the request itself? Those three have completely different fixes, and only two of them involve a proxy.

The rotation probe. Send six requests through the Random Rotation port, where each request can exit from a different IP, with a short pause between them:

def rotation_probe(url, n=6, pause=2.0):
    outcomes = []
    for _ in range(n):
        try:
            r = requests.get(url, proxies=PROXIES, timeout=30)
            refused = (r.status_code in (403, 429, 503)
                       or r.headers.get("cf-mitigated") == "challenge")
            outcomes.append("block" if refused else "ok")
        except requests.exceptions.RequestException:
            outcomes.append("error")
        time.sleep(pause)
    return outcomes

Read the result like this:

The class test. To tell "datacenter IPs are refused" from "this request shape is refused", fetch the same URL through the Scraping API twice: once on the default pool and once with proxy_type=premium, the residential pool. Both use json_response=true, so the API returns an envelope and the page itself arrives base64-encoded in body. Decode it before you search it for anything.

import base64


def api_fetch(url, **extra):
    params = {"url": url, "render_js": "false", "json_response": "true", **extra}
    r = requests.get(API, headers=HEADERS, params=params, timeout=120)
    r.raise_for_status()
    env = r.json()
    html = base64.b64decode(env["body"]) if env.get("body") else b""
    return env.get("status_code"), env.get("credits_used"), html


default_pool = api_fetch(URL)                       # 1 credit
premium_pool = api_fetch(URL, proxy_type="premium")  # 10 credits without JS
for label, (status, credits, html) in (("default", default_pool),
                                       ("premium", premium_pool)):
    print(label, status, credits, len(html), b"widget-42" in html)

If premium returns the real page and the default pool doesn't, the site is refusing the IP class. If both are refused, the IP was never the problem: the site is judging headers, TLS or behaviour, and a different proxy will not change the verdict. Retry with render_js=true (5 credits, or 25 on the premium pool) so a real browser makes the request.

Rate limits deserve their own fix. A 429 is defined in RFC 6585 as "too many requests in a given amount of time", and the response "MAY include a Retry-After header". If you get one, the wait is the instruction:

from email.utils import parsedate_to_datetime


def retry_after_seconds(response, default=60):
    value = response.headers.get("Retry-After", "").strip()
    if value.isdigit():
        return int(value)
    try:
        when = parsedate_to_datetime(value)  # the HTTP-date form
        return max(0, int(when.timestamp() - time.time()))
    except (TypeError, ValueError):
        return default

A Cloudflare page that reads "Error 1020: Access denied" is different. Cloudflare's 1020 documentation says a security rule configured by the site owner blocked the request and that it is returned with HTTP 403. That rule may match on country, ASN or request properties, so the class test above is the fastest way to see which.

Pacing and rotation strategy in depth live in how to avoid getting your proxy blocked, and backoff schedules in retry and backoff strategies for web scraping.

Status 200, the page title is right, the proxy is fine, and your parser comes back empty or throws AttributeError: 'NoneType' object has no attribute 'text'. That is a markup change, and no proxy setting will help. The question is where the value moved.

Take the exact value you wrote down from the browser and search the saved bytes for it. If it is there, print the path of tags that contains it, and check whether the same value also appears in structured data, which tends to survive redesigns better than CSS classes:

from bs4 import BeautifulSoup


def locate(html, known_value):
    soup = BeautifulSoup(html, "html.parser")
    for text in soup.find_all(string=lambda s: s and known_value in s):
        if text.parent.name in ("script", "style"):
            continue
        el, parts = text.parent, []
        while el is not None and el.name != "[document]":
            classes = "".join(f".{c}" for c in el.get("class", [])[:2])
            parts.append(el.name + classes)
            el = el.parent
        print(" > ".join(reversed(parts)))
    for tag in soup.find_all("script", type="application/ld+json"):
        if known_value in (tag.string or ""):
            print("also present in JSON-LD: parse that instead of the DOM")


html = Path("evidence").joinpath("20261030-071500.html").read_bytes()
locate(html, "$24.99")

Three outcomes, three fixes:

• The value is in the HTML under new class names. Rewrite the selector against something stable, such as a data- attribute, an itemprop, or the JSON-LD block. Anchoring rules are covered in XPath and CSS selectors for web scraping.
• The value is absent from the raw HTML but visible in the browser. The site moved that data to client-side rendering. Fetch it rendered (render_js=true on the Scraping API, the default) or find the XHR in the browser's Network tab and call that endpoint directly.
• An endpoint you parsed as JSON now returns HTML. Look at the saved body. If it is a challenge page, you are back in Branch 2. If it is a 404 page or a redirect to a new path, the API moved: find the new request in the Network tab.

If you'd rather not maintain selectors at all, extract_rules on the Scraping API returns an extracted object built from CSS selectors you pass per request. Don't combine it with json_response: extract_rules replaces the normal response, so there is no base64 body to decode.

Sometimes the proxy is accepted, the target serves real data from your laptop, and the job still fails on the server. Then the change happened on your side, even though nobody edited the scraper:

• An unpinned dependency upgraded. A rebuilt container pulled a new major version of a parser or HTTP client. Compare pip freeze today against the last good image.
• The cron environment differs from your shell. An HTTP_PROXY or HTTPS_PROXY variable set (or missing) on the server overrides what the code says, because requests reads them by default. Print requests.utils.get_environ_proxies(URL) inside the job.
• Disk or output. A full disk or a revoked database credential makes a scraper "return no data" while every request succeeds. Check the write path, not the fetch.

None of these is a proxy problem, and changing proxies will only add noise to the investigation.

Two rows say yes, one says partly. That ratio is the reason to run the tree before buying anything: most overnight breaks are fixed in your code or your dashboard.

Put freeze(), classify(), proxy_state() and rotation_probe() from above in one file with this entry point. It prints the branch and the next thing to run, so whoever is on call starts from an answer.

import sys

NEXT_STEP = {
    "proxy": "Branch 1: check the dashboard plan, credentials and whitelist IP",
    "blocked": "Branch 2: run rotation_probe(), then the premium class test",
    "site-changed": "Branch 3: run locate() with a value copied from the browser",
    "unclear": "Real data missing from a 200: run rotation_probe() first",
    "network": "Target unreachable: DNS, TLS or timeout, not the proxy",
}


def triage(url, known_value=None):
    record = freeze(url)
    html = b""
    if "exception" not in record:
        html = Path("evidence", f"{record['at']}.html").read_bytes()
    branch = classify(record, html, known_value)
    print("proxy:", proxy_state(PROXIES))
    print("branch:", branch, "->", NEXT_STEP[branch])
    if branch in ("blocked", "unclear"):
        print("rotation probe:", rotation_probe(url))
    return branch


if __name__ == "__main__":
    triage(sys.argv[1], sys.argv[2] if len(sys.argv) > 2 else None)

Run it as python triage.py https://shop.sparkproxy.io/products/widget-42 '$24.99'. Including proxy_state() in every run matters: a target can block you and the plan can lapse in the same week, and the cheaper fix should never hide behind the expensive one.

The reason this break cost you a morning is that it was silent until someone noticed missing rows. Two cheap changes help. Log the exception string and status of the first failure per run, not just a count, so the tree can run from the log alone. And alert on zero extracted rows, not only on exceptions, because a branch 3 failure never raises. For a complete detection layer with baselines and canary URLs, see the detection guide linked in the three-way split section; for running jobs on a schedule with alerting, how to schedule and automate web scrapers covers the setup.