Website Blocks Python Requests but Works in Browser: Fix

You paste a URL into Chrome and the page loads. You run requests.get() on the same URL, sometimes through the very same proxy, and get one of these instead:

Cloudflare documents the header check precisely: a challenge response "will have the cf-mitigated header present and set to challenge", and challenge is the only valid value (Cloudflare docs, Detect a Challenge Page response, updated May 5, 2026). Its 1020 page states that Cloudflare "returns it with an HTTP 403 status code" (Error 1020, updated Oct 14, 2026). Use these exact strings in code, not the status code alone, because a 200 with a challenge body is also a block. If you need a fuller set of detectors for silent blocks, the guide on detecting when your scraper is blocked covers that ground.

The confusing part of this bug is the proxy. If the browser and the script both exit through the same IP, how can the site tell them apart?

Because an HTTP proxy carrying an https:// request just opens a CONNECT tunnel and relays encrypted bytes. The proxy never rewrites your TLS handshake, your HTTP version or your headers, so the target sees exactly what your client produced. Our explainer on what an HTTP tunnel is walks through the mechanics. The practical consequence is blunt: the IP is shared, everything above the IP is yours, and the site is reading everything above the IP.

A browser carries four things requests lacks, roughly in the order anti-bot systems check them:

1. A full header set in a fixed order. Chrome sends a dozen headers, including sec-ch-ua and sec-fetch-*, in the same sequence every time.
2. A browser TLS ClientHello. Its cipher list, extensions and ALPN hash to a JA3/JA4 value that matches Chrome.
3. HTTP/2 with Chrome's SETTINGS frame. requests speaks only HTTP/1.1.
4. Cookies earned on an earlier page, sometimes by running JavaScript.

Most people fix all four at once by switching libraries and never learn which one mattered. That works until it stops working, and then they have nothing to go on. Test them one at a time instead.

We pointed four clients at the public fingerprint echo https://tls.peet.ws/api/all on 2026-10-30, from Python 3.14.2 on Windows Server 2022, using requests 2.34.2 (the current PyPI release, uploaded 2026-05-14) and curl_cffi 0.17.0 (released on GitHub 2026-10-26, last commit 2026-10-27). These are our own measurements; your JA4 for requests will differ with your Python and OpenSSL build, but the pattern will not.

Read the first two rows together. Swapping the User-Agent changed the header and left the TLS fingerprint byte-for-byte identical. That is why "just set a Chrome User-Agent" so often does nothing: the site may have stopped reading at the handshake. The last two rows show the reverse. curl_cffi keeps Chrome's cipher prefix (8daaf6152771) whether it speaks h2 or HTTP/1.1, which lets you toggle HTTP/2 on its own.

The header capture was just as telling. Default requests sent exactly this, in this order:

Host: tls.peet.ws
User-Agent: python-requests/2.34.2
Accept-Encoding: gzip, deflate, zstd
Accept: */*
Connection: keep-alive

Five lines. No browser has sent a request that short in years.

Before touching code, run the cheapest test there is. Open the target URL in a fresh incognito window, with DevTools open on the Network tab and "Preserve log" ticked.

• The page loads with no interstitial and the document request returns 200 directly. The browser has no secret head start. The difference is in the request itself, so go to Probe 1.
• You see "Just a moment..." or a spinner flash first, or a 403 followed by a 200. The browser ran JavaScript and earned a cookie before it got the page. Skip to Probe 4, and expect that a pure HTTP client will not be enough.

Then check the proxy usage. If your script uses a rotating endpoint and your browser holds one connection open, the browser keeps one exit IP for the whole visit while your script presents a new one per request. That alone breaks any cookie-based clearance. Repeat the test with a sticky session (covered in the proxy section below) before blaming fingerprints.

Now copy the browser's request. In DevTools, right-click the document request and choose Copy as cURL. Run it, then strip it down:

# 1. The browser's exact request, replayed by curl
curl 'https://www.sparkproxy.io/about/' -H 'user-agent: Mozilla/5.0 ...' -H 'sec-ch-ua: ...' \
  -x http://USERNAME:PASSWORD@gateway.sparkproxy.io:11002 \
  -s -o /dev/null -w '%{http_code}\n'

# 2. Same, with every -H removed
curl 'https://www.sparkproxy.io/about/' \
  -x http://USERNAME:PASSWORD@gateway.sparkproxy.io:11002 \
  -s -o /dev/null -w '%{http_code}\n'

If the first returns 200 and the second returns 403, the site cares about headers and does not care much about curl's TLS. That is very good news, because headers are the one layer requests can fix without changing libraries. If both return 403, headers are not the whole story. Note that curl is not a browser either: the Windows build above uses Schannel and produced its own JA4, so a curl pass tells you the site is tolerant of non-browser TLS.

Two separate problems hide here: which headers you send, and in what order. Setting headers= on a call fixes the first and makes the second worse. We captured what requests emits when you pass a Chrome header dict per request:

Host, user-agent, Accept-Encoding, accept, Connection, sec-ch-ua, sec-ch-ua-mobile, ...

requests keeps its own four defaults in their original slots and appends your extra keys after them. So sec-ch-ua lands after Connection, a sequence Chrome never produces. To control order you have to clear the session defaults first. When we did that, a second surprise appeared: with no Accept-Encoding set, Python's http.client inserted Accept-Encoding: identity right after Host. Set it yourself.

import requests

# Header set and order curl_cffi 0.17.0 sends for chrome142 (captured at tls.peet.ws)
CHROME_HEADERS = {
    "sec-ch-ua": '"Chromium";v="142", "Google Chrome";v="142", "Not_A Brand";v="99"',
    "sec-ch-ua-mobile": "?0",
    "sec-ch-ua-platform": '"Windows"',
    "upgrade-insecure-requests": "1",
    "user-agent": (
        "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
        "(KHTML, like Gecko) Chrome/142.0.0.0 Safari/537.36"
    ),
    "accept": (
        "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,"
        "image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7"
    ),
    "sec-fetch-site": "none",
    "sec-fetch-mode": "navigate",
    "sec-fetch-user": "?1",
    "sec-fetch-dest": "document",
    "accept-encoding": "gzip, deflate, br, zstd",
    "accept-language": "en-US,en;q=0.9",
}

session = requests.Session()
session.headers.clear()            # drop python-requests defaults and their slots
session.headers.update(CHROME_HEADERS)

resp = session.get("https://www.sparkproxy.io/about/", timeout=20)
print(resp.status_code, resp.headers.get("cf-mitigated"))

Advertising br and zstd means the server may answer with them, and requests only decodes those if urllib3 has the codecs. Install them with pip install "urllib3[brotli,zstd]", or drop both tokens from the header. Otherwise a "fixed" request returns binary garbage and looks like a new kind of block.

Keep the sec-ch-ua brand version, the User-Agent version and the platform consistent with each other. A Chrome 142 UA next to a Chrome 120 client hint is a mismatch some vendors score on its own. If this request passes where the default one failed, you are done: the site was checking headers, and you can stay on requests.

If the ordered Chrome headers still get a 403, the next suspect is the handshake. The test is a controlled swap: send the same headers, over the same HTTP/1.1, with a Chrome ClientHello instead of OpenSSL's. curl_cffi can do exactly that, because default_headers=False stops it adding its own header set and http_version="v1" holds the protocol constant.

from curl_cffi import requests as cffi

# CHROME_HEADERS from the Probe 1 snippet
resp = cffi.get(
    "https://www.sparkproxy.io/about/",
    impersonate="chrome142",
    default_headers=False,   # send only our headers, nothing extra
    headers=CHROME_HEADERS,
    http_version="v1",       # keep HTTP/1.1 so only TLS changes
    timeout=20,
)
print(resp.status_code, resp.headers.get("cf-mitigated"))

The only variable that differs from the Probe 1 request is the TLS fingerprint. If this returns 200 where requests returned 403, the site is fingerprinting the handshake, and no header or proxy change will fix requests. Move the scraper to curl_cffi. How JA3 and JA4 are computed is covered in what TLS fingerprinting is, and the day-to-day usage of the library (sessions, async, picking an impersonate target) is in web scraping with curl_cffi.

One version note. In curl_cffi 0.17.0 the BrowserType enum runs from chrome99 up to chrome152, plus Safari 26/27 and Firefox 155 profiles, and the bare "chrome" alias resolved to Chrome 152 in our capture. Pin a named target so a library upgrade does not silently change your fingerprint, and bump it on purpose.

requests cannot negotiate HTTP/2 at all; our capture shows HTTP/1.1 and an ALPN of h1 inside its JA4. A real Chrome on a modern site speaks h2. Some anti-bot rules do not care, others treat "claims to be Chrome, speaks HTTP/1.1" as a contradiction, and some also hash the h2 SETTINGS frame. curl_cffi's Chrome profile sends 1:65536;2:0;4:6291456;6:262144|15663105|0|m,a,s,p as its Akamai h2 fingerprint, which is the browser value it copies.

Isolating this layer is a one-argument change:

from curl_cffi import requests as cffi

URL = "https://www.sparkproxy.io/about/"

h1 = cffi.get(URL, impersonate="chrome142", http_version="v1", timeout=20)
h2 = cffi.get(URL, impersonate="chrome142", timeout=20)  # negotiates h2 via ALPN

print("HTTP/1.1:", h1.status_code, "| HTTP/2:", h2.status_code)

Both requests use the Chrome handshake and Chrome headers. If v1 is blocked and the default passes, the site wants HTTP/2 from a browser, so stay on curl_cffi with its default protocol. Do not try to bolt HTTP/2 onto requests with an adapter; you would still have OpenSSL's TLS. The HTTP/2 vs HTTP/1.1 breakdown explains what the h2 fingerprint contains if you need to go deeper.

Some sites refuse a cold deep link. The browser that "works" had visited the homepage, accepted a consent banner, or followed a link, and it arrives at the target carrying cookies and a Referer. A script that jumps straight to /product/123 has none of that.

Test it with a warm-up visit inside one session, sending the headers a same-site click would send:

from curl_cffi import requests as cffi
from urllib.parse import urlsplit

TARGET = "https://www.sparkproxy.io/about/"
origin = "{0.scheme}://{0.netloc}/".format(urlsplit(TARGET))

s = cffi.Session(impersonate="chrome142")
first = s.get(origin, timeout=20)
print("homepage:", first.status_code, "cookies:", list(s.cookies.keys()))

resp = s.get(
    TARGET,
    headers={"referer": origin, "sec-fetch-site": "same-origin"},
    timeout=20,
)
print("target:", resp.status_code)

If the warm session passes and the cold request fails, keep one session per crawl path, visit an entry page first, and reuse it. Our guide on handling cookies and sessions covers persistence and expiry.

Look at the cookie names the homepage set. Names like cf_clearance, _abck or datadome mean the cookie was produced by JavaScript, and an HTTP client cannot mint it. Copying one out of your browser rarely survives either: Cloudflare says the clearance cookie "is securely tied to the specific visitor and device it was issued to, preventing reuse across machines" (Cloudflare docs, Clearance). That is the point where you stop tuning HTTP and use a browser.

Running the probes by hand is fine once. When a scraper breaks at 3 a.m., you want one command that tells you the layer. This script holds everything constant except one variable per step, and the first step that passes names the culprit.

"""Usage: python diagnose.py https://www.sparkproxy.io/about/"""
import os
import sys
from urllib.parse import urlsplit

import requests
from curl_cffi import requests as cffi

TARGET = sys.argv[1] if len(sys.argv) > 1 else "https://www.sparkproxy.io/about/"
PROXY = os.environ.get("SPARK_PROXY")  # e.g. http://USER:PASS@gateway.sparkproxy.io:11002
PROXIES = {"http": PROXY, "https": PROXY} if PROXY else None
ORIGIN = "{0.scheme}://{0.netloc}/".format(urlsplit(TARGET))
IMP = "chrome142"

CHROME_HEADERS = {
    "sec-ch-ua": '"Chromium";v="142", "Google Chrome";v="142", "Not_A Brand";v="99"',
    "sec-ch-ua-mobile": "?0",
    "sec-ch-ua-platform": '"Windows"',
    "upgrade-insecure-requests": "1",
    "user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
                  "(KHTML, like Gecko) Chrome/142.0.0.0 Safari/537.36",
    "accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
    "sec-fetch-site": "none",
    "sec-fetch-mode": "navigate",
    "sec-fetch-user": "?1",
    "sec-fetch-dest": "document",
    "accept-encoding": "gzip, deflate",
    "accept-language": "en-US,en;q=0.9",
}
MARKERS = ("just a moment", "attention required", "access denied",
           "sorry, you have been blocked", "captcha-delivery.com", "press & hold")
BLOCK_CODES = {401, 403, 429, 503}


def verdict(resp):
    """Return 'PASS', 'BLOCK' or 'OTHER' plus a detail string."""
    if resp.headers.get("cf-mitigated") == "challenge":
        return "BLOCK", "cf-mitigated: challenge"
    body = resp.text[:20000].lower()
    hit = next((m for m in MARKERS if m in body), None)
    if resp.status_code in BLOCK_CODES or hit:
        return "BLOCK", f"{resp.status_code} {hit or ''}".strip()
    if resp.status_code >= 400:  # a 404 is a wrong URL, not a block
        return "OTHER", f"{resp.status_code}, check the URL"
    return "PASS", f"{resp.status_code}, {len(resp.text)} chars"


def requests_ordered():
    s = requests.Session()
    s.headers.clear()
    s.headers.update(CHROME_HEADERS)
    return s.get(TARGET, proxies=PROXIES, timeout=20)


def cffi_warm():
    s = cffi.Session(impersonate=IMP, proxies=PROXIES)
    s.get(ORIGIN, timeout=20)
    return s.get(TARGET, headers={"referer": ORIGIN, "sec-fetch-site": "same-origin"},
                 timeout=20)


PROBES = [
    ("baseline: requests defaults", None,
     lambda: requests.get(TARGET, proxies=PROXIES, timeout=20)),
    ("+ Chrome headers, ordered", "HEADERS",
     requests_ordered),
    ("+ Chrome TLS (HTTP/1.1)", "TLS",
     lambda: cffi.get(TARGET, impersonate=IMP, default_headers=False,
                      headers=CHROME_HEADERS, http_version="v1",
                      proxies=PROXIES, timeout=20)),
    ("+ HTTP/2", "HTTP/2",
     lambda: cffi.get(TARGET, impersonate=IMP, proxies=PROXIES, timeout=20)),
    ("+ warm-up cookies", "COOKIES",
     cffi_warm),
]

errors = 0
for name, layer, probe in PROBES:
    try:
        state, detail = verdict(probe())
    except Exception as exc:  # proxy or TLS errors should not stop the run
        state, detail = "ERROR", f"{exc.__class__.__name__}: {str(exc)[:70]}"
        errors += 1
    print(f"{state:5}  {name:30} {detail}")
    if state == "OTHER":
        print("Stop: not a block signature. Fix the URL or request first.")
        break
    if state == "PASS":
        print("Layer:", layer or "none, the plain request already passes")
        break
else:
    if errors == len(PROBES):
        print("Every probe errored: check the proxy string first (a 407 means bad credentials).")
    else:
        print("Layer: JavaScript or IP. Use a real browser (see the Scraping API section).")

The script uses gzip, deflate so it runs without the brotli and zstd extras. Read the output top to bottom:

Run it once without SPARK_PROXY and once with it. If every probe fails through the proxy and something passes without it, the exit IP is part of the problem, which is the one case a different proxy pool can fix.

Being honest about this saves a lot of wasted plan upgrades.

The one row where proxy configuration commonly causes the bug is the cookie row. With a rotating endpoint, every request leaves from a new IP, so a session the site tied to IP A arrives from IP B and gets challenged again. Use Random Rotation for independent single-page fetches and Sticky Sessions, which hold one exit IP for five minutes, for any flow that depends on cookies. On SparkProxy the two are separate ports on the same gateway and credentials:

from curl_cffi import requests as cffi

ROTATING = "http://USERNAME:PASSWORD@gateway.sparkproxy.io:11000"  # Random Rotation
STICKY = "http://USERNAME:PASSWORD@gateway.sparkproxy.io:11002"    # Sticky Sessions (5 min)

# Independent catalogue pages: a fresh IP per request is fine
r = cffi.get("https://www.sparkproxy.io/blog/", impersonate="chrome142",
             proxies={"https": ROTATING}, timeout=20)

# Warm-up then target: keep one IP so the cookies stay valid
s = cffi.Session(impersonate="chrome142", proxies={"https": STICKY})
s.get("https://www.sparkproxy.io/", timeout=20)
r = s.get("https://www.sparkproxy.io/about/", timeout=20)
print(r.status_code)

Don't confuse a proxy failure with a site block. With wrong credentials, curl_cffi raises curl: (7) CONNECT tunnel failed, response 407 and requests raises a ProxyError; we got both while testing the snippets above with placeholder credentials. A 407 comes from the proxy, never from the target, and no fingerprint work will change it.

The gateway accepts HTTP, HTTPS and SOCKS5 with user:pass or IP whitelist auth, from a pool of 1M+ premium IPs in 80+ countries, and plans start from $75/month with unlimited bandwidth (pricing read from sparkproxy.io, USD, October 2026). Because billing is by plan and concurrency rather than traffic, running the diagnosis script repeatedly does not add a per-GB cost. A 24-hour free trial needs no credit card. The trade-off between the two rotation modes is covered in more depth in what a sticky session proxy is.

If the diagnosis ends at "nothing passes" and the homepage sets a JS-minted cookie, the remaining fix is to run JavaScript. You can drive Playwright yourself, or send the URL to the SparkProxy Scraping API, which renders it in a headless browser on its side. Here render_js is on (it defaults to true), proxy_type=premium selects the premium pool, and json_response=true returns an envelope whose body is base64, so decode it before checking for block markers:

import base64
import requests

resp = requests.get(
    "https://scrape.sparkproxy.io/api/v1",
    headers={"X-API-Key": "YOUR_API_KEY"},
    params={
        "url": "https://www.sparkproxy.io/about/",
        "render_js": "true",        # default; 5 credits per request
        "proxy_type": "premium",
        "json_response": "true",    # envelope: job_id, status_code, duration_ms, credits_used, body
    },
    timeout=120,
)
resp.raise_for_status()   # fail loudly on an API-level error before parsing
env = resp.json()
html = base64.b64decode(env["body"]).decode("utf-8", errors="replace")

blocked = "just a moment" in html.lower() or env["status_code"] >= 400
print(env["status_code"], env["credits_used"], "blocked" if blocked else "ok", len(html))

Pricing on the API is in credits: a rendered request costs 5, render_js=false is a plain fetch at 1 credit, and country_code adds 5 more. New accounts get 1,000 free credits, enough to confirm whether rendering clears your target before you change any scraper code. The Scraping API docs list every parameter, and the credit pricing explainer works through cost per page. A sensible split is to keep curl_cffi for every URL the diagnosis says it can reach and send only the JavaScript-gated ones to rendering, because 1 credit versus 5 adds up fast on a large crawl.

Whatever layer you land on, write it down next to the target. Sites change their rules, and the next time this scraper breaks, rerunning the five probes and comparing against the old answer tells you in under a minute what moved.