Web Scraping With curl_cffi and TLS Impersonation

Python's requests links against whatever OpenSSL ships with your interpreter. That library builds a ClientHello no mainstream browser produces: a different cipher order, a different extension set, no GREASE padding, a different set of supported curves. An anti-bot service hashes that packet into a JA3 or JA4 fingerprint and knows you are automation on the very first request. Setting headers={"User-Agent": "...Chrome/124..."} changes nothing, because the block happened one layer below the header.

curl_cffi is a Python binding over curl-impersonate, a patched build of libcurl compiled against BoringSSL and nghttp2 with the same TLS and HTTP/2 settings a real browser uses. When you ask it to impersonate chrome124, it reproduces that browser's handshake closely enough that the resulting JA3, JA4, and Akamai HTTP/2 fingerprints match a genuine Chrome. Two things make it practical for scraping at volume:

• It is fast. The transport is C (libcurl), not a full browser, so you get browser-grade fingerprints at HTTP-client speed and memory cost.
• The API mirrors requests. from curl_cffi import requests gives you get, post, Session, cookies, and proxies with almost the same surface you already know.

Here is what curl_cffi does and does not fix, which frames everything below:

The two rows that say "No" are the reason the rest of this guide pairs curl_cffi with proxies and, for hard pages, with a rendering API.

There are two passive fingerprints at play, and most tutorials only talk about one.

The TLS fingerprint (JA3, JA3N, JA4). Every HTTPS connection opens with a ClientHello that lists TLS versions, cipher suites, extensions, elliptic curves, and signature algorithms. JA3 (Salesforce, 2017) concatenates those fields and hashes them with MD5. Its sorted cousin JA3N reorders extensions first, which defeats the extension shuffling and GREASE values Chrome inserts on purpose. JA4 (FoxIO, 2023) is the current standard: it records the protocol, the cipher and extension counts, the first ALPN value, and a sorted hash, so it is stable across a browser's random reordering. curl_cffi reproduces the cipher order, the extension list, the curves, and the GREASE placement of the target browser, which is why a static hand-built JA3 string often fails where curl_cffi passes.

The HTTP/2 fingerprint (Akamai). Once the TLS tunnel is up, HTTP/2 has its own tell. Akamai's fingerprint is built from the SETTINGS frame values, the initial WINDOW_UPDATE increment, the stream priority information, and the order of the pseudo-headers (:method, :authority, :scheme, :path). A library that only spoofs TLS still emits Python's HTTP/2 defaults here and gets flagged. curl_cffi patches the nghttp2 layer to send the browser's exact SETTINGS and header order, so both fingerprints line up.

Both checks are passive. No JavaScript runs, no CAPTCHA appears, no cookie is set. The server computes the hashes from packets you already sent and compares them to a table of known-good browsers and known-bad clients. That is why the block is instant and silent. The full JA3/JA4 explainer walks the ClientHello byte by byte if you want the deep version.

Install it from PyPI. curl_cffi ships prebuilt wheels for Linux, macOS, and Windows, so there is no C toolchain to set up:

pip install curl_cffi

Now send a request that reports its own fingerprint. tls.peet.ws is a public checker that returns your JA3, JA4, and Akamai fingerprints as JSON, which makes it the ideal first target:

from curl_cffi import requests

resp = requests.get(
    "https://tls.peet.ws/api/all",
    impersonate="chrome124",
)

data = resp.json()
print("Status:", resp.status_code)
print("JA3 hash:", data["tls"]["ja3_hash"])
print("JA4:", data["tls"]["ja4"])
print("HTTP/2 (Akamai):", data["http2"]["akamai_fingerprint"])

Run the same request with plain requests and the JA3 hash comes back as a Python-specific value that no browser produces. Run it with curl_cffi and impersonate="chrome124", and the hash matches a real Chrome. That single impersonate= argument is the whole trick. Everything else is applying it correctly.

The impersonate value picks which browser build to mimic. Use a current one and keep it consistent with your User-Agent.

The exact list depends on the version you installed, so check the curl_cffi docs for the targets your build supports.

from curl_cffi import requests

resp = requests.get(
    "https://www.sparkproxy.io/",
    impersonate="chrome131",
    # some releases also accept "chrome" to track the newest bundled build
)
print(resp.status_code)

Two points that thin tutorials miss:

• A stale target is itself a fingerprint. Pinning chrome99 or chrome110 in 2026 is almost as suspicious as not impersonating at all, because approximately nobody still runs that build. If a fleet of scrapers all present a two-year-old Chrome handshake, that pattern stands out. Bump the target when you upgrade the library.
• The User-Agent must match the target. If you impersonate chrome131 but send a requests default User-Agent or a Chrome 99 string, the mismatch between the handshake and the header is detectable on its own. curl_cffi sets browser-consistent default headers for you, so the safest move is to leave them alone unless you have a reason to override. If you do rotate the UA, keep it in the same family as the impersonate target. See rotating user agents for the wider pattern.

A Session persists cookies across requests and reuses the underlying TLS connection. Reuse matters for two reasons: it is faster (no fresh handshake per request), and opening a brand new connection for every single request is itself a behavioral pattern that stands out. Set the impersonation once on the session and every request inherits it:

from curl_cffi import requests

session = requests.Session(impersonate="chrome124")

# First call establishes cookies, e.g. a login or a consent gate
session.get("https://www.sparkproxy.io/login")

# Later calls reuse those cookies and the same connection + fingerprint
resp = session.get("https://www.sparkproxy.io/dashboard")
print(resp.status_code, len(resp.content))

You can still override per request when a specific call needs a different browser profile:

resp = session.get("https://www.sparkproxy.io/", impersonate="safari17_0")

For hundreds or thousands of URLs, AsyncSession runs concurrent requests on asyncio while keeping every request impersonated. Use max_clients to cap concurrency so you do not open more sockets than the target (or your proxy plan) tolerates:

import asyncio
from curl_cffi.requests import AsyncSession

urls = [f"https://www.sparkproxy.io/p/{i}" for i in range(50)]

async def scrape_all():
    async with AsyncSession(impersonate="chrome124", max_clients=10) as s:
        tasks = [s.get(u) for u in urls]
        results = await asyncio.gather(*tasks, return_exceptions=True)
        for url, r in zip(urls, results):
            if isinstance(r, Exception):
                print(url, "failed:", r)
            else:
                print(url, r.status_code)

asyncio.run(scrape_all())

Because the transport is libcurl, this scales to real concurrency without the memory footprint of running that many headless browsers. That is the core advantage of doing fingerprint work at the HTTP layer.

curl_cffi takes the same proxies dictionary requests uses, and it supports HTTP, HTTPS, and SOCKS5 schemes. Pass it per request or set it on the session:

from curl_cffi import requests

proxies = {
    "http": "http://USERNAME:PASSWORD@gate.sparkproxy.io:8000",
    "https": "http://USERNAME:PASSWORD@gate.sparkproxy.io:8000",
}

resp = requests.get(
    "https://www.sparkproxy.io/",
    impersonate="chrome124",
    proxies=proxies,
    timeout=20,
)
print(resp.status_code)

For SOCKS5, mind where DNS resolves. socks5:// resolves the hostname on your machine, which can leak the target through your local DNS. socks5h:// resolves it at the proxy, which is what you almost always want when scraping:

proxies = {"https": "socks5h://USERNAME:PASSWORD@gate.sparkproxy.io:8000"}
resp = requests.get("https://www.sparkproxy.io/", impersonate="chrome124", proxies=proxies)

Here is the insight that separates a scraper that survives from one that gets blocked on day two: the TLS fingerprint and the IP address are two independent signals, and fixing one does nothing for the other.

curl_cffi makes your handshake look like Chrome. It does not change where the connection comes from. A flawless chrome131 fingerprint arriving from a datacenter subnet that anti-bot vendors already flag will still be blocked on IP reputation alone. The fix is to combine impersonation with residential proxies, so the network layer looks like a home connection and the transport layer looks like a browser at the same time:

from curl_cffi import requests

session = requests.Session(
    impersonate="chrome131",
    proxies={"https": "http://USERNAME:PASSWORD@gate.sparkproxy.io:8000"},
)

resp = session.get("https://www.sparkproxy.io/pricing")
print(resp.status_code)

Match the rotation to the job. Use a rotating residential endpoint for wide, single-page fetches where a fresh IP per request is fine. Use a sticky session when you need several requests (a login, then paginated data) to come from the same IP, so the sequence looks like one user. SparkProxy's residential pool supports both from the same gateway credentials.

Never assume the impersonation worked. Prove it. Point curl_cffi at a fingerprint checker and read the values back:

from curl_cffi import requests

resp = requests.get("https://tls.peet.ws/api/all", impersonate="chrome124")
fp = resp.json()

print("JA3 hash:", fp["tls"]["ja3_hash"])
print("JA4:", fp["tls"]["ja4"])
print("Akamai HTTP/2:", fp["http2"]["akamai_fingerprint"])
print("User-Agent seen:", fp["http_version"], fp.get("user_agent"))

What to confirm:

• The JA3 or JA4 hash matches a known Chrome build. Compare it against a database like the one at ja3.zone, or against a request from your own real Chrome.
• The Akamai HTTP/2 fingerprint is present and browser-shaped, not the default your language runtime emits. This is the check the TLS-only libraries fail.
• The User-Agent the server saw matches the browser you impersonated. A chrome124 handshake next to a Python UA is a giveaway.

Run the same call once with plain requests and once with curl_cffi and put the two JSON blobs side by side. Seeing the JA3 hash change from a Python value to a Chrome value is the moment the technique clicks.

curl_cffi is an HTTP client. It does not run JavaScript. If a page renders its content client-side (a React or Vue app that ships an empty shell and hydrates from an XHR call), or if it throws a JavaScript challenge (a Cloudflare interstitial, a DataDome puzzle), a perfect TLS fingerprint gets you a mostly empty HTML document or a challenge page, not the data.

You have two options when that happens. First, check whether the site loads its data from a JSON endpoint you can call directly with curl_cffi, which is often faster and cleaner than rendering anything. If the content genuinely needs a browser, hand the request to the SparkProxy Scraping API with render_js on. The API runs a real headless browser with its own browser-grade fingerprint, solves the render, and returns the finished HTML, so you do not maintain browser infrastructure yourself:

import requests  # plain requests is fine here; the API drives the browser

resp = requests.get(
    "https://scrape.sparkproxy.io/api/v1",
    headers={"X-API-Key": "YOUR_API_KEY"},
    params={
        "url": "https://www.sparkproxy.io/pricing",
        "render_js": "true",       # execute JavaScript in a real browser
        "premium_proxy": "true",   # exit through a residential IP
        "stealth": "true",         # extra anti-bot layers
        "country_code": "US",      # geo-target the request
    },
    timeout=90,
)
print(resp.text)

That matches the documented contract at the Scraping API docs: base URL https://scrape.sparkproxy.io/api/v1, auth via the X-API-Key header, and render_js, premium_proxy, stealth, and country_code as parameters. Use this decision table to pick a tool per page:

The clean architecture is curl_cffi first for everything it can reach cheaply, with the rendering API as the fallback for the pages that truly need a browser. For a walk-through of the hardest of those, see bypassing Cloudflare.

A few habits keep a curl_cffi scraper healthy in production.

Retry on transport errors. Timeouts, resets, and TLS errors raise CurlError, which is stable across versions. Retry those; do not retry a clean HTTP 404:

from curl_cffi import requests, CurlError

def fetch(url, proxies=None, target="chrome124", attempts=3):
    for i in range(attempts):
        try:
            r = requests.get(url, impersonate=target, proxies=proxies, timeout=20)
            if r.status_code < 400:
                return r
            print(f"HTTP {r.status_code} on attempt {i + 1}")
        except CurlError as e:
            print(f"transport error on attempt {i + 1}: {e}")
    return None

Track version drift. The single most common cause of a scraper that "used to work" is a browser fingerprint that aged out. When you pip install --upgrade curl_cffi, raise your impersonate target to a build the new version bundles. Treat the target string as a value you maintain, not a constant you set once.

Reach for custom fingerprints only when you must. Recent curl_cffi lets you pass a raw ja3= string and an akamai= string to craft a fingerprint the built-in targets do not cover. It is powerful and easy to get wrong, because a malformed custom fingerprint is more suspicious than a stock one. Prefer a named impersonate target and only hand-roll when you are matching a very specific client.

Respect the target. Impersonation hides the fact that you are automated; it does not grant permission. Read robots.txt, keep concurrency reasonable, cache what you have already fetched, and scrape public data within the site's terms.