cURL vs Python Requests for Web Scraping (2026)

The table splits into two groups. The bottom half is ergonomics and you can work around any of it. The top half, the TLS row, is the one you cannot code around.

Before your request line is sent, before the User-Agent header exists in the conversation, your client sends a TLS ClientHello. That message is not generic. It carries an ordered list of cipher suites, an ordered list of extensions, the supported groups (elliptic curves), signature algorithms, ALPN protocols, and the TLS versions you accept. Every one of those lists is decided by the TLS library your client links against and by how that client configures it.

Chrome does not produce the same ClientHello as Firefox, and neither produces the same one as Python. A CDN sitting in front of a target reads the ClientHello, hashes the relevant fields, and gets a short string. If that string belongs to a known automation stack, the request can be scored, throttled, or dropped at the edge with a 403 before any application code runs. Your residential IP does not help. Your carefully rotated User-Agent does not help, and a Chrome UA string paired with a Python handshake is a stronger bot signal than an honest Python UA, because real Chrome never handshakes that way.

This is why a scraper that runs fine against a small site returns 403 the moment you point it at a Cloudflare or Akamai property, and why swapping in fresh proxies changes nothing. The background on the mechanism is in the explainer on what TLS fingerprinting is.

You can see your own fingerprint in about ten seconds. Run both of these and compare the JSON:

# curl's handshake
curl -s https://tls.peet.ws/api/all | python -m json.tool | head -40

# Requests' handshake, same endpoint
import requests

r = requests.get("https://tls.peet.ws/api/all", timeout=30)
data = r.json()
print(data["tls"]["ja3"])
print(data["tls"]["ja4"])
print(data["http_version"])

Two things jump out. The JA3 and JA4 strings differ between the two tools, and neither matches the value a real Chrome produces. The http_version line is the second tell: curl reports h2 on a default modern build, Requests reports HTTP/1.1.

JA3, published by Salesforce in 2017, builds a string from five ClientHello fields joined with commas: TLS version, cipher suites, extensions, elliptic curves, and elliptic curve point formats. It then takes the MD5 of that string. A Python Requests JA3 starts out looking like 771,4865-4866-4867-49195-... before hashing.

JA3 has a weakness that matters in 2026. Chrome 110, released in February 2023, began permuting the order of ClientHello extensions on every connection, specifically to stop servers from hardcoding assumptions about it. A single Chrome install therefore emits many different JA3 hashes, and static JA3 blocklists got noisy overnight.

JA4, from FoxIO, was designed around that. The JA4 TLS client fingerprint is a readable three-part string such as t13d1516h2_8daaf6152771_e5627efa2ab1, and it sorts the extension and cipher lists before hashing. Permutation stops helping. The first segment alone leaks plenty: t for TCP, 13 for TLS 1.3, d for a present SNI, then the cipher count, the extension count, and the negotiated ALPN value (h2, or 00 when ALPN is absent).

That ALPN field is worth staring at. A client that never negotiates HTTP/2 lands in a different JA4 bucket than every real browser on the internet, purely from the two characters at the end of the first segment. Requests, which cannot speak HTTP/2 at all, is permanently in that bucket.

There is a third layer above TLS. Once an HTTP/2 connection is up, the client sends a SETTINGS frame, an initial WINDOW_UPDATE, and its pseudo-headers in a particular order. Akamai's HTTP/2 fingerprint hashes exactly those values. Chrome sends its pseudo-headers as :method, :authority, :scheme, :path, and most libraries do not. So there are three independent fingerprint surfaces, and a Requests scraper fails the first one and never reaches the third.

The stack is the problem. Requests delegates connection handling to urllib3, which builds an ssl.SSLContext from Python's standard library, which wraps OpenSSL. Python's ssl module exposes set_ciphers() and little else. There is no API to reorder extensions, to add the GREASE values BoringSSL emits, to control supported_groups ordering, or to send the ALPS extension Chrome sends. You can narrow the cipher list, which turns your JA3 hash into a different non-browser hash. That is not impersonation, it is a new anomaly.

import ssl
import requests
from requests.adapters import HTTPAdapter
from urllib3.poolmanager import PoolManager

class CipherAdapter(HTTPAdapter):
    def init_poolmanager(self, connections, maxsize, block=False, **kw):
        ctx = ssl.create_default_context()
        ctx.set_ciphers("ECDHE+AESGCM:ECDHE+CHACHA20")
        kw["ssl_context"] = ctx
        self.poolmanager = PoolManager(
            num_pools=connections, maxsize=maxsize, block=block, **kw
        )

s = requests.Session()
s.mount("https://", CipherAdapter())
# JA3 changes. It still is not Chrome's, and JA4 still reports no HTTP/2.

curl is in a different position because it is C, and because it supports several TLS backends chosen at build time. That is exactly what curl-impersonate exploits. It compiles curl against BoringSSL (or NSS for the Firefox targets), patches the extension order and the default cipher list, sets the browser's HTTP/2 SETTINGS values, and ships wrapper binaries:

# Chrome-shaped handshake from the command line
curl_chrome116 https://tls.peet.ws/api/all

# compare against stock curl on the same box
curl -s https://tls.peet.ws/api/all | grep -o '"ja4":"[^"]*"'

The Python answer is curl_cffi, which binds that patched libcurl through CFFI and hands you a Requests-shaped API. Notice what that means: the fix for Python's fingerprint is to stop using Python's TLS stack and call curl's instead.

from curl_cffi import requests as cffi_requests

r = cffi_requests.get(
    "https://tls.peet.ws/api/all",
    impersonate="chrome131",
    proxies={"https": "http://user:pass@dc.sparkproxy.io:10000"},
    timeout=30,
)
print(r.json()["tls"]["ja4"])

Install, target selection, verification, and proxy pairing are covered end to end in the guide on web scraping with curl_cffi. The point for this comparison is narrower: impersonation is a curl capability, and Python borrows it rather than owning it.

A TLS handshake costs a round trip or two. On a scrape of ten thousand pages from one host, redoing it every time is the difference between minutes and hours.

Requests

A bare requests.get() opens a connection, uses it, and drops it. A Session keeps a urllib3 PoolManager alive across calls, reusing sockets and carrying cookies. The defaults surprise people under threads:

import requests
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry

session = requests.Session()
adapter = HTTPAdapter(
    pool_connections=20,   # number of distinct host pools to keep
    pool_maxsize=50,       # connections kept PER pool
    max_retries=Retry(total=3, backoff_factor=0.5,
                      status_forcelist=[429, 500, 502, 503, 504]),
)
session.mount("https://", adapter)
session.mount("http://", adapter)

pool_maxsize defaults to 10. Run 50 worker threads against one host with that default and urllib3 logs Connection pool is full, discarding connection: sparkproxy.io. Connection pool size: 10 on every overflow. Nothing crashes. The extra connections are opened, used once, and thrown away, so you silently pay a fresh handshake on 40 of every 50 requests. Set pool_maxsize to at least your thread count. That one line is the most commonly missed speedup in a threaded Requests scraper.

curl

libcurl keeps a connection cache per easy handle, and the CLI has one handle per invocation. So this reuses a single connection for all three pages:

curl -s https://www.sparkproxy.io/ https://www.sparkproxy.io/pricing https://www.sparkproxy.io/docs \
     -o p1.html -o p2.html -o p3.html

And this does not, because each iteration is a new process with a cold cache:

# one full TLS handshake per URL
while read -r u; do
  curl -s "$u" -o "out/$(basename "$u").html"
done < urls.txt

The fix is -Z (--parallel, added in curl 7.66.0), which runs transfers concurrently inside one process and reuses connections:

curl -Z --parallel-max 20 -s -w '%{http_code} %{url_effective}\n' \
     -K urls-config.txt -o '#1.html'

Feed it a config file with one url = "..." line per target. On a large list this often beats the equivalent Python thread pool, because there is no GIL and no per-request object churn.

Requests speaks HTTP/1.1 and nothing else. urllib3 has no HTTP/2 implementation in its stable line, so the ceiling belongs to the transport, not to Requests itself. There is no flag to turn on.

curl has supported HTTP/2 since 7.33.0 when built against nghttp2, and libcurl's default for HTTPS became HTTP/2 in 7.62.0. Check your build:

curl --version | head -2
# curl 8.x.y (x86_64-pc-linux-gnu) libcurl/8.x.y OpenSSL/3.x nghttp2/1.6x.x
# Features: alt-svc AsyncDNS HSTS HTTP2 HTTP3 ...

curl -sI --http2 https://www.sparkproxy.io/ | head -1
# HTTP/2 200

HTTP/3 arrives through --http3, which needs a QUIC-capable build (ngtcp2, quiche, or OpenSSL's QUIC API) and is still labelled experimental in the curl manual. It is rarely worth chasing for scraping.

Two practical consequences follow. Multiplexing means many requests to one host share a single connection, a real win when you pull hundreds of pages from the same domain and no win at all when you fan across hundreds of domains, since each needs its own connection anyway. And, as covered above, negotiating h2 puts you in the same ALPN bucket as browsers instead of standing out. If you want HTTP/2 in Python without leaving the Requests-style API, that is httpx territory, compared in detail in Requests vs httpx.

Both route through a proxy in one line, but the mental models differ and each has a trap.

curl

# HTTP proxy with inline credentials
curl -x http://user:pass@dc.sparkproxy.io:10000 https://www.sparkproxy.io/ip

# same thing with credentials kept out of the URL
curl -x http://dc.sparkproxy.io:10000 -U user:pass https://www.sparkproxy.io/ip

# SOCKS5 with DNS resolved BY the proxy (note the h)
curl -x socks5h://user:pass@dc.sparkproxy.io:10000 https://www.sparkproxy.io/ip

# skip the proxy for internal hosts
curl -x http://dc.sparkproxy.io:10000 \
     --noproxy '*.internal,localhost' https://www.sparkproxy.io/ip

The trap: -U is the proxy credential flag, while lowercase -u sends credentials to the target site. Get them backwards and you see a 407 Proxy Authentication Required that makes no sense. The second trap is the port default. Leave the port off and curl assumes 1080, the SOCKS port, not 8080. Always write it. There's a fuller flag reference in the curl proxy guide.

curl also reads http_proxy, HTTPS_PROXY, ALL_PROXY, and NO_PROXY from the environment. It deliberately ignores an uppercase HTTP_PROXY, because in a CGI process that variable is populated from an attacker-controlled Proxy: request header, the flaw catalogued as httpoxy (CVE-2016-5385 and siblings).

Requests

import requests

proxies = {
    "http":  "http://user:pass@dc.sparkproxy.io:10000",
    "https": "http://user:pass@dc.sparkproxy.io:10000",
}
r = requests.get("https://www.sparkproxy.io/ip", proxies=proxies, timeout=30)
print(r.text)

The trap here catches everyone once: the dictionary key is the scheme of the target URL, not the scheme of the proxy. "https": "http://..." is the normal, correct configuration for an HTTP proxy carrying HTTPS traffic through a CONNECT tunnel, as defined in RFC 9110. People write "https": "https://..." and get a handshake error against a proxy that never offered TLS on its own port.

SOCKS needs an extra install, then the socks5h scheme for proxy-side DNS, documented under proxies in the Requests advanced guide:

pip install "requests[socks]"

proxies = {"https": "socks5h://user:pass@dc.sparkproxy.io:10000"}

Set them once on a Session and every call inherits them. Add trust_env=False when you want certainty that an ambient HTTPS_PROXY on the build machine is not quietly overriding you:

session = requests.Session()
session.trust_env = False
session.proxies.update(proxies)

For rotation logic on top of this, see how to rotate proxies in Python.

curl streams by default. It writes bytes to stdout or to a file as they arrive, so a 4 GB export costs a few kilobytes of RAM:

curl -L --limit-rate 2M -C - -o data/export.csv \
     https://www.sparkproxy.io/exports/daily.csv

-C - resumes a partial file by sending a Range header, --limit-rate throttles, and -L follows redirects. There is no memory decision to make.

Requests does the opposite by default. resp.content reads the whole body into memory before you touch it, which is fine for a 40 KB HTML page and fatal for a large file. Opt in:

import requests

with requests.get(url, stream=True, timeout=(5, 60)) as r:
    r.raise_for_status()
    with open("data/export.csv", "wb") as f:
        for chunk in r.iter_content(chunk_size=1 << 16):
            f.write(chunk)

Two details are worth knowing. The with block matters: under stream=True the connection is not released back to the pool until the body is consumed or the response is closed, so a loop that fetches 200 streamed responses and abandons them will exhaust the pool and stall. And iter_content() decodes gzip for you, while resp.raw.read() does not unless you pass decode_content=True, a classic source of "why is my saved HTML binary garbage".

The timeout tuple above is the other Requests default worth changing. Requests applies no timeout at all unless you ask, so an unresponsive proxy hangs the call forever. curl has no default transfer timeout either, but it exposes --connect-timeout and --max-time, and --max-time is the one that saves cron jobs.

This is curl's genuine superpower, and it has nothing to do with speed. Open DevTools in Chrome, right-click any request in the Network tab, choose Copy as cURL, and you have a byte-accurate replay of what the browser sent: every header, every cookie, the exact body. Paste it into a terminal and within seconds you know whether the endpoint works outside the browser or whether something in the session is required.

curl 'https://www.sparkproxy.io/api/pricing' \
  -H 'accept: application/json' \
  -H 'referer: https://www.sparkproxy.io/pricing' \
  --compressed

From there you delete one header at a time until it breaks. That tells you the minimum viable request, and only then do you port it to Python. Doing it in reverse, guessing headers in Python first, wastes hours.

curl has two more debugging tools with no Requests equivalent. --trace-ascii dumps the full wire conversation including the TLS negotiation summary, and --libcurl writes out C source that reproduces the transfer:

curl -v --trace-ascii trace.log https://www.sparkproxy.io/
curl --libcurl repro.c -x http://dc.sparkproxy.io:10000 https://www.sparkproxy.io/ip

Requests wins the moment the job becomes stateful. Cookie jars that persist across a login flow, response objects you feed straight to BeautifulSoup or lxml, retry policies with backoff, branching on r.status_code, structured error handling, tests. Writing that in bash is possible and unpleasant.

The honest workflow most teams land on: prototype in curl, port to Requests, and when the port starts getting blocked, replace the transport rather than the language. curl_cffi.requests is close enough to a drop-in that the diff is usually the import line plus an impersonate= argument.

Both tools share a ceiling. Neither renders JavaScript, clears a challenge page, or maintains a residential IP pool. Once a target puts a real anti-bot in front of the content, you are maintaining browser builds and fingerprint targets as a side project.

The SparkProxy Scraping API collapses that into one HTTP call. It runs a real Chromium, presents a browser handshake, rotates the IP, and returns HTML, Markdown, JSON, a screenshot, or a PDF. Because it is a plain GET, it works identically from either tool, which is the point.

curl -G "https://scrape.sparkproxy.io/api/v1" \
  -H "X-API-Key: YOUR_API_KEY" \
  --data-urlencode "url=https://www.sparkproxy.io/pricing" \
  --data-urlencode "render_js=true" \
  --data-urlencode "premium_proxy=true" \
  --data-urlencode "country_code=US" \
  -o pricing.html

import requests

resp = requests.get(
    "https://scrape.sparkproxy.io/api/v1",
    headers={"X-API-Key": "YOUR_API_KEY"},
    params={
        "url": "https://www.sparkproxy.io/pricing",
        "render_js": "true",
        "premium_proxy": "true",
        "country_code": "US",
        "wait_for": "#pricing-table",
        "format": "md",
    },
    timeout=90,
)
print(resp.status_code, len(resp.text))

format also takes json, and extract_rules lets you pass CSS selectors so you get parsed fields back instead of raw HTML. Keep curl and Requests for soft targets where they still work. The cost comparison against running your own pool is in web scraping API vs self-managed proxies.