Requests vs httpx for Web Scraping (2026)

Both libraries send HTTP requests and hand you a response object with .status_code, .text, and .json(). If all you do is fetch one URL and read the body, they behave almost identically. The differences show up the moment you scale up, add concurrency, route through proxies, or care about defaults. Here is the head-to-head, scoped to scraping.

The short version: reach for httpx when you need async concurrency or HTTP/2, and stay on requests when you want the simplest possible dependency with the widest compatibility. The rest of this guide explains why, and where each choice costs you.

httpx was designed to feel like requests. The top-level calls match, the response object matches, and most requests code ports over by changing the import. A minimal GET looks the same in both:

# requests
import requests
r = requests.get("https://httpbin.org/get", params={"q": "proxies"}, timeout=30)
print(r.status_code, r.json())

# httpx
import httpx
r = httpx.get("https://httpbin.org/get", params={"q": "proxies"}, timeout=30)
print(r.status_code, r.json())

That similarity is the single biggest reason httpx gets adopted: the learning cost is near zero if you already know requests. The differences are in the details you don't see in a one-liner. requests groups a reusable connection under requests.Session(); httpx calls the same thing httpx.Client() and encourages you to use it as a context manager so it closes cleanly. httpx returns a slightly stricter response, and it changes two defaults (timeouts and redirects) that silently alter behavior when you port a real scraper. We cover both below.

This is the difference that decides most projects. requests is synchronous only. Every call blocks the thread until the response comes back, so to fetch 500 URLs concurrently with requests you reach for threads:

from concurrent.futures import ThreadPoolExecutor
import requests

def fetch(url):
    return requests.get(url, timeout=30).status_code

with ThreadPoolExecutor(max_workers=20) as pool:
    results = list(pool.map(fetch, urls))

Threads work, but each one carries real memory overhead, and the GIL plus context switching caps how far you scale on one box. httpx ships a real async client. One event loop, one thread, thousands of requests in flight:

import asyncio
import httpx

async def fetch(client, url):
    r = await client.get(url, timeout=30)
    return r.status_code

async def main(urls):
    limits = httpx.Limits(max_connections=50)
    async with httpx.AsyncClient(limits=limits) as client:
        tasks = [fetch(client, u) for u in urls]
        return await asyncio.gather(*tasks, return_exceptions=True)

asyncio.run(main(urls))

httpx.AsyncClient is why teams migrate. It handles I/O-bound scraping, which is almost pure waiting, far more efficiently than a thread pool, and it does so with a fraction of the memory. If you want the production patterns (bounding concurrency with a semaphore, sizing the pool, isolating failures so one bad task doesn't cancel the batch), our async web scraping in Python guide goes deep on exactly that. The takeaway here: if concurrency is on your roadmap, httpx removes the reason to bolt threads onto requests later.

requests speaks HTTP/1.1 and nothing newer. There is no flag to change that, because urllib3 underneath it is HTTP/1.1 only. httpx supports HTTP/2, but it's off by default and needs an extra package:

# pip install "httpx[http2]"
import asyncio
import httpx

async def main():
    async with httpx.AsyncClient(http2=True) as client:
        r = await client.get("https://httpbin.org/get")
        print(r.http_version)   # "HTTP/2" if the server negotiated it

asyncio.run(main())

Be honest about what HTTP/2 buys a scraper, because most comparisons oversell it. HTTP/2 multiplexes many requests over a single connection, so it shines when you hammer one host with many requests: you skip repeated connection setup and TLS handshakes. If you scrape 5,000 pages from the same domain, that's a real saving. If you scrape one page each from 5,000 different domains, multiplexing barely helps, because you open a fresh connection per host anyway. HTTP/2 also lets a server present a more browser-like protocol footprint, which can matter against anti-bot systems that flag HTTP/1.1 clients. Neither library supports HTTP/3 yet in 2026.

Two default differences cause more porting bugs than anything else, and both favor learning httpx's behavior before you trust it.

Timeouts. requests has no default timeout. If you forget to pass timeout=, a stalled server or a dead proxy can hang the call forever, which is a classic way a scraper silently freezes overnight. httpx applies a default timeout of 5 seconds to connect, read, write, and pool. That's safer, but it flips a subtle trap: 5 seconds is aggressive for scraping through residential proxies, where a slow target can legitimately take longer. Ported code that "worked on requests" starts throwing httpx.ReadTimeout. The fix is to raise it deliberately:

import httpx

# Loosen httpx's 5s default for slow proxied targets
timeout = httpx.Timeout(30.0, connect=10.0)
with httpx.Client(timeout=timeout) as client:
    r = client.get("https://httpbin.org/delay/8")

Redirects. requests follows redirects automatically. httpx does not, since version 0.20. A login flow or a 301 to the canonical URL that "just worked" on requests returns a 302 body on httpx until you opt in:

r = httpx.get("https://httpbin.org/redirect/2", follow_redirects=True)

Neither default is wrong. requests optimizes for "it works out of the box," httpx for "no surprises hidden from you." But if you migrate a scraper and skip these two lines, you'll chase phantom bugs.

Both libraries reuse TCP connections, and both share the same gotcha: the top-level helpers (requests.get(), httpx.get()) create a client, use it once, and throw it away, so you get no connection reuse across calls. For any real scraper, hold a client open.

requests does this with a Session and an HTTPAdapter to size the pool:

import requests
from requests.adapters import HTTPAdapter

session = requests.Session()
session.mount("https://", HTTPAdapter(pool_connections=20, pool_maxsize=20))
r = session.get("https://httpbin.org/ip", timeout=30)

httpx does it with a Client and a Limits object, which separates total connections from idle keepalive connections:

import httpx

limits = httpx.Limits(max_connections=100, max_keepalive_connections=20)
with httpx.Client(limits=limits, timeout=30) as client:
    r = client.get("https://httpbin.org/ip")

The mental model is the same, the knobs differ. httpx's split between max_connections and max_keepalive_connections gives you finer control over how many idle sockets you hold open, which matters when you fan out across many hosts and don't want to pin thousands of keepalive connections.

No scraper runs without proxies, and this is where the two libraries diverge most, including a change that breaks a lot of outdated tutorials.

requests takes a proxies dict keyed by scheme, with credentials inline in the URL:

import requests

proxies = {
    "http":  "http://sp_user:sp_pass@dc.sparkproxy.io:10000",
    "https": "http://sp_user:sp_pass@dc.sparkproxy.io:10000",
}
r = requests.get("https://httpbin.org/ip", proxies=proxies, timeout=30)
print(r.json())

For a SOCKS5 proxy, install the extra and use the socks5h:// scheme so DNS resolves through the proxy (avoiding a DNS leak):

# pip install "requests[socks]"
proxies = {
    "http":  "socks5h://sp_user:sp_pass@dc.sparkproxy.io:10800",
    "https": "socks5h://sp_user:sp_pass@dc.sparkproxy.io:10800",
}
r = requests.get("https://httpbin.org/ip", proxies=proxies, timeout=30)

httpx used to accept the same proxies dict, and it no longer does. This is the gotcha. proxies was deprecated in httpx 0.26 and removed entirely in 0.28. Code copied from a 2023 tutorial that passes httpx.Client(proxies=...) now raises TypeError on a current install. The replacement is proxy= for one proxy across all traffic:

import httpx

# httpx 0.28+: a single proxy for everything, note proxy= not proxies=
with httpx.Client(
    proxy="http://sp_user:sp_pass@dc.sparkproxy.io:10000",
    timeout=30,
) as client:
    r = client.get("https://httpbin.org/ip")
    print(r.json())

When you need different proxies per URL pattern (the thing the old dict did), use mounts, which maps a URL pattern to a transport. It's more verbose but far more capable, since you can route by scheme or host and bypass the proxy for specific domains:

import httpx

mounts = {
    "all://": httpx.HTTPTransport(
        proxy="http://sp_user:sp_pass@dc.sparkproxy.io:10000"
    ),
    "all://www.sparkproxy.io": None,   # go direct for this host, no proxy
}
with httpx.Client(mounts=mounts, timeout=30) as client:
    r = client.get("https://httpbin.org/ip")

SOCKS in httpx uses the same proxy= argument with a socks5:// URL (install httpx[socks]):

# pip install "httpx[socks]"
with httpx.Client(proxy="socks5://sp_user:sp_pass@dc.sparkproxy.io:10800") as client:
    r = client.get("https://httpbin.org/ip")

So requests gives you one simple dict, and httpx trades that for a proxy/mounts split that's stricter but supports per-host routing natively. If you rotate proxies rather than pin one, the pattern is the same in both libraries: build a new client (or swap the dict) per IP. Our guide on how to rotate proxies in Python covers the rotation loop for each.

It's tempting to ask "which is faster," but the honest answer has conditions. For a single synchronous request, the two are close, and requests is sometimes marginally faster because it carries less abstraction per call. Anyone who tells you httpx is dramatically faster at plain synchronous GETs is usually measuring noise. The real speedups from httpx come from two specific capabilities:

• Async concurrency. httpx.AsyncClient on an event loop overlaps hundreds of requests in one thread. For I/O-bound scraping this crushes a synchronous requests loop and beats a thread pool on memory. This is the big one.
• HTTP/2 multiplexing. Against a single host, reusing one connection for many requests removes repeated handshakes. Against many different hosts, the benefit shrinks toward zero.

Two rules follow. First, if your workload is synchronous and low-volume, don't switch libraries expecting a speed bump; you won't get a meaningful one. Second, always benchmark on your own targets, not on a generic "N requests to httpbin" test, because your proxy latency and the target's response time dominate the result far more than the client does. The client is rarely your bottleneck. Your proxies and the target usually are.

This is a genuine trade-off, not a tie. requests is one of the most-downloaded Python packages ever, sits at a stable 2.32.x, and is effectively feature-complete. It still receives security fixes (the 2.32 line patched real CVEs), but it isn't adding async or HTTP/2 by design. That stability is a feature: a scraper written on requests five years ago still runs, and the surface won't shift under you.

httpx is younger, maintained by Encode (the group behind Starlette and Uvicorn), and still pre-1.0 at 0.28.x. Active development means new capabilities, but it also means occasional breaking changes between minor versions. The removal of proxies in 0.28 is the perfect example: a breaking change in a 0.x release that quietly invalidated a lot of published code. That's the price of momentum. Pin your httpx version in production and read the changelog before you upgrade, or a minor bump can break your proxy setup.

Put plainly: requests optimizes for "won't surprise you in three years," httpx for "keeps gaining features now." Neither is abandoned. They're at different points in their lifecycle.

There's no universal winner, only a best fit for your constraints. Map your situation to the table.

The pattern: requests is the safe default for straightforward, synchronous work and maximum stability. httpx is the upgrade when async, HTTP/2, or a modern typed codebase pays for the pre-1.0 churn. And if your actual problem is getting blocked, changing HTTP client won't fix it, which is the last section.

Here's the point both camps skip. Whichever library you pick, your requests carry a TLS handshake fingerprint (JA3/JA4) that does not look like a real browser. Anti-bot systems like Cloudflare and DataDome read that fingerprint before they ever see your headers or your IP, and they block both requests and httpx on that signal alone. A clean residential proxy does not change your TLS fingerprint. This is why a scraper that works locally suddenly returns 403 on a protected target, and why swapping requests for httpx makes no difference against it. Background on the mechanism is in what is TLS fingerprinting.

You have two ways out. Use a client that impersonates a browser's TLS stack, such as curl_cffi. Or stop managing clients, proxies, and fingerprints yourself and call an API that handles all of it. The SparkProxy Scraping API runs the browser, rotates residential IPs, and presents a real fingerprint behind one endpoint. You send a URL and get HTML or structured data back:

import requests

resp = requests.get(
    "https://scrape.sparkproxy.io/api/v1",
    headers={"X-API-Key": "YOUR_API_KEY"},
    params={
        "url": "https://www.sparkproxy.io/pricing",
        "render_js": "true",       # runs a headless browser for you
        "premium_proxy": "true",   # residential IP pool
        "country_code": "US",
    },
    timeout=90,
)
print(resp.status_code, len(resp.text))

Because the endpoint mirrors a normal HTTP call, you can drive it with either library, and the async story stays intact: fan out API calls concurrently with httpx.AsyncClient when you need throughput.

import asyncio
import httpx

async def scrape(client, url):
    r = await client.get(
        "https://scrape.sparkproxy.io/api/v1",
        headers={"X-API-Key": "YOUR_API_KEY"},
        params={"url": url, "render_js": "true", "premium_proxy": "true"},
        timeout=90,
    )
    return r.status_code

async def main(targets):
    async with httpx.AsyncClient() as client:
        return await asyncio.gather(*(scrape(client, u) for u in targets))

asyncio.run(main(["https://www.sparkproxy.io/", "https://www.sparkproxy.io/pricing"]))

Keep requests or httpx for direct scraping of soft targets. Reach for the API when the fingerprinting and block-rate arms race, not the HTTP client, is what's costing you time. The cost math behind that call is in web scraping API vs self-managed proxies.