Async Web Scraping in Python: A Concurrency Guide

Web scraping is almost pure waiting. A single request spends a few milliseconds building the message, then 200 to 2,000 milliseconds sitting idle while the network and the target server do their work. That idle time is the opportunity. If you can start the next request during it, you collapse total runtime from the sum of every request to roughly the slowest one.

You have three ways to overlap that waiting:

Threads work fine for a few dozen URLs, and they let you keep using synchronous libraries. They stop scaling when you want thousands of connections open at once, because every OS thread carries a real stack and the GIL still serializes Python bytecode between them. asyncio runs all of that cooperative work on one thread, so ten thousand pending requests cost memory measured in kilobytes, not gigabytes.

The rule of thumb: if your bottleneck is waiting on the network, reach for asyncio. If it's CPU (parsing giant HTML trees, decompressing, running a headless browser), async alone won't save you, and you'll need to offload that work off the loop or spread it across processes. More on that in the distributed scraper guide.

asyncio runs one event loop on one thread. The loop holds a set of tasks and runs one at a time until that task hits an await on something that isn't ready yet, usually network I/O. At that point the task yields control back to the loop, which picks another ready task and runs it. Nothing is preemptive. A coroutine keeps the CPU until it voluntarily awaits.

That single fact explains every async scraping bug you'll hit. Concurrency only happens at await points. If a coroutine runs a chunk of code with no await in it, the loop cannot switch away, and every other task waits. Call something that blocks (a synchronous HTTP request, time.sleep, a heavy CPU parse) and you've frozen the entire scraper while pretending it's concurrent.

Keep three words in mind: coroutines pause at await. Design around that and async is straightforward. Fight it and you get a program that looks concurrent and runs sequentially.

Here's the baseline everyone starts with, fetching a list of URLs one at a time:

import httpx

def scrape_sequential(urls):
    results = []
    with httpx.Client(timeout=15) as client:
        for url in urls:
            r = client.get(url)
            results.append((url, r.status_code, len(r.text)))
    return results

For 100 pages at 500 ms each, that's about 50 seconds of mostly-idle waiting. The async version fetches them together:

import asyncio
import httpx

async def fetch(client, url):
    r = await client.get(url)
    return url, r.status_code, len(r.text)

async def scrape_all(urls):
    async with httpx.AsyncClient(timeout=15) as client:
        tasks = [fetch(client, url) for url in urls]
        return await asyncio.gather(*tasks)

results = asyncio.run(scrape_all(urls))

asyncio.gather schedules every coroutine and waits for all of them, returning results in the original order. One AsyncClient is shared across all tasks, which matters: it reuses TCP connections and TLS sessions from a pool instead of paying a fresh handshake for every request.

There's a catch this snippet hides. If urls has 5,000 entries, gather launches 5,000 requests at the same instant. That's a great way to exhaust your file descriptors, saturate the target, and earn a wall of 429 responses or an outright ban. Real scrapers cap concurrency.

asyncio.Semaphore is a counter that a coroutine must acquire before it runs and releases when it's done. Set it to 20 and at most 20 requests are ever in flight, no matter how many tasks you queue. This is the single most important control in a concurrent scraper.

import asyncio
import httpx

async def fetch(client, sem, url):
    async with sem:                      # blocks here until a slot is free
        r = await client.get(url)
        return url, r.status_code, len(r.text)

async def scrape_all(urls, concurrency=20):
    sem = asyncio.Semaphore(concurrency)
    async with httpx.AsyncClient(timeout=15) as client:
        tasks = [fetch(client, sem, url) for url in urls]
        return await asyncio.gather(*tasks)

The async with sem block is the governor. Tasks past the limit park at that line and wake up as slots free. You still create all 5,000 task objects (cheap), but only concurrency of them touch the network at once.

Pick the number deliberately. It should be the smallest of: what your target tolerates before rate-limiting, what your connection pool allows, and what your own machine can handle in open sockets. Twenty is a sane default for a single site. Two hundred is reasonable across many domains with a generous proxy pool behind them. Start low, watch your error rate, and raise it only while responses stay clean.

Two libraries dominate async HTTP in Python. Both are good. They differ in feel and features.

httpx has a requests-style API, supports HTTP/2, and exposes pool tuning through a Limits object. It's the gentler on-ramp if you already know requests.

import httpx

limits = httpx.Limits(max_connections=100, max_keepalive_connections=20)
async with httpx.AsyncClient(http2=True, limits=limits, timeout=15) as client:
    r = await client.get("https://www.sparkproxy.io")

aiohttp is older, extremely fast, and configures its pool through a TCPConnector. It gives you per-host limits and DNS caching out of the box.

import aiohttp

connector = aiohttp.TCPConnector(limit=100, limit_per_host=20, ttl_dns_cache=300)
async with aiohttp.ClientSession(connector=connector) as session:
    async with session.get("https://www.sparkproxy.io") as r:
        body = await r.text()

Note the shape difference: aiohttp responses are context managers, so you read the body inside a second async with. Forget that and you'll leak connections back into a pool that never frees them.

One rule applies to both. Create the client or session once and share it across every task. A common mistake is opening a fresh AsyncClient inside each coroutine, which throws away connection pooling and keepalive entirely, so every request pays a full TCP and TLS handshake. If you need proxy configuration inside these clients, the Python requests and aiohttp proxy guide covers the exact proxy= and auth syntax for each.

Your semaphore limit and your pool limit have to agree, or one of them is a lie. If the semaphore allows 100 concurrent tasks but the pool only holds 20 connections, 80 tasks sit blocked waiting for a connection, and your real concurrency is 20. If the pool is larger than the semaphore, the extra capacity is wasted.

A clean setup for hitting one site hard: semaphore 20, max_connections or limit at 20 to 25, and a per-host cap that matches. For a broad crawl across many domains, raise total connections to 100 or 200 but keep limit_per_host low, around 5 to 10, so no single site sees an abusive burst. The keepalive pool should be large enough that connections get reused between waves of requests instead of being torn down and rebuilt.

asyncio.gather has a default that bites hard at scale. If any task raises, gather propagates that first exception immediately and leaves the rest of your tasks dangling. One bad URL out of 5,000 and you lose the whole batch.

The fix is one keyword. return_exceptions=True tells gather to catch exceptions and return them as ordinary results, so every task finishes and you sort out the failures afterward.

async def scrape_all(urls, concurrency=20):
    sem = asyncio.Semaphore(concurrency)
    async with httpx.AsyncClient(timeout=15) as client:
        tasks = [fetch(client, sem, url) for url in urls]
        outcomes = await asyncio.gather(*tasks, return_exceptions=True)

    ok, failed = [], []
    for url, outcome in zip(urls, outcomes):
        if isinstance(outcome, Exception):
            failed.append((url, repr(outcome)))
        else:
            ok.append(outcome)
    return ok, failed

Now a httpx.ConnectTimeout on one URL becomes a row in failed, not a crash. You can retry the failed list, log it, or feed it back into the queue.

On Python 3.11 and newer you also have asyncio.TaskGroup, which is the structured-concurrency option. A TaskGroup guarantees every child task is awaited and cancels siblings if one fails. Use it when a single failure genuinely means the whole unit of work is void (for example, all pages of one paginated result must succeed together). Use gather(return_exceptions=True) when tasks are independent and partial success is fine, which is the usual scraping case.

async def scrape_group(client, urls):
    results = []
    async with asyncio.TaskGroup() as tg:      # Python 3.11+
        for url in urls:
            tg.create_task(fetch_into(client, url, results))
    return results

Transient failures are normal: a timeout, a dropped connection, a 429, a 503. Retrying immediately makes it worse, because you're adding load exactly when the target is struggling. Back off exponentially and add jitter so a whole batch of failures doesn't retry in lockstep.

import asyncio
import random
import httpx

async def fetch_with_retry(client, sem, url, attempts=4):
    for attempt in range(attempts):
        try:
            async with sem:
                r = await client.get(url, timeout=15)
            if r.status_code == 429:
                wait = float(r.headers.get("Retry-After", 2 ** attempt))
                await asyncio.sleep(wait + random.uniform(0, 0.5))
                continue
            r.raise_for_status()
            return url, r.text
        except (httpx.TransportError, httpx.HTTPStatusError):
            if attempt == attempts - 1:
                raise
            backoff = 2 ** attempt + random.uniform(0, 1)
            await asyncio.sleep(backoff)

Three details matter here. The sleep is asyncio.sleep, never time.sleep, so other tasks keep running while this one waits. The 429 branch reads the server's own Retry-After header before falling back to exponential timing, because the server knows better than your formula. And the jitter (random.uniform) staggers retries so 200 failed tasks don't all wake at the same millisecond and repeat the pileup.

This is the mistake that turns a "concurrent" scraper into a sequential one wearing a costume, and it's the single most common reason async scrapers run no faster than the loop version.

Remember the loop runs on one thread and only switches at await. Any synchronous call with no await freezes every task until it returns. Three usual suspects:

# All three block the entire event loop:
import time, requests
time.sleep(2)                       # use: await asyncio.sleep(2)
requests.get(url)                   # use: await client.get(url) with httpx/aiohttp
soup = BeautifulSoup(huge_html)     # CPU-bound parse, blocks every other task

The first two are easy: swap time.sleep for asyncio.sleep and swap requests for an async client. The third is subtle. Parsing a large HTML document with lxml or BeautifulSoup is CPU work, and while it runs, no other request can proceed. On a page-heavy scrape, parsing can dominate and quietly serialize everything.

The fix is to push CPU-bound work off the loop onto a thread pool with asyncio.to_thread (Python 3.9+):

from bs4 import BeautifulSoup

def parse(html):
    soup = BeautifulSoup(html, "lxml")
    return [a["href"] for a in soup.select("a[href]")]

async def fetch_and_parse(client, sem, url):
    async with sem:
        r = await client.get(url, timeout=15)
    links = await asyncio.to_thread(parse, r.text)   # runs off the loop
    return url, links

Now the parse happens on a worker thread while the loop keeps fetching. For genuinely heavy parsing across many cores, move it to a ProcessPoolExecutor instead, since the GIL still caps thread-based CPU work. A fast sanity check for any async scraper: if wall-clock time barely improves over the sequential version, you're almost certainly blocking the loop somewhere.

When targets sit behind Cloudflare, DataDome, or aggressive rate limits, running raw async requests through your own proxies becomes a second full-time job: rotation, headers, JavaScript rendering, CAPTCHA handling. The SparkProxy Scraping API handles that layer, and your async code just orchestrates calls to it. The endpoint is https://scrape.sparkproxy.io/api/v1, authenticated with an X-API-Key header. See the Scraping API docs for the full parameter list.

The concurrency lesson stays exactly the same: bound it with a semaphore, isolate failures with return_exceptions=True, and honor the API's own 429 signal. On a rate or concurrency limit the API returns status 429 with a retry_after_seconds field, which your backoff should read directly.

import asyncio
import httpx

API = "https://scrape.sparkproxy.io/api/v1"
HEADERS = {"X-API-Key": "YOUR_API_KEY"}

async def scrape_one(client, sem, target, attempts=4):
    params = {
        "url": target,
        "render_js": "true",       # headless Chromium for JS-heavy pages
        "premium_proxy": "true",   # route through residential IPs
        "country_code": "US",
        "json_response": "true",   # wrap result + metadata in JSON
    }
    for attempt in range(attempts):
        async with sem:
            r = await client.get(API, params=params, headers=HEADERS, timeout=60)
        if r.status_code == 429:
            body = r.json()
            await asyncio.sleep(float(body.get("retry_after_seconds", 2 ** attempt)))
            continue
        r.raise_for_status()
        return target, r.json()
    raise RuntimeError(f"exhausted retries for {target}")

async def scrape_many(targets, concurrency=10):
    sem = asyncio.Semaphore(concurrency)
    async with httpx.AsyncClient() as client:
        tasks = [scrape_one(client, sem, t) for t in targets]
        return await asyncio.gather(*tasks, return_exceptions=True)

results = asyncio.run(scrape_many(target_urls))

Set the semaphore to match your plan's concurrency allowance rather than guessing. Because each rendered request with a premium proxy costs more credits (rendering is 5 credits, a premium residential proxy adds more on top), tight concurrency control here protects your budget as well as your success rate. Use a longer client timeout than for raw fetching, since render_js=true spins up a real browser and legitimately takes several seconds per page.

For pulling large volumes this way without hitting walls, pair this pattern with the tactics in scraping high-volume data without rate limiting.

If you run your own proxy pool instead of the API, you spread concurrent tasks across many exit IPs so no single address looks abusive. The simplest reliable approach is a round-robin cursor guarded by a lock, so two coroutines never grab the same slot in a way that skews the rotation.

import asyncio
from itertools import cycle

class ProxyPool:
    def __init__(self, proxies):
        self._cycle = cycle(proxies)
        self._lock = asyncio.Lock()

    async def next(self):
        async with self._lock:
            return next(self._cycle)

async def fetch_rotating(pool, sem, url):
    proxy = await pool.next()
    async with sem:
        async with httpx.AsyncClient(proxy=proxy, timeout=15) as client:
            r = await client.get(url)
            return url, r.status_code

Two caveats. First, spinning up a new AsyncClient per proxy loses pooling, so for heavy rotation keep a small dict of long-lived clients keyed by proxy and reuse them. Second, rotating on every request breaks sites that pin a session to an IP; for those, keep the same proxy for a logical session and rotate between sessions. The mechanics, sticky sessions, and retry-on-ban logic are covered in depth in how to rotate proxies in Python. If you'd rather not run pool logic at all, the web scraping API versus self-managed proxies comparison lays out the tradeoff.