Python requests and aiohttp: Proxies for Async Scraping

Synchronous scraping spends most of its time waiting. Each requests.get() blocks the thread until the server responds, nothing else can run in that thread during the wait.

aiohttp is non-blocking. When one request is awaiting a server response, Python's event loop switches to another coroutine and starts the next one. With 50 concurrent coroutines, those 500 URLs that take 190 seconds synchronously complete in roughly four seconds.

The speedup math is straightforward:

Synchronous (requests):        500 URLs × 0.38s = 190 seconds
Async (aiohttp, 50 workers):   190s ÷ 50 workers = ~4 seconds

Residential proxies slow both numbers (Oxylabs benchmarks at 0.41s in the Global pool), but the multiplier advantage is the same. What changes is where you set the concurrency ceiling: targets with aggressive rate limiting may force you to run 10 workers instead of 50, which still means a 10x speedup over synchronous code.

The gain isn't free CPU time. It's I/O concurrency, Python isn't computing faster, it's not sitting idle waiting for packets.

[CITATION CAPSULE] "With a median success rate of 99.88%, rotating datacenter proxies very rarely fail on their own. The median response time was 0.38 seconds, the fastest participant completed requests over four times quicker than the slowest.", ProxyWay Proxy Market Research 2024

asyncio event loop

---

requests accepts a proxies dict that maps URL schemes to proxy addresses. This is the fastest path to route a synchronous request through a proxy:

import requests

proxy = "http://user:password@gate.example.com:7000"

proxies = {
    "http": proxy,
    "https": proxy,
}

response = requests.get(
    "https://httpbin.org/ip",
    proxies=proxies,
    timeout=30,
)
print(response.json())

Only include the schemes you'll actually use. If all your targets are HTTPS, the "http" key is optional.

For backconnect rotating proxies, a single hostname that assigns a different IP per request, the proxies dict stays the same across every call. The provider rotates server-side:

PROXY_ENDPOINT = "http://user:pass@rotating.example.com:9001"
proxies = {"http": PROXY_ENDPOINT, "https": PROXY_ENDPOINT}

urls = [
    "https://example.com/page-1",
    "https://example.com/page-2",
    "https://example.com/page-3",
]

for url in urls:
    r = requests.get(url, proxies=proxies, timeout=30)
    print(r.status_code, url)

This synchronous pattern is fine for lists under ~50 URLs or when a target enforces strict per-IP concurrency limits that make parallelism counterproductive. For anything larger, move to aiohttp.

backconnect proxy setup

---

aiohttp (v3.13.5 as of 2026) supports HTTP and HTTPS-tunnelled proxies natively through the proxy keyword. No extra library required for standard HTTP/HTTPS targets.

Per-request proxy, assign a specific proxy to each call:

import aiohttp
import asyncio

PROXY = "http://gate.example.com:7000"

async def fetch(session, url):
    async with session.get(url, proxy=PROXY) as response:
        return await response.text()

async def main():
    async with aiohttp.ClientSession() as session:
        html = await fetch(session, "https://httpbin.org/ip")
        print(html)

asyncio.run(main())

Session-level proxy, applied to every request in the session automatically:

async def main():
    async with aiohttp.ClientSession(
        proxy="http://gate.example.com:7000"
    ) as session:
        async with session.get("https://httpbin.org/ip") as resp:
            print(await resp.json())

asyncio.run(main())

Session-level is cleaner when all requests in a batch share one proxy endpoint. Per-request is required when you rotate across a proxy list and each task needs a different endpoint, you can mix both: set a session default, then override it per-task.

One useful flag: ClientSession(trust_env=True) makes aiohttp read HTTP_PROXY and HTTPS_PROXY environment variables automatically. Handy for local testing without changing code:

export HTTP_PROXY="http://localhost:8080"
python scraper.py

[CITATION CAPSULE] "aiohttp supports plain HTTP proxies and HTTP proxies upgradeable to HTTPS via the HTTP CONNECT method. Contrary to the requests library, it won't read environment variables by default, pass trust_env=True to enable.", aiohttp 3.13.5 Official Documentation

---

Most paid proxy services require a username and password. Two patterns handle this in aiohttp.

Option 1: Credentials embedded in the proxy URL

import os
import aiohttp
import asyncio

user = os.environ["PROXY_USER"]
pwd  = os.environ["PROXY_PASS"]
proxy = f"http://{user}:{pwd}@gate.example.com:7000"

async def main():
    async with aiohttp.ClientSession() as session:
        async with session.get("https://httpbin.org/ip", proxy=proxy) as resp:
            print(await resp.json())

asyncio.run(main())

Pull credentials from environment variables, never hardcode them in source files. The f-string builds the authenticated URL at runtime.

Option 2: aiohttp.BasicAuth object

import aiohttp
import asyncio

proxy_auth = aiohttp.BasicAuth("username", "s3cur3p@ss")

async def main():
    async with aiohttp.ClientSession() as session:
        async with session.get(
            "https://httpbin.org/ip",
            proxy="http://gate.example.com:7000",
            proxy_auth=proxy_auth,
        ) as resp:
            print(await resp.json())

asyncio.run(main())

BasicAuth separates the credential concern from the URL. This matters when rotating through multiple proxy hostnames that share the same credentials, you instantiate BasicAuth once and reuse it across every session.get() call without reconstructing the URL each time.

Both approaches are functionally equivalent. The URL-embedded pattern is shorter; BasicAuth is more explicit and easier to audit.

proxy credential management

---

Proxy rotation in aiohttp means each concurrent task picks a different proxy from a pool. The simplest pattern assigns proxies by index modulo pool size:

import aiohttp
import asyncio

PROXIES = [
    "http://user:pass@dc1.example.com:7000",
    "http://user:pass@dc2.example.com:7000",
    "http://user:pass@dc3.example.com:7000",
]

async def fetch(session, url, proxy):
    try:
        async with session.get(
            url,
            proxy=proxy,
            timeout=aiohttp.ClientTimeout(total=30),
        ) as resp:
            return {"url": url, "status": resp.status, "proxy": proxy}
    except Exception as e:
        return {"url": url, "error": str(e), "proxy": proxy}

async def scrape(urls):
    async with aiohttp.ClientSession() as session:
        tasks = [
            fetch(session, url, PROXIES[i % len(PROXIES)])
            for i, url in enumerate(urls)
        ]
        return await asyncio.gather(*tasks)

urls = [f"https://httpbin.org/get?page={i}" for i in range(12)]
results = asyncio.run(scrape(urls))

for r in results:
    print(r)

Twelve URLs across three proxies produces four requests per endpoint, balanced without a proxy manager library.

For backconnect rotating services (one hostname, different IP per request), skip the list entirely and pass the same endpoint to every task. The provider's infrastructure distributes IPs internally, and you don't need client-side rotation at all.

[UNIQUE INSIGHT] Combining client-side round-robin with a backconnect rotating endpoint produces double rotation: your code cycles through three endpoints, each of which already rotates IPs internally. You only need this architecture when splitting traffic across multiple provider sub-users with separate quotas, otherwise it's unnecessary complexity.

[CITATION CAPSULE] "Providers like Bright Data, Smartproxy, and Webshare sell static datacenter lists with an optional rotating layer on top, in addition to regular rotating endpoints.", ProxyWay Proxy Market Research 2024

proxy pool architecture

---

asyncio.gather fires all tasks immediately. For 1,000 URLs, that's 1,000 simultaneous connections, enough to breach TCPConnector limits or trigger IP blocks on the target.

asyncio.Semaphore caps the number of concurrently running coroutines. When all N slots are occupied, new tasks wait until a slot frees:

import aiohttp
import asyncio

PROXIES = [
    "http://user:pass@dc1.example.com:7000",
    "http://user:pass@dc2.example.com:7000",
]

MAX_CONCURRENT = 20  # at most 20 live requests at once

async def fetch(session, semaphore, url, proxy):
    async with semaphore:  # blocks when 20 slots are occupied
        try:
            async with session.get(
                url,
                proxy=proxy,
                timeout=aiohttp.ClientTimeout(total=30),
            ) as resp:
                body = await resp.text()
                return {"url": url, "status": resp.status, "bytes": len(body)}
        except aiohttp.ClientProxyConnectionError as e:
            return {"url": url, "error": f"proxy_fail: {e}"}
        except asyncio.TimeoutError:
            return {"url": url, "error": "timeout"}

async def main(urls):
    semaphore = asyncio.Semaphore(MAX_CONCURRENT)
    connector = aiohttp.TCPConnector(limit=MAX_CONCURRENT)  # match the semaphore
    async with aiohttp.ClientSession(connector=connector) as session:
        tasks = [
            fetch(session, semaphore, url, PROXIES[i % len(PROXIES)])
            for i, url in enumerate(urls)
        ]
        return await asyncio.gather(*tasks)

urls = [f"https://example.com/item/{i}" for i in range(100)]
results = asyncio.run(main(urls))
success = sum(1 for r in results if "status" in r)
print(f"Completed: {success}/{len(urls)}")

A few alignment rules worth following:

• Keep TCPConnector(limit=N) equal to or greater than your semaphore value. If the connector allows 100 connections but the semaphore only permits 20, the connector limit is irrelevant, but setting them out of sync adds confusion during debugging.
• For targets with per-IP rate limits, lower concurrency per proxy endpoint rather than overall. Ten tasks through one endpoint is safer than fifty.
• TCPConnector has a limit_per_host parameter (default: 0, unlimited) for fine-grained control when you're hitting the same domain through multiple proxy IPs.

Concurrency guide (starting points, tune from here):
┌────────────────────────────┬──────────────────┐
│ Target sensitivity         │ MAX_CONCURRENT   │
├────────────────────────────┼──────────────────┤
│ Public APIs / CDN-served   │ 50, 100           │
│ E-commerce product pages   │ 20, 50            │
│ Login-protected / JS-heavy │ 5, 20             │
│ Social media endpoints     │ 3, 10             │
└────────────────────────────┴──────────────────┘

---

Proxies fail. IPs get blocked, connections drop, timeouts fire. A scraper without retry logic stops at the first failure.

aiohttp raises distinct exceptions for proxy-related failures:

• aiohttp.ClientProxyConnectionError, the proxy itself is unreachable
• aiohttp.ClientConnectorError, proxy connects but the target is unreachable
• asyncio.TimeoutError, the request exceeded ClientTimeout
• aiohttp.ClientResponseError, the server responded with an error status

Here's an exponential backoff retry that also rotates the proxy on each failed attempt:

import aiohttp
import asyncio
import random
import logging

logging.basicConfig(level=logging.INFO)
log = logging.getLogger(__name__)

PROXIES = [
    "http://user:pass@dc1.example.com:7000",
    "http://user:pass@dc2.example.com:7000",
    "http://user:pass@dc3.example.com:7000",
]

async def fetch_with_retry(session, url, retries=3):
    proxy = random.choice(PROXIES)
    for attempt in range(retries):
        try:
            async with session.get(
                url,
                proxy=proxy,
                timeout=aiohttp.ClientTimeout(total=30),
            ) as resp:
                if resp.status == 200:
                    return await resp.text()
                log.warning("status=%d url=%s proxy=%s attempt=%d", resp.status, url, proxy, attempt)
                # Non-200: rotate proxy before retrying
                proxy = random.choice(PROXIES)
        except (aiohttp.ClientProxyConnectionError, asyncio.TimeoutError) as e:
            log.warning("error=%s url=%s proxy=%s attempt=%d", type(e).__name__, url, proxy, attempt)
            await asyncio.sleep(2 ** attempt)  # 1s, 2s, 4s
            proxy = random.choice(PROXIES)
    log.error("exhausted retries url=%s", url)
    return None

async def main():
    urls = [f"https://example.com/product/{i}" for i in range(50)]
    async with aiohttp.ClientSession() as session:
        tasks = [fetch_with_retry(session, url) for url in urls]
        results = await asyncio.gather(*tasks)
    success = sum(1 for r in results if r is not None)
    print(f"Success: {success}/{len(urls)}")

asyncio.run(main())

Three points on this pattern:

1. random.choice(PROXIES) selects a new proxy on each retry. If you maintain a round-robin list for even distribution, rotate deterministically instead, PROXIES[(i + attempt) % len(PROXIES)], so every proxy gets coverage across the full URL batch.

1. The 2 ** attempt sleep (1s, 2s, 4s) means a temporarily blocked IP usually clears before retry 3. In practice, we've found this recovers from transient rate-limit responses on most targets without burning proxy quota on rapid-fire retries.

1. Log every retry with the attempted proxy URL, HTTP status, and exception type to a JSONL file. After 1,000 URLs you can rank proxies by retry rate and retire consistently underperforming IPs before they drag your success rate below the 99.88% median seen in benchmark testing.

[UNIQUE INSIGHT] Catching aiohttp.ClientSSLError separately from ClientProxyConnectionError matters when scraping HTTPS targets through a proxy that performs TLS inspection. The two failures need different remediation: SSL errors usually point to proxy configuration, not IP blocking.

proxy error taxonomy

---

The gap between requests and aiohttp for proxy-backed scraping isn't a full rewrite. The proxy syntax is nearly identical: proxies={"https": ...} in requests becomes proxy=... in aiohttp. What changes is the async foundation underneath, one event loop, dozens of concurrent connections, and wall-clock times that shrink from minutes to seconds.

Start with the session-level proxy for batches sharing a single endpoint. Add Semaphore the moment you hit rate limits. Add the retry-and-rotate pattern when you need production-grade resilience. Each step adds about 10, 15 lines of code and meaningfully improves throughput or reliability.

The 99.88% success rate on rotating datacenter proxies isn't a ceiling, it's a floor. Matching the right concurrency level to your proxy tier and target sensitivity is how you stay above it.