Rate Limited Despite Rotating Proxies: Why It Happens

Throttling doesn't always arrive as a tidy 429, so first match what you see against the forms it takes. Each line below comes from a primary spec or vendor doc, cited inline.

The textbook version, from the example in RFC 6585, section 4:

HTTP/1.1 429 Too Many Requests
Content-Type: text/html
Retry-After: 3600

The same RFC's example body reads "I only allow 50 requests per hour to this Web site per logged in user." Note the last four words. The limit in the standard's own example is per user, not per IP.

nginx returns 503 by default. The limit_req_status directive in nginx's ngx_http_limit_req_module documents its default as limit_req_status 503;. If your scraper treats 503 as "server flaky, retry harder", you may be hammering a rate limiter without knowing it.

Some APIs use 403. GitHub's REST docs say that when you exceed the primary rate limit "you will receive a 403 or 429 response, and the x-ratelimit-remaining header will be 0".

Some servers just drop you. RFC 6585, section 7.2, says servers "are not required to use the 429 status code" and that under heavy load "it may be more appropriate to just drop connections". Timeouts and resets that start at a predictable request count belong in this bucket.

It may not be the target at all. If you send traffic through the SparkProxy Scraping API, its documented 429 bodies are JSON, and they look like this:

{ "error": "Rate limit exceeded", "retry_after_seconds": 60, "limit": 60 }
{ "error": "Concurrency limit reached", "active": 3, "limit": 3, "retry_after_seconds": 5 }

Those are limits on your own API key. No proxy rotation touches them.

Every rate limiter keeps a counter per something. RFC 6585 is explicit that the something is up to the server: it "does not define how the origin server identifies the user, nor how it counts requests", and it "might identify the user by its authentication credentials, or a stateful cookie".

The tools sites actually deploy confirm how wide that choice is. Here is what three common ones let an operator count on, read from each vendor's own documentation on 2026-10-30:

Two consequences follow. A Cloudflare rule keyed on a cookie, a JA4 hash or an AS number ignores your rotation completely. And an AWS WAF rule keyed on ASN counts every exit inside one network as a single client, however many addresses you cycle through. Rotation spreads requests across one dimension; the limiter may be counting along a different one.

The IETF's draft RateLimit header fields (draft-ietf-httpapi-ratelimit-headers-11, which expires 24 November 2026) even give servers a way to say so. Its pk parameter "conveys the partition key associated to the request", and the draft's example policy is named "peruser". It's still a draft, and few sites send it, but when you see one it answers the whole question.

Work down this table and stop at the first check that explains the failure. The first three cost a handful of requests; only the fourth spends real traffic.

With the Scraping API, a 429 can be the API's own limit or the target's, so separate them. Run in plain HTTP mode (render_js=false, 1 credit) with transparent_status_code=true, which the docs describe as mirroring "the target page's HTTP status code in the API response" and which only applies when render_js=false.

import requests

API = "https://scrape.sparkproxy.io/api/v1"
r = requests.get(
    API,
    headers={"X-API-Key": "YOUR_API_KEY"},
    params={
        "url": "https://www.sparkproxy.io/blog/",
        "render_js": "false",              # defaults to true (5 credits); false is 1 credit
        "proxy_type": "premium",
        "transparent_status_code": "true",  # mirror the target's status code
    },
    timeout=60,
)

if r.status_code == 429:
    try:
        body = r.json()
    except ValueError:
        body = {}
    if "retry_after_seconds" in body:
        print("API-key limit:", body.get("error"), "wait", body["retry_after_seconds"], "s")
    else:
        print("Target returned 429; Retry-After =", r.headers.get("Retry-After"))
else:
    print(r.status_code)

The JSON keys used for the check are the ones in the API's documented 429 bodies shown above. A concurrency 429 means you are running more parallel jobs than your key allows. Lower your worker count; adding exits won't help.

For raw proxy users, the equivalent question is whether the response came from the gateway or the site. A gateway problem shows up as a 407 or a failed CONNECT rather than a 429, and our proxy error codes reference covers that family.

This one catches more people than any exotic fingerprinting. HTTP clients reuse connections, and through a proxy an HTTPS request travels inside a CONNECT tunnel. Reuse the tunnel and you reuse whatever exit the gateway assigned to it.

We measured this on 2026-10-30 with Python 3.14.2 and requests 2.32.5, pointing a requests.Session at a local logging proxy. Ten HTTPS requests through one Session opened one CONNECT tunnel. Five bare requests.get() calls opened five. Five Session requests sent with a Connection: close header also opened five.

Whether that pins your exit depends on whether your rotating gateway picks a new exit per request or per connection, which you should verify rather than assume. This script does it:

import requests

PROXY = "http://USER:PASS@gateway.sparkproxy.io:11000"   # Random Rotation port
P = {"http": PROXY, "https": PROXY}
PROBE = "https://api.ipify.org"

with requests.Session() as s:
    s.proxies = P
    reused = [s.get(PROBE, timeout=20).text.strip() for _ in range(5)]

fresh = [requests.get(PROBE, proxies=P, timeout=20).text.strip() for _ in range(5)]

print("one Session :", reused, "unique:", len(set(reused)))
print("fresh each  :", fresh, "unique:", len(set(fresh)))

If the first line shows one address and the second shows five, your "rotation" has been a single IP all along. Open a new connection per logical request (a fresh Session, or Connection: close), and the IP-keyed case may simply go away.

The 429 you already have is free evidence. Print every header that hints at the window or the key:

import requests

def explain(r: requests.Response) -> None:
    print("status:", r.status_code)
    for name, value in r.headers.items():
        lname = name.lower()
        if lname in ("retry-after", "ratelimit", "ratelimit-policy") or lname.startswith("x-ratelimit"):
            print(f"  {name}: {value}")
    for c in r.cookies:
        print("  set-cookie:", c.name)

How to read the output:

• Retry-After: 3600 on a rotated request is a strong hint. A pure per-IP limiter has no reason to make a brand-new address wait an hour.
• RateLimit-Policy with pk= is the server naming its partition key outright.
• x-ratelimit-remaining that keeps falling across different exits means the counter follows something you carry. GitHub documents this pattern for authenticated calls: requests made with your token "count towards your personal rate limit of 5,000 requests per hour", while unauthenticated ones "are associated with the originating IP address" and get 60.
• A cookie set on the first response that you keep returning is the prime suspect for Check 4.

Now trip the limit once on purpose, then change exactly one variable per follow-up request. Use Sticky Sessions (port 11002, which holds an exit for up to 5 minutes) to keep the IP fixed, and Random Rotation (port 11000) to change it. Probe a single URL you are allowed to fetch, honour any Retry-After you receive, and stop as soon as you have an answer.

import time
import requests

URL = "https://www.sparkproxy.io/blog/"            # the page that gets throttled
STICKY = "http://USER:PASS@gateway.sparkproxy.io:11002"
ROTATE = "http://USER:PASS@gateway.sparkproxy.io:11000"
UA = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 Chrome/141.0 Safari/537.36"

def client(proxy, cookies=None):
    s = requests.Session()
    s.proxies = {"http": proxy, "https": proxy}
    s.headers["User-Agent"] = UA
    if cookies:
        s.cookies.update(cookies)
    return s

def exit_ip(s):
    return s.get("https://api.ipify.org", timeout=20).text.strip()

def throttled(r):
    return r.status_code in (403, 429, 503) or r.headers.get("x-ratelimit-remaining") == "0"

base = client(STICKY)
start_ip = exit_ip(base)
for n in range(1, 301):
    r = base.get(URL, timeout=30)
    if throttled(r):
        print(f"tripped after {n} requests on {start_ip}: {r.status_code}")
        break
    time.sleep(1)
else:
    raise SystemExit("never throttled at 1 req/s; the limit is not your problem")

jar = base.cookies.get_dict()
trials = {
    "same IP, fresh cookies": client(STICKY),
    "new IP, same cookies": client(ROTATE, jar),
    "new IP, fresh cookies": client(ROTATE),
}
for name, s in trials.items():
    r = s.get(URL, timeout=30)
    print(f"{name:24} -> {r.status_code}  via {exit_ip(s)}")

The script prints the exit for each trial so you can confirm the IP really did or didn't change, rather than trusting the port. If same IP, fresh cookies shows a different address from start_ip, the 5-minute window rolled over; rerun.

Read the three results against this table:

The last row needs one more round. Change the client fingerprint while keeping everything else (see the fingerprint fix below); if that passes, it's the fingerprint. If not, log failures by /24 as shown under the subnet fix. If you're logged in or sending an API token, assume the account until proven otherwise.

Keyed on the session cookie

Bind the cookie jar to the exit, and rotate them as one unit. An identity is "this exit plus this jar", and when either changes, both should. With Random Rotation, that means no persistent jar at all: a fresh client per request. With Sticky Sessions, keep one jar for the life of the exit and throw it away when the exit changes:

import requests

STICKY = "http://USER:PASS@gateway.sparkproxy.io:11002"

class Identity:
    """One exit IP plus one cookie jar. Replace both together."""

    def __init__(self):
        self.s = requests.Session()
        self.s.proxies = {"http": STICKY, "https": STICKY}
        self.ip = self.s.get("https://api.ipify.org", timeout=20).text.strip()

    def still_valid(self):
        return self.s.get("https://api.ipify.org", timeout=20).text.strip() == self.ip

def fetch(ident, url):
    if not ident.still_valid():
        ident = Identity()          # exit rolled over: start a clean jar too
    return ident, ident.s.get(url, timeout=30)

Calling still_valid() before every page doubles your request count, so in production check it every few pages or on a timer under 5 minutes. For multi-step flows that store state in cookies (carts, location pickers), keep the whole flow inside one identity. Our guide to handling cookies and sessions in scrapers covers jar mechanics, and what a sticky session proxy is explains the session model itself.

Keyed on an API key or logged-in account

Here a proxy can't help, and rotating accounts to escape a per-account quota usually breaks the site's terms. The fix is to spend the quota you have well: read the remaining budget from the headers, pause until the reset time, and cut requests you don't need with caching.

import time
import requests

def get_within_quota(s: requests.Session, url: str) -> requests.Response:
    while True:
        r = s.get(url, timeout=30)
        remaining = r.headers.get("x-ratelimit-remaining")
        reset = r.headers.get("x-ratelimit-reset")       # epoch seconds, GitHub-style
        if r.status_code in (403, 429) and remaining == "0" and reset:
            time.sleep(max(0, int(reset) - int(time.time())) + 1)
            continue
        if r.status_code == 429 and r.headers.get("Retry-After", "").isdigit():
            time.sleep(int(r.headers["Retry-After"]))
            continue
        return r

If the quota is too small for the job, ask for a higher one. GitHub, for example, documents a 15,000-per-hour limit for GitHub Apps owned by an Enterprise Cloud organization. That is a commercial conversation, not a proxy setting.

Keyed on a TLS or HTTP fingerprint

Cloudflare lists JA3 and JA4 as rate-limit characteristics. Every request your Python script sends carries the same TLS handshake, so if a site counts by JA4, all your rotated exits land in one small bucket labelled "this HTTP library". The workable fix isn't randomising fingerprints. It's sending one that mainstream browsers also send, so your requests sit in the same bucket as ordinary visitors. curl_cffi (0.17.0 on PyPI, released 2026-10-26) does this with its impersonate option:

from curl_cffi import requests as creq

PROXY = "http://USER:PASS@gateway.sparkproxy.io:11000"
r = creq.get(
    "https://www.sparkproxy.io/blog/",
    impersonate="chrome",
    proxies={"http": PROXY, "https": PROXY},
    timeout=30,
)
print(r.status_code)

Rerun the last row of the experiment table with this client. If it now passes, the fingerprint was the key. Our explainer on TLS fingerprinting shows how JA3 and JA4 are built.

Keyed on the subnet or ASN

A limiter keyed on /24 or AS number sees your rotation as a few busy blocks. You spot it in your logs: failures cluster by block while exits elsewhere in the pool keep succeeding at the same moment.

import ipaddress
from collections import defaultdict

def by_block(log):
    """log: iterable of (exit_ip, status). Prints throttle rate per /24."""
    stats = defaultdict(lambda: [0, 0])
    for ip, status in log:
        block = str(ipaddress.ip_network(f"{ip}/24", strict=False))
        stats[block][0] += 1
        stats[block][1] += status in (403, 429, 503)
    for block, (total, bad) in sorted(stats.items(), key=lambda kv: -kv[1][1]):
        print(f"{block:18} {bad:4}/{total:<4} throttled")

If a few blocks carry nearly all the failures, budget requests per block instead of per IP. Our mixed-subnet datacenter proxies guide covers measuring a pool's spread and the arithmetic of range-level throughput, so we won't repeat it. An ASN-keyed limit is harder: every datacenter exit belongs to a hosting network, so the honest options are a lower rate or a different source of traffic.

Keyed on the path, or counting everything

AWS WAF's "count all" option, or a Cloudflare rule keyed only on path, limits the endpoint itself. Nobody gets more than N requests per window, whoever they are. Nothing you rotate helps. Slow down, spread the crawl over a longer schedule, and use the general throttling patterns in scraping high-volume data without rate limiting and our retry and backoff strategies.

Only one of nine rows is a clean "yes". That ratio is why buying more IPs is such a common, and expensive, non-fix.

Match the port to the key you found:

• Random Rotation on gateway.sparkproxy.io:11000 for stateless page fetches against an IP-keyed limit. Open a fresh connection per request so the rotation reaches the target.
• Sticky Sessions on gateway.sparkproxy.io:11002 when the site ties state to a cookie. One exit plus one jar for up to 5 minutes, then both get replaced.
• SOCKS5 on port 13000 if your tooling needs it. HTTP, HTTPS and SOCKS5 are all supported, with either user:pass or IP whitelist authentication.

Proxy plans start from $75/month (USD, read from sparkproxy.io on 2026-10-30), billed as packs with unlimited bandwidth, so the unit you size is concurrency rather than gigabytes. The pool is 1M+ premium IPs across 80+ countries on 1 Gbps ports, and there's a 24-hour free trial without a credit card. Longer terms take 5% off for 3 months, 10% for 6 and 15% for 12.

For the Scraping API, two documented behaviours matter for this problem. In rendered mode, the session_id docs state that "each request gets a fresh browser profile regardless of the session_id value", so no cookie carries over between calls and a cookie-keyed counter starts from zero each time. In plain mode, render_js=false costs 1 credit and transparent_status_code=true shows you the target's real status. render_js defaults to true at 5 credits, country_code adds 5, and new accounts get 1,000 free credits. Run Check 1 first, though: the API has its own per-key limit, and that 429 isn't the target's.