How to Bypass AWS WAF When Web Scraping

One plain statement, then we move on. Circumventing an access control can breach a site's terms of service and, in some jurisdictions, computer-misuse law. Everything below is for data you are permitted to access: public pages, your own properties, a client's site with written authorization, or a target whose terms allow automated collection. If a site has told you no, the answer is no.

So this guide contains no payload-level filter evasion, no CAPTCHA-solving walkthrough, no token replay, and no human-farm referrals. Those techniques attack the security control itself, which is both the legally risky path and, in practice, the one that burns your IP pool fastest.

What's left is more useful anyway. AWS WAF blocks a lot of well-behaved automation by accident, because of how it was configured rather than because of anything the scraper did wrong. The skill worth having is diagnosing which rule fired and adjusting so it stops firing. That's avoidance, and it's durable.

The three questions worth answering first

• Does the target publish an API, a bulk export, or a data feed? If yes, use it and stop reading.
• Are you being blocked on your IP, on your client, or on your rate? These have completely different fixes, and the response tells you which.
• Is the volume you want actually necessary? Most scraping projects request ten times more than they use.

This is the part most guides get wrong, and it explains why advice about AWS WAF is so inconsistent across the web. Cloudflare Bot Management, DataDome and Kasada are vendor-operated systems with defaults the vendor controls. AWS WAF is a set of building blocks the site owner assembles themselves.

A site owner creates a web ACL and attaches it to a CloudFront distribution, an Application Load Balancer, an API Gateway stage, an AppSync API, a Cognito user pool, an App Runner service, or Verified Access. Inside the web ACL they add rules in priority order. Each rule inspects something and applies one of five rule actions: Allow, Block, Count, CAPTCHA, or Challenge. Evaluation stops at the first terminating action.

That means two sites can both be "protected by AWS WAF" and behave nothing alike. One might run a single geo-match rule. Another might run the full core rule set plus IP reputation plus Bot Control at the targeted tier. Your scraper meets whatever that particular team built, on whatever budget they had.

The budget shapes the config, and that helps you

Rules cost web ACL capacity units. The base price for a web ACL includes up to 1,500 WCUs, going over that incurs tiered additional fees, and the hard maximum is 5,000. Now look at what the popular managed rule groups cost:

The core rule set alone eats 700 of the included 1,500. Bot Control is cheap in WCUs but carries its own subscription and per-request charges, and the targeted tier costs more again. The practical consequence: a great many AWS WAF deployments are the core rule set plus one or two IP lists, with no bot detection at all. When one of those blocks you, it is not because it decided you were a bot. It's because you tripped a generic rule.

Test that assumption before you spend a week engineering around a bot defense that isn't there.

Here's the full menu, ordered roughly by how often scrapers run into each one.

Why datacenter ranges die before you do anything

Two of those layers judge you purely on your exit IP, with zero behavioral input. The Anonymous IP list rule group contains exactly two rules, and both default to Block:

• AnonymousIPList covers sources known to anonymize client information: Tor nodes, temporary proxies, and other masking services.
• HostingProviderIPList covers IP ranges belonging to web hosting and cloud providers, on the reasoning that they are less likely to source genuine end-user traffic.

That second rule is the one that quietly kills most scraping stacks. Rent a VPS anywhere, point a crawler at a site running this rule group, and you get a 403 on request number one. No fingerprinting was involved. Your ASN was the whole story, which is why understanding what a datacenter ASN is and what IP reputation actually measures matters more here than any header trick.

One line in the AWS documentation is worth reading twice: the HostingProviderIPList "IP list does not include AWS IP addresses." A scraper running on EC2 is not caught by that specific rule, while the same code on a competing cloud is. That's an oddity of the list, not a strategy. Where Bot Control is enabled, the SignalKnownBotDataCenter rule and the signal:cloud_service_provider:aws label still flag AWS traffic. Useful for diagnosis, not for hiding.

Core rule set rules that catch honest clients

The core rule set is aimed at OWASP-style attacks, but four of its rules routinely block ordinary crawlers:

None of that is bot detection. A crawler that accumulates cookies across thousands of pages and never prunes them will eventually cross 10 KB and start collecting 403s that look exactly like a ban. The URI path limit is 1,024 bytes and the request body limit is 8,192 bytes for ALB and AppSync targets, so deep faceted URLs and large GraphQL payloads hit the same wall.

AWS WAF tells you which mechanism stopped you, if you actually look. The signature is the status code plus the x-amzn-waf-action response header, and the documented action behavior is precise about it.

Two subtleties save real debugging time.

First, 405 for CAPTCHA and 202 for Challenge are counterintuitive on purpose, and both are unusual enough to be a reliable fingerprint. If your logs show a wall of 202s with empty bodies, you are not looking at a broken server. You are looking at an AWS WAF Challenge action.

Second, the interstitial only appears when the request carries an Accept header containing text/html. An API client asking for JSON gets the status code and the header with no body at all. That's why "the endpoint returns an empty 202" is such a common and confusing bug report. A minimal probe settles it:

import requests

def probe_waf(url, timeout=20):
    """Read-only probe. Identifies AWS WAF actions from status + header."""
    r = requests.get(url, headers={"Accept": "text/html"}, timeout=timeout,
                     allow_redirects=False)
    action = r.headers.get("x-amzn-waf-action")
    verdict = {
        (405, "captcha"): "CAPTCHA action, no valid aws-waf-token",
        (202, "challenge"): "Challenge action, browser challenge required",
    }.get((r.status_code, action))

    if verdict is None:
        verdict = "Block action (rule unknown)" if r.status_code == 403 else "no WAF action"

    print(f"{r.status_code} action={action} -> {verdict}")
    return r.status_code, action

A 403 with no x-amzn-waf-action header is the ambiguous case, and it's also the most common. Narrow it down by changing one variable at a time: same request from a residential IP, then same IP with a browser User-Agent, then same everything at a tenth of the rate. Whichever change clears it names the layer.

A rate-based rule statement counts requests per aggregation key over a rolling window and applies an action when the count crosses a limit. The exact numbers matter for planning a crawl:

• Evaluation window: 60, 120, 300, or 600 seconds. The default is 300.
• Minimum rate limit: 10. AWS lowered this floor, so a site can legitimately rate-limit at ten requests per window.
• Aggregation: source IP by default, or a forwarded IP from a header such as X-Forwarded-For, or up to five custom keys built from headers, cookies, query arguments, URI path, HTTP method, or label namespaces.
• Precision: AWS applies rate limiting near the limit you set, without guaranteeing an exact match.
• Check frequency: independent of the window setting. AWS WAF checks the rate frequently and looks back across the window each time.

Do the arithmetic on the defaults. A limit of 300 over a 300-second window is one request per second, sustained, per aggregation instance. That's the shape of most rate-based rules you'll meet, and it's a perfectly workable crawl rate for anything except a bulk archive job.

Two consequences that change how you schedule

The block is not instant and it is not instantly lifted. Because the window looks back 300 seconds by default, a burst that trips the rule keeps you blocked for as long as that burst stays inside the window. Retrying after 5 seconds accomplishes nothing except keeping the count high. Waiting out the full window is what clears it.

And if the aggregation key is a custom key rather than the IP, rotating proxies does not help at all. A rule aggregating on a session cookie, an Authorization header, or a URI path counts your requests regardless of which exit you used. Before you scale a proxy pool to solve a rate problem, confirm the rate is actually per-IP. A single request against a known-good path, sent from a fresh exit while the old one is still blocked, answers that in ten seconds.

If the site pays for AWS WAF Bot Control, the picture changes. There are two protection levels and the gap between them is large.

Common tier: static analysis, no JavaScript involved

The common tier classifies bots from request data alone. Several of its rules block by default and will end a naive scraper immediately:

• CategoryHttpLibrary matches requests generated by the HTTP libraries of programming languages. Python requests, Go's net/http, Node's axios: recognized and blocked when unverified.
• SignalNonBrowserUserAgent matches User-Agent strings that don't look like a browser.
• CategoryScrapingFramework matches known web scraping frameworks.
• SignalAutomatedBrowser matches indicators that the client browser is automated.
• SignalKnownBotDataCenter matches data center ranges typically used by bots.
• CategoryAI blocks AI bots, and unusually it applies regardless of whether the bot is verified.

Verified bots are the escape hatch, and it's a legitimate one. Bot Control verifies self-identifying bots such as search engine crawlers and labels them bot:verified, and the category rules skip them. AWS also supports Web Bot Authentication, where a crawler cryptographically signs its requests and earns the label bot:web_bot_auth:verified. When that label is present, the category rules and TGT_TokenAbsent do not match. If you operate a crawler with a real published identity, this is the path designed for you.

Targeted tier: tokens, sessions, and the trap in proxy rotation

The targeted tier adds browser interrogation, fingerprinting, and behavior heuristics. Its rules are prefixed TGT_, and the machine-learning ones TGT_ML_. Everything here hangs off the token: an encrypted value stored in a cookie named aws-waf-token, holding challenge and CAPTCHA solve timestamps plus browser interrogation results.

Here is the part almost nobody writes about, and it inverts the standard scraping advice:

All measured over the last 5 minutes, on a single token. Read the country and ASN rows again: more than one is enough to earn a label. A rotating proxy pool combined with a persisted cookie jar is therefore worse than no rotation at all, because it presents one session identity arriving from many networks in many countries inside five minutes, a pattern no human produces.

The rest of the targeted rules fill in the behavioral picture. TGT_VolumetricIpTokenAbsent issues a Challenge once a client sends five or more tokenless requests in 5 minutes. TGT_VolumetricSession applies CAPTCHA to sessions that are abnormally busy against AWS WAF's learned baseline, and TGT_VolumetricSessionMaximum blocks outright. TGT_SignalBrowserInconsistency applies CAPTCHA when browser interrogation data contradicts itself, which is precisely what a partially patched headless browser produces. TGT_ML_CoordinatedActivity at Low, Medium and High confidence targets distributed, low-intensity crawls spread across many IPs, and AWS notes these rules can take up to 24 hours to become effective after enablement, because the baseline has to be learned first.

The takeaway is not "defeat the token." It's that against a targeted-tier deployment, the honest architecture and the effective one are the same: one session, one exit IP, one country, one ASN, a modest rate, and a browser that isn't lying about what it is.

Work down this list. Stop at the first one that gets you the data.

1. The official API or data feed. Faster, structured, versioned, and never challenged. Check /api, the developer docs, and the sitemap before writing a parser. If there's a paid tier, price it against a month of engineering time spent fighting a WAF.
2. Authenticated access under the terms. Many sites permit automation for account holders within stated limits. Read the terms, then stay inside them. Credentialed access usually means the CAPTCHA and Challenge paths are configured differently for you.
3. A negotiated arrangement. Email the site and ask. Identify your crawler, state your volume, offer a contact address and a fixed source range they can allowlist. This works far more often than engineers expect, and an allowlist rule sits at priority 0, above everything else discussed here.
4. A published, verifiable crawler identity. A descriptive User-Agent with a URL explaining who you are, plus Web Bot Authentication signing where the target supports it. That's what lets a site allow you deliberately.
5. Less volume. Cache aggressively, crawl incrementally with conditional requests, and fetch only pages that changed. A daily delta of 400 pages never trips a rule that a nightly full re-crawl of 40,000 pages trips every time.
6. Consistent, well-reputed exit IPs and an honest client fingerprint, so no rule ever fires. That's the last resort in this list, not the first move.

Where Bot Control is enabled, the failure mode is contradiction rather than identity. AWS WAF's targeted rules explicitly look for browser inconsistency, and TGT_SignalBrowserInconsistency exists for exactly that. A request claiming to be Chrome 130 while performing a Python TLS handshake is neither a Chrome request nor a Python request. It's an anomaly, and anomalies score worse than either honest option.

You have two consistent stories available. Pick one and commit.

Story one: an identified, honest client

Best for public data collection at modest volume, and the only story compatible with route 3 or 4 above.

import requests

session = requests.Session()
session.headers.update({
    # Say who you are and give them a way to reach you.
    "User-Agent": "SparkProxyResearchBot/1.0 (+https://www.sparkproxy.io/bot; ops@sparkproxy.io)",
    "Accept": "text/html,application/xhtml+xml;q=0.9,*/*;q=0.8",
    "Accept-Language": "en-US,en;q=0.9",
    "Accept-Encoding": "gzip, deflate, br",
    "Connection": "keep-alive",
})

resp = session.get("https://www.sparkproxy.io/docs/scraping-api/", timeout=30)
print(resp.status_code, resp.headers.get("x-amzn-waf-action"))

That client satisfies NoUserAgent_HEADER, won't match UserAgent_BadBots_HEADER, and gives the site's operations team something to allowlist. Bot Control's common tier will still classify it as an HTTP library, which is correct, because it is one.

Story two: a real browser, consistently

Best when the target genuinely requires JavaScript to render, or where a Challenge action is in play and only a browser can mint aws-waf-token.

from curl_cffi import requests as cffi

session = cffi.Session(impersonate="chrome124")   # Chrome TLS + HTTP/2 profile
session.proxies = {
    "http":  "http://user:pass@gate.sparkproxy.io:10000",
    "https": "http://user:pass@gate.sparkproxy.io:10000",
}
resp = session.get("https://www.sparkproxy.io/")
print(resp.status_code)

impersonate="chrome124" aligns the TLS ClientHello and the HTTP/2 SETTINGS frame with the User-Agent, so the JA3 and JA4 fingerprint matches the claim. Header order matters as much as header values, which is why hand-assembling headers on top of a plain HTTP client rarely produces a coherent result.

Whichever story you tell, hold the network side steady. One session gets one exit IP, in one country, on one ASN, for its whole life. That's how real browsers behave, it's what the TGT_TokenReuse rules are measuring, and it's the same discipline that keeps you out of trouble on Cloudflare-protected sites too.

A 403, a 405 with x-amzn-waf-action: captcha, or a 202 with x-amzn-waf-action: challenge is a decision, not a transient error. Retrying it immediately from a fresh IP is the single most damaging thing a scraper can do. It converts one flagged request into a flagged subnet and, where Bot Control is running, feeds TGT_ML_CoordinatedActivity exactly the distributed pattern it's trained to catch.

Handle each signal differently:

import random, time
import requests

COOLDOWN = 300   # matches the default 300s rate window and token immunity time

def fetch(url, session, attempt=1, max_attempts=4):
    r = session.get(url, timeout=30, allow_redirects=False)
    action = r.headers.get("x-amzn-waf-action")

    if r.status_code == 200 and not action:
        return r.text

    if r.status_code in (405, 202) and action in ("captcha", "challenge"):
        # An interactive gate. Not a retry case. Park this target.
        raise PermissionError(f"AWS WAF {action} gate on {url}; out of scope")

    if r.status_code == 403:
        # Terminating Block. Stop this worker, do not rotate and hammer.
        raise PermissionError(f"AWS WAF blocked {url}; diagnose before retrying")

    if r.status_code == 429 or r.status_code >= 500:
        if attempt >= max_attempts:
            raise RuntimeError(f"gave up on {url} after {attempt} attempts")
        wait = float(r.headers.get("Retry-After") or random.uniform(0, 2 ** attempt))
        time.sleep(wait)                       # full jitter, capped by attempt
        return fetch(url, session, attempt + 1, max_attempts)

    raise RuntimeError(f"unexpected {r.status_code} for {url}")

Three rules of thumb sit behind that code. Full jitter beats fixed backoff, because synchronized retries across workers recreate the burst that caused the problem. Cooldowns should be sized to the mechanism: 300 seconds matches both the default rate-based evaluation window and the default token immunity time, and the minimum challenge immunity time is also 300 seconds, so shorter waits are wasted. A circuit breaker per host, rather than per request, keeps one blocked target from poisoning your whole pool. Our retry and backoff strategies guide goes deeper on the queue mechanics.

Managing residential exits, session stickiness, browser rendering and current TLS profiles yourself is ongoing work. The SparkProxy Scraping API collapses those into request parameters, which helps for permitted collection where a target needs rendering or a clean residential exit.

curl -G "https://scrape.sparkproxy.io/api/v1" \
  -H "X-API-Key: YOUR_API_KEY" \
  --data-urlencode "url=https://www.sparkproxy.io/pricing" \
  --data-urlencode "render_js=true" \
  --data-urlencode "premium_proxy=true" \
  --data-urlencode "country_code=US" \
  --data-urlencode "session_id=aws-waf-run-01" \
  --data-urlencode "wait_for=.pricing-table" \
  --data-urlencode "json_response=true"

The same call in Python, with each parameter mapped to the layer it addresses:

import requests

API = "https://scrape.sparkproxy.io/api/v1"

def fetch_rendered(url, session_label):
    env = requests.get(
        API,
        headers={"X-API-Key": "YOUR_API_KEY"},
        params={
            "url": url,
            "render_js": "true",          # real Chromium; can satisfy a Challenge action
            "premium_proxy": "true",      # residential exit, not a HostingProviderIPList range
            "country_code": "US",         # one country per session, per TGT_TokenReuseCountry
            "session_id": session_label,  # sticky profile: same IP, same cookie jar
            "wait_for": ".pricing-table",
            "json_response": "true",      # envelope with status_code, credits_used, meta
        },
        timeout=120,
    ).json()
    return env

session_id is the parameter that matters most here. It pins a browser profile and its exit so one logical session stays on one IP, one country and one ASN, which is exactly the shape the TGT_TokenReuse rules reward. premium_proxy=true routes through residential address space rather than the hosting ranges HostingProviderIPList blocks. render_js=true runs a genuine browser, the only thing that can complete a silent challenge and hold aws-waf-token.

Cost it before you scale. Per the published pricing, a plain HTTP fetch is 1 credit, headless rendering is 5, a premium proxy with rendering is 25, and country_code and stealth add 5 credits each. Turn the expensive parameters on for targets that need them, and leave them off for the ones that don't, which is most of them. If you're weighing this against running your own pool, the scraping API vs self-managed proxies comparison lays out the trade.