How to Bypass Fastly Next-Gen WAF

Signal Sciences was an independent product before Fastly acquired it, and the acquisition kept the architecture intact under a new name. The documentation now calls it the Next-Gen WAF, but the control panel, the agent, the modules and the vocabulary are all still Signal Sciences. If you've read a block page or a job log that mentions "sigsci" anywhere, this is the system you're meeting.

Here is the mental model that matters. A traditional WAF runs a request against a ruleset and returns a verdict inline. Signal Sciences does something closer to tagging. Every request that passes through it gets inspected and tagged with zero or more signals. A signal is a label, not a verdict. "This request contains a cross-site scripting payload" is a signal. "This request came from a datacenter IP range" is a signal. "This client claims to be a search bot but isn't" is a signal. The WAF collects those tags, and a separate decision layer decides what to do with them based on how many of which kind it has seen from a given source over a given window.

That separation is the thing to internalize. A request can be tagged and still allowed through. Tagging is not blocking. The system can watch you generate anomaly signals for an hour and never stop a single request, because the operator has it in logging mode, or because the signals you're generating are not the kind that count toward an automatic block. Understanding which tags lead to action, and under which configuration, is the entire game. The rest of this guide is about that map.

For read-only scraping of public pages, the good news is that you are rarely generating the signals the WAF cares about most. You are not sending SQL injection strings or Log4J exploitation payloads. The signals you do generate are reputational and behavioral: where your traffic comes from, how fast it arrives, and whether your request shape looks like a human's browser. Those are the levers, and most of them are not things a header rewrite can touch.

Fastly documents several ways to run the Next-Gen WAF, and the mode an operator chose changes where inspection happens relative to you and what you can infer. This is more operationally useful than it sounds, because the deployment mode determines whether you are talking to Fastly's edge or to the origin's own infrastructure.

Edge WAF. This runs on Fastly's Edge Cloud Platform across their global network of POPs, and it is Fastly-managed. The documentation describes both out-of-band configurations (using Compute and CDN) and in-band configurations (using the CDN). For you as a fetcher, the key fact is that inspection and the block decision happen on Fastly's edge, before your request ever reaches the origin. The block comes back from a POP close to your exit IP, which tends to mean a fast response. It also means your TLS and HTTP/2 handshake is being terminated and observed by well-resourced edge infrastructure, not a single origin box.

On-Prem WAF (formerly Core WAF). This is customer-managed and runs in the origin's own environment. Fastly documents two shapes. The first is the module and agent pair: a Next-Gen WAF module installs as a plugin to a web server or at the application layer, and a separate agent process makes the decisions. Fastly describes this as a "fail-open design, meaning the module verifies agent availability and allows all traffic when the agent is down." The second is reverse-proxy mode, which needs only the agent and "does not require modifications to your web server or code," and which Fastly describes as a "fail-close design, meaning all traffic is blocked when the agent is down." Fastly also notes that only the NGINX dynamic module supports WebSocket inspection.

The fail-open detail is worth sitting with, but not for the reason it first suggests. It is a documented property of the module topology, not an invitation. You cannot make an operator's agent fall over, and you shouldn't try. The point is architectural: in the module topology, the module's only job is to forward request data to the agent and enforce whatever the agent decides. Fastly describes the module as a lightweight interface whose sole role is to pass requests to the agent and receive and enforce the allow, log or block decision. The intelligence lives in the agent and the cloud engine behind it, not in the module sitting in the web server.

Cloud WAF. Also Fastly-managed, this runs in Fastly's cloud infrastructure as a load balancer in front of at least three Next-Gen WAF agents, each in reverse-proxy mode on separate redundant machines. Standing it up requires uploading a TLS certificate and reconfiguring DNS to point at Fastly. For you, this behaves like the edge case: the origin's DNS resolves to Fastly, inspection happens before the real origin, and the block is served by Fastly's infrastructure.

PaaS and A10. Fastly also documents running the agent inside supported platform-as-a-service vendors, and an integration with A10 Networks Thunder and vThunder application delivery controllers. These are niche from a fetcher's seat and behave like the agent-in-reverse-proxy case.

The modules themselves span a wide set: NGINX (both a dynamic module and a Lua module), Apache across Debian, Red Hat, Ubuntu and Windows, IIS, HAProxy including the SPOE variant, plus application-layer modules for .NET, .NET Core, Golang, Java (Dropwizard, Jetty, Netty, Servlet Filter and Weblogic flavors), a Kong plugin, and Node.js. The breadth matters because it tells you the WAF can sit at almost any layer of a target's stack. You cannot assume it lives at the CDN. It might be a plugin inside the application server itself, inspecting the request after TLS termination and after the load balancer, which means edge-level tricks like SNI games buy you nothing.

What leaks, across all of these, is less than with challenge-based vendors. There is no client-side script phoning telemetry home by default, so you are not leaking a browser automation fingerprint to a JavaScript collector the way you would against a bot manager. What you are leaking is the request itself: your IP and its reputation, your headers, your TLS and HTTP/2 fingerprint, your path, and your rate. The agent reads those and tags accordingly.

Fastly splits signals into two families. System signals are the ones Fastly ships, which the documentation describes as signals "that we create to track common attacks, anomalies, and behaviors," with over three dozen of them out of the box. Custom signals are ones the operator writes to track behavior specific to their own application. System signals are themselves divided into three types: attack, anomaly, and informational.

The distinction between attack and anomaly is the hinge of this whole system, so be precise about it. Attack signals mark requests that carry something offensive: Attack Tooling, AWS SSRF, Cross-Site Scripting, Log4J JNDI, GraphQL Max Depth and the rest. Anomaly signals mark requests that are merely unusual or suspicious in origin or shape: Datacenter Traffic, Malicious IP Traffic, SearchBot Imposter, Abnormal Path, Duplicate Header Names. A read-only scraper fetching public HTML almost never earns an attack signal, because it isn't sending exploit payloads. It very easily earns anomaly signals, because it often comes from a datacenter IP, sometimes from an IP with a dirty history, and sometimes while spoofing a search-engine user agent.

Now the part most write-ups get wrong. By default, those anomaly signals do not block you on their own. Fastly's attack-threshold documentation states plainly that "requests that have only been tagged with anomaly and custom signals are not counted towards flagging thresholds." The automatic IP blocklist is driven by attack signals, not anomaly signals.

Here is how the escalation actually works. Attack thresholds count attack signals per IP address over rolling windows. The documented defaults are 50 attacks in one minute, 350 in ten minutes, and 1,800 in one hour. When an IP crosses any of those, the system flags and blocklists it, and by default malicious traffic from that IP is blocked or logged for 24 hours. Whether flagged traffic is actually blocked or merely logged depends on the Protection mode, also called Agent mode. In Blocked mode, attack-tagged requests from a flagged IP are blocked. In Logging mode, also labeled Not Blocking, they are recorded but allowed through. There is also an Immediate blocking setting that skips the window entirely and blocks any request carrying at least one attack signal, or specific CVE virtual-patch signals, with the documentation naming support for signals such as Pentest, CVE, XXE and AWS-SSRF.

So a block escalates along one of three paths. Path one: you send actual attack payloads, cross the attack threshold, and get your IP blocklisted for a day. For honest scraping, this path should never fire, and if it does, you have a bug sending malformed requests. Path two, the one that catches scrapers: the operator wrote a custom rule or an advanced rate-limiting rule that acts on an anomaly or custom signal. A site can add a rule that blocks any request tagged Datacenter Traffic, or rate-limits anything tagged SearchBot Imposter, and now an anomaly signal that would have been harmless becomes a hard block. Path three: the operator runs in Logging mode while tuning, sees nothing blocked, then flips to Blocking, and traffic that sailed through yesterday dies today with no change on your end.

That last point is why a scraper against a Signal Sciences site can look stable for days and then collapse. You didn't change. The operator flipped Agent mode, or shipped a custom rule keyed on the anomaly signal you were generating the whole time. Build your retry logic assuming the decision layer is a moving target.

Fingerprinting Signal Sciences from the client side is harder than fingerprinting a challenge vendor, and the reasons are instructive.

The clearest documented marker is the block response code. Fastly states that by default, blocked requests receive a 406 agent response code, behaving like an HTTP 406 Not Acceptable. A sudden 406 on a site whose normal responses are 200, especially a 406 with a short or empty body and no challenge markup, is the single best hint that you're looking at Signal Sciences. Treat it as a hint, not proof, for two reasons. First, 406 is a legitimate content-negotiation status that unrelated servers also emit. Second, the code is configurable: the documentation says the supported blocking response codes are 301, 302, and 400 through 599, and that operators can change the default from 406 to any of those. A security team that has read the same docs you have will often move it to 403 or a custom code precisely to muddy this signal.

What you will usually not see is as telling as what you will. There is no JavaScript interstitial as part of the core WAF block. The default block is an immediate agent response with a status code, not a scripted challenge that reloads into content. There is no signature cookie chain to complete, the way Imperva hands you visid_incap and reese84 and expects them back. And, in our testing against sites known to run the Next-Gen WAF, there is no reliable vendor-branded response header on the block the way Imperva exposes X-Iinfo. That absence is itself diagnostic: signal-driven blocking with a bare status code, no cookie to replay, and no challenge page points toward a Signal Sciences-style WAF rather than a challenge vendor. We mark the "no reliable response header" observation as ours, not a Fastly claim, because the module's internal headers (the X-SigSci family) are added to the request forwarded to the application, not to the response sent back to you, so you generally do not see them from the client side.

Because the marker set is thin, confirm the identity the way you would for any ambiguous stack: look at the full picture rather than one header, and corroborate across response code, body shape, timing and TLS behavior. If you maintain a detection routine, add 406-with-no-challenge as a weighted hint rather than a hard match. Our broader write-up on fingerprinting a stack covers the general method for telling WAFs apart from markers, and the same approach applies here.

Both of the vendors most often confused with this one already have dedicated guides on this site, so this section compresses the mechanics and sends you there for the full treatment rather than re-explaining them.

Imperva, formerly Incapsula, is a challenge-and-cookie machine. It hands your client a client-side JavaScript challenge, expects you to execute it and return a chain of cookies (visid_incap, incap_ses, and the reese84 token), and it exposes itself through the X-Iinfo header and incident IDs on its block pages. Beating it is about completing the cookie chain with a real browser and holding the resulting session. The full method is in the Imperva Incapsula guide. Signal Sciences does none of that by default: no challenge to execute, no cookie to earn, no signature header to match.

AWS WAF is a rule engine you assemble from managed rule groups, rate-based rules and the optional Bot Control tier, where the Bot Control and challenge actions can issue a silent challenge or a CAPTCHA and hand back an aws-waf-token cookie that later requests must carry. There, part of the job is satisfying a token-issuing challenge and keeping the token. The AWS WAF guide walks the rule types and the token flow. Signal Sciences overlaps with AWS WAF on the rate-and-rule idea, since both count behavior per source and act at thresholds, but it has no token to carry and its default block is a bare 406 rather than a challenge.

The mechanical summary: Imperva makes you prove you're a browser by running its code, AWS WAF can make you carry a token it issues, and Signal Sciences makes neither demand. It simply tags and counts, then blocks when a counter crosses a line the operator set. That means the techniques that beat a challenge vendor, headless browsers that solve interstitials and cookie jars that replay tokens, are not the techniques that matter most here. Against Signal Sciences, the request has to be clean and the source has to be reputable, because there is nothing to solve that would let a dirty source through anyway.

Everything below assumes you have permission to collect the data in question: you own the target, you have a contract, or you're within the site's stated terms and robots directives. With that settled, here's the order of operations that actually moves success rate against a Signal Sciences site, cheapest and highest-impact first.

Rung one: read the block before you touch anything. The worst move is to start randomizing headers against a block you haven't classified. Fetch the target, capture the status code and body, and decide what you're facing. A 406 with no challenge markup, or a configured 403 that appears only above a certain rate, points at signal-driven blocking. A full HTML challenge page that reloads points at something else layered on top. Our guide on detecting when your scraper is blocked covers building this classification step properly. Do it first, every time.

Rung two: stop generating anomaly signals you don't need to. This is where most of the win is, and almost none of it is header spoofing. Do not spoof a search-engine user agent: claiming to be Googlebot from an IP whose reverse DNS and ASN don't back it up is exactly what the SearchBot Imposter anomaly signal catches, and it converts a boring request into a flagged one. Keep your paths normal, since hitting admin paths or probing for files earns Abnormal Path. Send a coherent, ordinary header set rather than a thin or contradictory one, because duplicate or malformed headers have their own anomaly signal. The goal is to be unremarkable, not clever.

Rung three: fix the source, because origin is a signal you can't header away. Datacenter Traffic and Malicious IP Traffic are anomaly signals keyed entirely on where your request comes from. If you exit from a cloud provider's range, you are tagged Datacenter Traffic no matter how perfect your headers are, and if the operator blocks or rate-limits that tag, you're done. The fix is residential or mobile egress with clean reputation, and rotation that doesn't concentrate volume on any single IP. A rotating residential pool with per-session stickiness is the right default here. Our walkthrough on rotating proxies in Python covers the rotation mechanics.

Rung four: pace under the rate limit, not just under the attack threshold. The attack thresholds (50 a minute and up) are generous because they count only attack signals. The limit that actually catches clean scrapers is an operator's advanced rate-limiting rule, which can count ordinary requests per IP or per session over a window and block well below the attack numbers. You can't read that rule from outside, so treat rate as a tuning parameter: start slow, spread load across sessions and IPs, and back off hard on the first sign of a threshold response rather than hammering into a 24-hour blocklist.

Rung five: match TLS and HTTP/2 only when rung three is already solved. A library like curl_cffi or an equivalent that presents a real browser's TLS and HTTP/2 fingerprint does help, because a Python-default handshake from a residential IP is still an inconsistency. But fingerprint matching is rung five for a reason: it does nothing if your IP is already tagged Datacenter Traffic, and it is pure wasted effort if you haven't solved origin first. Our comparison of curl_cffi versus other TLS-capable clients covers the options.

Rung six: use a real browser only where the page needs it. If the target renders content with JavaScript, or if the operator has layered Fastly's separate bot-management product on top of the WAF (which can add a real client-side challenge), you need a genuine browser engine, not a bare HTTP client. That's a rendering and challenge concern, not a WAF-signal concern, and it's covered in the headless browser detection guide. Reach for it last, because it's the slowest and most expensive rung.

Notice the shape of this ladder. Against a challenge vendor, the browser and the cookie jar would be near the top. Against Signal Sciences, the source IP and the request rate are at the top, and the browser-level work is at the bottom, because the WAF is judging your reputation and behavior far more than your ability to run its script.

Honesty about limits is the difference between a guide and a sales pitch, so here are the things no amount of header tuning from a stateless client will solve.

IP origin and reputation. The Datacenter Traffic and Malicious IP Traffic signals are computed from your egress address and its history. A stateless client cannot make a datacenter IP look residential, and it cannot scrub the reputation of an address that other people burned before you. The only real answer is cleaner egress: residential or mobile IPs with good reputation and enough diversity that you're not re-triggering the same reputational flag. This is an infrastructure problem, not a code problem.

Operator rate limits you can't see. Advanced rate-limiting rules live in the operator's control panel. You don't know the count, the window, or the key (IP, session, header, or something custom) until you trip it. You can adapt by pacing and spreading load, but you cannot pre-solve a limit whose parameters are private. Treat every success rate as provisional and keep backoff in the loop.

Server-side custom signals and rules. Operators write custom signals to catch behavior specific to their application, and custom rules to act on any signal combination they like. These are invisible from outside by design. A site might flag a header you consider innocuous, or a request sequence that looks fine to you, because it correlated that pattern with abuse in its own logs. You can only discover these empirically, and a sophisticated operator can always write one you haven't seen.

A separate challenge layer, when present. The core Next-Gen WAF block is a status code, not a scripted challenge. But Fastly sells a distinct bot-management product that can add a real client-side challenge, and nothing stops an operator from putting a JavaScript challenge in front of the WAF. When that layer is present, you are back to needing a real browser to clear it, and the WAF discussion is secondary. Identify whether a challenge layer exists before you assume the bare-status-code model applies.

Edge-side inspection timing. With Edge WAF or Cloud WAF, inspection happens on Fastly's infrastructure before your request reaches the origin. You cannot route around the inspection by targeting the origin directly, because in those modes DNS and the edge are the front door. The request is seen and tagged no matter what you do at the application layer.

The through-line: Signal Sciences is beatable for permitted collection precisely to the extent that you stop looking like a problem (clean payloads, normal shape, no imposter signals) and start looking like ordinary traffic (reputable residential egress, human-plausible pacing). It is not beatable by pretending, from a dirty source at machine speed, to be something you aren't. There's no challenge to fool into letting a bad source through.

Running residential egress, session stickiness, current TLS profiles and a browser for the pages that need one is ongoing infrastructure. The SparkProxy Scraping API folds those into request parameters, which helps when a permitted target sits behind the Next-Gen WAF and you want clean origin plus optional rendering without maintaining the plumbing. New accounts get 1,000 free credits to test against your own target.

Start by reading the block, which means asking for the envelope rather than just the page. Set json_response=true, and the API returns a JSON object with job_id, status_code, duration_ms, credits_used and body. The one trap: the body field is base64-encoded, so you must decode it before you match any marker like a 406 signature. A plain fetch (render_js=false) costs 1 credit, which is all you need to classify a block.

curl -G "https://scrape.sparkproxy.io/api/v1" \
  -H "X-API-Key: YOUR_API_KEY" \
  --data-urlencode "url=https://www.sparkproxy.io/status" \
  --data-urlencode "render_js=false" \
  --data-urlencode "json_response=true"

In Python, decode the body before you look for the Signal Sciences block signature. Reading status_code from the envelope and the decoded body together is how you tell a 406 WAF block from an ordinary content-negotiation 406:

import base64
import requests

API = "https://scrape.sparkproxy.io/api/v1"
HEADERS = {"X-API-Key": "YOUR_API_KEY"}


def classify(url):
    env = requests.get(
        API,
        headers=HEADERS,
        params={
            "url": url,
            "render_js": "false",      # plain fetch, 1 credit, enough to read a block
            "json_response": "true",   # envelope: status_code, duration_ms, body
        },
        timeout=60,
    ).json()

    status = env.get("status_code")
    body = base64.b64decode(env.get("body", "")).decode("utf-8", "replace")

    looks_like_sigsci = status == 406 and "<form" not in body.lower()
    return {
        "status": status,
        "blocked": status in (403, 406, 429),
        "sigsci_hint": looks_like_sigsci,
        "ms": env.get("duration_ms"),
    }


if __name__ == "__main__":
    print(classify("https://www.sparkproxy.io/status"))

When classification says the page is clean but the source is the problem, add a reputable residential exit and a country. Here proxy_type=premium selects the premium pool, and country_code pins the exit geography (each adds credits on top of the fetch), so one logical session stays on consistent, clean egress rather than a datacenter range that would earn a Datacenter Traffic tag:

import base64
import requests

API = "https://scrape.sparkproxy.io/api/v1"
HEADERS = {"X-API-Key": "YOUR_API_KEY"}


def fetch_clean(url, country="US"):
    env = requests.get(
        API,
        headers=HEADERS,
        params={
            "url": url,
            "render_js": "false",       # no rendering needed for static HTML
            "proxy_type": "premium",    # premium pool, reputable residential egress
            "country_code": country,    # one country per session (+5 credits)
            "json_response": "true",
        },
        timeout=90,
    ).json()
    if env.get("status_code") != 200:
        return None
    return base64.b64decode(env.get("body", "")).decode("utf-8", "replace")


if __name__ == "__main__":
    html = fetch_clean("https://www.sparkproxy.io/pricing")
    print("ok" if html else "blocked or non-200")

For a page that only renders content with JavaScript, turn rendering on with render_js=true (the default, at 5 credits) and drive the interaction with js_scenario, which runs up to 50 steps and requires rendering. Rendering is also what clears a separate client-side challenge layer if the operator stacked one on top of the WAF:

import requests

API = "https://scrape.sparkproxy.io/api/v1"
HEADERS = {"X-API-Key": "YOUR_API_KEY"}


def fetch_rendered(url):
    params = {
        "url": url,
        "render_js": "true",            # real browser engine, 5 credits
        "proxy_type": "premium",        # residential exit
        "js_scenario": '{"steps":[{"wait":1500},{"scroll_y":2000}]}',
        "format": "md",                 # convert the rendered page to Markdown
    }
    resp = requests.get(API, headers=HEADERS, params=params, timeout=180)
    return resp.text


if __name__ == "__main__":
    print(fetch_rendered("https://www.sparkproxy.io/pricing")[:500])

Two cost notes that keep the bill sane. Batch mode needs render_js=false and bills the whole batch at 1 credit, so group static fetches when you can. And format only ever converts the returned content (html, md, mdx, json, screenshot, pdf); it is not the envelope. If you want the envelope with status_code and friends, that is json_response=true, and the two concerns are separate. Don't reach for extract_rules in the same call as json_response, because extract_rules replaces the response with an extracted object and the two don't compose.

Pricing starts at $75 per month (packs with unlimited bandwidth billed on the plan, not per GB), with a 24-hour free trial and no card required, if you move past the free credits. For permitted work against a Signal Sciences target, the parameters that matter are proxy_type=premium and a stable country_code for clean origin, plus render_js only on the pages that genuinely need it.