How to Scrape GraphQL APIs

If you have reverse engineered a site's private REST or XHR JSON calls before, GraphQL feels familiar and different at the same time. It is still the same idea: skip the rendered HTML and hit the data feed the front end already uses. The mechanics change. With REST you find a URL per resource and swap path segments or query params to get different data. With GraphQL there is a single URL, and the request body carries a query that names the exact fields to return. That single difference cascades into how you find the endpoint, how you page, and how you handle errors.

Here is the trade-off, dimension by dimension:

The last two rows trip up people who arrive from a REST background. A GraphQL server hands back 200 OK even when your query is malformed or unauthorized, and it puts the real problem in an errors array inside the JSON. If your scraper only checks the HTTP status, it will treat a failed query as a success and store empty rows. You have to read the body.

The "fields you ask for" row is the upside. Because you write the selection set, you can request the internal SKU, a raw price in cents, or a sponsored flag the UI never renders, and you can drop everything you don't need to shrink the response. This is a different technique from replaying a fixed REST endpoint that returns a hidden JSON blob. If your target turns out to serve plain REST/XHR JSON instead of GraphQL, that is its own workflow. If you are new to the broader discipline, our primer on what web scraping is covers the vocabulary used below.

One boundary up front. This technique targets the exact GraphQL operations the site's own front end runs in your normal browser. It is not credential theft or bypassing access controls. Scrape public data you can already see, respect the site's terms and robots.txt, and stay away from personal or authentication-gated data you have no right to.

The steps to scrape GraphQL endpoint data are the same on every site: find it, read it, replay it, page it. You are hunting for a single request, and it looks different from REST calls.

1. Open DevTools. F12 on Windows and Linux, Cmd+Option+I on macOS.
2. Click the Network tab, then the Fetch/XHR filter so images, fonts, and scripts drop out.
3. Reload the page or trigger the action that loads the data you want (scroll a feed, open a product, run a search).
4. Look for a POST whose Request URL ends in /graphql, /api/graphql, /graphql/query, or a gateway path like /gql. The request name is often literally graphql.

Two faster tells confirm it. First, click a candidate and open the Payload (or Request) tab: a GraphQL request body has query, and usually variables and operationName, keys. Second, use the Network search box (Ctrl+F inside the panel, not the page find) and type a value you can see on the page, like a product name. The search scans response bodies, so it points you straight at the operation that returned it. That last move is the fastest way to skip past dozens of tracking pings.

Some sites batch several operations into one array in a single POST. If the payload is a JSON array of {query, variables} objects rather than a single object, the server supports query batching. You can send one operation at a time when you replay it; you do not have to reproduce the batch.

Once you have the request, capture it exactly. Right-click it in the Network list and choose Copy then Copy as cURL. You now hold the endpoint, the method, every header, and the full body. A typical captured GraphQL body looks like this:

{
  "operationName": "ProductList",
  "variables": { "category": "proxies", "first": 20, "after": null },
  "query": "query ProductList($category: String!, $first: Int!, $after: String) {\n  products(category: $category, first: $first, after: $after) {\n    edges {\n      node { id name priceCents inStock }\n      cursor\n    }\n    pageInfo { endCursor hasNextPage }\n  }\n}"
}

Three parts matter for replay. The query is the operation text, with $category, $first, and $after as typed placeholders. The variables object fills those placeholders, and it is the thing you will change to page or filter. The operationName is a label; when a document defines several named operations, the server uses it to pick which one to run. Send it when the captured request sends it.

The Copy as cURL output also shows the headers. Most are noise. The ones that tend to matter are content-type: application/json, sometimes authorization, sometimes a CSRF or client header the server checks (more on that below). Strip the request down to the minimum set that still returns data, and add headers back only if the call breaks without them.

With the body in hand, replaying it in Python is a plain POST. Note the two things REST scrapers forget: send the payload as a JSON body (not URL params), and check the errors array yourself because the status will be 200 either way.

import requests

ENDPOINT = "https://www.sparkproxy.io/graphql"

QUERY = """
query ProductList($category: String!, $first: Int!, $after: String) {
  products(category: $category, first: $first, after: $after) {
    edges { node { id name priceCents inStock } cursor }
    pageInfo { endCursor hasNextPage }
  }
}
"""

resp = requests.post(
    ENDPOINT,
    json={
        "operationName": "ProductList",
        "query": QUERY,
        "variables": {"category": "proxies", "first": 20, "after": None},
    },
    headers={"Accept": "application/json"},
    timeout=30,
)
resp.raise_for_status()               # catches transport-level failures only
payload = resp.json()

if payload.get("errors"):             # GraphQL reports logic errors here, at HTTP 200
    raise RuntimeError(payload["errors"])

products = payload["data"]["products"]["edges"]

Now use the part of GraphQL that REST cannot do: trim the query to only the fields you need. This is where GraphQL query scraping pays off, because you control the selection set and decide the shape and size of every response. If you only want ids and prices, delete the rest of the selection set. The response shrinks, the server does less work, and your footprint gets smaller because you are asking for less than the full UI does.

QUERY = """
query ProductList($category: String!, $first: Int!, $after: String) {
  products(category: $category, first: $first, after: $after) {
    edges { node { id priceCents } }
    pageInfo { endCursor hasNextPage }
  }
}
"""

You can add fields the interface never shows, too, as long as they exist on the type. Ask for internalSku or costCents and if the schema exposes them, they come back. The next section shows how to learn what exists. When you scale this past a handful of requests, the concurrency and retry patterns in our guide to using proxies with Python requests and aiohttp apply directly, since every call here is one POST.

GraphQL ships a query language for its own schema. If a server leaves it on, one request returns every type, field, and argument. This is GraphQL introspection, and it is the fastest way to discover fields the captured query never touched.

query IntrospectionQuery {
  __schema {
    queryType { name }
    types {
      name
      kind
      fields { name }
    }
  }
}

Run it the same way you ran the product query, then filter out the built-in types whose names start with __:

introspection = """
query {
  __schema {
    queryType { name }
    types { name kind fields { name } }
  }
}
"""

r = requests.post(ENDPOINT, json={"query": introspection}, timeout=30)
schema = r.json()["data"]["__schema"]
public_types = [t["name"] for t in schema["types"] if not t["name"].startswith("__")]
print(public_types)

Read the fields on the type you care about and you know exactly what to add to your selection set. Many production servers disable introspection, though. When they do, the response is an error like GraphQL introspection is not allowed by the server. Do not treat that as a dead end. The site's own JavaScript bundle contains every operation the front end runs, so the queries you need are already sitting in the client code. Open the JS sources in DevTools and search for query or the operation name you saw in the Network tab, and you get the full text without introspection. If the site uses persisted queries, you may not even need the text (see the next-but-one section).

Most GraphQL APIs paginate with the connection pattern from the Relay spec: a field returns edges (each with a node and a cursor) plus a pageInfo object holding endCursor and hasNextPage. You page by passing first (page size) and after (the previous endCursor) until hasNextPage is false.

import time

def fetch_all_products(category, page_size=20, max_pages=50):
    items, after, pages = [], None, 0
    while pages < max_pages:
        r = requests.post(
            ENDPOINT,
            json={
                "operationName": "ProductList",
                "query": QUERY,
                "variables": {"category": category, "first": page_size, "after": after},
            },
            headers={"Accept": "application/json"},
            timeout=30,
        )
        r.raise_for_status()
        conn = r.json()["data"]["products"]
        items.extend(edge["node"] for edge in conn["edges"])

        info = conn["pageInfo"]
        if not info["hasNextPage"]:
            break
        after = info["endCursor"]     # feed the cursor back in for the next page
        pages += 1
        time.sleep(0.5)               # be polite; add jitter at scale
    return items

The max_pages cap and the sleep are not optional. A wrong hasNextPage read or a server that keeps returning a stable cursor will loop forever without them. Not every schema uses connections. Simpler APIs expose limit and offset arguments and return a flat list, in which case you increment offset by the page size until a page comes back empty:

query {
  products(category: "proxies", limit: 20, offset: 40) {
    id
    name
    priceCents
  }
}

Read the arguments in the captured query to know which shape you are dealing with before you write the loop. Getting it wrong is the most common reason a paginated scrape silently stops at page one or never stops at all.

Here is where GraphQL scraping surprises people. On many high-traffic sites, the request body has no query text at all. Instead it carries an extensions.persistedQuery object with a sha256Hash:

{
  "operationName": "ProductList",
  "variables": { "category": "proxies", "first": 20 },
  "extensions": {
    "persistedQuery": {
      "version": 1,
      "sha256Hash": "ecf4edb46db40b5132295c0291d62fb65d6759a9eedfa4d5d612dd5ec54a6b38"
    }
  }
}

This is Automatic Persisted Queries (APQ), an Apollo optimization. The client sends only the hash to save bandwidth. If the server already has that hash registered, it runs the stored query and returns data. If it does not, it replies with an error whose message is PersistedQueryNotFound, and the client resends the same request with the full query text added. The server stores it against the hash and answers.

You can ride the same protocol. The one rule that catches everyone: the hash is the SHA-256 of the exact query string the client used, whitespace and all. If you reformat the query, the hash no longer matches and registration fails.

import hashlib

def apq_extensions(query_text):
    digest = hashlib.sha256(query_text.encode("utf-8")).hexdigest()
    return {"persistedQuery": {"version": 1, "sha256Hash": digest}}

body = {
    "operationName": "ProductList",
    "variables": {"category": "proxies", "first": 20, "after": None},
    "extensions": apq_extensions(QUERY),
}

data = requests.post(ENDPOINT, json=body, timeout=30).json()

if data.get("errors") and data["errors"][0].get("message") == "PersistedQueryNotFound":
    body["query"] = QUERY                         # send the exact text once to register it
    data = requests.post(ENDPOINT, json=body, timeout=30).json()

products = data["data"]["products"]["edges"]

If introspection is off and the site uses APQ, you have a shortcut. You do not need the query text to keep scraping. Capture the hash and the variables from the Network tab and replay them directly, changing only variables to page or filter. The server runs the stored operation for you as long as that hash stays registered.

If a query returns data in your browser but 401, 403, or an errors entry like Not authorized in Python, you are missing an auth signal. Carry the same ones the captured request sent.

session = requests.Session()
session.headers.update({
    "Accept": "application/json",
    "Content-Type": "application/json",
    "Authorization": "Bearer " + token,   # when the API uses bearer tokens
})

# Some GraphQL gateways gate on a static client key or an app id instead:
# session.headers["x-api-key"] = "..."

Tokens usually live in one of three places: an Authorization: Bearer header, a session cookie the browser sends automatically, or a value embedded in the page's initial HTML or a __NEXT_DATA__/state blob that the front end reads and forwards. If it is cookie-based, use a requests.Session and prime it by first requesting a normal page on the site so the Set-Cookie lands, then post to /graphql on the same session.

There is also a GraphQL-specific gate worth knowing. Apollo Server 4 ships CSRF prevention that rejects requests that could be sent as a browser "simple request", which means a bare POST with no distinguishing header gets a 400. Sending Content-Type: application/json (which you already do for a JSON body) satisfies it, and some deployments instead expect an Apollo-Require-Preflight: true header. If you get a CSRF error despite a valid query, add that header. When the token itself is minted by client-side JavaScript and refreshes on a timer, replaying it from cold Python is fragile, and the cleaner path is to run the operation from inside a real browser session. The Scraping API section below does exactly that. For the wider set of blocking signals and how to stay under them, see how to avoid getting your proxy blocked.

At scale, the two things that break a direct Python POST are IP rate limits and the auth/CSRF/cookie dance. The SparkProxy Scraping API solves both by running your GraphQL operation from inside a real browser on a rotating IP. The trick is js_scenario with an evaluate step: render any same-origin page on the target, then fetch('/graphql') from within that page. Because the browser is already on the site, the session cookie, the CSRF header, and the same-origin policy are all satisfied for you.

import requests
import json

QUERY = """
query ProductList($first: Int!, $after: String) {
  products(category: "proxies", first: $first, after: $after) {
    edges { node { id name priceCents } }
    pageInfo { endCursor hasNextPage }
  }
}
"""

gql_body = json.dumps({"query": QUERY, "variables": {"first": 20, "after": None}})

evaluate_js = (
    "return await fetch('/graphql', {"
    "  method: 'POST',"
    "  headers: {'content-type': 'application/json'},"
    "  body: " + json.dumps(gql_body) +
    "}).then(r => r.text());"
)

body = {
    "url": "https://www.sparkproxy.io/proxies",   # any real page on the target origin
    "render_js": True,
    "premium_proxy": True,                    # residential exit IPs
    "country_code": "US",                     # fix the exit geography
    "js_scenario": {"instructions": [{"evaluate": evaluate_js}]},
}

r = requests.post(
    "https://scrape.sparkproxy.io/api/v1",
    headers={"X-API-Key": "YOUR_API_KEY", "Content-Type": "application/json"},
    json=body,
    timeout=90,
)
print(r.json())

The evaluate step returns the GraphQL response text straight back in the API result, so you parse the JSON from there. premium_proxy swaps to residential IPs when the endpoint rejects datacenter ranges, and country_code pins region-locked data to the right geography. Because the API assigns a fresh IP per call, the rotation and retry logic you would otherwise hand-roll is handled for you.

When the endpoint has no cookie or CSRF gate, you do not need the browser at all. Route a plain requests.post through SparkProxy proxies and keep the whole thing to one credit-cheap round trip per page. Deciding between the managed API and running the proxies yourself is a real trade-off, laid out in web scraping API vs self-managed proxies. Send high-volume, ungated GraphQL calls through the direct-proxy path, and reserve the browser render for the endpoints that fight back. The full parameter list is in the SparkProxy Scraping API docs.