How to Scrape JSON API Endpoints Behind a Site

When a product grid, a review list, or an infinite-scroll feed loads after the page appears, that data almost always arrives over an XHR or fetch call returning JSON. Parsing the rendered HTML means running a headless browser, waiting for the DOM to settle, then writing CSS or XPath selectors against markup that changes every redesign. Hitting the JSON endpoint skips all of that. You ask the same URL the browser asks, and you get back structured, typed fields.

The difference is not cosmetic. Here is the trade-off, dimension by dimension:

The completeness point is the one people underestimate. A product card might show a rounded price and a star rating, while the JSON behind it carries the raw price to four decimals, the internal SKU, stock counts per warehouse, and a sponsored flag the UI never renders. You are reading the same feed the front end reads, so you get everything the front end was handed. If you are new to the broader discipline, our primer on what web scraping is covers the vocabulary used below.

There is a boundary worth naming up front. This technique targets the exact endpoints the site's own front end calls in your normal browser. It is not credential theft or bypassing access controls. Scrape public data you can already see, respect the site's terms and robots.txt, and stay away from personal or authentication-gated data you have no right to.

---

Every browser ships the tool you need. The goal is to catch the request that carries the data you can see on the page.

1. Open DevTools. F12 on Windows and Linux, Cmd+Option+I on macOS.
2. Click the Network tab, then the Fetch/XHR filter. That hides images, fonts, and scripts so only the API-style calls remain.
3. Keep DevTools open and trigger the load. Reload the page, scroll to fire the next batch, click "load more", or type into a search box. Watch new rows appear.
4. Find the row that holds your data. Two fast ways: sort by Size (data payloads are usually the largest JSON responses), or use Network search.

That last trick is the one most guides skip and it is the fastest by far. Press Ctrl+F (or Cmd+F) inside the Network panel to open a search box that scans across every response body, not just the URLs. Type a value you can see on the page, say a product name or a price, and DevTools points you straight at the request whose response contains it. No guessing.

Once you have a candidate, click it and open the Preview or Response tab. If you see the JSON that matches the page, you found the endpoint. Now open the Headers tab and note four things: the Request URL, the method (GET or POST), the query string parameters, and the request headers. For a POST, also grab the request payload. Those pieces are the entire recipe you will replay.

---

Before writing any code, let the browser hand you a working command. Right-click the request and choose Copy > Copy as cURL (bash). You get the full call with every header and cookie the browser sent:

curl 'https://www.sparkproxy.io/api/v2/products?category=proxies&page=1&limit=50' \
  -H 'accept: application/json' \
  -H 'referer: https://www.sparkproxy.io/products' \
  -H 'x-requested-with: XMLHttpRequest' \
  -H 'user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64)'

Run that in a terminal. If it returns the same JSON you saw in DevTools, you have a reproducible request and the rest is cleanup. Now figure out which headers actually matter, because copying all of them verbatim into production is noisy and fragile. Most hidden APIs care about a small subset:

• accept: application/json tells the server to return JSON rather than an HTML fallback. Many endpoints gate on this.
• x-requested-with: XMLHttpRequest is a common AJAX marker that some frameworks require.
• referer and origin are checked by APIs that refuse calls not coming from their own pages.
• An auth header (authorization: Bearer ..., x-api-key: ...) or a session cookie, when the data is account-scoped.

Strip the request down to those, test again, and keep only what breaks the call when removed. Paste the cURL into a converter, or use curl_cffi if the site fingerprints TLS, but for most JSON endpoints plain requests is enough.

---

With the minimal header set known, the replay is short. This is the core loop for any endpoint that needs no login:

import requests

resp = requests.get(
    "https://www.sparkproxy.io/api/v2/products",
    params={"category": "proxies", "page": 1, "limit": 50},
    headers={
        "Accept": "application/json",
        "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64)",
        "Referer": "https://www.sparkproxy.io/products",
        "X-Requested-With": "XMLHttpRequest",
    },
    timeout=30,
)
resp.raise_for_status()

data = resp.json()
for item in data["items"]:
    print(item["name"], item["price"], item["sku"])

Passing the query parameters as a params dict instead of hard-coding them into the URL is not a style preference. It makes pagination and filtering trivial, because you change one value in the dict rather than rebuilding a string. resp.raise_for_status() turns a silent 403 into a loud exception so you notice an auth problem immediately instead of parsing an error page as if it were data.

When you scale this from one call to thousands, the network layer becomes the bottleneck. Running these requests concurrently with a proxy pool is its own topic, covered in our guide on async scraping with requests and aiohttp. The endpoint you found here plugs straight into that concurrency model, since each call is independent and stateless.

---

Endpoints tied to a logged-in view, a dashboard, or a "your account" widget need more than headers. Three patterns cover almost everything you will meet.

Cookies that ride along. Many APIs just need the session cookie a normal page visit sets. Use a requests.Session, load one page to collect cookies, then call the API through the same session so the cookies travel with it automatically:

import requests

s = requests.Session()
s.headers["User-Agent"] = "Mozilla/5.0 (Windows NT 10.0; Win64; x64)"

# 1. Warm the session so the server hands us its cookies.
s.get("https://www.sparkproxy.io/dashboard", timeout=30)

# 2. Now the JSON call carries those cookies with no extra work.
r = s.get(
    "https://www.sparkproxy.io/api/v2/usage",
    params={"range": "30d"},
    headers={"Accept": "application/json"},
    timeout=30,
)
r.raise_for_status()
print(r.json())

Bearer tokens embedded in the page. Single-page apps often print an access token or an API key into the initial HTML, inside a