How to Scrape Single Page Applications (SPAs)
Learn to scrape single page applications (React, Vue, Angular): find the hidden JSON API, read the __NEXT_DATA__ hydration payload, or render JS with an API.
To scrape single page applications built with React, Vue, or Angular, you have to change tactics. Fetch the URL and the HTML comes back nearly empty: a `
That is the whole response. There is no product grid in the HTML because the server did not build one. It shipped an empty container (the "app shell") plus a JavaScript bundle. The browser downloads that bundle, runs it, and only then does the app fetch data and paint the DOM. `requests.get()` and `curl` stop at the first step: they receive the shell and never run the JavaScript, so `soup.find(...)` returns `None` and your scraper looks broken.
This is the defining trait of a SPA. The page you see in the browser is assembled on the client, after load, by framework code. To scrape it, you either get the data the app fetches, or you run the JavaScript yourself.
For the broader family of JS-heavy pages (widgets, partial hydration, AJAX filters), see our guide on [scraping dynamic JavaScript websites](https://www.sparkproxy.io/blog/how-to-scrape-dynamic-javascript-websites). SPAs are the extreme case: almost nothing renders without JavaScript.
## How a SPA loads its data {#how-data-loads}
Once the bundle runs, a SPA gets its data one of two ways, and both are good news for a scraper.
Most apps call a backend API over `fetch` or `XMLHttpRequest`. Open the Network tab and you will see requests to endpoints like `https://www.sparkproxy.io/api/v2/products?page=1` returning clean JSON. That JSON is the data you want, minus all the HTML noise.
Server-rendered SPAs are the second case. Next.js, Nuxt, and Angular Universal often embed the first screen of data directly in the initial HTML so the page hydrates instantly and ranks in search. That data sits in a `<script>` tag as JSON, and you can read it with a single request and no browser at all.
Knowing which mechanism a site uses decides your approach. The Network tab and a quick search of the page source tell you within a minute.
## The two ways to scrape a SPA {#two-ways}
Every SPA scraping method is one of two ideas:
| Approach | How it works | Speed and cost | Breaks when |
|---|---|---|---|
| Browserless (get the JSON) | Read the embedded hydration JSON, or replay the XHR/GraphQL call the app makes | Fast and cheap, one HTTP request | The request is signed or obfuscated and there is no embedded state |
| Headless render | Load the app in real Chromium, wait for the content, read the finished DOM | Slower and heavier per request | Rarely; it survives obfuscation but costs more time and money |
Reach for browserless first. It is faster, cheaper, and easier to run at volume. Fall back to a headless browser when the app hides its API behind signed requests or renders only after interaction. The rest of this guide is those two paths in detail, plus the shortcut that sits between them.
## Strategy 1: Replay the underlying JSON API {#replay-api}
The fastest way to scrape a SPA is to skip the SPA and call the API it calls.
Find the endpoint first:
1. Open DevTools and go to the Network tab.
2. Click the Fetch/XHR filter so you only see data requests.
3. Reload the page, or trigger the action you want (search, next page, open a product).
4. Look for a request that returns JSON with your data. Click it, check the Response tab.
5. Right click the request and choose Copy as cURL to capture the exact URL, headers, and body.
Once you have the endpoint, replay it directly. No rendering required:
python
import requests
resp = requests.get(
"https://www.sparkproxy.io/api/v2/products",
params={"page": 1, "per_page": 50},
headers={"Accept": "application/json"},
timeout=15,
)
products = resp.json()["data"]
for p in products:
print(p["id"], p["title"], p["price"])
Pagination is usually just a query parameter you increment. One loop over `page` gives you the entire catalog, far faster than clicking through a rendered UI.
If the endpoint sits behind anti-bot protection, rate limits, or geo rules, route the same call through the SparkProxy Scraping API with `render_js=false`. That gives you clean IPs and header handling without paying for a browser you do not need:
bash
curl -G "https://scrape.sparkproxy.io/api/v1" \
-H "X-API-Key: YOUR_API_KEY" \
--data-urlencode "url=https://www.sparkproxy.io/api/v2/products?page=1" \
--data-urlencode "render_js=false"
Note that `render_js` defaults to `true`, so set it to `false` explicitly for a plain JSON fetch.
One caveat decides whether this strategy holds up. If the request carries a signature the client computes in JavaScript (an `Authorization` bearer minted on the fly, an `x-signature` header, or a GraphQL persisted query with a `sha256Hash` parameter), replaying it from Python is fragile and will break the next time the app changes its signing logic. Unsigned, public JSON endpoints are safe to replay. Signed ones are your cue to render instead.
For a deeper walkthrough of finding and calling these endpoints, see [scraping hidden JSON API endpoints](https://www.sparkproxy.io/blog/how-to-scrape-hidden-json-api-endpoints). If the app talks GraphQL, our [guide to scraping GraphQL APIs](https://www.sparkproxy.io/blog/how-to-scrape-graphql-apis) covers query replay and persisted queries.
## The shortcut: read the hydration payload {#hydration}
Here is the path most SPA scraping tutorials skip, and it is often the best one.
Server-rendered SPAs ship the first dataset inside the initial HTML so the framework can hydrate without a second round trip. The data is already in your `requests.get()` response. You do not need the API and you do not need a browser. You need one JSON parse or one regex.
Each framework has a known location:
| Framework | Where the JSON lives | Notes |
|---|---|---|
| Next.js | `<script id="__NEXT_DATA__">` | Pure JSON, easy to parse |
| Redux apps | `window.__INITIAL_STATE__` | Common convention, pure JSON |
| Vue / Vuex SSR | `window.__INITIAL_STATE__` | Same convention |
| Nuxt | `window.__NUXT__` | May be a JS expression, not always pure JSON |
| Angular Universal | `<script id="ng-state">` | Transfer State, JSON in a script tag |
| Apollo GraphQL | `window.__APOLLO_STATE__` | Normalized cache as JSON |
Next.js is the cleanest case. The page dataset sits in a script tag with a fixed id:
python
import json, requests
from bs4 import BeautifulSoup
html = requests.get("https://www.sparkproxy.io/shop").text
soup = BeautifulSoup(html, "html.parser")
blob = soup.find("script", id="__NEXT_DATA__").string
data = json.loads(blob)
products = data["props"]["pageProps"]["products"]
print(len(products), "products from one request")
For the `window.__INITIAL_STATE__` convention, pull it out with a regex:
python
import re, json
m = re.search(r"window\.__INITIAL_STATE__\s=\s(\{.?\})\s;?\s*", html, re.S)
state = json.loads(m.group(1))
Watch one detail: Nuxt's `window.__NUXT__` is sometimes a JavaScript function call rather than plain JSON, so `json.loads` can fail on it. When that happens, target the API instead or render the page. `__NEXT_DATA__`, `__INITIAL_STATE__`, and `__APOLLO_STATE__` are almost always clean JSON.
When a hydration payload exists, this is the cheapest scrape available: one GET, one parse, zero JavaScript executed. If the page is also behind anti-bot, fetch it through the Scraping API with `render_js=false` and parse the payload from the returned HTML the same way.
## Strategy 2: Render the SPA with a headless browser {#render}
When there is no embedded state and the API is signed, stop fighting it and let a real browser do the work. A headless Chromium loads the app, runs the bundle, makes the same signed requests the site expects, and hands you the finished DOM.
The SparkProxy Scraping API renders by default (`render_js=true`). The trick is to wait for the element that proves your data arrived, using `wait_for` with a CSS selector:
bash
curl -G "https://scrape.sparkproxy.io/api/v1" \
-H "X-API-Key: YOUR_API_KEY" \
--data-urlencode "url=https://www.sparkproxy.io/shop" \
--data-urlencode "render_js=true" \
--data-urlencode "wait_for=.product-card"
The response is the fully rendered HTML, so your normal parser works because the product grid is really there:
python
import requests
from bs4 import BeautifulSoup
resp = requests.get(
"https://scrape.sparkproxy.io/api/v1",
headers={"X-API-Key": "YOUR_API_KEY"},
params={
"url": "https://www.sparkproxy.io/shop",
"render_js": "true",
"wait_for": ".product-card",
},
timeout=60,
)
soup = BeautifulSoup(resp.text, "html.parser")
cards = soup.select(".product-card")
print(len(cards), "cards rendered")
Some SPAs only reveal data after an action: accepting a cookie banner, clicking a Load More button, or scrolling. Use `js_scenario` to script those steps before capture. Send it as a POST with a JSON body:
json
POST https://scrape.sparkproxy.io/api/v1
Content-Type: application/json
X-API-Key: YOUR_API_KEY
{
"url": "https://www.sparkproxy.io/shop",
"render_js": true,
"js_scenario": {
"instructions": [
{ "click": "#accept-cookies" },
{ "wait_for": ".product-card" },
{ "scroll": 1600 }
]
}
}
```
For heavily defended SPAs, add stealth for anti-detection layers and premium_proxy to route through residential IPs, plus country_code when the content is geo-specific. If you would rather run the browser yourself, our Playwright scraping guide covers the self-managed route.
Waiting for content and lazy loading
The most common rendering bug is capturing the page before its data lands. The fix is to wait for a signal, not a stopwatch.
Prefer wait_for with a selector that only exists once real content renders (.product-card, [data-loaded="true"], a specific table row). The Scraping API waits for that selector and gives up after 30 seconds, so you get the painted DOM instead of a spinner. Avoid fixed sleeps: a hardcoded 5 second wait is both too slow on fast loads and too short on slow ones. If you genuinely cannot key off a selector, wait accepts a small number of extra seconds (max 30) as a blunt fallback.
Lazy loading is the other trap. Many SPAs only fetch images and rows as you scroll toward them, so a top-of-page capture misses most of the list. Turn on auto-scroll with scroll=true, or script explicit scroll steps in js_scenario to trigger each batch. For endless feeds specifically, our infinite scroll scraping guide covers when to scroll the DOM versus call the paginated API behind it.
Handling client-side routing
SPAs use client-side routing: clicking a link calls the History API (pushState) and swaps the view without a full page reload. This changes how you target pages.
The good news is that most SPAs use real path-based URLs. A category page at https://www.sparkproxy.io/shop/category/shoes is a genuine URL you can request directly with render_js=true and a wait_for selector. You do not have to start at the homepage and click through. Feed the deep URL straight to the renderer.
Hash-based routing is the exception. If routes look like https://www.sparkproxy.io/shop#/category/shoes, the server never sees anything after the #, so requesting that URL returns the same app shell every time. Two fixes work: render the root URL and use a js_scenario click to navigate to the route before capture, or find the underlying API call for that route and use Strategy 1. Hash routing is a strong hint that replaying the API will be simpler than rendering.
When each strategy wins
| Situation | Best strategy |
|---|---|
| Page embeds `__NEXT_DATA__` or `__INITIAL_STATE__` | Read the hydration payload (one GET) |
| Clean, public JSON or XHR you can replay | Replay the API |
| Signed request, dynamic headers, persisted GraphQL hash | Headless render |
| Data appears only after click, scroll, or login | Headless render with `js_scenario` |
| Very high volume, simple JSON | Replay the API (cheapest per request) |
| No SSR and no clean API (some Angular and Vue apps) | Headless render |
The pattern is consistent: get the JSON without a browser whenever you can, and render only when the site forces you to. Many teams combine both, replaying APIs for the bulk of pages and rendering the handful that are signed or interaction-gated. For a wider comparison of running your own browser fleet versus an API, see web scraping API vs self-managed proxies.
Frequently asked questions
FAQ
Because a SPA ships an empty app shell plus a JavaScript bundle, and builds the visible page in the browser after load. View Source shows the raw server response, which runs no JavaScript, so the content is missing. The browser's Inspect or Elements panel shows the rendered DOM instead.
Often, yes. If the site embeds a hydration payload (__NEXT_DATA__, window.__INITIAL_STATE__) you can read the JSON from one HTTP request. If it calls a public JSON or XHR endpoint, you can replay that request directly. You only need a browser when the request is signed or the content appears after interaction.
__NEXT_DATA__ is a script tag Next.js embeds in the initial HTML that holds the page's data as JSON for hydration. Fetch the page, select script#__NEXT_DATA__, and run json.loads on its contents to get structured data without rendering. Nuxt, Redux, and Apollo apps use similar globals such as window.__NUXT__ and window.__APOLLO_STATE__.
Wait for a CSS selector that only exists once real content renders, rather than sleeping a fixed number of seconds. With the SparkProxy Scraping API, set render_js=true and wait_for=.your-selector; it waits up to 30 seconds for that element before capturing the DOM. Use auto-scroll for lazy-loaded lists.
Call the API when the endpoint is public and unsigned, because it is faster and cheaper. Render the page when the request carries a client-generated signature, uses a persisted GraphQL hash, or only fires after a click or scroll. A quick look at the request headers in the Network tab tells you which case you are in.
Use the Scraping API's js_scenario to script the interaction before capture. Chain instructions like { "click": "#load-more" }, { "wait_for": ".product-card" }, and { "scroll": 1600 }, then read the rendered HTML. This handles cookie banners, Load More buttons, and lazy-loaded content.
Get 20% off your first month
Premium datacentre proxies with unlimited bandwidth. Use the code at checkout.
Save up to 15% more on quarterly, half-yearly and yearly plans
Related articles
How Many Proxies Do I Need for Web Scraping?
How many proxies do I need? Size threads, IPs per target and Mbps from your real scraping volume, then match the number to a plan you should actually buy.
How to Check Proxy IP Fraud Score and Geo Accuracy
Test the pool before you buy. Check a proxy IP fraud score across scoring vendors, verify geolocation on three layers, and set honest pass or fail thresholds.
Enterprise Proxy Procurement: What Security and Legal Will Ask
Buying an enterprise proxy provider? The exact questions security, legal and procurement ask, the answers that pass, and the ones that end the deal.