Guides

How to Scrape JavaScript-Rendered Websites in 2026

Q: Why do requests and BeautifulSoup return empty data on some sites?

Because requests downloads only the initial HTML and never runs JavaScript. On React, Vue, or Angular sites, the content is fetched and rendered by JavaScript after load, so the raw HTML is an empty shell. You need XHR interception, a headless browser, or a scraping API with rendering to get the real data.

Q: What does render_js do in a scraping API?

render_js=true tells the API to load the target in a real headless browser, execute its JavaScript, and return the fully rendered HTML instead of the raw source. On the SparkProxy Scraping API it defaults to true, costs about 5x a non-rendered request, and pairs with wait_for so capture happens only after the content appears.

Scrape JavaScript-rendered websites reliably: why requests returns empty pages, and when to use XHR interception, a headless browser, or a render_js API.

S SparkProxyJul 22, 2026 - 14:30 1 17 min read

How to Scrape JavaScript-Rendered Websites in 2026

Try to scrape JavaScript-rendered websites with the same requests and BeautifulSoup code that works on static pages, and you'll usually get an empty shell back. The price, the product grid, or the reviews you wanted aren't in the HTML your scraper downloads, even though they're right there in your browser. This guide explains exactly why that happens and gives you three working ways to get the real, rendered data: intercepting the site's own JSON API, driving a headless browser, and calling a scraping API with render_js. Each comes with real code and a decision table so you pick the cheapest approach that actually works.

Why requests and BeautifulSoup return empty pages

A plain HTTP client downloads exactly one thing: the HTML the server sends in the initial response. On a static site that HTML already contains the content. On a modern site built with React, Vue, Angular, or Svelte, the initial HTML is mostly an empty container plus a bundle of JavaScript. The content gets fetched and painted into the page after the browser runs that JavaScript.

Here's the failure in code. This looks correct and returns nothing useful:

import requests
from bs4 import BeautifulSoup

resp = requests.get("https://www.sparkproxy.io/products")
soup = BeautifulSoup(resp.text, "html.parser")

# The product grid is rendered client-side, so this is empty
print(len(soup.select(".product-card")))  # -> 0

requests never executes JavaScript. It hands you the raw document, and for a client-rendered app that document often boils down to something like this:

<body>
  
  <script src="/static/js/main.8f3a.js"></script>
</body>

There's no product grid to parse because the grid doesn't exist until Chrome runs main.8f3a.js, calls the backend, and injects the results. This is the core reason your scraper "sees" a different page than you do. It isn't blocked and it isn't a bug in your selectors. The data simply hasn't been rendered yet. If you're new to the mechanics of extraction, our primer on what web scraping is covers the request-and-parse basics this guide builds on.

How to tell if a site is JavaScript-rendered

Before you reach for a heavier tool, confirm you actually need one. Rendering is expensive, so don't pay for it unless the page requires it. Three quick checks:

1. View the raw source, not the DOM. In your browser, open view-source:https://www.sparkproxy.io/products (the literal source), not the Elements panel. The Elements panel shows the rendered DOM and will always look complete. If the raw source is missing the data but the Elements panel has it, the page is client-rendered.

2. Compare lengths in code. Fetch the page and grep for a value you can see on screen:

import requests

html = requests.get("https://www.sparkproxy.io/products").text
print("Price found in raw HTML:", "$" in html and "product-card" in html)
print("HTML length:", len(html))  # a near-empty shell is often < 50 KB

3. Look for a framework fingerprint. Search the raw HTML for id="root", id="__next", __NEXT_DATA__, window.__NUXT__, or ng-version. These tell you which framework rendered the page, and sometimes hand you the data directly (more on that next).

If the content is present in the raw HTML, skip everything below and scrape it the cheap way with Python requests or aiohttp. Only the pages that fail these checks need the approaches that follow.

Free trial

Scraping at scale? Skip the blocks.

Fast, unblockable datacentre proxies with unlimited bandwidth.

Start Free Trial

Three ways to scrape dynamic websites

There are exactly three reliable strategies, and they trade off in a predictable way. Most guides jump straight to "install Selenium," but that's often the slowest and most fragile option. Work down this list and stop at the first one that gets you the data:

Approach	How it works	Relative speed & cost	Reliability	Best for
XHR interception	Call the JSON/API endpoint the page's JS already calls	Fastest, cheapest (plain HTTP)	High, until the API changes	Data behind a clean internal API
Headless browser	Run real Chromium (Playwright/Puppeteer), scrape the rendered DOM	Slow, heavy (RAM + CPU per page)	High, but you own scaling & anti-bot	Complex flows, when you run your own fleet
API-based rendering	Send the URL to a scraping API with `render_js=true`	Slower than raw HTTP, no infra to run	High, provider owns browsers & fingerprints	Rendering at scale without managing Chrome

The order matters. XHR interception is almost always the right first attempt because it skips rendering entirely. If that fails, you choose between running your own headless browsers or offloading rendering to an API, which mostly comes down to whether you want to maintain browser infrastructure. That build-versus-buy question is covered in depth in our scraping API vs self-managed proxies comparison.

Option 1: Intercept the hidden JSON/XHR endpoint

Here's the insight most tutorials skip: the JavaScript that renders the page has to get its data from somewhere. That "somewhere" is almost always a clean JSON API sitting one network call away. If you call that endpoint directly, you get structured data with no browser, no rendering cost, and no brittle CSS selectors.

Find it in about a minute:

Open DevTools and go to the Network tab.
Filter to Fetch/XHR.
Reload the page or trigger the action (scroll, click "load more").
Look for a request that returns JSON with the data you want. Copy its URL.

Then replicate that request in code. It usually looks like this:

import requests

# The endpoint the page's JavaScript actually calls
url = "https://www.sparkproxy.io/api/v1/products"
params = {"page": 1, "per_page": 48, "sort": "price_asc"}

resp = requests.get(url, params=params, headers={
    "Accept": "application/json",
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64)",
})

data = resp.json()
for item in data["results"]:
    print(item["name"], item["price"])

This is faster and an order of magnitude cheaper than rendering, and the output is already parsed. Two things to watch for:

Headers and tokens. Some APIs require a Referer, an X-CSRF-Token, or an auth header the front end sends. Copy those from the Network tab too. In Chrome, "Copy as cURL" on the request captures every header exactly.
Pagination. Internal APIs almost always paginate with page, offset, or a cursor. Loop until the response returns fewer items than the page size.

When you're hitting an internal API in a tight loop, you'll trip rate limits fast. Rotating IPs and pacing requests is its own topic, covered in how to scrape high-volume data without rate limiting.

XHR interception fails in three cases: the data is inlined into the initial JS bundle instead of fetched, the endpoint is protected by a signed token that's hard to reproduce, or the values you need are computed in the browser. When any of those apply, you have to actually render the page.

Option 2: Run a headless browser

A headless browser is real Chromium with no visible window. It runs the site's JavaScript exactly like a normal visit, so the DOM you scrape is the fully rendered page. Playwright is the current default for Python. Here's the pattern, including the all-important wait:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page()
    page.goto("https://www.sparkproxy.io/products", wait_until="networkidle")

    # Wait for the real content, not just page load
    page.wait_for_selector(".product-card", timeout=15000)

    cards = page.query_selector_all(".product-card")
    for card in cards:
        name = card.query_selector(".name").inner_text()
        price = card.query_selector(".price").inner_text()
        print(name, price)

    browser.close()

The single most common mistake here is scraping too early. goto() returns when the document loads, but the data arrives later over the network. Always wait for a selector that only exists once the content is present. Never sleep for a fixed number of seconds and hope, because that's slow on fast pages and still fails on slow ones.

To stay unblocked at any real volume you'll route the browser through proxies and, on tough targets, match its fingerprint. Playwright and Puppeteer both accept a proxy per context:

browser = p.chromium.launch(
    headless=True,
    proxy={
        "server": "http://gate.sparkproxy.io:8000",
        "username": "USER",
        "password": "PASS",
    },
)

The full setup, including rotating IPs across browser contexts, is in how to use proxies with Puppeteer.

Headless browsers work, but the cost is real. Each Chromium instance eats 300 MB to 700 MB of RAM, cold starts take seconds, and running dozens in parallel means orchestrating a browser fleet, handling crashes, and keeping fingerprints current as anti-bot systems evolve. If that infrastructure isn't your product, the third option removes it entirely.

Option 3: API-based rendering with render_js

A scraping API runs the headless browser for you. You send one HTTP request with the target URL, the provider renders it in a real browser on their side, and you get back the fully rendered HTML (or parsed data). No Chrome to install, no fleet to scale, no fingerprints to maintain.

With the SparkProxy Scraping API, rendering is a single parameter. The endpoint is https://scrape.sparkproxy.io/api/v1 and you authenticate with an X-API-Key header:

curl -H "X-API-Key: sk-your-key" \
  "https://scrape.sparkproxy.io/api/v1?url=https://www.sparkproxy.io/products&render_js=true"

The same thing in Python, returning the rendered HTML you can then parse with BeautifulSoup:

import requests
from bs4 import BeautifulSoup

resp = requests.get(
    "https://scrape.sparkproxy.io/api/v1",
    headers={"X-API-Key": "sk-your-key"},
    params={
        "url": "https://www.sparkproxy.io/products",
        "render_js": "true",
        "country_code": "US",       # geo-target the exit IP
        "wait_for": ".product-card" # don't capture until the grid exists
    },
)

soup = BeautifulSoup(resp.text, "html.parser")
print(len(soup.select(".product-card")))  # now returns the real count

render_js defaults to true on this API, so you turn it off for static targets to save credits. That cost difference is the number to keep in mind: a rotating-proxy request without JS costs 1 credit, but with JS rendering it's 5 credits, and a residential (premium_proxy) request with JS is 25. Rendering is 5x the cost of not rendering. That's the whole reason Option 1 exists and why you confirm a page is truly client-rendered before you turn rendering on.

You can also skip parsing entirely and ask for clean output with format:

params = {
    "url": "https://www.sparkproxy.io/products",
    "render_js": "true",
    "format": "md",  # html, md, mdx, json, screenshot, or pdf
}

Or pull specific fields server-side with extract_rules, so you never ship raw HTML back at all:

import json, requests

rules = {"products": {"selector": ".product-card", "type": "list",
                       "output": {"name": ".name", "price": ".price"}}}

resp = requests.get(
    "https://scrape.sparkproxy.io/api/v1",
    headers={"X-API-Key": "sk-your-key"},
    params={"url": "https://www.sparkproxy.io/products",
            "render_js": "true",
            "extract_rules": json.dumps(rules)},
)

Waiting for content: wait vs wait_for

Getting the timing right is what separates a scraper that works from one that returns half-empty pages. The SparkProxy API gives you two levers, and they solve different problems.

wait is a fixed delay in seconds (max 30) after load. Use it only when there's no stable element to key on, for example a page that streams updates. It's the blunt instrument.
wait_for takes a CSS selector and blocks until that element appears in the DOM. This is what you want almost every time, because it's tied to the actual content instead of a guessed duration.

# Precise: capture the moment the reviews list is in the DOM
params = {
    "url": "https://www.sparkproxy.io/products/42",
    "render_js": "true",
    "wait_for": "#reviews .review-item",
}

# Blunt fallback: no reliable selector, wait 4 seconds
params_fallback = {
    "url": "https://www.sparkproxy.io/live-feed",
    "render_js": "true",
    "wait": 4,
}

The same principle applies to headless browsers: page.wait_for_selector(".target") beats time.sleep(5) on both speed and reliability. Pick an element that appears only after your data loads, not one that's in the initial shell like a header or footer.

Handling SPAs and infinite scroll with js_scenario

A single page application often needs more than a wait. You have to dismiss a cookie banner, click a tab, scroll to trigger lazy loading, or type into a search box before the data you want exists. That's what js_scenario is for: a scripted sequence of browser actions the renderer runs before capturing the page.

Here's a POST request that closes a cookie banner, waits for the grid, scrolls to load more, then runs a search, all in one call:

import requests

scenario = {
    "instructions": [
        {"click": "#cookie-banner-close"},
        {"wait_for": ".product-card"},
        {"scroll": 800},
        {"fill": {"selector": "#search", "value": "residential proxy"}},
        {"click": "#search-submit"},
        {"wait_for": ".results-loaded"}
    ]
}

resp = requests.post(
    "https://scrape.sparkproxy.io/api/v1",
    headers={"X-API-Key": "sk-your-key", "Content-Type": "application/json"},
    json={"url": "https://www.sparkproxy.io/catalog",
          "render_js": True,
          "js_scenario": scenario},
)
print(resp.text)

For infinite scroll, the pattern is repeated scrolls with a short wait between each so new items can load. On the SparkProxy API, scroll is true by default and auto-scrolls to trigger lazy loading; for finer control, chain explicit scroll steps in a js_scenario. Doing the same thing in Playwright means a loop:

prev_height = 0
while True:
    page.mouse.wheel(0, 20000)
    page.wait_for_timeout(1000)
    height = page.evaluate("document.body.scrollHeight")
    if height == prev_height:   # nothing new loaded, we're done
        break
    prev_height = height

The stop condition matters. Scroll until the page height stops growing, not a fixed number of times, or you'll either miss items on long lists or waste requests on short ones.

Which approach should you use? Decision table

Match your situation to a row and start there. Fall back down the list only if the first choice fails:

Your situation	Start with	Why
Data loads from a clean JSON call you can see in the Network tab	XHR interception	Fastest and cheapest; structured data, no rendering
Internal API is token-signed or the data is inlined in the bundle	Rendering (browser or API)	You can't cleanly replay the request
You need clicks, form fills, or multi-step flows before data appears	`js_scenario` (API) or Playwright	Rendering plus scripted interaction
High volume, and running Chrome fleets isn't your core work	API-based `render_js`	Provider owns browsers, proxies, and fingerprints
You already operate a browser fleet and need deep custom control	Headless browser	Maximum control, you accept the maintenance
The page is actually static (passed the raw-HTML check)	Plain `requests` / aiohttp	Never pay for rendering you don't need

The through-line: rendering is the expensive fallback, not the default. Confirm the page needs it, try to skip it with an XHR call, and when you do render, let a selector tell you when the content arrived.

Common mistakes to avoid

A few errors show up again and again when people first scrape dynamic sites:

Rendering everything by default. Rendering costs roughly 5x a plain request and is far slower. Check whether the page is really client-rendered first.
Sleeping instead of waiting for a selector. Fixed sleep() is slow on fast pages and flaky on slow ones. Wait for the element that signals your data loaded.
Ignoring the Network tab. Teams spin up Selenium for data that was one clean JSON request away. Always look for the XHR endpoint first.
Scraping the shell. Reading .inner_text() right after goto() grabs the empty container. Wait for content-specific selectors.
No proxies at volume. A real browser still leaks one IP. Rotate IPs and geo-target so you don't get rate-limited or blocked after a few hundred requests.
Guessing the infinite-scroll count. Loop until the scroll height stops changing rather than scrolling a hard-coded number of times.

Get these six right and most "the data won't load" problems disappear.

Frequently asked questions

FAQ

Why do requests and BeautifulSoup return empty data on some sites?

Because requests downloads only the initial HTML and never runs JavaScript. On React, Vue, or Angular sites, the content is fetched and rendered by JavaScript after load, so the raw HTML is an empty shell. You need XHR interception, a headless browser, or a scraping API with rendering to get the real data.

How do I scrape a JavaScript website in Python without a browser?

Open DevTools, go to the Network tab, filter to Fetch/XHR, and find the JSON endpoint the page calls. Then request that URL directly with requests and read resp.json(). It's faster and cheaper than rendering because you skip the browser entirely and get structured data.

Is headless browser scraping slower than using requests?

Yes, significantly. A headless browser has to launch Chromium, run the site's JavaScript, and wait for network calls, which adds seconds per page and 300 MB or more of RAM per instance. Plain HTTP requests finish in milliseconds, which is why you intercept the XHR endpoint whenever you can.

Can I scrape a single page application (SPA) without Selenium?

Yes. Many SPAs load data from a JSON API you can call directly with no browser at all. When the SPA needs clicks or scrolling first, a scraping API with a js_scenario handles the interaction server-side, so you get rendered output without running or maintaining Selenium yourself.

What does render_js do in a scraping API?

render_js=true tells the API to load the target in a real headless browser, execute its JavaScript, and return the fully rendered HTML instead of the raw source. On the SparkProxy Scraping API it defaults to true, costs about 5x a non-rendered request, and pairs with wait_for so capture happens only after the content appears.

How do I wait for dynamic content to load before scraping?

Wait for a CSS selector that exists only after your data renders, not a fixed timer. Use wait_for=".product-card" on the SparkProxy API or page.wait_for_selector() in Playwright. Avoid sleep() because it's slow on fast pages and still fails on slow ones.

#scrape javascript rendered websites #scrape dynamic websites #scrape javascript website python #headless browser scraping #scrape single page application #render_js #web scraping

Limited-time · 50% off

Get 50% off your first purchase

Premium datacentre proxies with unlimited bandwidth. Use the code at checkout.

Offer ends soon — claim it before it's gone

CodeSPARK50

Claim Discount

About the Author

This guide was written by the SparkProxy Technical Team. SparkProxy builds proxy and data-collection infrastructure used by scraping and automation teams, including datacenter proxies, residential proxies, and the SparkProxy Scraping API with built-in JavaScript rendering. We publish these guides from hands-on work supporting customers who collect data from dynamic, JavaScript-heavy sites at scale. For the full parameter reference behind the code above, see the Scraping API documentation.