Why Your Scraper Returns an Empty Page (and How to Fix It)
Scraper returns empty page with a 200? A 5-step triage separates JS rendering, disguised block pages and geo or consent gates, with a fix and code for each.
When a scraper returns an empty page, the script almost never crashes. You get 200 OK, soup.select(".price") comes back as [], and the HTML you print is either a near-empty or a page full of markup that contains none of the numbers you can see in your browser. Three different causes produce that identical symptom, and each one needs a different fix, so guessing wastes days. This guide gives you a five-step triage, cheapest check first, that tells you which cause you have in a few minutes, then the fix for each.
The symptom, and the three causes behind it
Here is the script that brought most readers to this page:
import requests
from bs4 import BeautifulSoup
r = requests.get("https://shop.sparkproxy.io/products/widget-42", timeout=30)
print(r.status_code) # 200
soup = BeautifulSoup(r.text, "html.parser")
print(soup.select("span.price")) # []
print(soup.get_text(strip=True)[:200]) # "Loading..." or a cookie notice or nothing
Status 200 only says the server answered. It says nothing about which document it chose to send you. In practice the blank result comes from one of three places:
| Cause | What the server sent | One-line test | Fix |
|---|---|---|---|
| 1. JavaScript-rendered content | The real page shell; data arrives later via XHR | The visible value is absent from raw HTML but present after rendering | Render (headless browser or `render_js`), or call the JSON endpoint |
| 2. Disguised block page | A challenge or "access denied" document with status 200 | Title, `cf-mitigated` header or a tiny body gives it away | Change IP class, pacing or fingerprint |
| 3. Geo or consent interstitial | A cookie wall, region selector or a different country's site | Final URL, `` or currency differs from your browser | Send a locale, pick an exit country, carry consent cookies |
These overlap more than people expect. A React storefront (cause 1) can also be geo-gated (cause 3), and rendering it from a flagged IP adds cause 2 on top. That is why the order of checks matters: each step rules something out with a single request before you spend money on the next.
Step 1: Save the raw bytes and measure them
Do not debug from print(soup). BeautifulSoup repairs broken markup, your terminal truncates, and you end up reasoning about a document nobody sent. Write the exact bytes to disk and take four measurements:
import re
import requests
def snapshot(url, **kw):
r = requests.get(url, timeout=30, **kw)
with open("raw.html", "wb") as fh:
fh.write(r.content)
html = r.text
visible = re.sub(r"<script.*?</script>|<style.*?</style>|<[^>]+>", " ", html, flags=re.S)
return {
"status": r.status_code,
"bytes": len(r.content),
"script_tags": len(re.findall(r"<script\b", html)),
"visible_words": len(visible.split()),
"final_url": r.url,
}
print(snapshot("https://shop.sparkproxy.io/products/widget-42"))
Read the numbers as a shape, not as thresholds:
- Large body, many script tags, very few visible words points at cause 1. A client-rendered shell is mostly JavaScript bundles and a mount node.
- Small body, few scripts, a handful of words points at cause 2. Challenge and denial pages are short, single-purpose documents.
- Normal size and plenty of words, just not your words points at cause 3, or at a page that was legitimately redesigned.
Open raw.html in a browser with JavaScript disabled as well. You will see exactly what your parser saw, and half the time the answer is printed on screen: a spinner, a "verify you are human" box, or a language picker.
Scraping at scale? Skip the blocks.
Fast, unblockable datacentre proxies with unlimited bandwidth.
Step 2: Grep the raw HTML for one value you can see
Pick one concrete string from the page in your real browser: a price like 49.99, a SKU, a review author. Then search the saved bytes for it. This is the single most decisive test in the triage, and it costs nothing.
needle = "49.99"
raw = open("raw.html", encoding="utf-8", errors="replace").read()
print("in raw HTML:", needle in raw)
for m in re.finditer(re.escape(needle), raw):
print(raw[max(0, m.start() - 80): m.end() + 40].replace("\n", " "))
Three outcomes:
- Found inside normal markup. Your selector is wrong, not the fetch. Check the class name in
raw.htmlrather than in DevTools, because DevTools shows the live DOM after scripts ran. - Found inside a