How to Scrape Infinite Scroll and Load-More Pages

Infinite scroll and load-more buttons work the same way under the hood. When you near the bottom of the list (or click a button), client-side JavaScript fires an XHR or fetch request to a backend endpoint, gets back a batch of items, and appends them to the page. The user sees an endless stream. The browser sees a series of small network calls.

That means the data lives in exactly two places you can grab it from:

• The rendered DOM, after you make the browser scroll.
• The network request itself, if you can find and replay it.

Most tutorials only teach the first one. The second is usually faster, cheaper, and returns cleaner data because it hands you JSON instead of HTML you have to parse. The right move is to check for a usable API first, and fall back to browser scrolling only when there isn't one. Pick based on the page, not habit.

Use this when there's no reachable API: the endpoint needs signed tokens you can't reproduce, the response is pre-rendered HTML fragments instead of data, or you also need the page's visual state. A headless browser runs the site's real JavaScript, so lazy-loaded content appears exactly as it would for a human.

Here's a Playwright (Python) scraper that scrolls, extracts as it goes, and stops on its own:

import random
from playwright.sync_api import sync_playwright

def scrape_infinite_scroll(url, item_selector, id_attr="data-id", max_idle=3):
    results = {}
    with sync_playwright() as p:
        browser = p.chromium.launch(headless=True)
        page = browser.new_page()
        page.goto(url, wait_until="domcontentloaded")
        page.wait_for_selector(item_selector)

        idle_rounds = 0
        while idle_rounds < max_idle:
            height_before = page.evaluate("document.body.scrollHeight")

            # Extract what's on screen NOW, before nodes get recycled.
            for el in page.query_selector_all(item_selector):
                key = el.get_attribute(id_attr) or el.inner_text()[:120]
                results[key] = el.inner_text()

            page.mouse.wheel(0, height_before)
            page.wait_for_timeout(random.randint(800, 1600))

            height_after = page.evaluate("document.body.scrollHeight")
            idle_rounds = idle_rounds + 1 if height_after == height_before else 0

        browser.close()
    return list(results.values())

Three things make this reliable. It extracts inside the loop rather than once at the end (that matters for virtualized lists, covered below). It dedupes by a stable key, so re-reading the same rows costs nothing. And it stops after a few idle rounds where the page height stops growing, instead of guessing a scroll count.

If your stack is Node rather than Python, Puppeteer does the same job with page.evaluate and page.mouse.wheel. Our guide on using proxies with Puppeteer covers the browser-plus-proxy setup so your scroll sessions rotate IPs cleanly.

This is the better approach whenever it's available. The scroll is just a trigger for a network call, and that call almost always returns clean, structured JSON. If you talk to it directly, you skip the browser entirely.

Finding the endpoint takes about a minute:

1. Open DevTools, go to the Network tab, and filter by Fetch/XHR.
2. Scroll the page once. Watch for a request that returns a list of items.
3. Inspect it: the URL, the query params (page, offset, cursor, after, limit), the method, and any required headers (an auth token, a CSRF token, a referer).
4. Replay it in code, incrementing the pagination parameter until the data runs out.

Most APIs use one of two pagination shapes. Offset or page-number style:

import requests

def fetch_paged(base_url, per_page=50):
    page, out = 1, []
    while True:
        r = requests.get(base_url, params={"page": page, "per_page": per_page}, timeout=20)
        batch = r.json().get("items", [])
        if not batch:          # empty page means we're done
            break
        out.extend(batch)
        page += 1
    return out

Cursor or token style, common on modern feeds:

def fetch_cursor(base_url):
    cursor, out = None, []
    while True:
        params = {"limit": 50}
        if cursor:
            params["after"] = cursor
        data = requests.get(base_url, params=params, timeout=20).json()
        out.extend(data["results"])
        cursor = data.get("next_cursor")
        if not cursor or not data["results"]:   # null cursor means end
            break
    return out

This is the same idea behind scraping hidden JSON endpoints in general: the list you see rendered is a thin wrapper around an API response, and going to the source is far more stable than parsing markup that changes with every redesign. Copy any headers the endpoint requires (Authorization, x-csrf-token, referer), because some backends reject requests that arrive without them.

For deep pagination across thousands of pages, run the requests concurrently instead of one at a time. Our walkthrough on async scraping with Python requests and aiohttp shows how to fan out pagination without tripping rate limits.

The short version: try the API first. Reach for the browser when the endpoint is locked behind tokens you can't reproduce, when the response is HTML rather than data, or when you genuinely need the rendered page.

The most common bug in scroll scraping is stopping too early or looping forever. Never hardcode "scroll 10 times." Feeds grow, and a fixed count either misses new items or wastes requests on an empty page. Use a real stop signal instead. Three are reliable:

• Page height stops growing. document.body.scrollHeight is identical across a scroll plus a wait. This is what the code in Approach 1 uses.
• Item count stops increasing. The number of matched elements is the same between two rounds.
• The API tells you. An empty results array, has_more: false, a null next_cursor, or an HTTP 404 on the next page.

Combine a stop signal with an idle-round counter, so one slow network response doesn't end the run prematurely. Requiring two or three consecutive idle rounds before quitting is a good default. It costs a couple of extra scrolls and saves you from truncated datasets.

This is the gotcha that silently ruins scrapes, and most tutorials never mention it. Many high-performance feeds use list virtualization (also called windowing or DOM recycling). Libraries like react-window, react-virtualized, TanStack Virtual, Angular CDK virtual scroll, and vue-virtual-scroller render only the rows near the viewport, maybe 20 to 40 at a time, and recycle those same DOM nodes as you scroll. The list feels endless, but the DOM never holds more than a screenful.

Scrape the DOM once after scrolling to the bottom and you'll get only the last 20 items. Everything above scrolled out of the DOM as you went. That's why the Approach 1 code extracts inside the loop and dedupes: you have to harvest each batch while it's painted, before the library throws those nodes away.

You can spot a virtualized list quickly:

• The scroll container has a large inner spacer element that holds the total height, while the visible rows sit in a small window.
• The DOM item count stays roughly constant as you scroll, even though the scrollbar keeps shrinking.
• Rows carry inline transform: translateY(...) styles that reposition recycled nodes.

Two fixes. Extract incrementally and dedupe by a stable key, which the browser approach already does. Or, better, find the API. Pagination returns every record regardless of what's painted on screen, which is one more reason to check the Network tab first.

A load-more button is the same mechanism with an explicit trigger. Instead of scrolling, you click, wait for the new batch, and repeat until the button is gone. The pattern:

import random

def scrape_load_more(page, item_selector, button_selector="button.load-more"):
    results = {}
    while True:
        for el in page.query_selector_all(item_selector):
            key = el.get_attribute("data-id") or el.inner_text()[:120]
            results[key] = el.inner_text()

        btn = page.query_selector(f"{button_selector}:not([disabled])")
        if not btn or not btn.is_visible():
            break

        btn.click()
        page.wait_for_timeout(random.randint(900, 1800))
    return list(results.values())

Check all three end states, because sites signal "done" differently. The button might be removed from the DOM, set to disabled, or hidden with CSS. Some sites swap the button for a spinner during the fetch and bring it back afterward, so wait for either new items or the button's return before deciding you're finished.

Whether you scroll a browser or paginate an API, the target still sees a burst of requests from one client. Pace it like a human:

• Randomize your waits. A jittered 0.8 to 2 second pause between scrolls or clicks looks natural. Fixed millisecond intervals are an obvious bot signature.
• Cap concurrency on the API. Three to five parallel requests is plenty for most feeds. More than that and you invite a 429.
• Handle 429 properly. Back off exponentially and honor the Retry-After header instead of retrying immediately.
• Rotate IPs. Hammering one endpoint from a single address is the fastest route to a block. Our guide on how to avoid getting your proxy blocked covers header hygiene, fingerprint consistency, and rotation cadence that keep long scroll sessions alive.

Running and hiding your own browser fleet is the expensive part of Approach 1. The SparkProxy Scraping API renders JavaScript, scrolls, waits for elements, and rotates residential IPs for you, so you send one request and get back the fully loaded page. The endpoint is https://scrape.sparkproxy.io/api/v1, authenticated with an X-API-Key header.

The simplest version turns on rendering plus the built-in auto-scroll and waits for your items to exist:

import requests

r = requests.get(
    "https://scrape.sparkproxy.io/api/v1",
    headers={"X-API-Key": "YOUR_API_KEY"},
    params={
        "url": "https://www.sparkproxy.io/feed",
        "render_js": "true",
        "scroll": "true",          # auto-scroll to trigger lazy-load
        "wait_for": ".feed-item",  # wait until items are present
        "country_code": "US",
    },
)
html = r.text

When one auto-scroll pass isn't enough, use js_scenario to script repeated scroll-and-wait steps. Post the payload as JSON so the instruction list stays clean:

import requests

payload = {
    "url": "https://www.sparkproxy.io/feed",
    "render_js": True,
    "wait_for": ".feed-item",
    "block_resources": True,     # skip images and fonts, faster and cheaper
    "premium_proxy": True,
    "country_code": "US",
    "js_scenario": {
        "instructions": [
            {"wait_for": ".feed-item"},
            {"scroll": 5000},  {"wait": 1200},
            {"scroll": 10000}, {"wait": 1200},
            {"scroll": 15000}, {"wait": 1200}
        ]
    },
}

r = requests.post(
    "https://scrape.sparkproxy.io/api/v1",
    headers={"X-API-Key": "YOUR_API_KEY"},
    json=payload,
)
html = r.text

For a load-more button, swap the scroll steps for clicks:

{
  "js_scenario": {
    "instructions": [
      {"wait_for": ".results"},
      {"click": "button.load-more"}, {"wait": 1500},
      {"click": "button.load-more"}, {"wait": 1500},
      {"click": "button.load-more"}, {"wait": 1500}
    ]
  }
}

The same call from the command line:

curl -G "https://scrape.sparkproxy.io/api/v1" \
  -H "X-API-Key: YOUR_API_KEY" \
  --data-urlencode "url=https://www.sparkproxy.io/feed" \
  --data-urlencode "render_js=true" \
  --data-urlencode "scroll=true" \
  --data-urlencode "wait_for=.feed-item"

One caveat sets the right expectation. A single API request can scroll a bounded number of times, which makes it ideal for feeds that are a few pages deep or when you want the rendered HTML. For unbounded feeds with thousands of items, find the JSON endpoint and paginate it directly. Once you have the API, the browser render is wasted work. You can also pass extract_rules to get structured JSON back instead of raw HTML, which saves a parsing step. If you're deciding between a managed API and running your own browsers and proxy pool, our breakdown of a web scraping API vs self-managed proxies lays out the real cost math.