Scrapy vs Selenium: When to Use Each for Scraping

The Scrapy vs Selenium question gets framed as a head-to-head between two scraping tools. It isn't one. Scrapy describes itself as "an application framework for crawling web sites and extracting structured data." Selenium describes itself as a project for automating browsers, built first and foremost for testing web applications. One manages a crawl. The other drives a browser. They overlap only because both end up with HTML in a variable.

The pattern in that table is the whole article. Nearly every row where Selenium says "you write it" is a row where Scrapy already shipped a tested implementation. And the one row where Scrapy says "No" is the row that sends people to Selenium in the first place.

A Scrapy project is a set of components wired around an engine: a scheduler that holds pending requests, a downloader that fetches them concurrently, spider callbacks that parse and yield more requests, and item pipelines that validate, dedupe, and store results. The architecture overview in the official docs is worth ten minutes of your time, because it shows how much of a scraper you are not writing.

A Selenium script is a client for a protocol. Your code sends commands defined by the W3C WebDriver specification to a driver binary, which controls a browser. Selenium 4 dropped the legacy JSON Wire Protocol for full W3C compliance, and 4.6 added Selenium Manager so you stop hand-managing chromedriver versions. What Selenium gives you is fidelity: the page renders exactly as it does for a user, cookies and storage behave normally, and anything a human can click, you can click.

So the real question is not "which tool scrapes better." It's "is my bottleneck the crawl, or the page?" Those have different answers, and answering the wrong one is how teams end up running 40 Chrome instances to scrape static HTML.

If the line between fetching a list of pages and walking a link graph is fuzzy, our breakdown of web scraping vs web crawling sets up the vocabulary this comparison leans on.

Most of the time, it doesn't. Test before you commit to a browser, because this single check decides the architecture.

# Does the data exist in the raw HTML the server sends?
curl -s -A "Mozilla/5.0" "https://www.sparkproxy.io/pricing" | grep -c "Residential"

If that returns a non-zero count, Scrapy can have the page in milliseconds with no browser involved. If it returns 0, don't reach for Selenium yet. Open DevTools, go to the Network tab, filter to Fetch/XHR, and reload. Sites that render client-side almost always fetch their data from a JSON endpoint, and that endpoint is usually cleaner than the HTML:

# The page renders via JS, but the data comes from a JSON API you can call directly.
import scrapy

class ProductSpider(scrapy.Spider):
    name = "products"

    def start_requests(self):
        for page in range(1, 51):
            yield scrapy.Request(
                f"https://www.sparkproxy.io/api/products?page={page}",
                headers={"Accept": "application/json"},
                callback=self.parse_api,
            )

    def parse_api(self, response):
        for item in response.json()["results"]:
            yield {"sku": item["sku"], "price": item["price"]}

Three outcomes, three answers. Data in raw HTML means Scrapy. Data in an XHR endpoint means Scrapy pointed at that endpoint. Data that appears only after scripts mutate the DOM, or that sits behind a flow you must click through, means a browser. In our experience that third bucket is far smaller than teams assume, often under 10% of the URLs in a crawl.

This is where the two tools stop being comparable at all.

A Scrapy request in flight is a Request object and a socket. The framework keeps CONCURRENT_REQUESTS of them moving at once on a single process and a single core, because Twisted's event loop spends its time waiting on network I/O, not burning CPU. Scrapy also supports asyncio integration through the asyncio reactor, so async libraries drop into callbacks.

A Selenium session is a Chrome process tree: a browser process, a renderer per tab, GPU and network service processes, plus the driver binary. It parses HTML, builds a DOM, runs a JavaScript engine, computes layout, and paints. All of that work is thrown away the moment you read driver.page_source.

Don't take a number from a blog post, including this one. Measure your own targets:

# Total resident memory of one Selenium Chrome session (Linux)
ps -o rss= -C chrome | awk '{s+=$1} END {print s/1024, "MB"}'

# Scrapy reports peak memory and request counts in its final stats block
scrapy crawl products -s MEMUSAGE_ENABLED=True 2>&1 | grep -E "memusage/max|response_received_count|elapsed_time_seconds"

The ratio you get back is orders of magnitude, not percentages. That is the practical ceiling. On a box that runs one Scrapy process handling dozens of concurrent requests, you will fit a small number of Chrome sessions before you're swapping. Scaling Selenium horizontally is a real answer, and Selenium Grid exists for exactly that, but you're now paying for a fleet to do work a single Scrapy process could do if the page didn't need rendering.

CPU tells the same story from the other side. Rendering is compute-bound, so Selenium throughput scales with cores. Crawling is I/O-bound, so Scrapy throughput scales with how politely you're willing to hit the target and how many clean exit IPs you have.

A crawl is a graph traversal with failure. Scrapy treats it that way.

The scheduler holds pending requests with priorities. RFPDupeFilter fingerprints each request so you don't fetch the same URL twice, which matters the moment a site links to the same product from four categories. RetryMiddleware retries transient failures twice by default and gives up cleanly. AutoThrottle watches response latency and adjusts delay, so you back off when the server slows down instead of hammering it into a 429.

Then there's the part people discover the hard way. A long crawl will die: an OOM kill, a deploy, a laptop lid. Scrapy's jobs support persists the scheduler queue and the dupefilter to disk:

# Ctrl-C once for a graceful shutdown, then run the exact same command to resume
scrapy crawl products -s JOBDIR=crawls/products-2026-08

# settings.py for a large, polite, resumable crawl
CONCURRENT_REQUESTS = 32
CONCURRENT_REQUESTS_PER_DOMAIN = 8
AUTOTHROTTLE_ENABLED = True
AUTOTHROTTLE_TARGET_CONCURRENCY = 4.0
RETRY_TIMES = 3
RETRY_HTTP_CODES = [429, 500, 502, 503, 504, 522, 524, 408]
HTTPCACHE_ENABLED = True          # replay locally while you iterate on parsing
FEEDS = {"out/products-%(time)s.jsonl": {"format": "jsonlines"}}

None of that exists in Selenium, because none of it is a browser's job. Building the equivalent means a queue, a seen-URL set, a retry wrapper, a delay policy, and a checkpoint file. That's a weekend of work and a year of edge cases, and you'll do it worse than the framework that has shipped it since 2008.

Scrapy loses outright the moment state lives in the browser rather than in the response.

# A flow no HTTP client reproduces cheaply: render, wait, click, read the mutated DOM
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

driver = webdriver.Chrome()
driver.get("https://www.sparkproxy.io/dashboard")

WebDriverWait(driver, 15).until(
    EC.element_to_be_clickable((By.CSS_SELECTOR, "button#load-more"))
).click()

rows = WebDriverWait(driver, 15).until(
    EC.presence_of_all_elements_located((By.CSS_SELECTOR, "tr.usage-row"))
)
data = [r.text for r in rows]
driver.quit()

Cases that genuinely need this: multi-step logins with device checks, infinite scroll where the API signature is generated in JavaScript, canvas or WebGL output, drag-and-drop filters, and anything gated behind a challenge that runs browser APIs. Selenium is also the right call when you need to verify that a page works, which is what it was designed for and where its explicit-wait and Grid machinery earns its keep.

Worth knowing before you commit: for pure scraping, Playwright generally beats Selenium on speed, network interception, and per-context proxies. We compare them directly in Playwright vs Selenium for web scraping. If you're picking a browser tool today with no existing Selenium investment, read that first.

This difference is bigger than most comparisons admit, and it's the one that bites in production.

In Scrapy, the proxy is a per-request property. HttpProxyMiddleware reads request.meta["proxy"], so a downloader middleware can assign a different exit IP to every single request in flight:

# middlewares.py
import random

class RotateProxyMiddleware:
    ENDPOINTS = [
        "http://user-session1:pass@gateway.sparkproxy.io:11000",
        "http://user-session2:pass@gateway.sparkproxy.io:11000",
        "http://user-session3:pass@gateway.sparkproxy.io:11000",
    ]

    def process_request(self, request, spider):
        request.meta["proxy"] = random.choice(self.ENDPOINTS)

    def process_exception(self, request, exception, spider):
        retry = request.copy()
        retry.meta["proxy"] = random.choice(self.ENDPOINTS)   # different IP on retry
        retry.dont_filter = True
        return retry

In Selenium, the proxy is a property of the browser process, set in the options before launch:

from selenium import webdriver

opts = webdriver.ChromeOptions()
opts.add_argument("--proxy-server=http://gateway.sparkproxy.io:11000")
driver = webdriver.Chrome(options=opts)

Three consequences follow, and they shape your whole design:

• Rotating means restarting. Changing the exit IP means quitting the driver and launching a new one, which costs a cold browser start. The common workaround is a rotating gateway endpoint that hands you a fresh IP per connection, so the browser config never changes.
• Username and password auth is awkward. Chrome ignores credentials embedded in --proxy-server, so you either build a tiny extension, run an authenticating local forwarder, or use IP allowlisting on the proxy side.
• Sessions and IPs must stay pinned together. A logged-in browser that changes exit IP mid-session looks exactly like a hijacked account. Scrapy has no such constraint for stateless requests, which is precisely why per-request rotation is safe there.

The full setups are already documented, so this post won't repeat the mechanics. See how to use proxies with Scrapy for the middleware and authentication details, and how to integrate proxies with Selenium for the extension and forwarder patterns.

Neither tool is stealthy by default, and they fail differently.

Scrapy sends a plain HTTP client's TLS handshake, so its JA3/JA4 fingerprint doesn't match any browser. Anti-bot vendors flag that instantly regardless of how good your headers are. No amount of user-agent rotation fixes a fingerprint mismatch at the transport layer.

Selenium presents a genuine browser fingerprint, then leaks that it's automated. navigator.webdriver is true under the WebDriver spec, driver-launched Chrome carries default flags, and the automation banner and profile differences are detectable. Stealth patches exist and decay quickly. Our writeup on headless browser detection covers what the checks actually look for.

The honest summary: a browser gets you past challenges that require executing JavaScript, and gets you caught by automation-specific checks. An HTTP framework gets caught earlier but costs almost nothing per attempt. Your IP quality usually matters more than either.

This is the architecture most mature scrapers converge on, and it follows directly from the earlier finding that only a minority of URLs need rendering.

Scrapy owns the crawl: discovery, scheduling, dedupe, retries, throttling, pipelines, resume. A browser gets invoked only for the specific requests that fail the raw-HTML test. You flag those per request rather than routing everything through a browser.

The maintained way to do this is scrapy-playwright, which plugs in as a download handler. The community scrapy-selenium middleware has been dormant for years, so if you want an in-framework browser, Playwright is the practical choice even in a Selenium shop:

# settings.py
DOWNLOAD_HANDLERS = {
    "https": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
}
TWISTED_REACTOR = "twisted.internet.asyncioreactor.AsyncioSelectorReactor"

def parse_listing(self, response):
    for href in response.css("a.product::attr(href)").getall():
        yield response.follow(href, self.parse_product)     # plain HTTP, fast, cheap

    yield response.follow(
        "/reviews",
        self.parse_reviews,
        meta={"playwright": True},                          # only this one boots a browser
    )

If you're keeping Selenium, run it out-of-band instead. Scrapy writes the small set of render-needed URLs to a queue, and a separate worker pool of Selenium sessions drains it and posts results back into a pipeline. Two processes, two scaling knobs, one crawl state. That decoupling also means a browser crash never kills the crawl.

Run it in order and stop at the first match.

One rule covers the rest: never put a browser behind a request that a plain HTTP fetch would satisfy. That is the most expensive mistake in scraping architecture, and it's usually made once at the start and never revisited.

The hybrid pattern is correct, and it's also infrastructure you now own: a browser pool to keep alive, proxy rotation to maintain, and stealth patches to chase every time a vendor ships an update.

The SparkProxy Scraping API collapses the browser half into one HTTP call. Base URL https://scrape.sparkproxy.io/api/v1, authenticated with an X-API-Key header. Scrapy keeps the crawl graph, scheduling, dedupe, and pipelines; the API handles rendering, proxies, and retries for the pages that fight back:

import scrapy
from urllib.parse import urlencode

API = "https://scrape.sparkproxy.io/api/v1"

class HybridSpider(scrapy.Spider):
    name = "hybrid"

    def start_requests(self):
        params = {
            "url": "https://www.sparkproxy.io/pricing",
            "render_js": "true",        # headless Chromium runs on their side
            "premium_proxy": "true",    # residential exit IP
            "country_code": "US",
            "stealth": "true",
        }
        yield scrapy.Request(
            f"{API}?{urlencode(params)}",
            headers={"X-API-Key": "YOUR_API_KEY"},
            callback=self.parse,
        )

    def parse(self, response):
        yield {"plan": response.css(".plan-name::text").get()}

Static pages stay on render_js=false, which costs 1 credit and returns about three times faster than a rendered fetch. Reserve render_js=true and stealth=true for the minority that need them, and you get the hybrid split without running a browser fleet. Failed scrapes refund credits, and the API retries up to three times before giving up.

Whether that trade is worth it depends on your volume and how much engineering time your proxy layer eats. We work through that math in web scraping API vs self-managed proxies.