How to Build a Distributed Web Scraping System

A single Python process hits three ceilings fast. CPU is the first one: parsing HTML with lxml, or rendering JavaScript in a headless browser, burns cores. A machine running headless Chromium usually sustains 1 to 3 pages per second per instance. Network is the second: even with async I/O, one host's bandwidth and file-descriptor limits cap concurrency. The third ceiling is IP reputation, since every request from one box shares one exit path, and the target starts rate-limiting or blocking you.

Going distributed fixes all three at once. You spread parsing across many cores and hosts, you multiply available concurrency, and you spread requests across many exit IPs so no single address looks abusive. Twenty workers behind a shared fetch layer can hold a steady 40 to 60 pages per second on a cooperative site, and far more on simple static targets.

Resilience matters just as much as speed. When work lives in a central queue instead of a worker's memory, a crashed worker loses nothing. The queue holds the pending URLs, another worker picks them up, and the crawl keeps moving. That property, durable work plus stateless workers, is the whole point of the design below.

Every scraping-at-scale architecture, no matter which framework you pick, is built from the same six parts. Keep them separate and each one scales on its own.

The flow is a loop. A seeding step pushes start URLs into the frontier. Workers pop a URL, check the dedup store, call the fetch layer, parse the response, write records to the sink, and push any newly discovered links back into the frontier. Because the frontier and the dedup store are shared, you can run one worker or three hundred without changing a line of parsing code.

The single most common mistake is fusing the fetch layer into each worker. When every worker manages its own proxy list, rotation and rate limits become uncoordinated, and two workers routinely hit the same domain through the same IP at the same second. Keep the fetch layer as its own shared service. That decision is what the second half of this guide is about.

The broker is your frontier. Pick it based on crawl size and delivery guarantees, not hype.

For most teams Redis is the right default. It doubles as your dedup store and your rate-limiter backend, so you run one dependency instead of three. Reach for RabbitMQ when you need real routing, for example a separate queue per target domain with independent priorities. Reach for Kafka only when scraping is a permanent streaming pipeline that other systems consume, and you need to replay history.

If you already write Scrapy spiders, scrapy-redis is the fastest route to a distributed scraper. It swaps Scrapy's in-memory scheduler and dupefilter for Redis-backed versions. Every spider process on every machine pulls from the same Redis queue and shares the same seen-URL set, so you scale by starting more copies of the same spider.

Install it, then point Scrapy at Redis in settings.py:

# settings.py
SCHEDULER = "scrapy_redis.scheduler.Scheduler"
DUPEFILTER_CLASS = "scrapy_redis.dupefilter.RFPDupeFilter"
SCHEDULER_PERSIST = True                       # keep the queue on restart
SCHEDULER_QUEUE_CLASS = "scrapy_redis.queue.PriorityQueue"

REDIS_URL = "redis://10.0.0.5:6379/0"          # one Redis all workers share
CONCURRENT_REQUESTS = 32
DOWNLOAD_DELAY = 0.0                            # rate is enforced centrally, see below

The spider inherits from RedisSpider and reads its start URLs from a Redis key instead of a hardcoded list:

# spiders/products.py
from scrapy_redis.spiders import RedisSpider

class ProductSpider(RedisSpider):
    name = "products"
    redis_key = "products:start_urls"          # workers pop seeds from here

    def parse(self, response):
        yield {
            "url": response.url,
            "title": response.css("h1::text").get(),
            "price": response.css(".price::text").get(),
        }
        for href in response.css("a.next::attr(href)").getall():
            yield response.follow(href, callback=self.parse)

Now start the same spider on as many hosts as you want, then seed the crawl by pushing URLs into the shared key:

# on each worker machine
scrapy crawl products

# seed once from anywhere
redis-cli -h 10.0.0.5 lpush products:start_urls "https://www.sparkproxy.io/catalog?page=1"

Every worker competes for the same queue, the shared dupefilter stops two workers from crawling the same URL, and SCHEDULER_PERSIST means a restart resumes instead of starting over. To route Scrapy through a shared fetch layer instead of raw proxies, see the downloader middleware below.

When your crawl doesn't fit Scrapy's request-response shape, for example you need to fan out from an API index, call other services, or run heavy post-processing, Celery gives you more control. Each URL becomes a task, and Celery distributes tasks to workers across machines.

Define the app and a fetch-and-parse task:

# tasks.py
import os, requests
from celery import Celery
from selectolax.parser import HTMLParser

app = Celery("scraper", broker="redis://10.0.0.5:6379/0",
             backend="redis://10.0.0.5:6379/1")

API = "https://scrape.sparkproxy.io/api/v1"
KEY = os.environ["SPARKPROXY_API_KEY"]

@app.task(bind=True, max_retries=3, default_retry_delay=10, acks_late=True)
def scrape(self, url):
    r = requests.get(API, headers={"X-API-Key": KEY},
                     params={"url": url, "render_js": "false"}, timeout=60)
    if r.status_code in (429, 503):
        raise self.retry()                     # transient, put it back on the queue
    tree = HTMLParser(r.text)
    node = tree.css_first("h1")
    return {"url": url, "title": node.text() if node else None}

acks_late=True is the important flag. The task is only acknowledged after it finishes, so if a worker dies mid-fetch the broker redelivers the URL to another worker. Combine a discovery task with detail tasks to fan out:

@app.task
def discover(index_url):
    r = requests.get(API, headers={"X-API-Key": KEY},
                     params={"url": index_url}, timeout=60)
    tree = HTMLParser(r.text)
    for a in tree.css("a.product"):
        scrape.delay(a.attributes.get("href"))  # enqueue one task per product

Run workers with as many concurrent slots as the host can handle, then scale by adding hosts:

celery -A tasks worker --concurrency=16 --prefetch-multiplier=1 -Q celery

Keep --prefetch-multiplier=1 for scraping. Long, uneven task durations mean you don't want one worker hoarding a batch of URLs while others sit idle.

Dedup is the difference between a crawl that finishes and one that loops forever. The rule is simple: the seen-URL set must live in shared storage, never in a worker's memory. Scrapy-Redis handles this for you. With Celery you do it yourself, and Redis makes it a one-liner because SADD returns 0 when the member already exists:

import redis
r = redis.Redis(host="10.0.0.5")

def should_crawl(url):
    fp = canonical(url)
    return r.sadd("seen:products", fp) == 1     # 1 = new, 0 = duplicate

Canonicalize before you fingerprint, or you'll re-crawl the same page under a dozen URL variants. Strip tracking params, sort the query string, and drop the fragment:

from urllib.parse import urlsplit, urlunsplit, parse_qsl, urlencode

DROP = {"utm_source", "utm_medium", "utm_campaign", "gclid", "ref"}

def canonical(url):
    p = urlsplit(url.lower())
    q = [(k, v) for k, v in parse_qsl(p.query) if k not in DROP]
    return urlunsplit((p.scheme, p.netloc, p.path.rstrip("/"),
                       urlencode(sorted(q)), ""))

A plain Redis SET costs roughly 60 to 100 bytes per URL, so a set of 100 million URLs needs several gigabytes of RAM. Past that scale, switch to a Bloom filter with the RedisBloom module. It trades a tiny, tunable false-positive rate for a fraction of the memory:

# BF.RESERVE key error_rate capacity, then BF.ADD returns 1 if newly added
r.execute_command("BF.RESERVE", "seen:bloom", "0.001", "500000000")

def should_crawl_bloom(url):
    return r.execute_command("BF.ADD", "seen:bloom", canonical(url)) == 1

A Bloom filter can only say "definitely new" or "probably seen," so a 0.1% false-positive rate means you skip about 1 in 1000 real URLs. For a half-billion-URL crawl that tradeoff is almost always worth the memory saved.

Here is the anti-pattern that quietly wrecks distributed crawls. Each worker loads the same list of proxies and rotates through it independently. With ten workers and one hundred proxies, nothing stops two workers from picking the same IP for the same domain in the same second. Per-IP request rates spike in ways no single worker can see, and the target starts blocking addresses your fleet still thinks are healthy. Ban rates climb even though every worker is behaving "correctly" on its own.

The fix is architectural, not a bigger proxy list. Move IP selection out of the workers and into one shared fetch layer. Workers stop being proxy managers and become stateless fetch requesters. They send a URL to one endpoint, the fetch layer decides which IP to use, tracks which addresses are getting blocked, and enforces rotation globally. Now proxy scaling and worker scaling are independent: add workers for parsing throughput, and the fetch layer handles IP health for all of them at once.

You can build that shared layer yourself as a proxy gateway, or use a scraping API that already is one. If you're weighing the two, our comparison of a web scraping API vs self-managed proxies breaks down the cost and maintenance tradeoffs. Either way, the goal is the same: one place that owns proxy decisions, so a hundred workers never contradict each other. For the block-avoidance rules the fetch layer should enforce, see how to avoid getting your proxy blocked.

The SparkProxy Scraping API is a ready-made shared fetch layer. Every worker calls the same endpoint with the same API key, and the service handles proxy rotation, geo-targeting, and optional JavaScript rendering server-side. Your workers never touch an IP. That's exactly the stateless-worker property a distributed scraper wants.

The base endpoint is https://scrape.sparkproxy.io/api/v1, authenticated with an X-API-Key header. A single fetch helper, shared by your Scrapy middleware and your Celery tasks, keeps behavior identical everywhere:

import os, requests

API = "https://scrape.sparkproxy.io/api/v1"
KEY = os.environ["SPARKPROXY_API_KEY"]

def fetch(url, country=None, render=False, retries=3):
    params = {"url": url, "render_js": str(render).lower(),
              "json_response": "true"}          # envelope with status + credits
    if country:
        params["country_code"] = country        # ISO code, e.g. "US", "DE"
    for attempt in range(retries):
        r = requests.get(API, headers={"X-API-Key": KEY},
                         params=params, timeout=90)
        data = r.json()
        if data["status_code"] < 400:
            return data                          # body, status_code, credits_used
        # 4xx/5xx from the target: back off and let the API rotate the exit IP
    raise RuntimeError(f"failed after {retries} tries: {url}")

Because json_response=true returns an envelope with status_code, duration_ms, and credits_used, every worker reports the same fields, which makes fleet-wide monitoring trivial (more on that below). Turn on render_js only for pages that need it, since headless rendering costs 5 credits against 1 for a plain HTTP fetch. Add premium_proxy: "true" to route through the residential pool for the hardest targets, and country_code when a page is geo-gated.

Wiring this into Scrapy is a short downloader middleware so your Scrapy-Redis workers share the exact same fetch path:

# middlewares.py
from scrapy.http import HtmlResponse

class SparkProxyMiddleware:
    def process_request(self, request, spider):
        data = fetch(request.url, render=False)
        return HtmlResponse(url=request.url, body=data["body"],
                            encoding="utf-8", request=request)

Now both frameworks route through one endpoint, and IP decisions live in one place instead of scattered across every worker.

Even with a shared fetch layer, you often need to cap how hard the whole fleet hits a specific domain, to stay polite or to respect a target's limits. A per-worker delay can't do this, because ten workers each waiting one second still produce ten requests per second combined. The rate budget has to be global, which means it lives in Redis, not in worker code.

A token bucket keyed per domain is the clean solution. Every worker asks Redis for a token before fetching, and Redis refills the bucket at a fixed rate. Do the check-and-decrement in a Lua script so it's atomic across all workers:

-- rate.lua: KEYS[1]=bucket key, ARGV=rate, capacity, now, requested
local b = redis.call("HMGET", KEYS[1], "tokens", "ts")
local tokens = tonumber(b[1]) or tonumber(ARGV[2])
local ts = tonumber(b[2]) or tonumber(ARGV[3])
local delta = math.max(0, tonumber(ARGV[3]) - ts) * tonumber(ARGV[1])
tokens = math.min(tonumber(ARGV[2]), tokens + delta)
if tokens < tonumber(ARGV[4]) then return 0 end
tokens = tokens - tonumber(ARGV[4])
redis.call("HMSET", KEYS[1], "tokens", tokens, "ts", ARGV[3])
redis.call("EXPIRE", KEYS[1], 60)
return 1

import time
rate_script = r.register_script(open("rate.lua").read())

def take_token(domain, rate=5, capacity=10):
    ok = rate_script(keys=[f"rl:{domain}"],
                     args=[rate, capacity, time.time(), 1])
    return ok == 1                              # False means wait and retry

Set rate=5 and every worker together respects five requests per second to that domain, whether you run 3 workers or 300. This is the control single-machine tutorials never cover, because on one box a simple sleep is enough. For the broader playbook on staying under limits at volume, read how to scrape high-volume data without rate limiting.

A distributed scraper fails silently unless you watch it. Track five signals:

• Queue depth over time. If the frontier grows faster than workers drain it, you're under-provisioned or stuck in a discovery loop. Read it with redis-cli llen products:start_urls or Celery's inspect API.
• Pages per second and error rate by status. A spike in 403s or 429s means the target is fighting back and your fetch layer should rotate harder.
• Credits used. Since the fetch helper returns credits_used per call, sum it to catch a runaway crawl before the bill does.
• Dedup hit rate. A rising ratio of duplicates means your frontier is full of URLs already crawled, often a canonicalization bug.
• Worker liveness. Dead workers should not silently shrink your throughput.

For Celery, Flower gives you a live dashboard of tasks, workers, and failure rates out of the box. For anything larger, export counters to Prometheus and alert in Grafana. A minimal per-worker counter is enough to start:

from prometheus_client import Counter, start_http_server
PAGES = Counter("scraper_pages_total", "pages fetched", ["status"])
start_http_server(9100)                         # scrape this port with Prometheus

def record(data):
    PAGES.labels(status=str(data["status_code"])).inc()

Alert on two conditions above all: queue depth trending up for more than a few minutes, and error rate crossing a threshold. Those two catch most real failures, from a blocked domain to a crashed broker.

If your workers make async requests inside each process, pair this fleet design with connection-level concurrency. Our guide to async scraping with requests and aiohttp covers per-worker throughput that multiplies what each node in the fleet can do.