Scrapy Proxy Setup: Rotating, Middleware, and Auth

Scrapy is fast. That is the problem. A default spider with CONCURRENT_REQUESTS = 16 will hammer a target site from one IP, and most anti-bot systems flag a single address doing dozens of requests per second within the first minute. You get 403 Forbidden, 429 Too Many Requests, or a CAPTCHA wall, and the crawl dies.

Proxies spread those requests across many IPs so no single address crosses a site's detection threshold. Scrapy handles this through its downloader middleware layer. The built-in HttpProxyMiddleware reads one key, request.meta["proxy"], and routes the request through whatever proxy URL it finds there. Everything else in this guide is about setting that key correctly and at the right moment.

If you are new to rotation strategy in general, the broader patterns (round-robin, weighted, health checks) are covered in how to rotate proxies in Python. This guide focuses on the Scrapy-specific wiring.

Scrapy enables HttpProxyMiddleware by default at priority 750, so you do not need to register anything to use a proxy. You only need to populate request.meta["proxy"].

The simplest way is per request, in start_requests:

import scrapy

class QuotesSpider(scrapy.Spider):
    name = "quotes"
    start_urls = ["https://www.sparkproxy.io"]

    def start_requests(self):
        for url in self.start_urls:
            yield scrapy.Request(
                url,
                meta={"proxy": "http://your-proxy.sparkproxy.io:10000"},
                callback=self.parse,
            )

    def parse(self, response):
        yield {"url": response.url, "status": response.status}

The proxy scheme is http:// even when the target URL is HTTPS. The proxy uses HTTP CONNECT tunneling for HTTPS traffic, so http://your-proxy.sparkproxy.io:10000 is correct for both. This trips up almost everyone the first time.

If you want the same proxy on every request without repeating yourself, set it from the shell using the environment variables HttpProxyMiddleware respects:

export http_proxy="http://your-proxy.sparkproxy.io:10000"
export https_proxy="http://your-proxy.sparkproxy.io:10000"
scrapy crawl quotes

That is fine for a quick test, but it applies one static proxy to the whole process. For real crawls you want a middleware so you can rotate, authenticate, and drop bad proxies programmatically. That starts in the next section.

Most providers, SparkProxy included, protect proxies with a username and password. Scrapy proxy authentication has one sharp edge that costs people hours, so it is worth doing deliberately.

You can embed credentials directly in the proxy URL:

meta = {"proxy": "http://your-username:your-password@your-proxy.sparkproxy.io:10000"}

When the proxy URL contains credentials, HttpProxyMiddleware strips them out and sets a Proxy-Authorization header for you. That works for a single static proxy. The trap appears the moment you rotate: HttpProxyMiddleware will not overwrite an existing Proxy-Authorization header. If a retried or redirected request already carries the header from a previous proxy, your new proxy receives the old proxy's credentials and returns 407 Proxy Authentication Required.

The reliable fix is to set the header yourself and clear it whenever the proxy changes:

import base64

def proxy_auth_header(username: str, password: str) -> str:
    token = base64.b64encode(f"{username}:{password}".encode()).decode()
    return f"Basic {token}"

# In a middleware or start_requests:
request.meta["proxy"] = "http://your-proxy.sparkproxy.io:10000"
request.headers["Proxy-Authorization"] = proxy_auth_header(
    "your-username", "your-password"
)

Rule of thumb: if you change request.meta["proxy"] anywhere in your code, either set a fresh Proxy-Authorization in the same place or delete the stale one with request.headers.pop("Proxy-Authorization", None). The rotating middleware below does exactly that.

A scrapy rotating proxy setup assigns a different proxy per request. You write a custom downloader middleware, keep the proxy list in settings.py, and let Scrapy call process_request on every outgoing request.

settings.py:

ROTATING_PROXY_LIST = [
    "http://your-proxy-1.sparkproxy.io:10000",
    "http://your-proxy-2.sparkproxy.io:10001",
    "http://your-proxy-3.sparkproxy.io:10002",
]

PROXY_USER = "your-username"
PROXY_PASS = "your-password"

DOWNLOADER_MIDDLEWARES = {
    "myproject.middlewares.RotatingProxyMiddleware": 350,
    "scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware": 750,
}

Priority order is the whole game here. Scrapy runs process_request in ascending priority, so your middleware at 350 runs before the built-in HttpProxyMiddleware at 750. That means your middleware sets request.meta["proxy"] first, and HttpProxyMiddleware then applies it to the actual connection. Put your middleware at a higher number than 750 and it runs too late: the proxy is never applied and requests go out on your real IP. This is the number-one reason a "working" scrapy proxy middleware silently does nothing.

middlewares.py:

import base64
import random

class RotatingProxyMiddleware:
    def __init__(self, proxies, username, password):
        if not proxies:
            raise ValueError("ROTATING_PROXY_LIST is empty")
        self.proxies = proxies
        self.auth = "Basic " + base64.b64encode(
            f"{username}:{password}".encode()
        ).decode()

    @classmethod
    def from_crawler(cls, crawler):
        settings = crawler.settings
        return cls(
            proxies=settings.getlist("ROTATING_PROXY_LIST"),
            username=settings.get("PROXY_USER", ""),
            password=settings.get("PROXY_PASS", ""),
        )

    def process_request(self, request, spider):
        # Respect an explicit per-request proxy if one is already set.
        if request.meta.get("proxy"):
            return
        request.meta["proxy"] = random.choice(self.proxies)
        # Clear any leftover auth from a prior proxy, then set the current one.
        request.headers.pop(b"Proxy-Authorization", None)
        request.headers["Proxy-Authorization"] = self.auth

from_crawler is what lets the middleware read settings.py. Reading the list there instead of hardcoding it means you can swap proxy pools per environment without touching code. The request.meta.get("proxy") guard keeps a proxy you set manually in a spider (for a specific request) from being overwritten by the random pick.

For a session that must keep the same IP across a multi-step flow (login, then navigate), set the proxy once in the spider and let the guard above preserve it for that request chain.

Rotation alone is not enough. Proxies go down, get rate-limited, or get banned mid-crawl. Scrapy's built-in RetryMiddleware retries on connection failures and on the status codes in RETRY_HTTP_CODES, but by default it retries the same request, which can mean the same proxy. You want a retry to pick a fresh proxy and to treat soft bans (403, 429, CAPTCHA pages that return 200) as failures.

Add a middleware that detects a ban, drops the current proxy from the request, and reschedules it. Because the proxy is removed from meta, your RotatingProxyMiddleware assigns a new one on the retry.

middlewares.py:

from scrapy.downloadermiddlewares.retry import get_retry_request

BAN_STATUSES = {403, 429, 503}

class ProxyBanRetryMiddleware:
    def process_response(self, request, response, spider):
        banned = (
            response.status in BAN_STATUSES
            or b"captcha" in response.body[:20000].lower()
        )
        if not banned:
            return response

        # Drop the bad proxy so a fresh one is chosen on retry.
        request.meta.pop("proxy", None)
        request.headers.pop(b"Proxy-Authorization", None)

        new_request = get_retry_request(
            request, spider=spider, reason=f"proxy-ban-{response.status}"
        )
        return new_request or response  # None means retries exhausted

Register it, and make sure RetryMiddleware and the proxy middleware are all enabled:

DOWNLOADER_MIDDLEWARES = {
    "myproject.middlewares.RotatingProxyMiddleware": 350,
    "scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware": 750,
    "myproject.middlewares.ProxyBanRetryMiddleware": 560,
    "scrapy.downloadermiddlewares.retry.RetryMiddleware": 550,
}

RETRY_TIMES = 4
RETRY_HTTP_CODES = [403, 429, 500, 502, 503, 504]
DOWNLOAD_TIMEOUT = 20

get_retry_request (available since Scrapy 2.5) handles the retry counter and honors RETRY_TIMES, so you do not reimplement backoff yourself. When retries are exhausted it returns None, and you fall back to the original response so the failure is logged instead of vanishing.

There is a limit to what rotation and retries can fix. If a target consistently blocks datacenter IPs, no amount of retrying datacenter proxies will help. That is a signal to switch that job to residential exit IPs or the Scraping API. The fingerprinting and header signals that get proxies flagged are covered in how to avoid getting your proxy blocked.

Managing a proxy list, health-checking it, and evicting dead IPs is real work. A rotating gateway does it server-side: you point every request at a single endpoint, and the provider assigns a fresh exit IP per request from its pool. Your Scrapy config collapses to one proxy URL and no rotation middleware at all.

# settings.py
ROTATING_PROXY_LIST = ["http://your-proxy.sparkproxy.io:10000"]
PROXY_USER = "your-username"
PROXY_PASS = "your-password"

Keep the RotatingProxyMiddleware from earlier if you like (a one-item list just sets the same endpoint every time), or drop it and set the proxy once per request. Either way the gateway rotates the actual exit IP for you.

When you need the same IP for a sequence of requests (a login flow, a paginated session), most gateways support sticky sessions through a session token in the username, for example your-username-session-a1b2c3. Requests using that username keep the same exit IP until the session expires. Use rotating exits for stateless page fetches and a sticky session for anything that carries a cookie or login state.

A gateway is the simplest scrapy proxy setup that still scales, and it pairs well with Scrapy's concurrency. For high-throughput jobs, tune the settings in the next section and see how to scrape high volume data without rate limiting.

Proxies solve the IP problem. They do not render JavaScript, solve CAPTCHAs, or manage browser fingerprints. When a target needs a real browser or aggressive anti-bot handling, route the request through the SparkProxy Scraping API instead of a raw proxy. The API fetches the page (optionally rendering JS and using residential exits), and returns the HTML to Scrapy as a normal response.

The cleanest integration builds the API request in start_requests so your spider logic stays in parse:

import urllib.parse
import scrapy

API_ENDPOINT = "https://scrape.sparkproxy.io/api/v1"
API_KEY = "YOUR_API_KEY"  # from your SparkProxy dashboard

class ProtectedSpider(scrapy.Spider):
    name = "protected"
    targets = ["https://www.sparkproxy.io/some-protected-page"]

    def start_requests(self):
        for target in self.targets:
            params = urllib.parse.urlencode({
                "url": target,
                "render_js": "true",     # headless browser render
                "country_code": "us",    # geo-route the exit IP
                "premium_proxy": "true", # residential exit pool
            })
            yield scrapy.Request(
                f"{API_ENDPOINT}?{params}",
                headers={"X-API-Key": API_KEY},
                callback=self.parse,
                meta={"real_url": target},
            )

    def parse(self, response):
        # response.body is the rendered HTML of the target page.
        for quote in response.css("div.quote span.text::text").getall():
            yield {"source": response.meta["real_url"], "quote": quote}

Authentication is the X-API-Key header, not a proxy username. Because Scrapy sees the API URL as the request URL, response.url is the API endpoint, not your target, which is why the example stashes the real URL in meta. If you follow links from the page, resolve them against meta["real_url"] (or response.urljoin with that base), not against response.url, otherwise relative links point back at the API host.

For a decision framework on when a raw rotating proxy is cheaper than an API call and when the API earns its cost, see web scraping API vs self-managed proxies. The short version: use plain proxies for static, high-volume pages, and the API for the smaller set of pages that need JS rendering or heavy anti-bot bypass. Set render_js=false on the API for static pages to keep those requests cheaper.

A proxy layer changes what the "right" Scrapy settings are. Proxied requests are slower and less reliable than direct ones, so the defaults tuned for a fast local connection will hurt you.

# settings.py: a sane proxy-aware baseline
CONCURRENT_REQUESTS = 24
CONCURRENT_REQUESTS_PER_DOMAIN = 6
DOWNLOAD_TIMEOUT = 25
DOWNLOAD_DELAY = 0.5
AUTOTHROTTLE_ENABLED = True
AUTOTHROTTLE_TARGET_CONCURRENCY = 4.0
RETRY_TIMES = 4
RETRY_HTTP_CODES = [403, 429, 500, 502, 503, 504]
COOKIES_ENABLED = False

The COOKIES_ENABLED = False line is the subtle one. With rotation on, Scrapy's cookie jar is shared across requests by default, so a cookie set through proxy A can be sent through proxy B. On sites that bind sessions to an IP, that mismatch is itself a ban signal. Turn cookies off for stateless scraping, and when you do need a session, pin it to one IP with a sticky gateway session as shown earlier. If you also scrape with plain requests or aiohttp outside Scrapy, the same session-versus-rotation tradeoff is covered in using proxies with Python requests and aiohttp.