Retry and Backoff Strategies for Web Scraping

Retries exist because most scraping failures are temporary. A proxy drops a connection, a target returns 503 during a deploy, a request times out under load. Retry once and the second attempt usually works.

The trouble starts when you retry badly. Three failure modes show up again and again:

• Retry storms. Fixed delays synchronize your workers. If 200 tasks all fail at once and all wait exactly 2 seconds, they all fire again at exactly the same moment. You have not spread the load, you have rebuilt the spike.
• Retrying the unretryable. A 404 will still be a 404 on attempt five. A 401 from a bad API key will never fix itself. Retrying these wastes time and, on a metered API, wastes credits.
• Hammering a target that asked you to stop. A 429 Too Many Requests is an explicit instruction to back off. Retrying it immediately gets your IP range flagged harder.

A good retry policy answers four questions on every failure: is this worth retrying, how long should I wait, when do I give up, and is it safe to run the side effects again? The rest of this guide answers each one with code you can drop into a real scraper. For the wider picture on keeping request volume under a target's radar, see the companion guide on scraping high-volume data without rate limiting.

The first decision is classification. Retry transient failures, surface permanent ones immediately. Get this wrong and you either give up on recoverable errors or grind forever on ones that will never succeed.

The 403 row is the one people get wrong. A 403 from a missing credential is permanent. A 403 from an anti-bot layer is a soft block, and a single retry on a clean exit IP often clears it. Decide per target. With a rotating scraping API, a retried request already lands on a different IP, so a block-style 403 is cheap to test once. For the full list of what each proxy and HTTP status code means, the proxy error codes explainer is the reference to keep open.

Here is the classifier as code. Keep the retryable set in one place so the whole scraper agrees on the rules.

import requests

RETRYABLE_STATUS = {408, 425, 429, 500, 502, 503, 504, 530}
NON_RETRYABLE_STATUS = {400, 401, 402, 404, 422}

def is_retryable(exc: Exception | None, status: int | None) -> bool:
    # Network-level failures are almost always transient.
    if isinstance(exc, (requests.ConnectionError, requests.Timeout)):
        return True
    if status is None:
        return False
    if status in RETRYABLE_STATUS:
        return True
    # Everything else (auth, not found, bad params) is a bug, not bad luck.
    return False

Exponential backoff grows the wait after each failed attempt. Instead of a flat 2 seconds every time, you wait roughly base * 2 ** attempt: 1s, then 2s, then 4s, then 8s. The idea is simple. If the target is struggling, giving it geometrically more room each round is far kinder than a constant drumbeat, and it lets a brief outage clear before you give up.

BASE_DELAY = 1.0    # seconds
MAX_DELAY = 60.0    # cap so a high attempt count can't sleep for an hour

def backoff_delay(attempt: int) -> float:
    # attempt starts at 0 for the first retry
    return min(MAX_DELAY, BASE_DELAY * (2 ** attempt))

The cap matters more than it looks. Without min(MAX_DELAY, ...), attempt 12 would sleep for over an hour. Cap the per-attempt delay so a long tail of retries stays bounded, then bound the total run separately with a deadline (covered below).

Pure exponential backoff still has one fatal flaw for scrapers: it is deterministic. Two workers that fail at the same instant compute the exact same delay and retry in lockstep. That is the retry storm again, just on a growing schedule. Jitter fixes it.

Jitter means randomizing the wait so concurrent clients spread out instead of firing together. The canonical analysis is AWS's "Exponential Backoff And Jitter," which measured three variants and found that adding randomness sharply reduces contention and completes work faster under load (AWS Architecture Blog).

The one worth using by default is full jitter: pick a random wait anywhere between zero and the current exponential ceiling.

import random

def full_jitter_delay(attempt: int) -> float:
    ceiling = min(MAX_DELAY, BASE_DELAY * (2 ** attempt))
    return random.uniform(0, ceiling)

The three jitter styles trade off in ways worth knowing before you pick one:

Full jitter spreads retries across the whole window, so 200 workers no longer collide. There is a second option worth reaching for, decorrelated jitter, which grows the window off the previous delay rather than the attempt count. It tends to drain a backlog slightly faster:

def decorrelated_jitter(prev_delay: float) -> float:
    # Seed prev_delay with BASE_DELAY on the first retry.
    return min(MAX_DELAY, random.uniform(BASE_DELAY, prev_delay * 3))

The one thing you should not ship is exponential backoff with no jitter at all. That is the single most common mistake in scraping retry loops, and it is the reason a pool of otherwise clean IPs gets flagged in a synchronized wave. Randomize the wait, always.

When a server sends 429 or 503, it often includes a Retry-After header telling you exactly how long to wait. Per RFC 9110 Section 10.2.3, the value is either a number of seconds or an HTTP date. Ignoring it and using your own backoff is a good way to get blocked, because the server already told you the answer.

The SparkProxy Scraping API makes the hint explicit. On a rate-limit or concurrency 429, the JSON body carries a retry_after_seconds field, documented at /docs/scraping-api:

{ "error": "Rate limit exceeded", "retry_after_seconds": 60, "limit": 60 }

Parse both sources, preferring the explicit hint:

from email.utils import parsedate_to_datetime
from datetime import datetime, timezone

def retry_after_seconds(response) -> float | None:
    # 1) SparkProxy JSON hint (rate-limit or concurrency 429).
    try:
        body = response.json()
        if isinstance(body, dict) and "retry_after_seconds" in body:
            return float(body["retry_after_seconds"])
    except ValueError:
        pass
    # 2) Standard Retry-After header: delay-seconds or HTTP-date (RFC 9110).
    header = response.headers.get("Retry-After")
    if not header:
        return None
    if header.strip().isdigit():
        return float(header)
    try:
        when = parsedate_to_datetime(header)
        return max(0.0, (when - datetime.now(timezone.utc)).total_seconds())
    except (TypeError, ValueError):
        return None

Now the insight most tutorials miss. Obeying Retry-After exactly recreates the storm at a later time: if the server says "retry in 60 seconds," every client waits 60 seconds and then stampedes at the same instant. Treat the server hint as a floor and add a small jitter on top so the herd disperses:

def wait_for(attempt: int, response=None) -> float:
    computed = full_jitter_delay(attempt)
    hint = retry_after_seconds(response) if response is not None else None
    if hint is None:
        return computed
    # Obey the server, but add jitter so workers don't all wake on the
    # exact second the server named.
    return hint + random.uniform(0, 1.0)

Two limits keep a retry loop from running forever, and you want both.

A max attempt count bounds how many times you try a single request. Five or six is plenty for scraping. If six attempts across growing, jittered waits have not succeeded, the target is not having a bad second, it is having a bad hour.

A total deadline bounds wall-clock time across all attempts. This matters because backoff plus a large Retry-After can blow past any reasonable budget even under the attempt cap. A job that must finish in two minutes should not sleep for three because attempt four honored a 180-second hint.

import time

MAX_ATTEMPTS = 6
DEADLINE_SECONDS = 120

def within_budget(attempt: int, started_at: float) -> bool:
    if attempt >= MAX_ATTEMPTS:
        return False
    if time.monotonic() - started_at > DEADLINE_SECONDS:
        return False
    return True

Use time.monotonic(), not time.time(). The monotonic clock never jumps backward when the system clock is adjusted, so your deadline math stays correct.

You rarely need to hand-roll every loop. Two libraries cover most cases in Python. Reach for a library when you want the standard behavior with less code, and hand-roll (see the end-to-end example) when you need a circuit breaker and idempotency in the same loop.

Backoff buys time, but if the same exit IP keeps failing, waiting longer will not help. When a request is retryable and the failure looks IP-shaped (a block-style 403, repeated 429, or 530), change the exit IP on the next attempt.

With the SparkProxy Scraping API this is automatic. Each request selects a fresh exit IP server-side, so simply issuing the retry lands you on a different address. You do not maintain a pool or track which IP is burned.

If you run your own proxy list, rotate explicitly so a retry never reuses the address that just failed. itertools.cycle is not thread-safe on its own, so guard it with a lock:

import itertools, threading, time, requests

class ProxyPool:
    def __init__(self, proxies):
        self._lock = threading.Lock()
        self._cycle = itertools.cycle(list(proxies))

    def next(self) -> str:
        with self._lock:              # cycle isn't thread-safe without this
            return next(self._cycle)

pool = ProxyPool([
    "http://user:pass@dc1.sparkproxy.io:8000",
    "http://user:pass@dc2.sparkproxy.io:8000",
])

def fetch_with_rotation(url: str, max_attempts: int = 5):
    last = None
    for attempt in range(max_attempts):
        proxy = pool.next()           # a different IP each attempt
        try:
            r = requests.get(url, proxies={"http": proxy, "https": proxy}, timeout=30)
            if r.status_code in RETRYABLE_STATUS:
                last = r
                time.sleep(full_jitter_delay(attempt))
                continue
            return r
        except (requests.ConnectionError, requests.Timeout) as e:
            last = e
            time.sleep(full_jitter_delay(attempt))
    raise RuntimeError(f"exhausted {max_attempts} proxies for {url}: {last}")

For deeper pool patterns (round-robin, weighted, health checks, and async rotation), see how to rotate proxies in Python.

Retries and rotation assume the failure is local. Sometimes the target itself is hard down: a full outage, a WAF that has decided your traffic is hostile, an endpoint returning 503 for everyone. Retrying then is pure waste. You spend time, and on a metered API you spend credits, chasing a target that cannot answer.

A circuit breaker (the pattern Martin Fowler describes in CircuitBreaker) fixes this. It counts consecutive failures and, past a threshold, "opens" to fail fast without even trying, for a cooldown window. After the cooldown it goes "half-open" and lets one probe through: success closes it, failure re-opens it.

import time, threading

class CircuitBreaker:
    def __init__(self, fail_max: int = 5, reset_after: float = 30.0):
        self.fail_max = fail_max
        self.reset_after = reset_after
        self._fails = 0
        self._opened_at = None
        self._lock = threading.Lock()

    def allow(self) -> bool:
        with self._lock:
            if self._opened_at is None:
                return True                                  # closed
            if time.monotonic() - self._opened_at >= self.reset_after:
                self._opened_at = None                       # half-open probe
                self._fails = 0
                return True
            return False                                     # open: fail fast

    def record(self, ok: bool):
        with self._lock:
            if ok:
                self._fails = 0
                self._opened_at = None
            else:
                self._fails += 1
                if self._fails >= self.fail_max:
                    self._opened_at = time.monotonic()

Key the breaker on the target host, not the proxy. A breaker per exit IP defeats the point, because rotation would keep opening fresh breakers while the real problem is the destination. One breaker per target domain lets a healthy scraper skip a dead site in microseconds and keep working on everything else.

Here is the failure retry tutorials almost never mention, because they stop at the HTTP call. A request can succeed on the server and still look like a failure to your client. A read timeout fires after the target already sent the page. A connection resets after your database write committed but before the response reached you. Retry blindly and you fetch twice, write twice, and end up with duplicate rows.

The fix is idempotency: design the retryable unit so running it twice produces the same result as running it once. At the HTTP layer, GET is already idempotent, which is why read-heavy scraping is forgiving. Your pipeline's writes are not. Guard them.

Derive a stable key for the unit of work, then upsert on it so a retried write overwrites instead of inserting:

import hashlib

def idempotency_key(url: str, run_id: str) -> str:
    raw = f"{run_id}:{url}".encode()
    return hashlib.sha256(raw).hexdigest()

# When persisting, dedupe on the key so a retried write overwrites
# instead of inserting a second row (PostgreSQL):
#
#   INSERT INTO pages (idem_key, url, html, fetched_at)
#   VALUES (%s, %s, %s, now())
#   ON CONFLICT (idem_key) DO UPDATE
#     SET html = EXCLUDED.html, fetched_at = EXCLUDED.fetched_at;

If a retried fetch then hands off to a downstream API you do not own, send an Idempotency-Key header when the service supports one (Stripe and many payment and queue APIs do). The scheduling-side view of the same idea, watermarks plus upserts for incremental runs, is covered in how to schedule and automate web scrapers.

Now assemble the pieces: classification, backoff with jitter, the Retry-After floor, an attempt cap and deadline, a per-target circuit breaker, implicit IP rotation from the scraping API, and an idempotent write. This is the function a production scraper actually calls.

import time, requests

breaker = CircuitBreaker(fail_max=5, reset_after=30.0)

class NonRetryable(Exception): ...
class Retryable(Exception): ...

def resilient_fetch(url: str, run_id: str,
                    max_attempts: int = 6, deadline_s: float = 120) -> str:
    started = time.monotonic()
    for attempt in range(max_attempts):
        if not breaker.allow():
            raise NonRetryable(f"circuit open for target, skipping {url}")
        if time.monotonic() - started > deadline_s:
            raise Retryable(f"deadline exceeded after {attempt} attempts on {url}")

        try:
            resp = requests.get(
                "https://scrape.sparkproxy.io/api/v1",
                headers={"X-API-Key": "YOUR_API_KEY"},
                params={"url": url, "render_js": "true"},
                timeout=30,
            )
        except (requests.ConnectionError, requests.Timeout):
            breaker.record(ok=False)
            time.sleep(full_jitter_delay(attempt))
            continue

        if resp.status_code == 200:
            breaker.record(ok=True)
            # key + upsert make this safe even if the call already ran once
            persist(idempotency_key(url, run_id), url, resp.text)
            return resp.text

        if resp.status_code in {401, 402, 404, 422}:
            breaker.record(ok=True)          # target is fine, our request is wrong
            raise NonRetryable(f"{resp.status_code} for {url}")

        # 429 / 5xx / 530: transient. Back off, honor the server hint, retry.
        # Each new SparkProxy request lands on a fresh exit IP automatically.
        breaker.record(ok=False)
        time.sleep(wait_for(attempt, resp))

    raise Retryable(f"gave up on {url} after {max_attempts} attempts")

Read the control flow once and the design goals fall out. Network errors and transient status codes retry with jittered backoff. A 429 waits at least as long as the server asked. Non-retryable statuses raise immediately and do not touch the breaker's failure count, because the target is healthy and the fault is yours. Repeated real failures trip the breaker so the next call to a dead target returns in microseconds. And the successful write is keyed, so nothing this function does can double-insert. That is a retry layer you can leave running unattended.