Proxy Failover and Redundancy: Design for Failure

These get used interchangeably in provider marketing and they are not interchangeable in code.

Load balancing is a steady-state concern: distribution algorithms, health checks, ejection windows. That machinery is a separate subject. Failover is the exception path, and it starts where the balancer runs out of healthy candidates, or where the thing that failed is not an individual IP at all.

The practical test for whether you have real failover: unplug your primary path right now and time how long until data flows again. If nobody knows the answer, you have redundancy on paper.

Most proxy failure handling collapses everything into one branch: request failed, retry, then rotate. That branch is wrong for four of the five modes below, because the correct response depends on how much of your capacity the failure implicates.

Two of these deserve a closer look because they are the ones people misdiagnose.

A subnet block looks exactly like a single IP block on the first request. The distinguishing signal is correlation: if failures cluster by /24 rather than scattering across the pool, you are dealing with range-level classification, and every retry against a sibling IP is teaching the target that the whole range belongs to one actor. Group your failures by network prefix before you decide. Ranges get scored as units, which is the entire reason subnet proxies exist as a distinct product.

Then there is a sixth thing that impersonates all five: the target itself is down. A site returning 502 through every proxy you own is not a proxy failure, and failing over to a second provider just spends money to observe the same outage twice. Keep one control request per target per interval on a different path. Without a control you cannot separate "our path is broken" from "their site is broken", and that distinction decides whether failover helps at all.

def classify(exc, status, body, ip):
    if status in (407, 401, 402):
        return "auth_or_billing"          # stop, page a human
    if status in (403, 429) or looks_like_captcha(body):
        return "range_block" if prefix_fail_rate(ip) > 0.6 else "ip_block"
    if isinstance(exc, (ConnectionRefusedError, ssl.SSLError)):
        return "endpoint_down"
    if isinstance(exc, TimeoutError):
        return "route_or_target"          # needs the control request to split
    if 500 <= (status or 0) < 600:
        return "target_side"
    return "unknown"

prefix_fail_rate(ip) is the whole trick: a rolling failure rate keyed on the first three octets rather than the full address. Full status code semantics are covered in proxy error codes explained, including which ones come from the proxy and which from the origin.

A retry is a second attempt on the same resource. A failover is an attempt on a different one. Standard retry advice, written for backend services, assumes failures are transient and not attributable to the caller's identity. Proxy failures invert that assumption: most of them are attributable to the identity, and the identity is the IP.

So the question is never "should I retry", it is "is this failure about the request or about the IP that sent it".

The Retry-After header is defined in RFC 9110 section 10.2.3 and accepts either delay-seconds or an HTTP-date, so parse both. The 429 status itself comes from RFC 6585 section 4. Backoff shapes, jitter, and per-attempt caps are covered in depth in retry and backoff strategies for web scraping.

One control matters more than the backoff curve: a retry budget expressed as a share of total traffic rather than as attempts per request. The Google SRE book chapter on cascading failures uses a client-side budget of roughly 10 percent of requests, and it is the right shape here. Per-request limits let a broad outage multiply your outbound volume by three or four at exactly the moment the far side is struggling. A global budget cannot.

class RetryBudget:
    def __init__(self, ratio=0.10, window=60):
        self.ratio, self.window = ratio, window
        self.attempts, self.retries = deque(), deque()

    def allow_retry(self, now):
        self._trim(now)
        return len(self.retries) < self.ratio * max(len(self.attempts), 1)

When the budget is exhausted, requests fail immediately instead of queueing. That is not data loss, it is data deferred, and the next section explains why the difference is the entire point.

A circuit breaker wraps a dependency and stops calling it once failures cross a threshold, described by Martin Fowler in the pattern's canonical write-up. Three states: closed (traffic flows), open (calls fail instantly without touching the network), half-open (a single probe decides whether to close again).

The reason this matters for proxies is arithmetic, not elegance. Say you run 40 concurrent workers against a proxy endpoint with a 90 second timeout, and the endpoint goes dark.

Without a breaker, every worker blocks the full 90 seconds before failing. Effective throughput drops to 40 failures per 90 seconds, about 0.44 per second, while new work keeps arriving at the normal rate. A queue that was draining at 25 requests per second now grows at nearly 25 per second. Ten minutes of outage leaves roughly 15,000 queued items and 40 workers asleep on dead sockets. That is fail-slow, and the outage outlives the outage.

With a breaker set to open after 20 consecutive failures, the first 20 requests eat 90 seconds each, spread across workers so about 45 seconds of wall clock, and everything after that fails in under a millisecond. Workers stay free. The queue either routes to a standby or marks rows as deferred at full speed. When the endpoint returns, one half-open probe closes the circuit and throughput resumes.

class Breaker:
    def __init__(self, threshold=20, cooldown=30):
        self.threshold, self.cooldown = threshold, cooldown
        self.fails, self.opened_at = 0, None

    def before(self, now):
        if self.opened_at is None:
            return "closed"
        if now - self.opened_at >= self.cooldown:
            return "half_open"        # exactly one caller may pass
        raise BreakerOpen("skip the network entirely")

    def record(self, ok, now):
        if ok:
            self.fails, self.opened_at = 0, None
        else:
            self.fails += 1
            if self.fails >= self.threshold:
                self.opened_at = now

Three details decide whether it helps or hurts.

Scope the breaker to the right key. One breaker per provider endpoint, not per IP. Per-IP health belongs to your ejection policy instead. A breaker keyed on IP will never trip during a provider outage, because no single IP accumulates enough failures to reach the threshold.

Let exactly one request through when half-open. Envoy's outlier detection docs show the production shape of this. Releasing the full queue at a recovering endpoint knocks it back down and produces the flapping pattern where you alternate between full stop and full blast every 30 seconds.

Never break on target-side status codes. A target returning 403 is not a reason to stop calling your proxy provider. Feed only transport failures and gateway errors into the breaker. Mixing the two is the most common way a breaker ends up worse than no breaker at all.

Signing with a second provider feels like redundancy. Whether it is depends entirely on whether the two fail independently, and independence has to hold at four layers.

The first layer catches most of it. Plenty of sellers resell the same underlying ranges, so two invoices can resolve to one network. When a target blocks an ASN, and that is routine because it is one rule instead of thousands, both "providers" die in the same second. What an ASN is and why targets score at that level is covered in what is a datacenter ASN.

Verifying this takes about five minutes with the Team Cymru IP-to-ASN mapping service, which answers over DNS:

# reverse the octets, then query origin.asn.cymru.com
for ip in 203.0.113.7 198.51.100.42; do
  rev=$(echo "$ip" | awk -F. '{ print $4"."$3"."$2"."$1 }')
  printf '%s -> ' "$ip"
  dig +short TXT "${rev}.origin.asn.cymru.com" | tr -d '"' | cut -d'|' -f1,3
done

Pull 100 live exits from each provider, map them, and compute the overlap. If the intersection of ASN sets is not empty, you have correlated failure, and the size of that intersection is the fraction of your "redundancy" that is fictional.

Redundancy tiers, roughly in order of how much correlated failure they leave behind:

Tier 4 matters because block decisions often key on the type signal rather than the individual address. A datacenter range and a residential range fail for different reasons, so they rarely fail together.

One more thing, and almost nobody does it. A standby you never send traffic to is not a standby, it is an assumption. Credentials expire. Credit balances hit zero. IP allowlists drift after an office move. The code path building the standby request rots through six months of refactors. Keep 3 to 5 percent of production traffic permanently on the secondary. It costs a rounding error, keeps all of that exercised, and hands you a live block-rate comparison you would otherwise run as a special project.

When failover has nowhere left to go, you choose between serving stale data and stopping. That decision belongs per dataset, not per pipeline, and it is driven by one number: the freshness budget, meaning how old a record can be before it becomes actively misleading.

A degradation ladder makes the choice explicit instead of emergent:

LADDER = [
    ("full",   lambda u: fetch(u, render_js=True, premium_proxy=True)),
    ("cheap",  lambda u: fetch(u, render_js=False)),
    ("cached", lambda u: cache.get(u)),          # returns (data, as_of)
    ("skip",   lambda u: None),                  # emit an explicit gap
]

def collect(url, budget_seconds):
    for name, step in LADDER:
        try:
            data = step(url)
        except Exception:
            continue
        if data is None:
            continue
        if name == "cached" and staleness(data) > budget_seconds:
            continue                             # too old to be useful
        return {"data": data, "tier": name,
                "as_of": as_of(data), "degraded": name != "full"}
    return {"data": None, "tier": "skip", "degraded": True}

Two rules make this safe. Every record carries as_of and degraded so downstream consumers decide for themselves rather than inheriting your judgment. And a skipped record emits an explicit gap marker rather than nothing, because a missing row and a row that was never attempted look identical in a database and completely different in a postmortem.

Disaster recovery vocabulary transfers to scraping cleanly once you translate the two terms. AWS defines RTO and RPO as maximum acceptable downtime and maximum acceptable data loss. For a collection pipeline:

• RTO is the time from "collection stopped" to "collection restored", detection included.
• RPO is not lost transactions, because you are not the system of record. It is the maximum acceptable age of the newest record in a dataset. Freshness, not durability.

Write the numbers down per dataset, because they determine the architecture rather than the other way around:

The mapping from RTO to automation is the useful part. If RTO is under 30 minutes, no human can be in the loop, because paging, acknowledging, and getting to a keyboard consumes it. Under 30 minutes means the failover has to be code that already exists and already ran this week. Above a few hours, an alert into a channel someone reads is a legitimate design, and building automatic multi-provider switching for it is wasted effort.

Then budget the RTO across its parts:

RTO  >=  MTTD (detect)  +  time to decide  +  MTTR (switch and drain)

A 5 minute health probe interval already spends up to 5 of a 15 minute RTO before anyone knows anything. Passive detection off live traffic, meaning a rolling failure rate that trips within 30 seconds, buys back most of that. Add the queue drain: if 15,000 items backed up during the outage and you clear 25 per second, that is another 10 minutes of catch-up before freshness is genuinely restored. RTO is met when the newest record is inside the RPO, not when the first request succeeds.

Blast radius is the fraction of your workload that one failure can remove. You cannot measure it after the fact unless you tagged requests beforehand, which makes this an instrumentation task rather than an analysis task.

Tag every request with its shared dependencies:

log_attempt(
    request_id=rid, target_host=host, provider="primary", pool="dc-eu",
    asn=asn_of(exit_ip), prefix=prefix24(exit_ip), country="DE",
    worker=worker_id, status=status, latency_ms=ms,
)

Then a single query answers "what do the failures have in common", which is the only question that matters in the first ten minutes of an incident:

SELECT provider, asn, prefix, country,
       COUNT(*) FILTER (WHERE status NOT BETWEEN 200 AND 299) AS failed,
       COUNT(*) AS total,
       ROUND(100.0 * COUNT(*) FILTER (WHERE status NOT BETWEEN 200 AND 299)
             / COUNT(*), 1) AS pct
FROM attempts
WHERE ts > now() - interval '15 minutes'
GROUP BY GROUPING SETS ((provider), (asn), (prefix), (country))
HAVING COUNT(*) > 50
ORDER BY pct DESC;

The GROUPING SETS clause gives you failure rate broken down by each dimension in one pass. If asn shows 98 percent and provider shows 51 percent, you have an ASN-level block spanning providers, and the two-provider setup you paid for is tier 2, not tier 4.

Sane targets: no single ASN should carry more than 40 percent of a critical dataset's traffic, and no single provider more than 70 percent. Those are not laws. They are the point where one block stops being an inconvenience. Spreading work so that no shared dependency dominates is a design property of the collection layer, and building a distributed web scraper walks through the sharding side of it.

Untested failover fails at roughly the rate of no failover, because the failure modes it hits are configuration drift and code rot, not the ones you designed for. The principles of chaos engineering formalize the fix: inject the failure deliberately, in production or in a production-shaped environment, on a schedule.

Four drills cover most of the ground.

The blackhole is the highest-value one and takes a single line:

# blackhole the primary endpoint on ONE worker host
echo "127.0.0.1 scrape.sparkproxy.io" | sudo tee -a /etc/hosts
# observe the breaker and the standby, then undo it
sudo sed -i '/scrape.sparkproxy.io/d' /etc/hosts

Run it on one worker first, never the fleet. You are testing whether the code path works, and one host proves that.

State pass criteria as numbers tied to your RTO. "Failover works" is not a criterion. "95th percentile time from injection to first successful request through the standby is under 15 minutes, and the newest prices record never exceeded 6 hours" is one, and it is the same pair of numbers from the RTO table. Record the measured value each run. When it drifts upward across two quarters, something rotted and you found it before a real outage did.

Baseline reliability expectations, so you know what failure rate is normal before you start injecting, are covered in understanding proxy uptime and reliability.

A managed scraping endpoint moves several of these layers server-side. The SparkProxy Scraping API retries internally with escalating navigation timeouts, 90 seconds on the first attempt and up to three automatic retries at 120 s and 180 s, so transient target slowness never reaches your code.

curl -G "https://scrape.sparkproxy.io/api/v1" \
  -H "X-API-Key: YOUR_API_KEY" \
  --data-urlencode "url=https://example.com/products" \
  --data-urlencode "render_js=false" \
  --data-urlencode "premium_proxy=true" \
  --data-urlencode "country_code=DE"

What still belongs to you is classifying the response, because the status codes map directly onto the failure taxonomy above:

The 429 and 530 rows are where teams go wrong. A 429 here is a limit on your account's concurrency, not a target's per-IP rate limit, so the fix is fewer parallel jobs rather than more rotation, and the response carries active, limit, and retry_after_seconds so you can compute the wait exactly. A 530 says the target failed, which means failing over to another provider will most likely reproduce it.

import time, requests

resp = requests.post(
    "https://scrape.sparkproxy.io/api/v1",
    headers={"X-API-Key": "YOUR_API_KEY", "Content-Type": "application/json"},
    json={"url": target, "render_js": True, "premium_proxy": True,
          "country_code": "DE", "json_response": True, "tag": "de-prices"},
    timeout=200,   # covers the server-side 90/120/180 retry ladder
)

if resp.status_code in (401, 402, 422):
    alert(f"fatal {resp.status_code}: {resp.text[:200]}")
    raise SystemExit(1)
if resp.status_code == 429:
    time.sleep(resp.json().get("retry_after_seconds", 5))
elif resp.status_code == 503:
    breaker.record(ok=False, now=time.time())     # fail over to standby
elif resp.status_code == 530:
    degrade(target)                               # target-side, drop a tier

Note the client timeout of 200 seconds. Setting it to 30 while the server is still working through its own retry ladder produces a client-side timeout on a request that would have succeeded, and you pay for the work either way.

For a hybrid setup, own_proxy accepts ip:port, ip:port:user:pass, or a full http://user:pass@host:port string, so tier 1 can be your own pool with the managed endpoint as the tier 2 standby. That gives you two paths with genuinely different failure characteristics without maintaining two integrations.

Everything above compresses to a short list. Print it, keep it in the repo, argue about the numbers.

1. Classify before you react. IP, range, endpoint, auth, region, or target. Six branches, not one.
2. Never retry the same IP on a 403. Do retry it on a 5xx.
3. Cap retries as a share of total traffic, around 10 percent, not as attempts per request.
4. One breaker per provider endpoint, tripped only by transport and gateway failures.
5. Verify provider independence by ASN, not by invoice.
6. Send 3 to 5 percent of production traffic to the standby every day.
7. Write RTO and RPO per dataset. Under 30 minutes means no human in the loop.
8. Tag every request with provider, ASN, prefix, and country so blast radius is a query.
9. Blackhole the primary endpoint on one worker, monthly, and record the recovery time.
10. Degrade with as_of and degraded on every record, and emit gap markers for skips.

The pipelines that survive bad weeks are not the ones with the most spare capacity. They are the ones where somebody already knows, to the minute, how long it takes to get data flowing again.