What Is Proxy Load Balancing? Algorithms and Health Checks

A proxy load balancer answers one question, repeatedly: given N candidate exit IPs and one outbound request, which IP gets it? Everything else in a balancer exists to make that answer better. The selection algorithm decides the default distribution. The health checker removes candidates that would fail. The ejection policy decides how long a bad candidate stays out. The affinity rule decides when the answer must stay the same as last time.

That is a different job from a classic web load balancer. HAProxy or NGINX in front of your own app servers protects resources you own: CPU, memory, connection slots. A proxy balancer protects a resource you do not own, the reputation and rate-limit budget of each exit IP against a third-party target. Overload a backend server and it gets slow. Overload an exit IP and the target bans it, sometimes for days, and no amount of extra hardware fixes that.

That distinction reshapes every algorithm choice below. A backend server recovers the instant load drops. A burned IP does not.

Proxy load balancing: the decision path

request
   |
   v
[ affinity rule ]  --- has session key? ---> pick IP by consistent hash
   |  no key
   v
[ health filter ]  --- drop ejected / cooling-down IPs
   |
   v
[ selection algo ] --- round-robin | weighted | least-conn | LRU
   |
   v
[ exit IP ] ---> target site
   |
   v
[ outcome feedback: 200 / 403 / 429 / timeout ] --> health state

The feedback arrow at the bottom is the part most homegrown pools are missing. Without it the balancer keeps confidently handing requests to IPs the target already blocked.

These two terms get used as synonyms and they should not be. Rotation is a policy about change: how often the IP presented to a target should differ from the last one. Load balancing is a policy about distribution: how total request volume spreads across the pool so no single IP carries an unfair share.

You can rotate badly and still be balanced. You can be perfectly balanced and rotate too slowly for the target. For the rotation side in depth, the guide to rotating proxies and per-IP request limits covers cadence, per-IP caps, and sticky windows. This article is about distribution and health.

The clean mental model: rotation decides when an IP is eligible to change, load balancing decides what it changes to. Both operate on the same underlying proxy pool.

Round-robin walks the list in order and wraps around. Every IP gets exactly one request per cycle. It is stateless apart from a cursor, it needs no measurement, and for a pool of identical datacenter IPs hitting one target it is genuinely fine.

import itertools
import threading

class RoundRobin:
    def __init__(self, proxies):
        self._cycle = itertools.cycle(proxies)
        self._lock = threading.Lock()

    def pick(self):
        with self._lock:
            return next(self._cycle)

pool = RoundRobin([
    "http://user:pass@45.61.10.11:8000",
    "http://user:pass@45.61.10.12:8000",
    "http://user:pass@45.61.10.13:8000",
])
print(pool.pick())  # 45.61.10.11
print(pool.pick())  # 45.61.10.12

The lock matters. itertools.cycle is not thread-safe in the way you need here: concurrent next() calls can interleave, hand the same proxy to two workers, and skip a third entirely. On a three-IP pool nobody notices. On a 40-worker crawler it produces visible skew.

Round-robin breaks down in three situations. First, when request costs differ wildly, since a cheap HTML fetch and a full JavaScript render are not equal load. Second, when IPs differ in quality, because equal shares to unequal IPs means your worst IP burns first while your best IP sits underused. Third, and most damaging, when the pool changes size. Add one IP to a round-robin cursor and the entire subsequent assignment sequence shifts, which is harmless for stateless requests and fatal for anything that needed to stay on the same IP.

Weighting fixes the "IPs are not equal" problem. Each IP gets a weight and receives a share of traffic proportional to it. A pool mixing a clean /24 datacenter block with older, more-flagged addresses should not treat both identically.

Naive weighted round-robin (repeat each IP weight times in a flat list) works but bursts: with weights 5, 1, 1 you send five consecutive requests through the same IP before touching the others. Bursting is exactly what per-IP rate limits punish. The smooth weighted round-robin algorithm used in the NGINX upstream module spreads the same ratio evenly instead.

class SmoothWeighted:
    """NGINX-style smooth weighted round-robin: same ratios, no bursts."""
    def __init__(self, peers):          # peers: {proxy: weight}
        self.peers = [{"p": p, "w": w, "cur": 0} for p, w in peers.items()]
        self.total = sum(peers.values())

    def pick(self):
        best = None
        for peer in self.peers:
            peer["cur"] += peer["w"]
            if best is None or peer["cur"] > best["cur"]:
                best = peer
        best["cur"] -= self.total
        return best["p"]

lb = SmoothWeighted({"ip_a": 5, "ip_b": 1, "ip_c": 1})
print([lb.pick() for _ in range(7)])
# ['ip_a', 'ip_a', 'ip_b', 'ip_a', 'ip_c', 'ip_a', 'ip_a']

Look at the output: ip_a still takes five of seven, but ip_b and ip_c are interleaved rather than stranded at the end of the cycle.

The interesting move is making weights dynamic. Static weights encode what you believed about your IPs on the day you configured them. A weight recalculated from the last 500 requests encodes what is true now.

def adaptive_weight(stats, floor=1, ceiling=20):
    """stats: {'ok': int, 'blocked': int, 'p50_ms': float}"""
    attempts = stats["ok"] + stats["blocked"]
    if attempts < 20:
        return 5                                  # not enough data, stay neutral
    success = stats["ok"] / attempts
    speed = min(1.0, 1500 / max(stats["p50_ms"], 1))
    score = (success ** 2) * speed                # square it: punish blocks hard
    return max(floor, min(ceiling, round(score * ceiling)))

Squaring the success rate is deliberate. An IP at 90% success is not "10% worse" than one at 100%, because those failures are usually the leading edge of a block rather than random noise. Squaring drops its weight to 0.81 of the maximum and keeps dropping fast as the rate slips.

Least connections sends each request to the candidate with the fewest requests currently in flight. HAProxy calls it balance leastconn; NGINX calls it least_conn. For backend servers with variable request durations it is usually the best default available, because it self-corrects: a slow server accumulates in-flight requests and automatically stops receiving new ones.

import threading
from contextlib import contextmanager

class LeastConnections:
    def __init__(self, proxies):
        self.inflight = {p: 0 for p in proxies}
        self._lock = threading.Lock()

    @contextmanager
    def lease(self):
        with self._lock:
            proxy = min(self.inflight, key=self.inflight.get)
            self.inflight[proxy] += 1
        try:
            yield proxy
        finally:
            with self._lock:
                self.inflight[proxy] -= 1

pool = LeastConnections(["ip_a", "ip_b", "ip_c"])
with pool.lease() as proxy:
    pass  # send the request through `proxy`

Here is the part that rarely gets said out loud: for proxies, least connections optimizes a metric that is not the binding constraint. In-flight socket count measures concurrency, and concurrency only becomes a problem when it exceeds what your plan allows (covered in understanding concurrent connections in proxies). What actually gets an IP banned is requests per IP per unit time against a specific target. An IP that just finished 200 fast requests in the last minute has zero connections in flight, so least-connections cheerfully picks it again, right as the target's rate limiter prepares a 429.

The proxy-appropriate variant is least-recently-used with a cooldown, keyed per target.

import time
from collections import defaultdict

class Cooling(Exception):
    """Whole pool is inside its per-IP cooldown for this host. Back off."""

class LruCooldown:
    """Pick the IP that has gone longest without touching THIS target."""
    def __init__(self, proxies, cooldown_s=8.0):
        self.proxies = list(proxies)
        self.cooldown = cooldown_s
        self.last_used = defaultdict(dict)        # host -> {proxy: timestamp}

    def pick(self, host, healthy):
        now = time.monotonic()
        seen = self.last_used[host]
        candidates = [p for p in self.proxies if p in healthy]
        # oldest touch first; never-used IPs sort to the front
        candidates.sort(key=lambda p: seen.get(p, 0.0))
        chosen = candidates[0]
        idle = now - seen.get(chosen, 0.0)
        if idle < self.cooldown:
            raise Cooling(self.cooldown - idle)
        seen[chosen] = now
        return chosen

Raising instead of returning a hot IP is the point. When every IP is still cooling for that host, the correct action is to slow down, not to pick the least-bad option. Pair it with the rules in retry and backoff strategies for web scraping.

One more algorithm is worth knowing because it costs almost nothing: power of two choices. Sample two IPs at random, take the better one. Michael Mitzenmacher showed in his 1996 doctoral work that this cuts the maximum load from roughly log n / log log n down to log log n, capturing most of the benefit of full least-loaded selection without scanning the whole pool. On pools of thousands of IPs, where a full min() scan per request is real overhead, it is the practical choice.

Everything above assumes any healthy IP will do. Often it will not. A logged-in session, a multi-step checkout flow, a paginated search carrying a server-side cursor: these break if request 4 arrives from a different IP than requests 1 through 3. You need affinity, meaning the same key always maps to the same IP.

The obvious implementation is proxies[hash(key) % len(proxies)]. It works until the pool changes size, and then it destroys nearly every existing mapping.

import hashlib

def modulo_map(key, n):
    h = int(hashlib.md5(key.encode()).hexdigest(), 16)
    return h % n

keys = [f"session-{i}" for i in range(10_000)]
before = {k: modulo_map(k, 100) for k in keys}
after = {k: modulo_map(k, 101) for k in keys}       # one IP added
moved = sum(1 for k in keys if before[k] != after[k])
print(f"{moved / len(keys):.1%} of sessions remapped")
# 99.0% of sessions remapped

Adding one IP to a hundred moved 99% of sessions. Every one of those sessions now presents a new IP mid-flow, which reads to the target exactly like account sharing or session hijacking.

Consistent hashing, introduced by Karger and colleagues at STOC 1997, fixes this. Hash both the IPs and the keys onto the same circular keyspace, then map each key to the first IP clockwise from it. Add or remove one node and only the keys in that node's arc move, roughly K/n of them instead of all of them.

import bisect
import hashlib

class HashRing:
    def __init__(self, proxies, vnodes=160):
        self.vnodes = vnodes
        self.ring = {}          # hash -> proxy
        self.sorted_keys = []
        for p in proxies:
            self.add(p)

    def _hash(self, value):
        return int(hashlib.md5(value.encode()).hexdigest()[:8], 16)

    def add(self, proxy):
        for i in range(self.vnodes):
            h = self._hash(f"{proxy}#{i}")
            self.ring[h] = proxy
            bisect.insort(self.sorted_keys, h)

    def remove(self, proxy):
        for i in range(self.vnodes):
            h = self._hash(f"{proxy}#{i}")
            self.ring.pop(h, None)
            idx = bisect.bisect_left(self.sorted_keys, h)
            if idx < len(self.sorted_keys) and self.sorted_keys[idx] == h:
                self.sorted_keys.pop(idx)

    def get(self, key):
        if not self.ring:
            raise RuntimeError("empty ring")
        h = self._hash(key)
        idx = bisect.bisect(self.sorted_keys, h) % len(self.sorted_keys)
        return self.ring[self.sorted_keys[idx]]

Run the same experiment against the ring:

pool = [f"ip_{i}" for i in range(100)]
ring = HashRing(pool)
before = {k: ring.get(k) for k in keys}
ring.add("ip_100")
after = {k: ring.get(k) for k in keys}
print(f"{sum(1 for k in keys if before[k] != after[k]) / len(keys):.1%} remapped")
# ~1.0% remapped

Ninety-nine percent versus one percent, for the same pool change. That is the entire argument for consistent hashing in a proxy balancer.

The vnodes=160 default is not arbitrary. It comes from libketama, the consistent hashing implementation memcached clients standardized on, and it exists because a ring with a single point per node distributes arcs very unevenly. With 160 virtual points per IP, load across a 100-IP pool typically lands within a few percent of uniform. Drop it to 10 and some IPs take twice the traffic of others.

Two practical notes. Choose the hash key carefully: hash the thing that must stay stable, usually an account ID, a session token, or a target domain, never the individual URL. And if you need affinity stronger than your own ring can guarantee across restarts and pool churn, provider-side sticky sessions hold one exit IP for a token with a defined TTL, which is a firmer guarantee than any client-side ring can make about IPs it does not control.

A selection algorithm is only as good as its candidate list. Feed it dead or blocked IPs and you get failures no matter how elegant the math is. Two mechanisms keep the list honest, and mature setups run both.

Passive health checking (also called in-band checking or outlier detection) observes real traffic. Every response updates the state of the IP that served it. It costs nothing extra and reflects genuine conditions against the real target, but it reacts only after a user-visible failure has already happened.

Active health checking (out-of-band) probes IPs on a timer, independently of real traffic. It catches a dead IP before a real request hits it, and it can bring an ejected IP back with evidence rather than a guess. It costs bandwidth and requests, and a probe against a neutral endpoint proves the tunnel works, not that a specific target still trusts the IP.

import asyncio
import aiohttp

async def probe(session, proxy, timeout=6):
    """Active check: verify the tunnel works and confirm the exit IP."""
    try:
        async with session.get(
            "https://api.ipify.org?format=json",
            proxy=proxy, timeout=aiohttp.ClientTimeout(total=timeout)
        ) as r:
            if r.status != 200:
                return proxy, False, f"http_{r.status}"
            data = await r.json()
            return proxy, True, data["ip"]
    except asyncio.TimeoutError:
        return proxy, False, "timeout"
    except aiohttp.ClientError as e:
        return proxy, False, type(e).__name__

async def sweep(proxies):
    async with aiohttp.ClientSession() as s:
        return await asyncio.gather(*(probe(s, p) for p in proxies))

Two details make this probe useful rather than decorative. It returns the observed exit IP, so you can detect an IP silently reassigned to a different address than the one you paid for. And it separates timeouts from client errors, because a timeout usually means a slow route while a connection reset usually means a dead endpoint, and those deserve different ejection weights.

The rule of thumb: passive checks decide who leaves, active checks decide who comes back. Passive signals carry target-specific truth. Active signals are the only ones you can safely gather while an IP sits ejected.

Ejection is where homegrown balancers usually fail, in one of two directions. Too eager and a single transient 503 removes a healthy IP, the reduced pool concentrates load on the survivors, they get knocked out too, and the pool collapses in under a minute. Too lax and blocked IPs stay in rotation for hours, burning budget and dragging the success rate down.

The pattern that solves both is a circuit breaker with three states: closed (in rotation), open (ejected), and half-open (one trial request allowed). Envoy's outlier detection uses the same shape, and its defaults are worth stealing: eject after 5 consecutive 5xx responses, a base ejection time of 30 seconds, and a hard cap of 10% of the pool ejected at once.

That last number is the one people skip, and it is the one that prevents the collapse. If more than a tenth of your pool looks broken, the likeliest explanation is that the target changed its defenses, not that your IPs simultaneously died. Ejecting them all makes the situation worse.

import time
import random

class Breaker:
    CLOSED, OPEN, HALF_OPEN = "closed", "open", "half_open"

    def __init__(self, base_ejection_s=30, max_ejection_s=600, threshold=5):
        self.state = self.CLOSED
        self.consecutive_failures = 0
        self.ejection_count = 0          # how many times this IP has been ejected
        self.open_until = 0.0
        self.base = base_ejection_s
        self.cap = max_ejection_s
        self.threshold = threshold

    def record(self, ok, status=None):
        if ok:
            self.consecutive_failures = 0
            if self.state == self.HALF_OPEN:
                self.state = self.CLOSED
                self.ejection_count = max(0, self.ejection_count - 1)  # slow forgiveness
            return
        # a hard block counts far more than a transient error
        weight = 3 if status in (403, 407, 451) else 1
        self.consecutive_failures += weight
        if self.consecutive_failures >= self.threshold:
            self.trip()

    def trip(self):
        self.state = self.OPEN
        self.ejection_count += 1
        backoff = min(self.cap, self.base * (2 ** (self.ejection_count - 1)))
        jitter = random.uniform(0.8, 1.2)           # avoid synchronized re-entry
        self.open_until = time.monotonic() + backoff * jitter
        self.consecutive_failures = 0

    def available(self):
        if self.state == self.OPEN and time.monotonic() >= self.open_until:
            self.state = self.HALF_OPEN
        return self.state in (self.CLOSED, self.HALF_OPEN)

Three design choices in there are worth defending.

Ejection time grows with repeat offenses. An IP ejected once gets 30 seconds. The same IP ejected a fourth time gets four minutes. Chronically bad IPs drift out of rotation on their own, with no separate blacklist to maintain, and a single bad minute costs an otherwise good IP almost nothing.

Jitter on the re-entry time stops synchronized recovery. Eject 12 IPs in the same second with a fixed 30-second timer and all 12 return in the same second, hit the same still-hostile target, and get ejected together again. That loop can run for hours. Multiplying the backoff by a random 0.8 to 1.2 breaks the lockstep.

Status codes are not weighted equally. A 403 or a 407 is evidence about the IP itself. A 500 or a 502 is usually evidence about the target's own servers, and ejecting your IPs because of the target's outage is self-harm. Timeouts sit in between and deserve weight 1.

Now the detail almost no tutorial mentions: ejection should be scoped per target, not globally. An IP blocked by one retailer is very often perfectly clean against every other site you crawl. A global breaker throws that capacity away.

from collections import defaultdict

class ScopedHealth:
    """Breaker state keyed by (proxy, target_host), plus a global tunnel breaker."""
    def __init__(self):
        self.scoped = defaultdict(Breaker)      # (proxy, host) -> Breaker
        self.tunnel = defaultdict(Breaker)      # proxy -> Breaker

    def healthy_for(self, proxies, host, max_ejected_pct=0.10):
        live = [p for p in proxies
                if self.tunnel[p].available() and self.scoped[(p, host)].available()]
        floor = max(1, int(len(proxies) * (1 - max_ejected_pct)))
        if len(live) < floor:
            # too much of the pool is out: the target changed, not our IPs.
            # keep the floor filled with the least-recently-ejected candidates.
            spare = sorted(
                (p for p in proxies if p not in live and self.tunnel[p].available()),
                key=lambda p: self.scoped[(p, host)].open_until,
            )
            live += spare[: floor - len(live)]
        return live

    def record(self, proxy, host, ok, status=None):
        self.scoped[(proxy, host)].record(ok, status)
        # only transport-level failures count against the tunnel itself
        if status is None and not ok:
            self.tunnel[proxy].record(False)
        elif ok:
            self.tunnel[proxy].record(True)

Two breakers per IP, one per target and one for the tunnel, is the smallest structure that gets both cases right. A proxy that cannot open a connection at all is globally unhealthy. A proxy that gets a 403 from one domain is unhealthy only there.

There are two places this logic can live, and the choice has real consequences.

Client-side balancing means your code holds the IP list and every process runs its own selection, health state, and ejection. You get complete control, per-target scoping, and custom weights. You also get a coordination problem the moment you run more than one process. Ten workers each keeping private health state means one blocked IP has to be independently rediscovered ten times, at a cost of ten failed requests. Cursors drift, weights disagree, and a rolling deploy resets every breaker to closed at once.

If you go this route, health state belongs in shared storage rather than in each process.

import redis, json, time

r = redis.Redis(decode_responses=True)

def eject(proxy, host, seconds):
    """Shared ejection: every worker sees it immediately."""
    r.setex(f"ejected:{proxy}:{host}", int(seconds), json.dumps({"at": time.time()}))

def healthy(proxies, host):
    keys = [f"ejected:{p}:{host}" for p in proxies]
    flags = r.mget(keys)
    return [p for p, flag in zip(proxies, flags) if flag is None]

One MGET per request batch, not per request. A shared breaker that costs a round trip on every single fetch is a worse problem than the one it solves.

Gateway-side balancing means you send everything to one endpoint and the provider runs the algorithms, the health checks, and the ejection across a pool far larger than yours. You lose per-target weight tuning. You gain health data aggregated across every customer hitting the same targets, which is a much stronger signal than anything a single pool can observe on its own. That architecture is covered in what a proxy gateway is.

The honest recommendation: run client-side balancing when your pool is small, static, and pointed at a handful of targets you understand deeply. Move to a gateway when the pool grows large enough that health tracking becomes its own maintenance project, which in practice happens somewhere around a few hundred IPs.

The SparkProxy Scraping API sits at the fully delegated end of that spectrum. You send a URL, the service selects the exit IP, checks health, retries on failure, and returns the response. There is no pool for you to balance.

curl -G "https://scrape.sparkproxy.io/api/v1" \
  -H "X-API-Key: YOUR_API_KEY" \
  --data-urlencode "url=https://example.com/products" \
  --data-urlencode "render_js=false" \
  --data-urlencode "json_response=true"

The response envelope carries accounting you would otherwise have to instrument yourself:

{
  "job_id": "abc123",
  "result_url": "https://scrape.sparkproxy.io/api/v1/files/job_abc123",
  "format": "html",
  "status_code": 200,
  "duration_ms": 3840,
  "meta": { "title": "Example Domain" },
  "credits_used": 1
}

Selection is steered by parameters rather than by your algorithm. premium_proxy=true routes through the residential pool, country_code pins the exit geography, and stealth=true (which requires render_js=true) adds evasion on hostile targets.

import requests

resp = requests.post(
    "https://scrape.sparkproxy.io/api/v1",
    headers={"X-API-Key": "YOUR_API_KEY", "Content-Type": "application/json"},
    json={
        "url": "https://example.com/search?q=laptop",
        "render_js": True,
        "premium_proxy": True,
        "country_code": "DE",
        "json_response": True,
        "tag": "de-price-sweep",
    },
    timeout=90,
)
print(resp.status_code, resp.headers.get("X-Credits-Used"))

Two operational details matter for balancing specifically. A 429 from this endpoint is a concurrency limit on your account, not a target's per-IP rate limit, so the fix is fewer parallel jobs rather than more rotation. And session_id is a label echoed into logs, not a sticky-session mechanism: it does not persist cookies or hold an exit IP, so build affinity on the residential product's sticky sessions instead of assuming that parameter provides it.

If you want to keep your own pool and delegate only the transport, own_proxy accepts ip:port, ip:port:user:pass, or a full http://user:pass@host:port string, which lets you run your own selection while still using the rendering stack.

chosen = pool.pick()          # your balancer decides

resp = requests.get(
    "https://scrape.sparkproxy.io/api/v1",
    headers={"X-API-Key": "YOUR_API_KEY"},
    params={"url": target, "render_js": "true", "own_proxy": chosen},
    timeout=90,
)
if resp.status_code == 530:               # scrape failed: page error or timeout
    health.record(chosen, host, ok=False)

Status 530 is the one to wire into your health recorder. It signals that the scrape itself failed rather than an API-level problem, which is exactly the passive signal an ejection policy needs.

Four numbers tell you whether any of this is doing its job. Track them per target, because aggregates hide the failure you care about.

Gini coefficient of request distribution. Sort per-IP request counts and measure inequality. Perfect balance is 0, one IP doing everything is 1. Above roughly 0.3 on a pool of equal-weight IPs your selection is skewed, usually by a locking bug or a health filter quietly removing most candidates.

Ejected fraction over time. A stable 2 to 5% is normal churn. A spike toward your 10% cap almost always means the target changed, so slow down rather than keep retrying.

Success rate of half-open trials. If trial requests after ejection fail more than half the time, your base ejection time is too short and you are re-admitting IPs before the target's own cooldown expires.

P99 wait for an available IP. A rising P99 with a flat success rate means the pool is too small for the request rate, and no algorithm fixes that.

def gini(counts):
    """0.0 = perfectly even distribution, 1.0 = one IP took everything."""
    xs = sorted(counts)
    n = len(xs)
    if n == 0 or sum(xs) == 0:
        return 0.0
    cumulative = sum((i + 1) * x for i, x in enumerate(xs))
    return (2 * cumulative) / (n * sum(xs)) - (n + 1) / n

print(gini([100, 100, 100, 100]))   # 0.0
print(gini([370, 10, 10, 10]))      # 0.66  -> something is badly skewed

Log the algorithm name and the health state with every request outcome. When the success rate drops next month you will want to know whether the balancer chose badly or the target simply got harder, and without that field you cannot tell the two apart.