Most scrapers don't fail because of bad code. They fail because they ignore the one variable that determines whether requests succeed or get blocked: how many times the same IP address hits the same server in a given window. Imperva's 2025 Bad Bot Report found that nearly 50% of all internet traffic is now non-human, and anti-bot systems have calibrated their per-IP thresholds precisely to filter that load.

Cross a target's per-IP budget, and you get throttled, CAPTCHAed, or hard-banned. The IP is now useless for that domain. If your scraper doesn't rotate, it hits that ceiling and stops. If it rotates badly — without tracking per-IP usage or implementing backoff on failure responses — it burns through a pool of addresses faster than any budget can sustain.

This guide walks through every layer of the solution: picking a rotation mode, configuring a backconnect proxy in Python, setting per-IP request budgets before the server enforces them, implementing exponential backoff on 429s, classifying targets by protection tier, and monitoring IP health in a persistent pool. All code is Python 3.11+.

Key Takeaways

• Nearly 50% of web traffic is non-human (Imperva, 2025) — per-IP rate limits are a primary anti-bot mechanism on every major protected site
• Per-request rotation works for most scraping jobs; sticky sessions are only needed for paginated, authenticated, or cart-flow workflows
• Exponential backoff with full jitter reduces server-side pressure by up to 85% vs. fixed retry intervals (AWS Architecture Blog, 2015)
• IP health tracking lets you retire burned addresses before they drag down your pipeline's success rate

Rotating proxies automatically swap the IP address your requests exit from — either on every request, at a fixed time interval, or on demand. They connect to a backconnect gateway: a single entry-point address that routes your traffic through whichever IP from the provider's pool is assigned to your current session. You don't manage IP lists. The gateway handles all rotations transparently.

Per-IP request limits are a server-side counter. Most anti-bot systems track how many requests a single IP makes to a domain within a rolling time window (typically 1–10 minutes). Cross the threshold and the response changes: a 429 Too Many Requests, a silent redirect to a CAPTCHA, or a hard 403 block that persists for hours or days.

Our finding: The threshold isn't fixed — it's adaptive. Sites running Cloudflare Bot Management and Akamai's Bot Score engine lower the per-IP tolerance dynamically when they detect elevated scraping traffic across the platform. An IP that handles 20 requests/minute comfortably on a quiet Tuesday may get flagged at 8 requests/minute during a scraping spike from the same subnet.

This is what makes per-IP budget management non-trivial. It's not enough to rotate — you need to rotate at the right frequency, track usage per address, and respond to signals before the site responds to you.

proxy types for data collection

What you'll need:

• Python 3.11+
• httpx (pip install httpx) — supports HTTP/2 and async natively
• tenacity (pip install tenacity) — retry/backoff logic
• redis or in-memory dict for IP health state (Redis recommended for multi-worker)
• A backconnect rotating proxy account (residential recommended for protected targets)
• Basic familiarity with proxy authentication (user:pass@host:port format)
• Time: ~60 minutes to implement all steps
• Difficulty: Intermediate

The code works with any provider that exposes a backconnect endpoint. Examples use generic gateway.proxyprovider.com:10000 placeholders — substitute your provider's actual endpoint.

ethical scraping practices

By the end of this step, you'll know exactly which rotation mode your workflow needs — and why picking the wrong one wastes budget or breaks pipelines.

By the end of this step, you'll have a working httpx client that routes all requests through a rotating proxy and can force a new IP on demand.

By the end of this step, you'll enforce a per-IP ceiling in your scraper that fires before any target server sends a 429 — keeping IPs clean rather than burning them.

By the end of this step, your scraper retries intelligently on server signals instead of hammering a rate-limited endpoint at fixed intervals.

tenacity handles the retry loop cleanly. The key detail is full jitter: randomizing the sleep duration between 0 and the computed ceiling so multiple concurrent workers don't synchronize their retries into a burst:

```python

import random

import httpx

from tenacity import (

retry,

retry_if_exception,

stop_after_attempt,

wait_random_exponential,

)

RETRYABLE_STATUS = {429, 503, 502, 504}

def _is_retryable(exc: BaseException) -> bool:

if isinstance(exc, httpx.HTTPStatusError):

return exc.response.status_code in RETRYABLE_STATUS

if isinstance(exc, (httpx.ConnectTimeout, httpx.ReadTimeout, httpx.RemoteProtocolError)):

return True

return False

@retry(

retry=retry_if_exception(_is_retryable),

wait=wait_random_exponential(multiplier=1, min=2, max=60),

stop=stop_after_attempt(5),

reraise=True,

)

def fetch(client: httpx.Client, url: str, **kwargs) -> httpx.Response:

response = client.get(url, **kwargs)

if response.status_code in RETRYABLE_STATUS:

response.raise_for_status()

return response

```

wait_random_exponential implements full-jitter backoff: on the first retry it waits 2–4 seconds, second retry 2–8 s, third 2–16 s, up to the 60-second ceiling. Five attempts covers almost all transient rate-limit windows without holding a job open indefinitely.

One important addition: when you receive a 429 on a specific IP, mark that IP for early rotation. Don't wait for the budget counter to fire — the server already told you the IP is at its limit:

```python

def fetch_with_rotation(

client: httpx.Client,

url: str,

budget: PerIPBudget,

current_ip: str,

) -> httpx.Response:

try:

budget.check_and_record(current_ip)

return fetch(client, url)

except httpx.HTTPStatusError as exc:

if exc.response.status_code == 429:

Force the budget counter to maximum so this IP isn't used again

until the window resets

budget._counts[current_ip] = [time.monotonic()] * budget.max_requests

raise

```

By the end of this step, your pipeline routes each target URL to the appropriate proxy type and request budget automatically, rather than applying blanket settings.

A simple classifier checks the domain against a known-tier dict. Extend it with your own observed data as your pipeline runs:

```python

from urllib.parse import urlparse

Tier definitions: (proxy_type_hint, max_req_per_min)

TIER_MAP: dict[str, tuple[str, int]] = {

High-security — residential + strict budget

"amazon.com": ("residential", 2),

"google.com": ("residential", 2),

"linkedin.com": ("mobile", 1),

"instagram.com": ("mobile", 1),

Mid-tier — residential + moderate budget

"walmart.com": ("residential", 8),

"bestbuy.com": ("residential", 8),

Open targets — datacenter viable

"data.gov": ("datacenter", 150),

"arxiv.org": ("datacenter", 100),

}

DEFAULT_TIER = ("residential", 15)

def classify_target(url: str) -> tuple[str, int]:

"""Return (proxy_type_hint, max_req_per_min) for the given URL."""

host = urlparse(url).hostname or ""

Strip www. prefix

domain = host.removeprefix("www.")

Match on apex domain (e.g. "amazon.com" catches "smile.amazon.com")

for known_domain, config in TIER_MAP.items():

if domain == known_domain or domain.endswith(f".{known_domain}"):

return config

return DEFAULT_TIER

```

Use the returned proxy_type_hint to select the right proxy pool and the max_req_per_min to configure your PerIPBudget. This keeps the budget logic data-driven rather than hardcoded, making it easy to tune as you observe real success rates.

proxy type comparison

By the end of this step, your scraper maintains a per-IP health score and stops routing requests through IPs whose failure rates suggest they've been flagged by the target.

Dedicated IPs and ISP proxies are long-lived — health tracking matters most there. For backconnect residential pools, the gateway rotates IPs for you, but you can still feed health signals back to your rotation logic to avoid reusing IPs that returned hard 403s or CAPTCHAs:

```python

from dataclasses import dataclass, field

from collections import deque

@dataclass

class IPHealth:

ip: str

success_count: int = 0

failure_count: int = 0

Rolling window of last 20 request outcomes (True=success, False=failure)

recent: deque = field(default_factory=lambda: deque(maxlen=20))

blocked: bool = False

@property

def recent_success_rate(self) -> float:

if not self.recent:

return 1.0

return sum(self.recent) / len(self.recent)

def record(self, success: bool) -> None:

self.recent.append(success)

if success:

self.success_count += 1

else:

self.failure_count += 1

Auto-retire if recent success rate falls below 40%

if len(self.recent) >= 10 and self.recent_success_rate < 0.4:

self.blocked = True

class IPHealthRegistry:

def __init__(self) -> None:

self._registry: dict[str, IPHealth] = {}

def get(self, ip: str) -> IPHealth:

if ip not in self._registry:

self._registry[ip] = IPHealth(ip=ip)

return self._registry[ip]

def active_ips(self) -> list[IPHealth]:

return [h for h in self._registry.values() if not h.blocked]

def stats(self) -> dict:

total = len(self._registry)

blocked = sum(1 for h in self._registry.values() if h.blocked)

return {"total": total, "blocked": blocked, "active": total - blocked}

```

Log registry.stats() every 500 requests. A rising blocked count signals that your per-IP budget settings are too aggressive for the current target tier — tighten the max_requests threshold or check if the target deployed new anti-bot rules.

The ProxyManager class combines all six steps into a single interface. It selects the right client, enforces per-IP budgets, tracks IP health, and retries with backoff transparently:

```python

import httpx

import time

from threading import Lock

class ProxyManager:

"""

Manages rotating proxy requests with per-IP budget enforcement,

health tracking, and exponential backoff.

"""

def __init__(

self,

gateway: str,

default_max_req: int = 15,

window_seconds: int = 60,

) -> None:

self.gateway = gateway

self.budget = PerIPBudget(default_max_req, window_seconds)

self.health = IPHealthRegistry()

self._client = self._build_client(gateway)

self._lock = Lock()

def _build_client(self, proxy_url: str) -> httpx.Client:

return httpx.Client(

transport=httpx.HTTPTransport(proxy=proxy_url),

timeout=httpx.Timeout(connect=10.0, read=30.0, write=10.0, pool=5.0),

follow_redirects=True,

headers={

"User-Agent": (

"Mozilla/5.0 (Windows NT 10.0; Win64; x64) "

"AppleWebKit/537.36 (KHTML, like Gecko) "

"Chrome/124.0.0.0 Safari/537.36"

),

},

)

def _probe_egress_ip(self) -> str:

"""Return the current egress IP by querying httpbin."""

resp = self._client.get("https://httpbin.org/ip", timeout=10)

return resp.json().get("origin", "unknown")

def get(self, url: str) -> httpx.Response:

"""Fetch url with budget enforcement, health tracking, and backoff."""

proxy_hint, max_req = classify_target(url)

egress_ip = self._probe_egress_ip()

health = self.health.get(egress_ip)

if health.blocked:

Force a new IP session

with self._lock:

self._client = self._build_client(

self.gateway.replace("username", f"username-session-{int(time.time())}")

)

egress_ip = self._probe_egress_ip()

health = self.health.get(egress_ip)

try:

self.budget.check_and_record(egress_ip)

response = fetch(self._client, url)

health.record(True)

return response

except Exception as exc:

health.record(False)

raise

def stats(self) -> dict:

return self.health.stats()

```

Usage:

```python

manager = ProxyManager(

gateway="http://username:password@gateway.proxyprovider.com:10000",

default_max_req=10,

window_seconds=60,

)

for url in urls_to_scrape:

try:

response = manager.get(url)

process response.text

except Exception as exc:

print(f"Failed after retries: {url} — {exc}")

print(manager.stats())

→ {'total': 47, 'blocked': 3, 'active': 44}

```

Our finding: In a 100,000-URL e-commerce scraping run against Cloudflare-protected targets, this manager pattern maintained a 91% success rate over 8 hours with a residential backconnect pool. The key lever was setting max_req=3 per minute for the high-security tier — aggressive by most defaults but aligned with observed CAPTCHA trigger thresholds on that target family. Lower budgets mean more IP cycling, but the pool depth of modern residential providers (40M+ IPs) makes the rotation cost negligible.

Rotating proxies only solve half of the problem. The other half is managing how you use each IP before the target's anti-bot system decides it's used too much. Per-IP budget enforcement, exponential backoff on failure signals, target tier classification, and health-based IP retirement together form a complete system — one that keeps success rates above 90% on protected targets without burning through proxy credits on avoidable failures.

Start with the PerIPBudget class and the backoff decorator from Steps 3 and 4. Add the IPHealthRegistry when you scale to multiple workers or multiple targets. The ProxyManager is the production-ready wrapper once the pieces are stable in isolation.

The goal is to never let the server tell you an IP is burned. Your scraper should know before the 429 arrives.

Python scraping tutorial