How to Avoid Getting Your Proxy Blocked

Sites use multiple independent detection layers. A proxy can fail any one of them regardless of how clean the IP is:

Changing the proxy only fixes the first row. The others require changes to your HTTP client or browser configuration.

---

Rate-based blocks are the most common and the easiest to fix. A target site counts requests per IP per time window, typically 10, 100 requests per minute before triggering a CAPTCHA or 429.

import time
import random
import requests

PROXIES = [
    "http://your-proxy-1.sparkproxy.io:10000",
    "http://your-proxy-2.sparkproxy.io:10001",
    "http://your-proxy-3.sparkproxy.io:10002",
]

def get(url: str, min_delay: float = 1.0, max_delay: float = 3.0) -> requests.Response:
    proxy = random.choice(PROXIES)
    resp = requests.get(
        url,
        proxies={"http": proxy, "https": proxy},
        timeout=15,
    )
    time.sleep(random.uniform(min_delay, max_delay))  # Human-like pacing
    return resp

Two rotation strategies and when to use each:

proxy rotation

---

This is the most overlooked aspect of proxy detection bypass. Every TLS client sends a "Client Hello" message at the start of an HTTPS connection. The combination of cipher suites, TLS extensions, elliptic curves, and signature algorithms in this message creates a fingerprint, JA3 (older) or JA4 (newer). This fingerprint is computed entirely from the network packet, not from any header your code sets.

Python's requests library uses urllib3 under the hood, which has its own distinctive TLS fingerprint that matches no browser. Even if you set User-Agent: Mozilla/5.0 (Chrome/124), the TLS fingerprint still says "Python/urllib3." Cloudflare and DataDome block on this mismatch.

The fix: curl_cffi, a Python binding to curl that impersonates the actual TLS fingerprint of Chrome, Firefox, or Safari:

pip install curl_cffi

from curl_cffi import requests as cffi_requests

# Impersonates Chrome 124's exact TLS fingerprint, HTTP/2, and headers
resp = cffi_requests.get(
    "https://httpbin.org/ip",
    proxies={"http": "http://your-proxy.sparkproxy.io:10000",
             "https": "http://your-proxy.sparkproxy.io:10000"},
    impersonate="chrome124",
)
print(resp.json())

Available impersonation targets (as of curl_cffi 0.7):

curl_cffi also handles HTTP/2 automatically, requests only does HTTP/1.1 by default, which is another detectable mismatch against modern browsers.

from curl_cffi import requests as cffi_requests

session = cffi_requests.Session(impersonate="chrome124")
session.proxies = {
    "http":  "http://your-proxy.sparkproxy.io:10000",
    "https": "http://your-proxy.sparkproxy.io:10000",
}

# All requests in this session use Chrome's TLS fingerprint
resp = session.get("https://example.com/login")

Even when your TLS fingerprint is correct, inconsistent HTTP headers reveal automation. A real Chrome 124 browser always sends a specific set of headers in a specific order. Sending User-Agent: Chrome/124 while omitting sec-ch-ua or sending Accept: / (Python default) instead of Chrome's real Accept header is a detectable anomaly.

Minimum consistent header set for Chrome 124 on Windows:

import requests

CHROME_HEADERS = {
    "User-Agent": (
        "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
        "AppleWebKit/537.36 (KHTML, like Gecko) "
        "Chrome/124.0.0.0 Safari/537.36"
    ),
    "Accept":           "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8",
    "Accept-Language":  "en-US,en;q=0.9",
    "Accept-Encoding":  "gzip, deflate, br",
    "sec-ch-ua":        '"Chromium";v="124", "Google Chrome";v="124", "Not-A.Brand";v="99"',
    "sec-ch-ua-mobile": "?0",
    "sec-ch-ua-platform": '"Windows"',
    "Sec-Fetch-Dest":   "document",
    "Sec-Fetch-Mode":   "navigate",
    "Sec-Fetch-Site":   "none",
    "Sec-Fetch-User":   "?1",
    "Upgrade-Insecure-Requests": "1",
}

session = requests.Session()
session.headers.update(CHROME_HEADERS)
session.proxies = {"http": "http://your-proxy.sparkproxy.io:10000",
                   "https": "http://your-proxy.sparkproxy.io:10000"}

Header order matters. Browsers send headers in a consistent, fixed order. HTTP/2 makes this especially detectable, the HPACK compression encodes header order into the frame. curl_cffi handles this automatically. With plain requests, you can use requests.structures.CaseInsensitiveDict only for storage; the actual wire order may vary by Python version.

Combining curl_cffi for TLS + correct headers is the most reliable approach:

from curl_cffi import requests as cffi_requests

resp = cffi_requests.get(
    "https://target-site.com",
    headers={
        "Accept-Language": "en-US,en;q=0.9",
        "Referer": "https://www.google.com/",
    },
    impersonate="chrome124",
    proxies={"http": "http://your-proxy.sparkproxy.io:10000",
             "https": "http://your-proxy.sparkproxy.io:10000"},
)

curl_cffi already sets the correct TLS fingerprint, HTTP/2, and base browser headers for the chosen impersonation target. You only need to add request-specific headers like Referer.

---

Sending 10 requests per second from the same proxy will trigger rate limiting before any fingerprint detection kicks in. Respecting the target site's intended crawl rate is both effective and ethical.

import time
import random

def polite_delay(min_s: float = 1.0, max_s: float = 4.0) -> None:
    """Sleep for a random human-like interval between requests."""
    time.sleep(random.uniform(min_s, max_s))

Guidelines:

To check a site's declared crawl rate:

import urllib.robotparser

rp = urllib.robotparser.RobotFileParser()
rp.set_url("https://example.com/robots.txt")
rp.read()

print(rp.can_fetch("*", "/products"))      # True or False
print(rp.crawl_delay("*"))                  # Returns float or None

If crawl_delay returns a value, respect it. Ignoring Crawl-delay is one of the most common reasons scraper IPs get added to permanent blocklists.

---

Sites use cookies to track session continuity. Two behaviors trigger blocks:

1. Rotating the proxy but keeping the same cookies. The site sees the same session ID arriving from two different IPs, a signal consistent with proxying. Fix: reset cookies when rotating to a new proxy.

2. Making requests with no cookies at all. Real browsers accumulate cookies progressively, homepage sets a session cookie, subsequent pages send it back. Scripts that skip directly to a product page with no cookies look like bots.

import requests
import random

PROXIES = [
    "http://your-proxy-1.sparkproxy.io:10000",
    "http://your-proxy-2.sparkproxy.io:10001",
]

def new_session(proxy_url: str) -> requests.Session:
    """Fresh session (no cookies) bound to one proxy."""
    session = requests.Session()
    session.proxies = {"http": proxy_url, "https": proxy_url}
    return session

def scrape_product(product_url: str) -> str:
    proxy   = random.choice(PROXIES)
    session = new_session(proxy)

    # Step 1: Hit the homepage to receive initial cookies
    session.get("https://example.com/", timeout=10)

    # Step 2: Now request the target page, cookies are sent automatically
    resp = session.get(product_url, timeout=10)
    return resp.text

Each call to new_session() creates a session with empty cookies. The homepage visit seeds the session with whatever cookies the site sets on first visit, making subsequent requests look like normal browsing.

---

Some proxies add X-Forwarded-For or X-Real-IP headers to outgoing requests, inadvertently revealing your real IP address to the target server. Always verify your proxy does not leak before using it in production.

import requests

def check_header_leak(proxy_url: str) -> dict:
    proxies = {"http": proxy_url, "https": proxy_url}
    resp = requests.get("https://httpbin.org/headers", proxies=proxies, timeout=10)
    headers = resp.json().get("headers", {})

    leak_headers = {
        k: v for k, v in headers.items()
        if k.lower() in ("x-forwarded-for", "x-real-ip", "via", "forwarded")
    }

    return {
        "proxy":        proxy_url,
        "leak_headers": leak_headers,
        "has_leak":     bool(leak_headers),
    }

result = check_header_leak("http://your-proxy.sparkproxy.io:10000")
print(result)
# {"proxy": "...", "leak_headers": {}, "has_leak": False}  ← clean proxy
# {"proxy": "...", "leak_headers": {"X-Forwarded-For": "203.0.113.1"}, "has_leak": True}  ← leaks real IP

If has_leak is True, switch to a proxy that strips these headers. SparkProxy datacenter proxies do not forward client IP headers.

---

Before deploying, test your setup against detection services:

from curl_cffi import requests as cffi_requests

def detection_test(proxy_url: str) -> None:
    proxies = {"http": proxy_url, "https": proxy_url}
    session = cffi_requests.Session(impersonate="chrome124")
    session.proxies = proxies

    tests = {
        "Exit IP":          "https://httpbin.org/ip",
        "Headers":          "https://httpbin.org/headers",
        "Cloudflare check": "https://www.cloudflare.com/cdn-cgi/trace",
    }

    for name, url in tests.items():
        try:
            resp = session.get(url, timeout=10)
            if name == "Cloudflare check":
                # Parse key=value text response
                data = dict(line.split("=", 1) for line in resp.text.strip().splitlines() if "=" in line)
                print(f"[Cloudflare] ip={data.get('ip')}  uag={data.get('uag', '')[:40]}")
            else:
                print(f"[{name}] {resp.status_code}: {resp.text[:120]}")
        except Exception as exc:
            print(f"[{name}] FAILED: {exc}")

detection_test("http://your-proxy.sparkproxy.io:10000")

https://www.cloudflare.com/cdn-cgi/trace returns the IP Cloudflare sees, the User-Agent string it received, and whether it thinks the request is from a bot. If uag matches your User-Agent header and ip matches the proxy IP, the request looks legitimate to Cloudflare.

---

---