How Many Proxies Do I Need for Web Scraping?

If you want one line to act on:

threads = peak_requests_per_second x avg_seconds_per_request x 1.3

The 1.3 covers retries and the dead time threads spend in backoff. Round up to the next plan tier, then run two sanity checks: do you have enough distinct IPs in flight for your hardest target, and does your plan's speed cap carry the bytes.

Most buyers who ask how many proxies they need discover their thread requirement is a single digit. Their real constraint is the nightly window they insist on finishing inside, or a bandwidth ceiling they never looked at.

There are two purchase models, and mixing them up is the root of most sizing confusion.

With a rotating gateway you never see an IP list. Workers point at one host and the pool assigns an exit per request or per session. SparkProxy runs this model: gateway.sparkproxy.io, port 11000 for HTTP and HTTPS, 11002 for sticky sessions, 13000 for SOCKS5, drawing on 1M+ datacenter IPs across 80+ countries including 50,000+ US IPs. Asking how many of that million you need is like asking how many highway lanes you reserved. You bought the number of cars you may drive at once, so this is a concurrency problem, not an inventory problem. If pooled exits are new to you, what a proxy pool actually is covers the architecture.

Nobody scrapes evenly across a month. You scrape in a window, and the window sets the peak rate. This table converts a monthly page target into the requests per second you have to sustain, for four common collection windows.

Read one row across. Ten million pages a month is a 3.9 req/s job or a 46 req/s job depending only on how long you let it run. Widening a 2-hour window to 8 hours cuts the concurrency requirement by 4x, which on most price lists is one or two tiers. If the data has no real freshness deadline, patience is the cheapest capacity you can buy.

Then inflate for failures. At an 85% success rate with two retries, budget 1.2x to 1.3x the request count you got from the page count. Blocked requests still hold a thread, a connection, and time.

Concurrency follows Little's Law: the number of requests in flight equals arrival rate times time in system. For scraping that reads as threads = req/s x seconds per request.

The second term surprises people. A plain HTML fetch through a datacenter proxy is usually a low single digit of seconds once you count DNS, TLS handshake, time to first byte, and transfer. A headless render of a heavy commercial page is several times that, because you wait on scripts, XHRs, and fonts rather than on the network.

Two rules fall out of that table. Cutting per-request latency is identical to buying threads: halve the duration, halve the requirement. Turning off image, font, and analytics loading in a headless browser often does more for capacity than a plan upgrade. And threads sitting in backoff still occupy a slot, so a scraper with a poor success rate needs more threads than the arithmetic suggests. Fixing blocks is cheaper than buying around them: how to avoid getting your proxy blocked is where to start.

Threads tell you how much traffic you can push. Distinct IPs per domain tell you how much a target will accept before it starts answering with 429s and challenge pages.

The formula is the same shape:

ips_in_flight = req_per_second_to_that_domain / safe_requests_per_second_per_ip

The denominator is not a fact anyone can hand you. It depends on the site, the endpoint, the hour, and how your headers and TLS stack look. Treat the numbers below as starting assumptions for a first calibration run, not as measurements. You replace them with your own within an hour of testing.

This is where a large rotating pool earns its price. At 0.1 req/s per IP, holding 10 req/s against one WAF-protected domain needs roughly 100 exits alive at once, and they have to keep changing through the hour. A short static list cannot do that at any thread count. A million-IP pool can. Rotating proxies and per-IP request limits covers the pacing mechanics.

The inverse matters just as much. If your work is session-pinned, more IPs buy you nothing beyond one per live session, so shop for sticky session support and session duration instead of pool size.

Unlimited bandwidth means no per-GB invoice. It does not mean infinite throughput. Every plan carries a speed ceiling, and on high-volume jobs that ceiling binds before the thread count does.

SparkProxy's Fair Usage speed caps are 25 Mbps on Starter, 50 on Core, 100 on Boost, 150 on Plus, 200 on Pro, 250 on Pro+, and up to 1 Gbps on Custom. Those are ceilings, not guaranteed rates. Here is what your job actually demands, computed as req/s x page bytes x 8:

Read the right-hand column before you buy. A 25 req/s job rendering full pages with images and fonts wants 500 Mbps, past every published tier and into Custom territory. The same 25 req/s fetching HTML only wants 24 Mbps and fits the cheapest plan. Blocking subresources is the difference between a $75 plan and a custom contract. Bandwidth in proxy services explains what is being metered.

Solo price monitor, 60,000 pages a month

Three competitor sites, a 2-hour nightly run, HTML only. That is 0.28 req/s, about 1 thread at 3 seconds per request, under 0.3 Mbps. A proxy plan here buys capacity you will never touch. The Scraping API at $49 for 250,000 credits covers it four times over on plain fetches, and 1,000 free credits with no card prove it first.

Retail intelligence, 2,000,000 pages a month

A 6-hour nightly window is 3.1 req/s, roughly 12 threads with headroom, about 3 Mbps at 120 KB a page. Twenty easy domains plus one hostile one. Threads and bandwidth sit far inside Starter, and the hostile target needs only about 10 exits alive at once, which the pool provides. Starter at $75, 100 threads, 5 whitelist slots. The one reason to move up is having more egress machines than whitelist slots.

SERP and marketplace collection, 20,000,000 pages a month

A 12-hour window is 15.4 req/s. At 3 seconds a request that is 46 threads, comfortably inside Starter. But at 250 KB a page the job needs 31 Mbps and Starter caps at 25, so bandwidth binds first. Core at $140 with 250 threads and 50 Mbps is the correct buy. Size on threads alone here and you spend a week debugging phantom slowness.

Rendered pages, 3,000,000 a month

An 8-hour window is 3.5 req/s. At 9 seconds per render that is 31 threads, trivial. At 2.5 MB per rendered page it is 69 Mbps, which needs Boost at $240 for its 100 Mbps ceiling. Strip images and fonts and the same job drops near 10 Mbps and fits Starter. Same volume, same deadline, a $165 monthly difference decided entirely by what your browser downloads.

Every SparkProxy plan is unlimited bandwidth with 30 days validity. The last column is arithmetic on the published caps under one stated assumption, 120 KB pages at 3 seconds each, so you can see which ceiling binds first.

The pattern holds on every tier: for light HTML work the megabit ceiling arrives before the thread ceiling. Flip to rendered pages and it inverts, since each request costs roughly 20x the bytes and 3x the time. Most vendor comparisons line up thread counts only, which is exactly why they mislead. See datacenter proxy pricing models for cost per successful request.

Whitelist slots are the quiet constraint. Slots are IP-authorised egress machines, so a Kubernetes deployment with 12 nodes on separate public IPs needs Core or higher regardless of throughput. Username and password auth avoids the issue entirely, and is the better default for anything that autoscales.

If you would rather not size threads at all, the Scraping API prices the same work in credits: 1 credit for a plain fetch, 5 for a JavaScript render, 10 for a screenshot or PDF.

Two honest reads. Under roughly a million plain pages a month credits usually win on total cost, because you are not paying for a browser fleet, rotation logic, or the engineer on call for them. Once you render millions of pages and already run headless browsers, self-managed proxies are cheaper per page. Budget 1.15x to 1.3x credits for retries either way, since a failed request still bills. Web scraping API versus self-managed proxies has the full trade-off.

Every number above depends on two measurements you can take in under an hour on a trial: how long your requests actually take, and how many bytes they actually move. Take them against your real targets, not against an example site.

import statistics, time, requests

GATEWAY = "http://USER:PASS@gateway.sparkproxy.io:11000"
PROXIES = {"http": GATEWAY, "https": GATEWAY}
TARGETS = ["https://your-target.example/page/1"]  # your real URLs

lat, size, ok = [], [], 0
for url in TARGETS * 100:
    t0 = time.perf_counter()
    try:
        r = requests.get(url, proxies=PROXIES, timeout=20)
        size.append(len(r.content))
        ok += r.status_code == 200
    except requests.RequestException:
        pass
    finally:
        lat.append(time.perf_counter() - t0)

n = len(lat)
print("p50 latency", round(statistics.median(lat), 2), "s")
print("p95 latency", round(sorted(lat)[int(n * 0.95)], 2), "s")
print("mean bytes ", int(statistics.mean(size)))
print("success    ", round(ok / n, 3))

Feed those four numbers straight into the sizing function:

def size_plan(pages_per_month, window_hours_per_month, p95_seconds,
              mean_bytes, success_rate):
    rps = pages_per_month / (window_hours_per_month * 3600) / success_rate
    threads = rps * p95_seconds * 1.3
    mbps = rps * mean_bytes * 8 / 1_000_000
    return round(rps, 2), int(threads + 1), round(mbps, 1)

print(size_plan(20_000_000, 360, 3.0, 250_000, 0.9))
# (17.15, 67, 34.3)  -> 67 threads and 34 Mbps: Core, capped by bandwidth

If you are sizing the Scraping API instead, the same calibration runs against https://scrape.sparkproxy.io/api/v1 with an X-API-Key header, and the number you record is credits per successful page rather than threads. A page that needs render=true costs 5 credits whether it succeeds or not, so measure your render-vs-fetch split before you pick a tier.

Use p95 latency rather than the median. Sizing on the median leaves you with a queue that grows through the run and a job that misses its window by morning. If your success rate is the term dragging the result down, fix that first: detecting when your scraper is blocked is a cheaper intervention than any upgrade.

Counting IPs on a rotating plan. There is no IP count to buy. Threads are what changes your invoice.

Sizing on the monthly total. A million pages a month is 0.4 req/s spread evenly and 4.6 req/s in a 2-hour window. Only one of those is on your bill.

Ignoring the speed cap because bandwidth is unlimited. Unlimited is about the invoice. Mbps is about whether the job finishes inside its window.

Buying threads to fix blocks. At a 60% success rate, more concurrency raises your block rate rather than your yield. Per-IP pacing, headers, and TLS behaviour are the fix.

Undercounting whitelist slots. Twelve worker nodes on separate public IPs will not fit into 5 slots. Buy the tier that fits, or switch to credential auth.

Buying for next year's peak. Plans run 30 days. Size for the next month, measure, then move.