How to Forecast Proxy Bandwidth Before You Buy

Take a sample of 200 real pages from your real targets through a real proxy, record the Content-Length of each response rather than the length of the decoded body, multiply by your requests per record and your record count, add a retry tax of one over your success rate, and round up to the next plan tier. That is the entire method, and it takes about an hour.

Skip the sample and you will do what most buyers do: pick a tier that sounds reasonable, blow through it in week three, and pay overage rates that are always worse than the tier you should have bought.

If the answer comes back volatile, meaning your worst plausible month is more than about twice your median month, stop forecasting and buy flat. Predictability has a price and it is usually lower than the cost of being wrong. SparkProxy's proxy plans are built for that case: unlimited bandwidth on every tier, priced by concurrent threads instead, from $75/mo at 100 threads up to $440/mo at 1000.

Everything below is one equation with five terms, four of which you can measure in an afternoon.

monthly_bytes = P * Q * S * (1 + r) * (1 + a)

P = records or pages you actually want, per month
Q = requests per record (pagination, redirects, API calls behind the page)
S = average COMPRESSED response size in bytes
r = retry rate, expressed as a fraction (0.25 means 25% of attempts are repeated)
a = subresource multiplier, 0 for plain fetches, much higher when rendering

monthly_GB = monthly_bytes / 1073741824

Buyers reliably get three of these wrong in the same direction. They use decompressed size for S, they set Q to 1, and they set r to 0. All three errors push the estimate down, which is why the first invoice is always a surprise and never a pleasant one.

The sections below fix each term in turn.

This is the error that costs the most and is the least known.

Almost every site serves HTML with gzip or brotli compression. Your proxy provider meters what crosses the wire, which is the compressed body plus protocol overhead. Your HTTP client, helpfully, decompresses the response before you see it. So len(response.content) in Python or response.text.length in Node reports the decompressed size, which for text-heavy HTML is substantially larger than what you were billed for.

Forecast with the decompressed figure and you over-buy. Forecast with an uncompressed sample from a site that does not compress and you under-buy on the ones that do. Either way you are estimating the wrong quantity.

Measure both and look at your own ratio:

import requests

PROXY = "http://USER:PASS@gateway.sparkproxy.io:11000"
proxies = {"http": PROXY, "https": PROXY}

wire_total = 0
decoded_total = 0
sampled = 0

for url in sample_urls[:200]:
    try:
        r = requests.get(url, proxies=proxies, timeout=20)
    except requests.RequestException:
        continue
    # Content-Length is the compressed body as sent. Missing on chunked responses.
    wire = int(r.headers.get("Content-Length", 0))
    if wire == 0:
        continue
    wire_total += wire
    decoded_total += len(r.content)
    sampled += 1

print(f"sampled: {sampled}")
print(f"avg wire bytes:    {wire_total / sampled:,.0f}")
print(f"avg decoded bytes: {decoded_total / sampled:,.0f}")
print(f"compression ratio: {decoded_total / wire_total:.2f}x")

Run that against your real targets. We are deliberately not printing a ratio here for you to copy, because it varies enormously by site and by content type, and a number you did not measure is exactly the kind of assumption that produces a wrong forecast. Measure it. It takes four minutes.

Two caveats the code comments hint at. Content-Length is absent on chunked transfer responses, so the sample above skips them, and if most of your targets are chunked you will need to count at the socket or use a proxy-side traffic report instead. And the headers themselves, plus TLS handshake overhead, are real bytes that the header value does not include. On small responses that overhead is a meaningful fraction, which is one reason vendor meters read slightly higher than a body-only calculation.

Our explainer on what bandwidth means in proxy services covers what different vendors include in the meter.

Q is almost never 1. Walk through what one "record" actually costs you.

A search or category page to find the item. Often several, because of pagination. A redirect, which is a separate request with its own response body, small but real. The detail page itself. Any XHR or API call the page makes that you also need. A retry of any of the above. Possibly a robots.txt fetch if your crawler is polite, though that is cheap and cacheable.

The reliable way to find Q is not to reason about it. It is to instrument one full record end to end and count:

from collections import Counter

counts = Counter()

def tracked_get(url, kind):
    counts[kind] += 1
    return requests.get(url, proxies=proxies, timeout=20)

# ... run your real extraction for exactly one record, tagging each call ...

total = sum(counts.values())
print(f"requests per record: {total}")
for kind, n in counts.most_common():
    print(f"  {kind}: {n}")

Do this for ten records across your different target types and take the mean. The distribution matters as much as the average: if one target type needs 14 requests per record and the others need 3, that target type deserves its own line in the forecast rather than being averaged away.

One more thing to check while you are in there. If the data you need arrives in a JSON endpoint the page calls, you may be able to skip the HTML entirely and cut both Q and S at the same time. That single discovery is worth more than any vendor negotiation.

Failed requests transfer bytes. A 403 page has a body. A CAPTCHA challenge has a body, often a large one with scripts attached. A timeout may have transferred most of a response before it gave up. On a per-gigabyte plan, every one of those is billed at the same rate as a successful page.

So the retry multiplier is not optional and it is not small. If your success rate is 80%, you make 1.25 attempts per successful record, which is a 25% surcharge on your entire bandwidth forecast. At 60% success it is 67%.

The nasty part is that this term is the least stable in the whole formula. Your page weight will not change much month to month. Your success rate can halve overnight when a target deploys a new bot defence, and it takes your bandwidth bill with it. Measure your current success rate with the method in what is proxy success rate and how to measure it, then forecast with a worse one than you measured.

Also make sure your retry logic is not making this worse than it needs to be. Retrying a 403 immediately, three times, at full page weight, is a way of paying triple for the same block. Retry and backoff strategies covers the patterns that fail cheaply.

A plain fetch pulls one document. A headless browser pulls the document plus every stylesheet, script, font, image, tracking pixel and video poster the page references. The multiplier between those two is the a term in the formula, and on a modern commercial page it is not a rounding error.

Do not guess it. Measure it, and measure it with and without resource blocking, because blocking is the lever:

from playwright.sync_api import sync_playwright

BLOCKED = {"image", "media", "font", "stylesheet"}
bytes_seen = 0

def on_response(response):
    global bytes_seen
    try:
        bytes_seen += int(response.headers.get("content-length", 0))
    except (TypeError, ValueError):
        pass

with sync_playwright() as p:
    browser = p.chromium.launch(proxy={
        "server": "http://gateway.sparkproxy.io:11000",
        "username": "USER",
        "password": "PASS",
    })
    page = browser.new_page()
    # Comment this line out to measure the unblocked baseline.
    page.route("**/*", lambda route: route.abort()
               if route.request.resource_type in BLOCKED else route.continue_())
    page.on("response", on_response)
    page.goto(TARGET_URL, wait_until="networkidle")
    print(f"bytes: {bytes_seen:,}")
    browser.close()

Run it twice, once with the route handler and once without, on ten representative pages. The gap between the two numbers is the largest single bandwidth lever available to you, and it costs one line of code to take.

Before you accept rendering as necessary at all, check whether you need it. Fetch the page plain, search the HTML for one of the values you are extracting, and if it is there, you never needed a browser. On a credit-based API the same decision shows up as price directly: on SparkProxy's Scraping API a plain fetch costs 1 credit and a JavaScript render costs 5, so establishing that render_js=false works is an 80% cut.

curl -G "https://scrape.sparkproxy.io/api/v1" \
  -H "X-API-Key: YOUR_API_KEY" \
  --data-urlencode "url=https://example.com/product/4471" \
  --data-urlencode "render_js=false" \
  --data-urlencode "premium_proxy=true" \
  --data-urlencode "country_code=US" \
  -D headers.txt -o plain.html

The X-Credits-Used header in headers.txt tells you what that fetch cost, and the 1,000 free credits are plenty to answer the rendering question across all your target types. Scraping API credit pricing has the full cost table.

You now have a number. Converting it into a purchase means knowing what a gigabyte costs on each billing model, and the spread is wide.

Published rates, read 23 September 2026 on each vendor's own page:

Two things to take from that table. Datacenter bytes are cheap and residential bytes are not, so the same forecast produces wildly different bills depending on which exit type your targets force you into. And the per-GB rate falls with commitment on every vendor that publishes a ladder, which means your forecast is also a negotiating position: under-forecast and you buy at the top of the ladder.

The break-even against a flat plan is simple arithmetic. SparkProxy Starter at $75/mo with unlimited bandwidth is worth it against a $0.50/GB datacenter plan above 150 GB a month, and against a $2.25/GB residential plan above 33 GB. Below those lines, metered is cheaper. Above them, flat is cheaper and also stops being a forecast at all. Are unlimited bandwidth proxies worth it works through that trade in more detail, and residential proxy pricing per GB covers the expensive half of the table.

A single number is a wish. Produce three and you have a decision.

The median month. Your measured page weight, your measured requests per record, your measured success rate, your planned volume. This is the number you quote to finance.

The bad month. Same volume, but success rate drops by a third and one target starts requiring rendering. This is the number you size the plan against, because it is a Tuesday away at all times.

The growth month. Your median month at whatever volume the roadmap promises in six months. This tells you whether the tier you are about to buy has a future or whether you are about to renegotiate in a quarter.

If the bad month is more than roughly twice the median month, metered pricing is a poor fit for your workload regardless of the headline rate, because you will either buy for the median and get burned or buy for the bad month and waste money most of the time. That volatility ratio, not the per-gigabyte price, is the real signal for choosing a billing model. Our breakdown of datacenter proxy pricing models has the wider argument.

Five things, in rough order of how often they show up.

A target adds a bot defence. Success rate falls, retries rise, bandwidth rises with them. This is the most common cause and the hardest to predict.

Scope creep in the extraction. Somebody asks for one more field, that field lives on a sub-page, and Q quietly goes from 3 to 5. A 67% bandwidth increase arrives with no architectural change anyone would think to flag.

A target starts requiring JavaScript. The a term switches on for a whole source. This is the single largest step change available in the formula.

Someone turns off resource blocking. Usually while debugging a rendering issue, usually not turned back on. Put the blocklist in code with a test around it, not in a config someone can edit at 2am.

Volume grows where you were not watching. The pipeline picked up 40% more sources because a source-discovery job worked better than expected. Good news, unbudgeted.

Every one of these is invisible on a flat unlimited plan and immediately expensive on a metered one. That asymmetry is the whole argument for flat pricing on volatile workloads, and the whole argument against it on stable ones.

Illustrative, using stated assumptions rather than a test we ran. Substitute your own measured values at every step.

A price-monitoring job covering 60,000 products, refreshed daily.

monthly_bytes = 1,800,000 * 1.4 * 42,000 * 1.18 * 1.0
              = 124,891,200,000 bytes
monthly_GB    = 124,891,200,000 / 1,073,741,824
              = 116.3 GB

So roughly 116 GB a month in the median case. Now the other two forecasts. In the bad month, success drops to 60%, giving a retry multiplier of 1.67 instead of 1.18, and one source of 8,000 products starts requiring rendering. That pushes the estimate well past 160 GB before the render overhead is even counted. In the growth month at 100,000 products, the median rises to about 194 GB.

Against a $0.50/GB datacenter rate, those three scenarios are roughly $58, $80-plus and $97 a month. Against a $2.25/GB residential rate they are $262, $360-plus and $437. Against SparkProxy Starter at $75 flat with unlimited bandwidth, they are $75, $75 and $75.

Which tells you the real decision. If this workload can run on datacenter exits, metered is competitive and you should shop on rate. If it needs residential exits, the flat plan is not just cheaper, it removes the forecast from your life entirely. And the forecast was only ever necessary because someone was going to bill you for bytes.

• Sampled at least 200 real pages from real targets through a real proxy, not from your laptop.
• Recorded Content-Length rather than decoded body length, and noted what fraction of responses were chunked and therefore skipped.
• Measured Q on at least ten full records, per target type, not averaged across all of them.
• Measured success rate, then forecast with a worse one.
• Confirmed for every target whether rendering is genuinely required, by grepping the plain HTML for a value you need.
• If rendering is required, measured the byte cost with and without resource blocking.
• Produced three forecasts: median, bad, growth.
• Checked the volatility ratio. If bad is more than twice median, priced a flat plan before a metered one.
• Asked the vendor two questions in writing: what exactly does the meter count, and what is the overage rate.

That last one catches people. Overage is where a cheap headline rate becomes an expensive invoice, and it is rarely on the pricing page. How much do proxies cost covers the wider price landscape once your forecast is done.