How to Reduce Proxy Costs Without Losing Success Rate

There is exactly one number worth optimizing, and it is not gigabytes. It is cost per successful record.

cost per record = monthly proxy spend / records successfully parsed

Teams that optimize gigabytes end up blocking assets a target site checks for, watching success rate slide, and celebrating a smaller bill while collecting less data. Cost per record catches that immediately, because the denominator moves too.

Before touching anything, pull three figures for the last full billing period:

Divide spend by records. A price monitoring job pulling 400,000 pages a month for $1,100 runs at $0.00275 per record. Once that number exists, every change below is measurable in a day instead of arguable in a meeting.

Read the invoice line by line, not just the total. Three charges hide below the base subscription: overage billed above your contracted rate, add-ons such as extra sub-users or static IP reservations, and a second product someone spun up for a proof of concept and never cancelled. On plenty of bills the fastest saving available is switching off a line nobody has queried in months.

If your bill is per gigabyte, work out your average transferred page weight too: total GB divided by requests billed. Most teams guess low by a factor of three. Our explainer on what bandwidth actually means in proxy services covers why the number your browser reports and the number your provider meters rarely match.

Ranked by how much a typical scraping workload saves against how long the change takes to ship.

Do levers 1 and 4 this week. Lever 1 is a purchasing decision and needs no code. Lever 4 usually pays for itself inside a sprint, and it tends to raise success rate rather than lower it.

Proxy vendors bill three ways, and they are not interchangeable. Buying the wrong one is the most expensive mistake in this category.

The break-even between per-gigabyte and flat-rate unlimited is simple arithmetic: take the monthly price of the flat plan and divide it by your contracted per-GB rate.

SparkProxy sells datacenter proxies per thread with unlimited bandwidth on a 30 day term:

Higher Pro and Pro+ tiers exist in the Fair Usage Policy at 1500 and 2000 threads, with 200 and 250 Mbps caps, and custom capacity goes to 1 Gbps. Those are quoted rather than list priced. The speed figure is a ceiling under the fair usage policy, not a guaranteed throughput rate.

Run the break-even against whatever your current per-GB rate happens to be:

Per-GB list prices differ sharply between vendors and tiers, and they move. Check the current published rate on your vendor's own pricing page rather than trusting a figure from a blog post, this one included. As of September 2026 the spread across the major residential networks is wide enough that the table above is only useful with your real contracted rate in it.

The honest caveat: this works only if datacenter IPs still pass on your targets. They will not on the hardest consumer sites. The winning structure for most teams is a split, with datacenter carrying the bulk of the catalog and residential reserved for the endpoints that genuinely need it. We cover that split in the hybrid datacenter and residential approach and in residential vs datacenter proxies.

Connection details for the datacenter gateway, if you want to test the split: gateway.sparkproxy.io on port 11000 for HTTP and HTTPS, port 11002 for sticky sessions, port 13000 for SOCKS5.

One trap in this lever: annual prepayment. A discount for paying twelve months up front is real money, and it is also the fastest way to lock in a billing model you have not proven against your own targets. Buy monthly until you have two billing periods of cost-per-record data, then commit.

On per-GB billing, the fetch path you choose changes the bill by more than an order of magnitude for the same 100,000 pages. The table below is arithmetic, not a benchmark. Substitute your own measured page weight before you act on it.

The span between the top and bottom rows is roughly 24x for identical extracted output. That is the whole argument for payload discipline.

Three changes carry most of the saving.

Block asset classes in the browser. Images, fonts, media and stylesheets are usually pure cost. Playwright makes this a short route handler:

BLOCK = {"image", "media", "font", "stylesheet"}

async def prune(route, request):
    if request.resource_type in BLOCK:
        await route.abort()
    else:
        await route.continue_()

await page.route("**/*", prune)

Test per target before shipping it. Some anti-bot vendors score whether a client fetched the expected stylesheet, and blocking too aggressively converts a bandwidth saving into a block. Start with images and media only, measure success rate for a day, then add fonts and stylesheets if nothing moves.

Cap the response body. If the data you need sits in the first 400 KB, do not download 6 MB of tracking payload:

import requests

PROXY = "http://USER:PASS@gateway.sparkproxy.io:11000"

with requests.get(
    url,
    proxies={"http": PROXY, "https": PROXY},
    headers={"Accept-Encoding": "gzip, br"},
    stream=True,
    timeout=20,
) as r:
    body = r.raw.read(400_000, decode_content=True)

You still pay for whatever crossed the wire before the socket closed, so the saving is partial. Always send an Accept-Encoding header offering gzip and brotli: scrapers built on raw socket libraries often request uncompressed HTML and pay several times the bytes for it.

Go to the JSON endpoint instead of the page. The largest single win here is usually not compression, it is skipping the HTML entirely. A listing page that renders at 1.4 MB is often backed by an internal API returning 30 KB of JSON with the same fields, better structured. Finding hidden JSON API endpoints is worth a day of DevTools work per target site, and it removes parsing fragility at the same time.

On credit-based scraping APIs, JavaScript rendering is priced as a multiple of a plain fetch, and rendering by default is how teams burn a monthly quota in nine days. The SparkProxy Scraping API charges 1 credit for a plain fetch, 5 for a JavaScript render, and 10 for a screenshot or PDF. Applied to the $99 Growth plan, which includes 1,000,000 credits a month and 100 concurrent requests:

If only 15% of a catalog genuinely needs rendering, the mixed rate works out at 1.6 credits per page, so rendering everything costs about 3.1x more than it should. The fix is a per-target flag rather than a global default:

from urllib.parse import urlparse

NEEDS_JS = {"target-a.example", "target-b.example"}

def fetch_mode(url: str) -> dict:
    host = urlparse(url).hostname or ""
    return {"render": "true"} if host in NEEDS_JS else {}

Populate NEEDS_JS empirically. Fetch each target once without rendering, check whether the fields you need are already in the raw HTML, and add the host only when they are genuinely absent. Re-run that audit quarterly: sites move content into and out of client-side rendering more often than scraper owners expect.

If your bill is mostly render spend, run the comparison of a scraping API against self-managed proxies. A managed API costs more per request and less per engineer-hour, so the winner depends on how many targets you maintain. The 1,000 free credits, no card required, are enough to price your own workload first.

A block page is a payload. It crossed the wire, so it is metered on per-GB plans, and on many credit-based plans a non-200 status still counts unless the vendor explicitly says otherwise. Ask that question in writing before you sign, because vendors differ and the policy is rarely on the pricing page.

Blind retry loops are where this gets expensive. Model 100,000 requests with a 12% failure rate and up to three retries:

The third row assumes those 12,000 failures split into 50% transient (retried up to three times, 70% recovering on each attempt), 30% hard 403s (escalated once), and 20% permanent 404s (never retried). Every row is arithmetic from stated assumptions, not a measurement of any network. Plug in your own split and the ranking holds.

The rule is that a retry must change something. Retrying the same URL through the same IP type with the same headers after a 403 will fail three more times and bill four times.

def retry_plan(status: int, body: bytes) -> str:
    if status in (429, 503):
        return "backoff"      # transient, wait and retry same config
    if status == 403 or b"Access Denied" in body[:2000]:
        return "escalate"     # change IP type or session, retry once
    if status == 404:
        return "drop"         # never retry
    return "backoff" if status >= 500 else "drop"

Cap attempts per URL at two escalations rather than an unbounded loop, and log the terminal status so you can tell a target that changed its defenses from one that never had the data.

Retry and backoff strategies covers the timing side, and detecting when your scraper is blocked covers the harder case: a soft block returning HTTP 200 with an empty result set. Soft blocks are the quiet cost sink here, because they never trigger a retry and they silently poison your records-parsed denominator.

The cheapest request is the one you skip. Most catalogs change far less than the crawl schedule assumes. A daily full re-crawl of a 400,000 page catalog where 3% of pages changed pays 97% of the bill for identical bytes.

Conditional requests are the low effort version. Store the ETag or Last-Modified value with each record and send it back:

etag = store.get(url)
headers = {"If-None-Match": etag} if etag else {}

r = requests.get(url, headers=headers, proxies=PROXIES, timeout=20)
if r.status_code == 304:
    return None    # a few hundred bytes instead of a full page
store.put(url, r.headers.get("ETag"))

Test this per target. Sites behind an aggressive edge or bot-management layer often strip or randomize validators, which turns every conditional request back into a full fetch. Where it works, the saving on unchanged pages is close to total.

The higher effort version is tiering crawl frequency by observed volatility. Track how often each URL's extracted fields actually change, then crawl the volatile 10% daily and the stable tail weekly or monthly. Incremental scraping and change detection walks through the bookkeeping. On a large stable catalog this is frequently the biggest single saving available, larger than every payload optimization combined, because it removes requests instead of shrinking them.

Where a target maintains real lastmod values, one sitemap fetch tells you which of 400,000 URLs are worth pulling. Spot-check the values first, because plenty of sites stamp every entry with today's date.

Price per unit is the item vendors defend hardest, and it is rarely the item worth pushing on. These five asks cost the vendor less and are worth more to you.

Send those in one message, in writing, before you discuss price at all. A provider that will not put its failed-request policy in an email has told you what the policy is.

Bring a number. "We are at 340 GB a month, next quarter forecasts at 500, here is our current effective rate" gets a different answer than "can you do better on price." Volume commitments do lower per-unit rates, but only when the vendor can see the volume.

Four cuts look like savings on the invoice and are not.

Do not cut concurrency below your deadline. Threads and bandwidth are separate resources. Dropping from a 500 thread plan to 100 to save $165 a month reduces no bytes at all, it just makes the same job take five times longer, and a job that misses its window carries a cost that dwarfs the saving. Concurrent connections in proxies explains how to size threads against target latency rather than against price.

Do not move to free proxy lists. Poor uptime, unknown operators sitting inside your TLS session, and IPs already burned on your targets. The success rate collapse alone makes cost per record worse.

Do not drop geo accuracy to buy a cheaper pool. If you collect localized pricing, an IP that geolocates to the wrong country returns clean HTTP 200 responses full of wrong data. That is worse than a block: a block is visible, bad data is not.

Do not remove monitoring to save requests. Canary fetches are a rounding error on any real bill, and they are what tells you a target changed before a week of records comes back empty.

Work through this before the next invoice, in order. It takes an afternoon.

1. Compute cost per successful record for the last full billing period.
2. Split spend by target domain. Most bills are dominated by two or three sites.
3. For the top domain, measure real transferred page weight and compare it against the fetch-path table above.
4. Check whether datacenter IPs pass on that domain. If they do, price the flat-rate break-even against your per-GB rate.
5. Count non-200 responses and retry attempts. Anything above roughly 15% of billed requests is a retry problem, not a proxy problem.
6. Ask the vendor in writing: are failed requests billed, does unused quota roll over, what is the notice period, what happens at the fair usage ceiling.
7. Test any replacement provider on your hardest target, not on a synthetic echo endpoint, and run it for at least 48 hours to catch time-of-day variation.

Point 7 is where most switches go wrong: a pool that looks identical on a synthetic endpoint behaves very differently against a real anti-bot stack at 09:00 on a weekday. For how pricing structures differ between vendors, see datacenter proxy pricing models.

The short version: fix the billing model first because it costs nothing to change, fix retries second because the fix is small and usually raises success rate, then work on payload and caching with per-target measurement at every step. Optimizing bytes before checking whether you are on the right plan is doing the hard work in the wrong order.