Proxies for SERP Scraping at Scale

A single search results page is trivial to fetch. If you have never built one, the mechanics live in our walkthrough of how to scrape Google search results, and the IP-layer background sits in what a SERP scraping proxy is. This article assumes you already have a working single-page fetch and now need to run it 300,000 times a day without watching the success rate rot.

Three properties of search engines make volume harder than it looks on other targets.

The defence is per-IP and behavioural, not per-request. Most e-commerce sites decide on a single request: bad TLS fingerprint, bad header order, blocked. Search engines score an IP across a window. An IP issuing 8 queries an hour with human-shaped gaps can survive for weeks. That same IP issuing 80 queries in two minutes and then going silent draws a challenge around request 30 and a hard block soon after. The individual request was fine. The pattern around it was not.

Query diversity is itself a signal. Real users search related things. A scraper firing site: operators, quoted long-tail strings and 40 unrelated head terms from one IP in five minutes resembles nothing human. Two scrapers with identical volume can post very different block rates purely because of how queries were sharded across the pool.

Failure is correlated, not independent. When a subnet gets flagged, every IP inside it degrades together. Teams model block probability as an independent coin flip per request, then get blindsided when success drops from 94% to 40% inside ten minutes. Plan for the cliff, not the slope.

Almost every capacity plan starts wrong because it counts keywords instead of counting fetches. The real row count is a product of four dimensions, and the multiplier compounds fast.

With depth folded into a single request, 50,000 keywords becomes 300,000 SERP fetches per day. Paginate ten pages per keyword instead and the identical tracking scope becomes 3,000,000 fetches, a tenfold infrastructure bill for the same output. That one decision usually dominates every other optimisation in this article.

Two sizing errors show up in almost every real project:

1. Treating locales as cheap. Adding one country to a rank tracker adds a complete copy of the keyword set, not a rounding error. City-level tracking multiplies again by the number of cities.
2. Forgetting the re-check tail. Flagged results, ranking anomalies and failed parses all generate a second pass. Budget 5% to 15% on top of nominal daily volume for that tail.

Write the final number down. Call it D, the daily fetch count. Everything below derives from it.

Concurrency is the number of requests in flight at once. It is not a preference, it is a consequence of throughput and latency, and Little's Law computes it exactly:

L = λ × W

L = concurrency (requests in flight)
λ = throughput (requests per second)
W = average end-to-end latency per request (seconds)

Worked example with D = 300,000 fetches spread evenly across 24 hours:

λ = 300,000 / 86,400   =  3.47 requests/second
W = 4.0 s                 (fetch + parse, no JS rendering)
L = 3.47 × 4.0         =  13.9   →  round up to 16 workers

Sixteen. Not 500. Teams routinely provision an order of magnitude more concurrency than the arithmetic calls for, then wonder why they are burning through IPs. Excess concurrency does not raise throughput once you are network-bound. It compresses the same requests into shorter bursts and makes your traffic shape look more automated than it needs to.

Two adjustments matter in practice. First, you are rarely spreading evenly across 24 hours. If the business wants every rank captured inside a 6 hour overnight window, λ becomes 13.9 req/s and L becomes 56. Compressing the collection window is the most expensive product decision in rank tracking, and it is usually made without anyone doing this multiplication. Second, if you render JavaScript, W climbs from roughly 4 seconds to 10 or 15, and L climbs linearly with it. Our deep dive on concurrent connections in proxies covers how providers meter that number.

Here is the part most guides skip. Concurrency and pool size are different numbers with different drivers. You can run 16 concurrent workers across 4 IPs or across 4,000. What sets the pool size is how many requests a single IP can issue per hour before it draws a challenge.

IPs required = (λ × 3600) / R

R = safe requests per IP per hour

Using λ = 3.47 req/s, which is 12,492 requests per hour:

Read that table twice. The same job needs 21 IPs or 2,499 IPs depending on one parameter you control through pacing and IP quality. That is why "how many proxies do I need for SERP scraping" has no fixed answer, and why a bigger pool rarely fixes a block problem that pacing caused.

R is not a published constant, so measure it. Run a calibration: take 20 IPs, assign each a different fixed hourly rate from 5 to 300, run for six hours, and record where the challenge rate crosses 2%. That crossing point, minus 30% headroom, is your R for that target and that IP class. Re-measure monthly, because it moves.

One more constraint, easily missed: the pool must be at least as large as your concurrency. If L is 56 and you hold only 21 IPs, several requests are permanently sharing an IP simultaneously, which is a loud automation tell on search engines regardless of your hourly rate. Take the larger of the two numbers.

Say R is 20 requests per IP per hour. The naive implementation sleeps 180 seconds between requests on each IP. That is a metronome, and a metronome is the most machine-like signal you can emit. Inter-arrival times from real users are roughly exponential, so sample them that way.

import random

def next_gap(rate_per_hour: int) -> float:
    """Exponential inter-arrival gap that holds a target hourly rate."""
    mean_gap = 3600.0 / rate_per_hour
    return max(random.expovariate(1.0 / mean_gap), 8.0)

for _ in range(5):
    print(round(next_gap(20), 1))
# 41.6   212.3   95.8   388.1   73.2

The exponential distribution preserves your average rate while destroying the periodicity, and the max(..., 8.0) floor stops it occasionally firing two requests a second apart on the same IP. Two further refinements are worth the effort:

• Shard by theme, not round robin. Give each IP a contiguous slice of semantically related keywords. An IP searching "running shoes", "best running shoes 2026" and "asics gel kayano review" reads like one person with one intent. Round-robin assignment sends every IP on a random walk across your entire keyword universe, which is exactly what a human never does.
• Respect a diurnal curve. Traffic from a residential IP at 04:00 local time is unusual. Weight the scheduler toward the target locale's waking hours wherever the collection window allows it.

That per-IP discipline is the same principle covered in our guide to rotating proxies and per-IP request limits, applied to a target that scores behaviour far more aggressively than the average site.

If a fraction p of requests fail and you retry until success, the expected attempts per successful result is:

E[attempts] = 1 / (1 - p)

p = 0.05  →  1.05x
p = 0.12  →  1.14x
p = 0.30  →  1.43x
p = 0.50  →  2.00x

A 12% block rate looks tolerable at roughly 14% extra volume. The formula is misleading in one important way: it assumes retries are independent. They are not. If your retry lands on the same IP that just got challenged, its conditional failure probability is far above the pool average, and you can burn three attempts on a dead IP for every genuine failure. Three rules keep amplification honest:

1. Never retry on the failing IP. Move the request to a different IP, preferably a different subnet.
2. Cap attempts at 3, then push to a dead-letter queue. Unbounded retries turn a partial outage into a self-inflicted flood that burns the whole pool.
3. Back off globally, not just locally. When pool-wide success drops below 70% inside a 5 minute window, halve concurrency for 15 minutes. Per-request exponential backoff does nothing when the problem is systemic, because every worker independently decides to wait a little and then all of them resume at once.

At scale the economically correct answer is almost always a tiered pipeline rather than one type. Run the daily bulk sweep on datacenter IPs, detect which queries get challenged, and re-run only those on residential. If 85% of volume clears on datacenter, you pay residential rates on 15% of traffic instead of 100%. That split is the pattern described in the hybrid datacenter plus residential approach, and it is the biggest cost lever after request reduction.

Subnet diversity matters more than raw count. Two hundred IPs spread across 40 distinct /24 blocks outperform 2,000 IPs sitting in 3 blocks, because search engines act on network ranges when they act at all. Ask a provider for the subnet distribution, not just the pool size.

Every request you avoid is free, permanently. Work this list before touching the proxy budget.

• Fetch 100 results per request. Depth is the largest multiplier in the sizing table. Pulling positions 1 to 100 in one response instead of ten paginated responses is a 90% cut in fetches for full top-100 tracking.
• Turn off JavaScript rendering where the static HTML already carries the data. Organic result blocks are present in the raw HTML for most query types. Rendering costs both latency (roughly 4 seconds becomes 12) and, on a credit-metered API, five times the credits per call.
• Block images, fonts and media on the fetches that genuinely need a browser. That cuts transferred bytes hard, and bytes are the metered unit on per-GB residential plans.
• Split cadence by volatility. Volatile head terms deserve daily checks. A long tail of stable, low-volume terms is fine weekly. Splitting a 50,000 keyword set into 10,000 daily and 40,000 weekly cuts monthly fetches by roughly 62% with almost no loss of signal.
• Deduplicate before dispatch. Identical query, locale and device tuples requested twice in the same window are pure waste. A Redis set keyed on sha1(query|locale|device|date) catches them for the cost of one round trip.
• Cache negative results. A query returning zero results does not need re-checking six times a day.

The SparkProxy Scraping API handles proxy selection, rotation and rendering behind one endpoint, which removes most of the pool bookkeeping above. The base endpoint is https://scrape.sparkproxy.io/api/v1 and it authenticates with an X-API-Key header.

A single geo-targeted SERP fetch with no rendering:

curl -G "https://scrape.sparkproxy.io/api/v1" \
  -H "X-API-Key: sk-YOUR_API_KEY" \
  --data-urlencode "url=https://www.google.com/search?q=running+shoes&num=100&gl=us&hl=en" \
  --data-urlencode "render_js=false" \
  --data-urlencode "country_code=us" \
  --data-urlencode "json_response=true" \
  --data-urlencode "tag=rank-daily-2026-08-18"

The tag parameter accepts up to 128 characters and is worth setting on every request. It is how you attribute credit spend to a specific job when several collection pipelines share one key.

Structured extraction avoids shipping full HTML back to your parser:

curl -X POST "https://scrape.sparkproxy.io/api/v1" \
  -H "X-API-Key: sk-YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://www.google.com/search?q=proxy+api&num=100&gl=gb&hl=en",
    "render_js": false,
    "country_code": "gb",
    "extract_rules": {
      "titles": "h3",
      "result_links": "div.g a @href"
    }
  }'

Now the part that matters at volume. The API enforces 60 requests per minute and a concurrency limit of 3 parallel requests on the default tier, returning HTTP 429 with retry_after_seconds and an active count. Your client should obey both limits rather than fight them:

import asyncio, httpx

API = "https://scrape.sparkproxy.io/api/v1"
HEADERS = {"X-API-Key": "sk-YOUR_API_KEY"}
CONCURRENCY = 3                      # match your plan's parallel limit
sem = asyncio.Semaphore(CONCURRENCY)

async def fetch_serp(client, query: str, gl: str = "us", attempt: int = 1):
    params = {
        "url": f"https://www.google.com/search?q={query}&num=100&gl={gl}&hl=en",
        "render_js": "false",
        "country_code": gl,
        "json_response": "true",
        "tag": "rank-daily",
    }
    async with sem:
        r = await client.get(API, headers=HEADERS, params=params, timeout=90)

    if r.status_code == 429:                       # throttle, not a failure
        wait = float(r.json().get("retry_after_seconds", 5))
        await asyncio.sleep(wait)
        return await fetch_serp(client, query, gl, attempt)

    if r.status_code == 530 and attempt < 3:       # target error or timeout
        await asyncio.sleep(2 ** attempt)
        return await fetch_serp(client, query, gl, attempt + 1)

    r.raise_for_status()
    return r.json()

async def main(keywords, gl="us"):
    async with httpx.AsyncClient() as client:
        tasks = [fetch_serp(client, k, gl) for k in keywords]
        return await asyncio.gather(*tasks, return_exceptions=True)

Two details there are load-bearing. A 429 is a throttle, so it must not consume one of your three attempts, otherwise a busy minute looks like a failed keyword. A 530 means the scrape itself failed against the target and is the status worth backing off on. A 500 refunds credits, so it costs you time rather than money.

For large batches, the webhook path removes the need to hold connections open at all. Supply callback_url and the API returns 202 immediately, then POSTs the result when the job finishes:

curl -G "https://scrape.sparkproxy.io/api/v1" \
  -H "X-API-Key: sk-YOUR_API_KEY" \
  --data-urlencode "url=https://www.google.com/search?q=serp+api&num=100" \
  --data-urlencode "render_js=false" \
  --data-urlencode "callback_url=https://hooks.sparkproxy.io/serp-results" \
  --data-urlencode "tag=batch-0818"

The webhook payload carries job_id, result_url, status_code, duration_ms and credits_used, which is enough to reconcile spend per job without a separate billing query. At 300,000 fetches a day the webhook pattern also flattens your memory profile, since nothing is waiting on an open socket for 4 seconds at a time.

Model in credits, then map credits onto your plan. Per the API's published pricing, base costs are 1 credit for a rotating proxy without JS, 5 credits with JS, 10 for a premium proxy without JS and 25 for premium with JS, plus 5 credits each for country_code, stealth or a js_scenario.

The spread between the cheapest and most expensive row is 35x for the same million pages. That is why the tiered pipeline pays for itself immediately. If 85% of queries succeed on the 6 credit configuration and 15% need the 35 credit one, your blended cost is 10.35 credits per request, about 70% below running everything premium.

Two structural savings are easy to miss:

• Apply geo only where it changes the answer. Add country_code for locale-sensitive queries and drop it everywhere else. On a 1M page job that one flag is worth 5,000,000 credits.
• Batch the no-JS calls. Multiple comma-separated URLs sent with render_js=false cost 1 credit total for the batch and return a results array with per-URL status, HTTP status and body. For high-volume, no-render SERP collection this is the single largest lever in the whole pricing table.

For a self-managed residential pool the metered unit is bandwidth rather than credits:

GB = (pages × avg_page_KB) / 1,048,576

1,000,000 pages × 320 KB  =  305 GB
Monthly cost = 305 × your $/GB rate

At an illustrative $3/GB that is roughly $915 per million pages, before counting the engineering time to run pool health, pacing and rotation yourself. Blocking images and stylesheets and skipping rendering are what hold the average page near 320 KB instead of 900 KB, and that ratio maps straight onto the invoice.

Track these per hour, per IP class and per target locale. Aggregate averages hide precisely the failures that matter.

The fourth one is the metric nobody instruments and the one that costs the most. A search engine will sometimes return a valid 200 with a stripped-down result set instead of blocking outright. Your success rate stays at 99%, your parser stays green, and your rank data quietly becomes wrong. Assert on shape, not just on status:

def is_degraded(results: list, expected: int = 100) -> bool:
    """A 200 carrying 12 results where 100 were requested is a soft block."""
    return len(results) < expected * 0.5

# Alert when the degraded fraction crosses 2% in a rolling hour.

Pair that with a control group: 50 stable keywords whose top-10 you already know, re-checked hourly from a separate IP class. When the control group drifts, the problem is your collection rather than the rankings. This is standard practice in datacenter proxy rank tracking setups, and it catches silent data corruption that no HTTP-level monitor will ever see.

Run the four numbers as a weekly review alongside your credit spend. Block rates drift, R drifts, and page weights drift. A capacity model built once and never re-measured is wrong within a quarter.