How to Scrape Google Search Results Without Blocks

Google doesn't block you for scraping one page. It blocks you for looking like automation across many pages. Three signals do most of the work.

Rate and cadence. A single IP firing 60 identical queries a minute at fixed intervals is the clearest bot tell there is. Humans pause, retype, and vary their timing. Uniform cadence gets flagged faster than a high raw count.

IP reputation. Requests from cloud ASNs (AWS, GCP, Azure) start with a worse reputation than residential ISP addresses. Datacenter IPs without careful header work hit "unusual traffic" screens quickly, while residential and mobile IPs last longer because they look like real subscribers.

Request fingerprint. The TLS handshake (JA3/JA4), HTTP/2 header order, and the Sec-CH-UA client hints all have to match a real Chrome build. A bare requests.get() sends a fingerprint no browser produces, so the IP barely matters once that mismatch is visible.

When enough signals stack up, Google returns one of three things: HTTP "429 Too Many Requests", a "Our systems have detected unusual traffic from your computer network" page (HTTP "200" with no results), or a reCAPTCHA challenge. All three mean the same thing: slow down and look more human. The full detection surface and how to counter each signal is covered in how to avoid getting your proxy blocked.

Scraping the public search results page sits in contested territory, so set your own guardrails before you write a line of code.

• Only collect public SERP data. Query results as an anonymous user sees them. Don't scrape logged-in, personalized, or paywalled surfaces.
• Respect rate limits and back off. If Google returns a "429" with a Retry-After header, honor it. Ignoring back-off signals is both rude and self-defeating; it accelerates the block.
• Know the Terms of Service. Google's ToS restrict automated access outside its own interface. Scraping public data is not a criminal act in most jurisdictions, but it can breach the contract, so weigh the risk for your use case.
• Prefer the official API when it fits. Google's Programmable Search Engine JSON API gives 100 free queries a day, then $5 per 1,000 up to 10,000/day. It's a real google search results api, just narrower and pricier at scale than scraping the live SERP. For your own site's ranking data, Search Console API carries zero ToS risk.

If your goal is rank tracking or market research, keep volume proportional to need and cache aggressively. Two teams pulling the same 5,000 keywords hourly is wasteful; most rank data only needs a daily or weekly refresh.

The SERP is just a URL with query parameters. Get these right and you control language, region, device, and pagination without any hidden state.

A complete URL for US English results looks like this:

https://www.google.com/search?q=running+shoes&hl=en&gl=us&num=10

The num=100 caveat (this is where old tutorials break). For years, scrapers set num=100 to pull 100 organic results in one request and cut request count by 10x. In September 2025 Google stopped honoring num=100 reliably; it now caps most responses near 10 results regardless of the parameter. If a guide tells you to grab 100 results per page, it predates that change. Plan for 10 results per request and paginate with start (0, 10, 20, ...) instead. Budget your credits and pacing around that reality, because it roughly 10x'd the request count for deep-SERP collection overnight.

Cadence is the single biggest lever you control. The goal is variance, not just delay.

import random
import time

def human_delay(base=25, jitter=0.5):
    """Sleep base seconds ± 50%, drawn uniformly. Never a fixed interval."""
    low = base * (1 - jitter)
    high = base * (1 + jitter)
    time.sleep(random.uniform(low, high))

A fixed time.sleep(25) between queries is itself a fingerprint: real users never hit the millisecond-perfect interval. The random.uniform version above spreads each gap between roughly 12 and 38 seconds, which reads far more like a person refining searches.

Two more pacing rules that matter in production:

• Cap queries per IP. For residential IPs, keep it near 100 to 200 queries per IP per day spread across hours. Above that, CAPTCHA rates climb steeply.
• Randomize order and refinements. Don't send 500 keywords in perfect alphabetical order at a steady beat. Shuffle the queue and vary start depth so no two sessions look identical.

If you're managing your own proxy pool, you own all of this. If you use a scraping API, you offload IP rotation and cadence-per-IP to the provider and only control your own concurrency and retry logic, which is most of why teams switch. That trade is broken down in web scraping API vs self-managed proxies.

Google localizes results heavily. A query for "plumber" or "best bank" returns completely different SERPs in London versus New York. If your data has to reflect a specific market, the request has to originate from that market, and the URL has to declare it.

Two layers work together:

1. Exit IP location. Route the request through a proxy in the target country so Google sees local traffic. Without this, gl alone gives you an approximation, not the real local SERP.
2. URL geo parameters. Set gl to the country and, for city-level precision, encode a uule location string. Keep hl consistent with the market's language.

Mismatched layers produce garbage: a UK exit IP sending gl=us returns a blend that matches neither market. Keep the IP country, gl, and hl aligned. This is exactly the setup rank trackers rely on, covered in datacenter proxies for SEO rank tracking.

Instead of assembling proxies, headers, TLS fingerprints, and a headless browser yourself, you can send one request to the SparkProxy Scraping API. It rotates the proxy, renders the page, and returns the SERP HTML. You pass the Google search URL as the url parameter and authenticate with the X-API-Key header.

A minimal cURL request for a geo-targeted, residential-routed SERP:

curl -G "https://scrape.sparkproxy.io/api/v1" \
  -H "X-API-Key: YOUR_API_KEY" \
  --data-urlencode "url=https://www.google.com/search?q=running+shoes&hl=en&gl=us&num=10" \
  --data-urlencode "render_js=true" \
  --data-urlencode "country_code=us" \
  --data-urlencode "premium_proxy=true"

The same call in Python:

import requests
import urllib.parse

query = "running shoes"
serp_url = "https://www.google.com/search?" + urllib.parse.urlencode({
    "q": query,
    "hl": "en",
    "gl": "us",
    "num": 10,
})

resp = requests.get(
    "https://scrape.sparkproxy.io/api/v1",
    headers={"X-API-Key": "YOUR_API_KEY"},
    params={
        "url": serp_url,
        "render_js": "true",       # run a real browser so SERP features hydrate
        "country_code": "us",      # exit through a US IP
        "premium_proxy": "true",   # residential pool, harder to block
    },
    timeout=90,
)
resp.raise_for_status()
html = resp.text

Each parameter maps to one of the blocking signals from earlier. render_js=true runs Chromium so JavaScript-driven SERP features load and the request carries a real browser fingerprint. country_code=us fixes the exit location. premium_proxy=true routes through residential IPs that survive Google's reputation checks. On the toughest days, add stealth=true, which layers a homepage pre-warm, a forced Google referrer, and longer idle delays on top.

Watch the credit cost, because these stack: premium_proxy with render_js is 25 credits, country_code adds 5, and stealth adds another 5, so a fully loaded SERP request runs about 35 credits. Turn off what a given target doesn't need. Many keyword-only queries succeed without stealth.

For deep SERPs, loop over start with pacing between requests:

def scrape_serp_pages(query, pages=3, gl="us", hl="en"):
    results_html = []
    for page in range(pages):
        serp_url = "https://www.google.com/search?" + urllib.parse.urlencode({
            "q": query, "hl": hl, "gl": gl, "num": 10, "start": page * 10,
        })
        r = requests.get(
            "https://scrape.sparkproxy.io/api/v1",
            headers={"X-API-Key": "YOUR_API_KEY"},
            params={
                "url": serp_url,
                "render_js": "true",
                "country_code": gl,
                "premium_proxy": "true",
            },
            timeout=90,
        )
        if r.status_code == 200:
            results_html.append(r.text)
        human_delay(base=25)   # from the pacing section
    return results_html

A modern Google SERP is not a list of ten blue links. It's a mixed layout of organic results, ads, and features, and each needs its own extraction logic.

The honest caveat first: Google's CSS class names are obfuscated and rotate often (.g, .tF2Cxc, .yuRUbf and friends change without notice). Don't hard-code brittle class chains. Prefer structural anchors (the result container that holds an