How to Scrape Google Shopping Product Data

Public data is a narrower lane than most people assume, so get the framing right before you write a line of code.

In the United States, the Ninth Circuit's ruling in hiQ Labs v. LinkedIn (2022) held that scraping data that is publicly accessible, meaning no login and no authentication wall, generally does not violate the Computer Fraud and Abuse Act. Google Shopping results sit behind no login, so they fall on the public side of that line. That decision is about unauthorized access, not a blanket license. Google's Terms of Service separately restrict automated access, so scraping can still breach a contract even where it does not break the CFAA. Those are two different legal questions, and "public" answers only the first one.

Guardrails that keep a project defensible:

• Collect public product data only: title, price, merchant, rating, product ID. Never anything behind a login, and never personal data about shoppers or reviewers.
• Rate-limit yourself and back off on errors so you are not degrading the service.
• Don't republish copyrighted assets (merchant product images, full review text) beyond what fair use allows.
• If the data feeds a commercial product, run it past a lawyer. This guide is engineering guidance, not legal advice.

Price monitoring, catalog research, and competitive intelligence are common, legitimate uses of public Shopping data. For the business context around that, see How E-commerce Companies Use Proxies for Competitive Intelligence.

A Google Shopping card and its product page expose a consistent set of fields. The DOM around them shifts constantly, but the fields themselves are stable. Here is the reference set worth pulling, with the durable way to reach each one as of mid-2026:

The product ID is the anchor for everything. It is the numeric identifier Google assigns to a catalog product (for example 13068762482273657031), and it is the same across every merchant selling that item. Store it as your primary key, and every price you collect for that product hangs off it.

There are two surfaces, and confusing them is the number-one reason a Shopping scraper collects the wrong data.

The Shopping results grid. Add tbm=shop to a normal Google search and you get the Shopping tab: a grid of product cards for your query. The canonical URL is https://www.google.com/search?tbm=shop&q=. Google has been migrating this surface to a newer udm=28 parameter alongside the classic tbm=shop, so if one returns an unfamiliar layout, try the other. Each card gives you a title, one merchant's price, a rating, and a link into the product page.

The product page. Click any card and you land on https://www.google.com/shopping/product/. This is the page that matters for price comparison, because it lists every merchant selling that product, each with its own price, condition, and total after shipping and tax. The grid shows you one offer; the product page shows you all of them.

Most tutorials scrape only the grid and stop. If your goal is genuine price comparison, the grid is just the discovery step that hands you product IDs, and the product page is where the dataset actually lives. We will build both.

Google runs one of the toughest anti-bot stacks on the public web, and Shopping adds parsing traps on top. Four things break naive scrapers:

Obfuscated, rotating class names. The inner elements of a Shopping card carry short randomized classes like .a8Pemb or .tAxDx that Google rotates on its own schedule. Hard-code one and your parser silently returns empty prices within days. The durable move is to anchor on the semantic container class (sh-dgr__grid-result for the grid) and extract fields by content pattern, not by brittle inner selectors.

The consent interstitial. From EU and UK exit IPs, Google serves a cookie-consent wall on consent.google.com before any results. Like a soft block, it returns a normal HTTP 200 with a real page, so a scraper that trusts the status code stores the consent form instead of products. You have to detect it and route around it.

CAPTCHA and the "sorry" page. Fire too many requests from one IP and Google returns its /sorry/index reCAPTCHA interstitial. Plain datacenter IP ranges trip this fast, which is why a single static proxy dies quickly. Rotating residential exits blend in and last far longer.

JavaScript rendering. Shopping paints prices, ratings, and lazy images with JavaScript. A raw HTTP fetch returns a shell with the data missing. You need a real browser to get the final DOM.

A managed scraping API absorbs the rendering, rotation, and CAPTCHA layers for you. The class-name churn is yours to solve in parsing, because it lives in the HTML. For the proxy-side theory behind ban avoidance, How to Avoid Getting Your Proxy Blocked goes deep.

The SparkProxy Scraping API takes a target URL and handles the proxy, rotation, browser rendering, and anti-bot layer for you. You send one request; you get the rendered HTML back. For Google Shopping, three parameters carry the weight:

• render_js=true: Shopping paints prices and ratings with JavaScript, so a raw fetch misses them. Rendering with a real Chromium browser gets the final DOM.
• premium_proxy=true: routes through residential IPs, which survive Google's defenses where datacenter IPs get flagged into the CAPTCHA wall.
• country_code: the ISO alpha-2 code of the exit country (US, GB, DE). It sets where the request appears to come from, which also decides whether you hit the EU consent interstitial.

The base endpoint is https://scrape.sparkproxy.io/api/v1 and auth is a single header, X-API-Key. A minimal request in cURL:

curl -G "https://scrape.sparkproxy.io/api/v1" \
  -H "X-API-Key: YOUR_API_KEY" \
  --data-urlencode "url=https://www.google.com/search?tbm=shop&q=wireless+earbuds&gl=us&hl=en" \
  --data-urlencode "render_js=true" \
  --data-urlencode "premium_proxy=true" \
  --data-urlencode "country_code=US"

The full parameter list and response fields live in the Scraping API docs. If you are weighing this against running your own proxy pool, Web Scraping API vs Self-Managed Proxies lays out the trade-off.

Start with one query. Wrap the request so every call carries the Shopping-specific parameters, and give it a generous timeout since a rendered request runs a real browser.

import requests
from urllib.parse import urlencode

API = "https://scrape.sparkproxy.io/api/v1"
API_KEY = "YOUR_API_KEY"

def shop_url(query: str, gl: str = "us", hl: str = "en", start: int = 0) -> str:
    params = {"tbm": "shop", "q": query, "gl": gl, "hl": hl}
    if start:
        params["start"] = start
    return "https://www.google.com/search?" + urlencode(params)

def fetch(target: str, country: str = "US") -> str:
    resp = requests.get(
        API,
        headers={"X-API-Key": API_KEY},
        params={
            "url": target,
            "render_js": "true",        # prices/ratings are JS-rendered
            "premium_proxy": "true",    # residential IPs survive Google's defenses
            "country_code": country,    # exit country, also gates the consent wall
        },
        timeout=90,
    )
    resp.raise_for_status()
    return resp.text

Before you trust the HTML, check whether Google handed you a consent wall or a CAPTCHA. Both return HTTP 200, so raise_for_status() will not catch them. Scan the body for the telltale markers:

def is_blocked(html: str) -> bool:
    """Google returns HTTP 200 for its consent and CAPTCHA pages, so the status lies."""
    markers = (
        "consent.google.com",
        "Before you continue to Google",
        "/sorry/index",
        "recaptcha",
        "unusual traffic from your computer network",
    )
    return any(m in html for m in markers)

Now a single fetch is honest: it either returns a real results page or tells you it was blocked so you can retry.

For parsing at scale, use selectolax (a C-backed HTML parser) rather than the pure-Python default. Install it with pip install selectolax.

The key move is to anchor on the semantic grid container and pull each field by content pattern, not by the randomized inner classes. The container sh-dgr__grid-result is a semantic BEM class Google keeps far more stable than the short obfuscated ones inside it:

import re
from selectolax.parser import HTMLParser

PRICE_RE = re.compile(r"[$£€]\s?[\d][\d.,]*")
PRODUCT_ID_RE = re.compile(r"/shopping/product/(\d+)")

def parse_grid(html: str) -> list[dict]:
    tree = HTMLParser(html)
    rows = []
    for card in tree.css("div.sh-dgr__grid-result"):
        link = card.css_first('a[href*="/shopping/product/"]')
        href = link.attributes.get("href", "") if link else ""
        pid = PRODUCT_ID_RE.search(href)
        text = card.text(separator="\n", strip=True)
        price = PRICE_RE.search(text)
        rating = card.css_first('[aria-label*="out of 5"]')
        rows.append({
            "product_id": pid.group(1) if pid else None,
            "title": link.text(strip=True) if link else None,
            "price": price.group(0) if price else None,
            "rating": (rating.attributes.get("aria-label") if rating else None),
            "url": ("https://www.google.com" + href) if href.startswith("/") else href,
        })
    return rows

Two things worth knowing. Matching price by a currency-symbol regex survives class-name rotation that would break .a8Pemb-style selectors, which is the whole point. And the product link both names the item and carries the product ID in its href, so a single node gives you the title and the join key at once.

If you would rather not maintain a parser at all, the SparkProxy Scraping API can extract fields server-side with the extract_rules parameter. You pass a map of field names to selectors, and the response comes back as JSON keyed by your names:

import json
import requests

rules = {
    "title": "div.sh-dgr__grid-result a[href*='/shopping/product/']",
    "price": "div.sh-dgr__grid-result span.a8Pemb",
    "merchant": "div.sh-dgr__grid-result div.aULzUe",
}

resp = requests.get(
    "https://scrape.sparkproxy.io/api/v1",
    headers={"X-API-Key": "YOUR_API_KEY"},
    params={
        "url": "https://www.google.com/search?tbm=shop&q=wireless+earbuds&gl=us&hl=en",
        "render_js": "true",
        "premium_proxy": "true",
        "country_code": "US",
        "extract_rules": json.dumps(rules),
    },
    timeout=90,
)
data = resp.json()

Check the docs for the exact extract_rules syntax your plan exposes. The trade is the same one, just moved: server-side rules mean you ship no parser, but you still update the selectors when Google rotates them, so keep the container class as your anchor there too.

Here is the gotcha that silently corrupts Shopping price datasets: the price and currency you see depend on locale, not just the exit IP. Two dials control it, and they must agree.

gl sets Google's country, which drives currency. gl=us returns USD prices, gl=gb returns GBP, gl=de returns EUR. If you rotate exit IPs across countries and leave gl unset, Google infers the country from the IP, and your dataset ends up mixing currencies for the same product. A price series that jumps from $59 to £59 is not a price change, it is a locale change, and it will wreck any comparison you build on top of it.

hl sets the interface language. Set it alongside gl (for example gl=us&hl=en) to keep the DOM strings, and therefore your selectors and text patterns, consistent across runs.

country_code should match gl. The API's country_code sets the exit IP country; the URL's gl sets Google's locale. Keep them aligned (country_code=US with gl=us) so Google does not detect a mismatch and redirect you. Matching them is also the cleanest way to sidestep the EU consent wall: a US exit with gl=us never sees it. If you must scrape from an EU locale, inject a consent cookie instead:

import json
import requests

resp = requests.get(
    "https://scrape.sparkproxy.io/api/v1",
    headers={"X-API-Key": "YOUR_API_KEY"},
    params={
        "url": "https://www.google.com/search?tbm=shop&q=espresso+machine&gl=de&hl=de",
        "render_js": "true",
        "premium_proxy": "true",
        "country_code": "DE",
        "cookies": json.dumps([{"name": "SOCS", "value": "CAI", "domain": ".google.com"}]),
    },
    timeout=90,
)

Pick one canonical (gl, hl, country_code) triple per market and hold it constant across every run, so a price change in your data reflects a real price change. If you are building a price-comparison feed, this consistency is the whole game, and we get into it in Datacenter Proxies for Price Comparison Websites.

The grid gives you one price per product. To scrape product prices across every merchant on Google, follow the product ID into the product page, which lists all offers for that item.

def product_page_url(product_id: str, gl: str = "us", hl: str = "en") -> str:
    return (f"https://www.google.com/shopping/product/{product_id}"
            f"?gl={gl}&hl={hl}")

def parse_offers(html: str) -> list[dict]:
    tree = HTMLParser(html)
    offers = []
    for row in tree.css("tr.sh-osd__offer-row"):
        text = row.text(separator="\n", strip=True)
        price = PRICE_RE.search(text)
        merchant = row.css_first(".sh-osd__seller-link, .b5ycib")
        offers.append({
            "merchant": merchant.text(strip=True) if merchant else None,
            "price": price.group(0) if price else None,
        })
    return offers

Each sh-osd__offer-row is one merchant's offer, and the price still resolves cleanly through the same currency regex. Join the offers back to the product ID from the grid, and you have a per-product table of every seller and their price, which is exactly the raw material a price-comparison product runs on. Note that sh-osd__ and b5ycib are the current selectors as of mid-2026; verify them against a live fetch before a big run, because Google rotates these too.

The classic tbm=shop grid paginates with a start offset, in steps of roughly 60 results per page. Request &start=60, &start=120, and so on, and stop when a page returns no cards:

import time
import random

def scrape_query(query: str, gl: str = "us", max_pages: int = 5) -> list[dict]:
    results, seen = [], set()
    for page in range(max_pages):
        target = shop_url(query, gl=gl, start=page * 60)
        html = fetch(target)
        if is_blocked(html):
            time.sleep(2 ** page + random.random())
            continue
        cards = parse_grid(html)
        if not cards:
            break
        for card in cards:
            pid = card.get("product_id")
            if pid and pid not in seen:      # dedupe repeated products
                seen.add(pid)
                results.append(card)
    return results

One limit to plan around: Google caps how deep Shopping pagination goes, often just a few pages per query, and it repeats products near the tail. Broad queries like "headphones" leave most of the catalog unreachable. The fix is to narrow: split by brand, price band, or attribute (headphones over-ear, headphones under $50), run each narrow query to its cap, then dedupe on product ID. Ten targeted queries surface far more of the catalog than one broad one.

At volume, three things keep the pipeline healthy: retries on soft blocks, backoff so you do not spike a single IP, and modest concurrency. With a scraping API the provider rotates the exit IP for you, so your concurrency ceiling is your plan's rate limit, not the number of proxies you own. Keep worker counts sane (5 to 15 is plenty) and let retries absorb the occasional block.

from concurrent.futures import ThreadPoolExecutor, as_completed

def fetch_with_retry(target: str, attempts: int = 3) -> str | None:
    for i in range(attempts):
        html = fetch(target)
        if not is_blocked(html) and "sh-dgr__grid-result" in html:
            return html
        time.sleep(2 ** i + random.random())   # exponential backoff + jitter
    return None

def scrape_many(queries: list[str], workers: int = 8) -> list[dict]:
    out = []
    with ThreadPoolExecutor(max_workers=workers) as pool:
        futures = {pool.submit(scrape_query, q): q for q in queries}
        for fut in as_completed(futures):
            out.extend(fut.result())
    return out

The backoff matters more than it looks. Jitter (random.random()) staggers retries so a batch of failures does not retry in lockstep and re-trigger the same CAPTCHA. Persist results as you go rather than holding everything in memory, and stamp each row with the locale you pinned and a timestamp:

import csv

def save_csv(rows: list[dict], path: str = "google_shopping.csv") -> None:
    if not rows:
        return
    with open(path, "w", newline="", encoding="utf-8") as f:
        writer = csv.DictWriter(f, fieldnames=list(rows[0].keys()))
        writer.writeheader()
        writer.writerows(rows)

For a running price tracker, write to a database keyed on (product_id, merchant, scraped_at) with the (gl, hl) locale attached to every row. That gives you a clean time series where each price point is comparable because the locale was held constant. The proxy-side patterns that keep a high-volume run alive are covered in How to Avoid Getting Your Proxy Blocked.