How to Scrape Amazon Product Data at Scale

Scraping publicly visible data sits in a narrower lane than most people assume, so get the framing right before you write a line of code.

In the United States, the Ninth Circuit's ruling in hiQ Labs v. LinkedIn (2022) held that scraping data that's publicly accessible, meaning no login and no authentication wall, generally does not violate the Computer Fraud and Abuse Act. That decision is about unauthorized access, not a blanket license. Amazon's Conditions of Use separately prohibit automated data collection, so scraping can still be a breach of contract even where it isn't a CFAA violation. Those are two different legal questions, and "public" answers only the first one.

Practical guardrails that keep a project defensible:

• Collect public product data only: title, price, ASIN, ratings. Never anything behind a login, and never personal data about buyers or reviewers.
• Don't hammer the servers. Rate-limit yourself and back off on errors so you aren't degrading the service.
• Don't republish copyrighted assets (product images, full review text) beyond what fair use allows.
• If the data feeds a commercial product, run it past a lawyer. This guide is engineering guidance, not legal advice.

Price monitoring, catalog research, and competitive intelligence are common, legitimate uses of public Amazon data. If you want the business context around that, we cover it in How E-commerce Companies Use Proxies for Competitive Intelligence.

A public Amazon product page (a "detail page", reachable at /dp/) exposes a consistent set of fields. The DOM locations drift, but the fields themselves are stable. Here's the reference set worth pulling, with selectors that work as of mid-2026:

The ASIN is the anchor for everything. It's the 10-character identifier (for example B0CHX1W1XY) that uniquely names a product in a given marketplace. Store it as your primary key and every other field hangs off it.

Amazon runs one of the more aggressive anti-bot stacks on the public web. Four things break naive scrapers:

The Robot Check page. When Amazon suspects automation, it serves a challenge page (titled "Robot Check", sometimes called the "Dogs of Amazon" page) at /errors/validateCaptcha. The trap: it often returns HTTP 200. If your code trusts the status code, response.ok is True, you save the "page", and you've stored a CAPTCHA instead of a product. A 200 that contains a Robot Check is still a block. You have to inspect the body.

Rate limiting. Fire too many requests from one IP in a short window and Amazon returns 503s or a "Sorry, something went wrong" interstitial. The threshold isn't published and it moves.

IP bans. Plain datacenter IP ranges get flagged quickly on Amazon, which is why a single static proxy dies fast. Residential IPs blend in and last far longer. Rotating the exit IP per request is the difference between a scraper that runs for an hour and one that runs for a week.

Shifting layouts. Amazon A/B tests page templates constantly, so the price you want might sit in #corePrice_feature_div on one request and #apex_desktop on the next. Hard-coding a single selector guarantees silent breakage. You match a list of candidates instead.

A managed scraping API absorbs the first three of those for you. The layout problem is yours to solve in parsing, because it lives in the HTML. If you want the proxy-side theory behind ban avoidance, How to Avoid Getting Your Proxy Blocked goes deep on it.

The SparkProxy Scraping API takes a target URL and handles the proxy, rotation, browser rendering, and anti-bot layer for you. You send one request; you get the rendered HTML back. For Amazon, three parameters carry the weight:

• render_js=true: Amazon paints price and stock with JavaScript, so a raw fetch misses them. Rendering with a real Chromium browser gets the final DOM.
• premium_proxy=true: routes the request through residential IPs, which survive Amazon's defenses where datacenter IPs get flagged.
• country_code: the ISO alpha-2 code of the marketplace you're targeting (US for amazon.com, GB for amazon.co.uk, DE for amazon.de). It sets the exit country, which drives the default regional price and currency.

The base endpoint is https://scrape.sparkproxy.io/api/v1 and auth is a single header, X-API-Key. A minimal request in cURL:

curl -G "https://scrape.sparkproxy.io/api/v1" \
  -H "X-API-Key: YOUR_API_KEY" \
  --data-urlencode "url=https://www.amazon.com/dp/B0CHX1W1XY" \
  --data-urlencode "render_js=true" \
  --data-urlencode "premium_proxy=true" \
  --data-urlencode "country_code=US"

Full parameter list and response fields live in the Scraping API docs. If you're weighing this against running your own proxy pool, Web Scraping API vs Self-Managed Proxies lays out the trade-off.

Start with one product. Wrap the request so every call carries the Amazon-specific parameters, and give it a generous timeout since a rendered request runs a real browser.

import requests

API = "https://scrape.sparkproxy.io/api/v1"
API_KEY = "YOUR_API_KEY"

def fetch_product(asin: str, country: str = "US") -> str:
    resp = requests.get(
        API,
        headers={"X-API-Key": API_KEY},
        params={
            "url": f"https://www.amazon.com/dp/{asin}",
            "render_js": "true",       # Amazon renders price/stock with JS
            "premium_proxy": "true",   # residential IPs survive Amazon's defenses
            "country_code": country,   # marketplace + regional pricing
        },
        timeout=90,
    )
    resp.raise_for_status()
    return resp.text

Before you trust the HTML, check whether Amazon handed you a Robot Check. Because that page returns 200, raise_for_status() won't catch it. Scan the body for the telltale markers:

def is_blocked(html: str) -> bool:
    """Amazon returns HTTP 200 for its Robot Check page, so the status code lies."""
    markers = (
        "Robot Check",
        "Enter the characters you see below",
        "/errors/validateCaptcha",
        "Type the characters you see in this image",
    )
    return any(m in html for m in markers)

Now a single fetch is honest: it either returns a real product page or tells you it was blocked so you can retry.

For parsing at scale, use selectolax (a C-backed HTML parser) rather than the pure-Python default. It parses Amazon's large DOM roughly an order of magnitude faster than html.parser, which matters when you're processing tens of thousands of pages. Install it with pip install selectolax.

The key move for price is trying a list of selectors instead of betting on one, because Amazon serves different price layouts to different sessions:

import re
from selectolax.parser import HTMLParser

PRICE_SELECTORS = [
    "#corePrice_feature_div .a-offscreen",
    "#corePriceDisplay_desktop_feature_div .a-offscreen",
    ".a-price .a-offscreen",
    "#priceblock_ourprice",     # legacy layout
    "#priceblock_dealprice",    # legacy deal layout
]

def _first_text(tree, selectors):
    for sel in selectors:
        node = tree.css_first(sel)
        if node and node.text(strip=True):
            return node.text(strip=True)
    return None

def extract_asin(url: str) -> str | None:
    m = re.search(r"/(?:dp|gp/product)/([A-Z0-9]{10})", url)
    return m.group(1) if m else None

def parse_product(html: str, url: str) -> dict:
    tree = HTMLParser(html)
    title = tree.css_first("#productTitle")
    rating = tree.css_first("#acrPopover")
    reviews = tree.css_first("#acrCustomerReviewText")
    avail = tree.css_first("#availability")
    brand = tree.css_first("#bylineInfo")
    image = tree.css_first("#landingImage")
    return {
        "asin": extract_asin(url),
        "title": title.text(strip=True) if title else None,
        "price": _first_text(tree, PRICE_SELECTORS),
        "rating": rating.attributes.get("title") if rating else None,
        "review_count": reviews.text(strip=True) if reviews else None,
        "availability": avail.text(strip=True) if avail else None,
        "brand": brand.text(strip=True) if brand else None,
        "image": (image.attributes.get("data-old-hires")
                  or image.attributes.get("src")) if image else None,
    }

A few things worth knowing. The rating lives in the title attribute of #acrPopover as a string like "4.6 out of 5 stars", so parse the number out downstream. The hi-res image URL is in data-old-hires, while src is often a smaller thumbnail. And #availability returns free text ("In Stock", "Only 3 left in stock, order soon", "Currently unavailable"), so normalize it into a boolean plus a raw string rather than parsing it into an enum you'll have to keep extending.

Maintaining CSS selectors across Amazon's layout churn is the tax on self-parsing. The SparkProxy Scraping API can do the extraction server-side with the extract_rules parameter: you pass a map of field names to selectors, and the API returns JSON keyed by your names. This turns the scraping endpoint into a lightweight amazon product data api where the response is already structured.

import json
import requests

rules = {
    "title": "#productTitle",
    "price": ".a-price .a-offscreen",
    "rating": "#acrPopover@title",       # @attr pulls an attribute value
    "reviews": "#acrCustomerReviewText",
    "availability": "#availability",
}

resp = requests.get(
    "https://scrape.sparkproxy.io/api/v1",
    headers={"X-API-Key": "YOUR_API_KEY"},
    params={
        "url": "https://www.amazon.com/dp/B0CHX1W1XY",
        "render_js": "true",
        "premium_proxy": "true",
        "country_code": "US",
        "extract_rules": json.dumps(rules),
    },
    timeout=90,
)
data = resp.json()   # {"title": "...", "price": "$249.00", "rating": "4.6 out of 5 stars", ...}

The @title suffix pulls an attribute instead of text, which is how you grab the rating out of #acrPopover. Check the docs for the exact extract_rules syntax your plan exposes. The trade is the same one from the previous section, just moved: server-side rules mean you don't ship a parser, but you still update the selectors when Amazon changes them. For a small, fixed field set like price monitoring, extract_rules is usually the lower-maintenance path.

Here's the gotcha almost no Amazon scraping tutorial mentions, and it silently corrupts price datasets: Amazon localizes price, delivery estimates, and stock by the shopper's delivery location. The same ASIN can show one price for a New York ZIP and a different price, or "unavailable", for a rural one. If you scrape the same product from rotating IPs without pinning a location, your price history will jitter for reasons that have nothing to do with Amazon actually changing the price.

There are two layers to control:

Marketplace / country. country_code sets the exit country, which selects the marketplace default and currency. Match it to the domain: country_code=US with amazon.com, country_code=DE with amazon.de. Mismatching them triggers redirects and interstitials.

Postal code, within a marketplace. Amazon stores the delivery ZIP in the "Deliver to" location (the glow pill in the top nav). To pin it, drive the location picker with a browser automation scenario using the js_scenario parameter, so every request in a batch localizes to the same ZIP:

import json
import requests

zip_scenario = {
    "instructions": [
        {"click": "#nav-global-location-popover-link"},
        {"wait_for": "#GLUXZipUpdateInput"},
        {"fill": ["#GLUXZipUpdateInput", "10001"]},
        {"click": "#GLUXZipUpdate input[type=submit]"},
        {"wait": 2},
    ]
}

resp = requests.get(
    "https://scrape.sparkproxy.io/api/v1",
    headers={"X-API-Key": "YOUR_API_KEY"},
    params={
        "url": "https://www.amazon.com/dp/B0CHX1W1XY",
        "render_js": "true",
        "premium_proxy": "true",
        "country_code": "US",
        "js_scenario": json.dumps(zip_scenario),
    },
    timeout=120,
)

Pick one canonical ZIP per marketplace and hold it constant across every run, so a price change in your data reflects a real price change and not a location change. If you're building a price-comparison feed, this consistency is the whole game; we get into it in Datacenter Proxies for Price Comparison Websites.

Search pages let you discover ASINs by keyword. Each result card is a div with a non-empty data-asin attribute inside div.s-main-slot. Sponsored and spacer rows carry an empty data-asin, so filter those out.

from urllib.parse import quote_plus
from selectolax.parser import HTMLParser

def search_url(keyword: str, page: int) -> str:
    return f"https://www.amazon.com/s?k={quote_plus(keyword)}&page={page}"

def parse_search(html: str) -> list[dict]:
    tree = HTMLParser(html)
    rows = []
    for card in tree.css("div.s-main-slot div[data-asin]"):
        asin = card.attributes.get("data-asin", "")
        if not asin:                       # skip sponsored / spacer rows
            continue
        title = card.css_first("h2 a span")
        price = card.css_first(".a-price .a-offscreen")
        rows.append({
            "asin": asin,
            "title": title.text(strip=True) if title else None,
            "price": price.text(strip=True) if price else None,
        })
    return rows

To walk every page, request &page=N and stop when a page returns no cards or the "next" control is disabled:

def scrape_all_pages(keyword: str, max_pages: int = 7) -> list[dict]:
    results = []
    for page in range(1, max_pages + 1):
        html = fetch_product_url(search_url(keyword, page))  # same API call as fetch_product
        if is_blocked(html):
            continue
        page_rows = parse_search(html)
        if not page_rows:
            break
        results.extend(page_rows)
        tree = HTMLParser(html)
        nxt = tree.css_first("a.s-pagination-next")
        classes = (nxt.attributes.get("class") or "") if nxt else "disabled"
        if nxt is None or "s-pagination-disabled" in classes:
            break
    return results

One hard limit to plan around: Amazon caps how deep organic search pagination goes, often around 7 pages per query. You can't page past it, so broad queries like "headphones" leave most of the catalog unreachable. The fix is to narrow: split by category, brand, or price band (Amazon's URL filters like &rh=... and &low-price=), run each narrow query to its page cap, then dedupe ASINs. Ten targeted queries surface far more of the catalog than one broad one.

At volume, three things keep the pipeline healthy: retries on soft blocks, backoff so you don't spike a single IP, and modest concurrency. With a scraping API the provider rotates the exit IP for you, so your concurrency ceiling is your plan's rate limit, not the number of proxies you own. Keep worker counts sane (5 to 15 is plenty) and let retries absorb the occasional block.

import time
import random
from concurrent.futures import ThreadPoolExecutor, as_completed

def fetch_with_retry(asin: str, attempts: int = 3) -> str | None:
    for i in range(attempts):
        html = fetch_product(asin)
        if not is_blocked(html) and "productTitle" in html:
            return html
        time.sleep(2 ** i + random.random())   # exponential backoff + jitter
    return None

def scrape_catalog(asins: list[str], workers: int = 8) -> list[dict]:
    out = []
    with ThreadPoolExecutor(max_workers=workers) as pool:
        futures = {pool.submit(fetch_with_retry, a): a for a in asins}
        for fut in as_completed(futures):
            html = fut.result()
            if html is None:
                continue
            asin = futures[fut]
            out.append(parse_product(html, f"https://www.amazon.com/dp/{asin}"))
    return out

The backoff matters more than it looks. Jitter (random.random()) staggers retries so a batch of failures doesn't retry in lockstep and re-trigger the same rate limit. Persist results as you go rather than holding everything in memory, so a crash at ASIN 40,000 doesn't cost you the first 39,999. A flat CSV is enough to start:

import csv

def save_csv(rows: list[dict], path: str = "amazon_products.csv") -> None:
    if not rows:
        return
    with open(path, "w", newline="", encoding="utf-8") as f:
        writer = csv.DictWriter(f, fieldnames=list(rows[0].keys()))
        writer.writeheader()
        writer.writerows(rows)

For a running price tracker, append a scraped_at timestamp and the ZIP you pinned to each row, and write to a database keyed on (asin, scraped_at). That gives you a clean time series where every price point is comparable because the location was held constant. The general proxy patterns behind high-volume collection are in Using Datacenter Proxies for Web Scraping.