How to Scrape Ecommerce Prices Across Multiple Sites

Prices are facts, and facts are not copyrightable. That's the starting point, but it isn't the whole answer, so frame the project correctly before you write code.

In the United States, the Ninth Circuit's ruling in hiQ Labs v. LinkedIn (2022) held that scraping data that's publicly accessible, with no login and no authentication wall, generally does not violate the Computer Fraud and Abuse Act. That covers unauthorized access, not a blanket license to do anything with the data. Most retailers' terms of service separately forbid automated collection, so a scrape can breach a contract even when it doesn't break the CFAA. Those are two different questions, and "the price is public" only answers the first.

Guardrails that keep a price-monitoring project defensible:

• Collect public listing data only: price, currency, availability, product identifiers. Never anything behind a login or a checkout, and never buyer data.
• Rate-limit yourself and back off on errors. You are reading a shelf tag, not stress-testing a checkout.
• Don't copy protected assets (full descriptions, photography) beyond fair use. You want the number, not the marketing.
• If the feed drives pricing decisions in a commercial product, get legal sign-off. This is engineering guidance, not legal advice.

Competitor price monitoring, catalog research, and repricing are common, legitimate uses of public price data. For the business context around that, see How Ecommerce Companies Use Proxies for Competitive Intelligence.

Write a scraper for one store and you solve one DOM. Write a price monitor for twenty stores the same way and you now own twenty scripts that each break on a different Tuesday. The failure isn't the parsing. It's everything around it.

Four problems appear the moment you go multi-site, and none of them exist when you scrape a single retailer:

This is why a multi-site price scraper is data engineering, not a parsing exercise. The value lives in the normalization and matching layers, the parts most tutorials skip because they only ever show one site. Price comparison sites solve exactly this problem at scale, and the infrastructure side of it is covered in Datacenter Proxies for Price Comparison Websites. If you're focused on a single retailer like Amazon instead, that has its own detail-page quirks that we cover separately .

The mistake is putting per-site logic in code. Every store you add then means a new function, a new file, a new thing to test. Instead, describe each target as data. One generic engine reads the config and runs any site.

A target registry is just a dictionary (load it from YAML or a database in production). Each entry says where to scrape, in what currency, from what country, and how to read the price:

# targets.py: everything site-specific lives here, not in the engine
TARGETS = {
    "store-us": {
        "country": "US",
        "currency": "USD",
        "decimal": ".",                       # 1,299.00
        "extract": {
            "price":    ".product-price .amount",
            "currency": ".product-price .symbol",
            "stock":    ".availability",
            "gtin":     "meta[itemprop=gtin13]@content",
        },
    },
    "store-de": {
        "country": "DE",
        "currency": "EUR",
        "decimal": ",",                       # 1.299,00
        "extract": {
            "price":    "span[itemprop=price]@content",
            "stock":    "link[itemprop=availability]@href",
            "gtin":     "meta[itemprop=gtin13]@content",
        },
    },
}

# The product catalog: one canonical item, its URL on each store.
CATALOG = {
    "nova-headphones-x1": {
        "store-us": "https://store-us.sparkproxy.io/p/nova-headphones-x1",
        "store-de": "https://store-de.sparkproxy.io/p/nova-headphones-x1",
    },
}

The store-us.sparkproxy.io hosts are placeholders. Swap in the real retailer URLs you're tracking. The point is that adding a retailer is a config edit, not a code change, and every site flows through the same extraction and normalization path.

One endpoint scrapes any of those sites. You send a URL and a few flags, SparkProxy handles the browser, the proxy, and the anti-bot layer, and you get back either raw HTML or fields already extracted by CSS selector.

The base URL is https://scrape.sparkproxy.io/api/v1 and auth is the X-API-Key header. A minimal request:

curl -X POST "https://scrape.sparkproxy.io/api/v1" \
  -H "X-API-Key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://store-us.sparkproxy.io/p/nova-headphones-x1",
    "render_js": true,
    "format": "json",
    "country_code": "US"
  }'

The parameters that matter for price scraping:

Credits scale with what you turn on: a plain fetch is 1 credit, JavaScript rendering is 5, and premium residential with JS is 25, per the Scraping API docs. Price pages usually don't need the heaviest tier. Start light and only add premium_proxy or stealth for the specific sites that fight back.

Instead of pulling raw HTML and parsing it yourself, hand the CSS selectors to the API and get structured fields back. The extract_rules you send are exactly the extract block from each registry entry, so the engine stays generic:

import requests

API = "https://scrape.sparkproxy.io/api/v1"
KEY = "YOUR_API_KEY"

def fetch_fields(url, extract_rules, country):
    r = requests.post(
        API,
        headers={"X-API-Key": KEY, "Content-Type": "application/json"},
        json={
            "url": url,
            "render_js": True,
            "format": "json",
            "country_code": country,
            "extract_rules": extract_rules,
        },
        timeout=90,
    )
    r.raise_for_status()
    return r.json()

def scrape_target(site, product_key):
    cfg = TARGETS[site]
    url = CATALOG[product_key][site]
    data = fetch_fields(url, cfg["extract"], cfg["country"])
    return data.get("extracted", {})

The response wraps everything in a JSON envelope with status_code, credits_used, and the extracted object holding your named fields. One function, driven entirely by config, now scrapes every site in the registry. What comes back is still messy, though. A raw price field might be "$1,299.00", "1.299,00 €", or "EUR 1299". Comparable numbers come next.

This is the section most guides never write, and it's where multi-site price scraping actually lives. A price string is a locale artifact, not a number. The single most common bug in a price monitor is parsing 1.299,00 (German for 1,299.00) as one point two nine nine.

The fix is to know each site's decimal convention up front (you already stored it in the registry) and parse against it:

import re
from decimal import Decimal, InvalidOperation

def parse_price(raw, decimal="."):
    """Turn a messy price string into a Decimal.
    decimal=',' for locales like de-DE ('1.299,00'); '.' for en-US ('1,299.00').
    """
    if not raw:
        return None
    s = re.sub(r"[^\d.,]", "", raw)          # strip symbols, letters, spaces
    if decimal == ",":
        s = s.replace(".", "").replace(",", ".")   # 1.299,00 -> 1299.00
    else:
        s = s.replace(",", "")                       # 1,299.00 -> 1299.00
    try:
        return Decimal(s)
    except InvalidOperation:
        return None

# store-us: "$1,299.00" -> Decimal('1299.00')
# store-de: "1.299,00 €" -> Decimal('1299.00')

Two more normalization rules earn their keep once you have real data:

• Per-unit price. A 500g bag at 8.00 and a 1kg bag at 14.00 are not comparable until you compute price per unit. If the site exposes unit and quantity, store a unit_price alongside the shelf price so a 1kg-versus-500g comparison is honest.
• Use Decimal, never float. Money in binary floating point drifts. 0.1 + 0.2 is not 0.3. Parse to Decimal, store as a fixed-precision numeric, and you'll never chase a rounding ghost in a repricing rule.

Normalize at write time, not at read time. Every row that lands in your store should already be a clean number in a known currency. Downstream queries stay simple and fast.

The same product often carries a different price depending on where the shopper appears to be. Retailers localize by IP: currency, tax-inclusive versus exclusive display, and sometimes genuinely different price points per market. If you always scrape from one country, you see one region's shelf and miss the variance you're trying to track.

country_code pins the scrape to a location. Loop the countries you care about for the same URL:

def scrape_by_country(url, extract_rules, countries):
    out = {}
    for cc in countries:
        data = fetch_fields(url, extract_rules, cc)
        out[cc] = data.get("extracted", {})
    return out

prices = scrape_by_country(
    "https://store-us.sparkproxy.io/p/nova-headphones-x1",
    TARGETS["store-us"]["extract"],
    ["US", "GB", "DE", "AU"],
)

Two things to keep straight. First, currency is not the same as country: a .com store might quote USD to a US IP and GBP to a UK IP from the same URL, so always capture the currency you actually saw, never assume it from the domain. Second, converting everything to one base currency for comparison is fine for a dashboard, but store the original currency and amount as observed. Convert on read using the exchange rate for that date. If you overwrite the original with a converted figure, you've destroyed the fact and can never re-derive it. Geo targeting is one of the main reasons teams use proxies for this work, covered in Using Datacenter Proxies for Web Scraping.

You can't compare prices until you know two listings are the same product. This is the second problem tutorials skip, and it's harder than parsing. "Nova Headphones X1" on one store is "Nova X1 Wireless Headphones (Black)" on another and "NOVA-X1" in a third. Title matching alone will burn you.

Match on identifiers first, text last:

1. GTIN / UPC / EAN. The global trade item number is the gold key. If two listings share a GTIN, they are the same product, full stop. Many stores publish it in meta[itemprop=gtin13] or in embedded JSON-LD (Product.gtin13). Pull it during extraction, as the registry above does.
2. MPN plus brand. Manufacturer part number scoped to a brand is nearly as reliable when GTIN is missing.
3. Normalized title as a last resort. Lowercase, strip punctuation, sort tokens, then fuzzy-match. Treat any text match as a candidate that needs review, not a confirmed pair.

def product_identity(fields):
    """Prefer stable identifiers; fall back to a normalized title."""
    gtin = (fields.get("gtin") or "").strip()
    if gtin:
        return ("gtin", gtin)
    mpn, brand = fields.get("mpn"), fields.get("brand")
    if mpn and brand:
        return ("mpn", f"{brand.lower()}:{mpn.lower()}")
    title = re.sub(r"[^a-z0-9 ]", "", (fields.get("title") or "").lower())
    return ("title", " ".join(sorted(title.split())))

Build your catalog around one canonical product key (as in the registry), then attach each retailer's URL to it. Now every price you collect already knows which product it belongs to, and cross-site comparison is a group-by instead of a guessing game.

More often is not better. It costs credits, raises your block risk, and for most catalogs it collects noise. Match cadence to how fast prices actually move, and to what a stale number costs you.

A tiered schedule beats one blanket interval:

Drive it with cron or any scheduler, iterating the catalog by tier. Stagger start times so you don't hammer one retailer with a burst:

# Runs hourly for hot items; a daily job handles warm, weekly for cold.
import random, time

def run_tier(products, jitter=(1, 6)):
    for key in products:
        for site in CATALOG[key]:
            fields = scrape_target(site, key)
            store_observation(key, site, fields)   # see next section
            time.sleep(random.uniform(*jitter))    # spread the load

The jittered sleep matters more than it looks. Predictable, evenly spaced requests are a bot signature. A little randomness spreads load and reads more like organic traffic.

A price is only useful over time. "Competitor is at 89.99" tells you little. "Competitor dropped from 109.99 to 89.99 over three days, matching our promo" is a decision. That means append-only history, never a single overwritten "current price" column.

The core table is a fact table: one row per observation, never updated, never deleted.

CREATE TABLE price_observations (
    id           BIGSERIAL PRIMARY KEY,
    product_key  TEXT        NOT NULL,   -- your canonical product id
    site         TEXT        NOT NULL,   -- which retailer
    url          TEXT        NOT NULL,
    price        NUMERIC(12,2),          -- normalized, no symbol
    currency     CHAR(3)     NOT NULL,   -- ISO 4217, as observed
    country      CHAR(2)     NOT NULL,   -- where we scraped from
    in_stock     BOOLEAN,
    observed_at  TIMESTAMPTZ NOT NULL DEFAULT now()
);

-- Fast "latest price per product+site" and history queries.
CREATE INDEX ON price_observations (product_key, site, observed_at DESC);

The composite fact worth internalizing: a price observation is only meaningful as the tuple (product, site, currency, country, timestamp). Drop any one of those and the number becomes ambiguous. The same 89.99 means different things in USD-from-US versus EUR-from-DE, and a price with no timestamp is a rumor.

The insert is trivial because normalization already happened upstream:

import psycopg2
from datetime import datetime, timezone

def store_observation(product_key, site, fields):
    cfg = TARGETS[site]
    price = parse_price(fields.get("price"), cfg["decimal"])
    conn = psycopg2.connect("dbname=prices")   # pool this in production
    with conn, conn.cursor() as cur:
        cur.execute(
            """INSERT INTO price_observations
               (product_key, site, url, price, currency, country,
                in_stock, observed_at)
               VALUES (%s,%s,%s,%s,%s,%s,%s,%s)""",
            (product_key, site, CATALOG[product_key][site], price,
             cfg["currency"], cfg["country"],
             "in" in (fields.get("stock") or "").lower(),
             datetime.now(timezone.utc)),
        )

Append-only storage gives you every downstream feature for free: price history charts, drop alerts, promo detection, and lowest-ever tracking are all just queries over this one table. If you ever need lifetime lows or price-change frequency, the data is already there.

Twenty retailers means twenty different anti-bot postures. One store waves everyone through, another runs Cloudflare, a third rate-limits by IP within minutes. This is the failure mode that quietly kills price monitors: they run clean for a week, then half the sites start returning empty prices and nobody notices until the dashboard goes flat.

Route around it per site rather than globally:

• Start cheap, escalate per target. Use a plain render for the easy stores. Add premium_proxy (residential IPs) only for the ones that block datacenter ranges, and stealth for the aggressive few. Escalating globally just burns credits.
• Detect soft blocks, don't trust HTTP 200. A block often returns 200 with a challenge page and no price. If extracted.price is empty on a page that should have one, treat it as a block and retry with a heavier tier, not as "out of stock."
• Respect 429 with backoff. The API returns retry_after_seconds on rate limits. Honor it. Retry 530 (scrape failed) once with stealth on.

def scrape_with_escalation(site, product_key, max_tries=3):
    cfg = TARGETS[site]
    url = CATALOG[product_key][site]
    tiers = [
        {"render_js": True},
        {"render_js": True, "premium_proxy": True},
        {"render_js": True, "premium_proxy": True, "stealth": True},
    ]
    for i in range(max_tries):
        r = requests.post(
            API,
            headers={"X-API-Key": KEY, "Content-Type": "application/json"},
            json={"url": url, "format": "json",
                  "country_code": cfg["country"],
                  "extract_rules": cfg["extract"], **tiers[i]},
            timeout=120,
        )
        if r.status_code == 429:
            time.sleep(r.json().get("retry_after_seconds", 30))
            continue
        fields = r.json().get("extracted", {})
        if fields.get("price"):          # got a real price, done
            return fields
    return None                          # exhausted tiers; log and move on

The escalation ladder keeps average cost low while still clearing the hard sites. For the wider playbook on avoiding blocks, see How to Avoid Getting Your Proxy Blocked. And if you're weighing whether to run this on a managed API or your own proxy pool, Web Scraping API vs Self-Managed Proxies lays out the tradeoff. At multi-site scale, the API usually wins because you're maintaining one integration instead of one anti-bot workaround per retailer.