How to Scrape Shopify Stores: Products & Variants

The lazy way to scrape Shopify stores is to load each product page in a headless browser and pick prices out of the DOM. It works, it's slow, and it breaks every time the merchant changes themes. There's a far cleaner path most tutorials skip: nearly every Shopify store publishes its entire catalog as structured JSON at a public, unauthenticated URL. This guide shows you how to pull the whole catalog, every variant, every price, and every collection from that endpoint with no JavaScript rendering, then how to fall back to a real browser and residential IPs on the minority of stores that hide it behind Cloudflare.

Product listings, prices, and availability are public facts, and facts aren't copyrightable. That's the starting point, not the whole answer, so set the project up correctly before you write a scraper.

In the United States, the Ninth Circuit's ruling in hiQ Labs v. LinkedIn (2022) held that scraping data that's publicly accessible, with no login and no authentication wall, generally does not violate the Computer Fraud and Abuse Act. That covers unauthorized access, not a license to do anything you like with the result. Each Shopify store is an independent merchant with its own terms of service, and many forbid automated collection, so a scrape can breach a contract even when it clears the CFAA. Those are two separate questions.

A few guardrails keep a Shopify catalog project defensible:

• Collect public listing data only: title, handle, variant, price, SKU, availability. Never touch /cart, /checkout, /account, or anything behind a login.
• Read the store's robots.txt and respect its Disallow rules and any Crawl-delay.
• Rate-limit yourself and back off on errors. You're reading a shelf tag, not load-testing a checkout.
• Never collect buyer data, and don't republish protected assets (full descriptions, photography) beyond fair use.
• If the feed drives a commercial product, get legal sign-off. This is engineering guidance, not legal advice.

Catalog research, competitor price tracking, and assortment monitoring are common, legitimate uses of public product data. For the wider business context, see How Ecommerce Companies Use Proxies for Competitive Intelligence.

Before you build anything, confirm the target actually runs on Shopify. Two quick checks settle it.

First, the response headers. Shopify's storefront edge stamps requests with a handful of internal headers that other platforms don't send:

curl -sI "https://store.sparkproxy.io" \
  | grep -i -E "x-shopid|x-shardid|x-sorting-hat-shopid|x-storefront-renderer"

Second, and definitively, probe the JSON endpoint. If the site is Shopify and hasn't disabled it, this returns a JSON object with a products array:

curl -s "https://store.sparkproxy.io/products.json?limit=1" | head -c 200

Throughout this guide, https://store.sparkproxy.io stands in for the store you're scraping. Substitute the merchant's real domain, or its canonical myshopify.com subdomain, which answers the same endpoints. The HTML source is a weaker signal but worth knowing: Shopify themes load assets from cdn.shopify.com and expose a Shopify and ShopifyAnalytics JavaScript global on the page.

Here's the part most guides miss. Shopify's storefront ships a legacy, read-only JSON feed of the public catalog at /products.json. It's on by default, needs no API key, and returns fully structured product objects. No headless browser, no DOM parsing, no selector that snaps the next time the merchant reskins.

One request gives you this shape (trimmed):

{
  "products": [
    {
      "id": 74920381,
      "title": "Trail Runner Jacket",
      "handle": "trail-runner-jacket",
      "vendor": "Northline",
      "product_type": "Outerwear",
      "tags": ["waterproof", "mens"],
      "updated_at": "2026-08-08T14:20:11-04:00",
      "options": [{ "name": "Size", "values": ["S", "M", "L"] }],
      "variants": [
        {
          "id": 553001,
          "title": "S",
          "option1": "S",
          "sku": "TRJ-BLK-S",
          "price": "149.00",
          "compare_at_price": "199.00",
          "available": true,
          "updated_at": "2026-08-08T14:20:11-04:00"
        }
      ],
      "images": [{ "src": "https://cdn.shopify.com/s/files/....jpg" }]
    }
  ]
}

Every field you want for catalog and price work is right there: product identity, variant options, price, sale price, SKU, and an availability flag. Pulling one page in Python is four lines:

import requests

def fetch_products_page(store, page, limit=250):
    url = f"{store}/products.json"
    r = requests.get(url, params={"limit": limit, "page": page}, timeout=30)
    r.raise_for_status()
    return r.json()["products"]

store = "https://store.sparkproxy.io"
print(len(fetch_products_page(store, 1)), "products on page 1")

This endpoint is a public JSON API hiding in plain sight, which is exactly the class of target covered in How to Scrape Hidden JSON API Endpoints. The difference here is that it's the same on every Shopify store, so one scraper works across thousands of them.

/products.json is paginated, and the two knobs matter. limit caps at 250 items per request (the default is a stingy 30, so always set it). Paging is page-based, not cursor-based: increment page until the products array comes back empty. That empty array is your stop condition, since this endpoint gives no total count.

def fetch_all_products(store, limit=250):
    products, page = [], 1
    while True:
        batch = fetch_products_page(store, page, limit)
        if not batch:
            break
        products.extend(batch)
        page += 1
    return products

A store with 4,000 products is 16 requests at limit=250, done in seconds. Compare that to rendering 4,000 product pages in a browser. The endpoints break down like this:

Sometimes you don't want the whole store, just one category. Shopify mirrors the products feed per collection at /collections//products.json. To enumerate the handles, hit /collections.json first, which lists every published collection with the same pagination rules.

def fetch_collections(store, limit=250):
    handles, page = [], 1
    while True:
        r = requests.get(f"{store}/collections.json",
                         params={"limit": limit, "page": page}, timeout=30)
        batch = r.json().get("collections", [])
        if not batch:
            break
        handles += [c["handle"] for c in batch]
        page += 1
    return handles

def fetch_collection_products(store, handle, limit=250):
    products, page = [], 1
    while True:
        r = requests.get(f"{store}/collections/{handle}/products.json",
                         params={"limit": limit, "page": page}, timeout=30)
        batch = r.json().get("products", [])
        if not batch:
            break
        products += batch
        page += 1
    return products

Scraping by collection keeps your dataset aligned with how the merchant merchandises the store, which is handy for assortment analysis: you can see which SKUs sit in "Sale" versus "New Arrivals" without inferring it from tags.

A Shopify product is a container; the sellable unit is the variant. A t-shirt with three sizes and four colors is one product and up to twelve variants, each with its own price, SKU, and availability. Flatten to one row per variant so your price data is actually comparable.

from decimal import Decimal

def variant_rows(product):
    rows = []
    for v in product["variants"]:
        rows.append({
            "product_id": product["id"],
            "product_title": product["title"],
            "handle": product["handle"],
            "variant_id": v["id"],
            "variant_title": v["title"],
            "sku": v.get("sku") or "",
            "price": Decimal(v["price"]),
            "compare_at_price": Decimal(v["compare_at_price"]) if v.get("compare_at_price") else None,
            "on_sale": bool(v.get("compare_at_price")),
            "available": v["available"],
            "updated_at": v["updated_at"],
        })
    return rows

Three gotchas trip people up on this endpoint, and none of them are documented on the store:

• Price is a string in major currency units. You get "149.00", not 14900 cents. That's the opposite of Shopify's Admin API, which returns cents. Parse with Decimal, never float, so 0.10 + 0.20 doesn't drift.
• There is no exact inventory count. The storefront feed exposes a boolean available, not inventory_quantity. If you need a number, you can't get it here; infer in-stock versus out-of-stock from available and move on.
• There's no currency code in the JSON. /products.json gives you the amount but not the currency. Pull the currency once from the store's product page (its JSON-LD offers.priceCurrency) or the Shopify.currency.active global, and attach it to every row. A price of 149.00 is meaningless until you know it's USD, GBP, or EUR.

That last point matters most when you compare prices across stores. For a full cross-site price pipeline (currency normalization, product matching, time-series storage), see How to Scrape Ecommerce Prices Across Multiple Sites.

Shopify Markets lets a merchant show different prices, currencies, and even catalog availability per country. A store selling into the US, UK, and Germany might quote 149.00 USD, 129.00 GBP, and 139.00 EUR for the same variant. The /products.json feed returns the base market's pricing by default, so if you scrape from one location you capture one market and silently miss the rest.

The fix is to route each pass through an exit in the target country. Set country_code per market and scrape the catalog once for each:

def scrape_markets(store, markets=("US", "GB", "DE")):
    per_market = {}
    for country in markets:
        rows, page = [], 1
        while True:
            target = f"{store}/products.json?limit=250&page={page}"
            data = fetch_shopify_json(target, country=country)
            batch = data.get("products", [])
            if not batch:
                break
            for product in batch:
                rows += variant_rows(product)
            page += 1
        per_market[country] = rows
    return per_market

Tag every row with the country you scraped it from, because a price only means something paired with its market. Then read the currency code from that market's product page JSON-LD (offers.priceCurrency), since the JSON feed still won't hand you one. That's the gap between a price table that looks right and one that's genuinely comparable across borders.

The JSON feeds cover what's published now. For URL discovery and change detection, add the sitemap. Shopify's /sitemap.xml is a sitemap index that points to child files, including sitemap_products_1.xml, which lists every product URL with a lastmod timestamp and image references.

import requests, xml.etree.ElementTree as ET

NS = {"s": "http://www.sitemaps.org/schemas/sitemap/0.9"}

def product_urls(store):
    index = requests.get(f"{store}/sitemap.xml", timeout=30).text
    root = ET.fromstring(index)
    children = [loc.text for loc in root.iterfind(".//s:loc", NS)
                if "sitemap_products" in loc.text]
    urls = []
    for child in children:
        body = requests.get(child, timeout=30).text
        node = ET.fromstring(body)
        urls += [loc.text for loc in node.iterfind(".//s:loc", NS)]
    return urls

The lastmod field is the quiet win. On a daily run, only re-scrape products whose lastmod moved since your last crawl. That turns a full re-scrape into a diff, cutting request volume by an order of magnitude on a large catalog and keeping you well under any rate limit.

Running the requests above straight from your own IP works until it doesn't. Hammer one address and a store's edge, often Cloudflare, starts returning 429s and then challenge pages. Routing through the SparkProxy Scraping API makes the anti-bot problem one parameter instead of a proxy fleet you maintain.

The base is https://scrape.sparkproxy.io/api/v1, authenticated with an X-API-Key header. The key move for Shopify: because /products.json is static JSON, set render_js=false. That's a plain HTTP fetch, roughly 3x faster and just 1 credit per request instead of 5 for a rendered page.

curl -s -G "https://scrape.sparkproxy.io/api/v1" \
  -H "X-API-Key: YOUR_API_KEY" \
  --data-urlencode "url=https://store.sparkproxy.io/products.json?limit=250&page=1" \
  --data-urlencode "render_js=false"

Here's a small wrapper that all the later code reuses:

import requests

API = "https://scrape.sparkproxy.io/api/v1"
API_KEY = "YOUR_API_KEY"

def api_get(target, render=False, residential=False, country=None):
    params = {"url": target, "render_js": str(render).lower()}
    if residential:
        params["premium_proxy"] = "true"
    if country:
        params["country_code"] = country
    return requests.get(API, headers={"X-API-Key": API_KEY},
                        params=params, timeout=90)

The parameters that earn their keep on Shopify targets:

A minority of stores put Cloudflare or a bot manager in front of the JSON feed, or disable it outright. The symptom is a request that used to return {...} now returning a 403, a 429, or an HTML challenge page whose body opens with and contains Just a moment.... A JSON endpoint that hands you HTML is a challenge, full stop.

Detect it, then escalate. Tier one is the cheap plain-HTTP call. If that looks blocked, retry with a real browser and a residential exit in the store's country, which is what clears the challenge:

import json

def looks_blocked(r):
    if r.status_code in (403, 429, 503):
        return True
    body = r.text.lstrip()
    if not body.startswith("{"):          # JSON endpoint returned non-JSON
        return True
    markers = ("Just a moment", "cf-chl", "Attention Required", "Access denied")
    return any(m in r.text for m in markers)

def fetch_shopify_json(target, country="US"):
    r = api_get(target, render=False)                 # Tier 1: 1 credit
    if looks_blocked(r):
        r = api_get(target, render=True,              # Tier 2: 25 credits
                    residential=True, country=country)
    r.raise_for_status()
    return json.loads(r.text)

The escalation ladder, in one table:

If a store disables the JSON feeds completely (tier three), fall back to the product page HTML, which still carries a JSON-LD Product block with price, currency, and availability:

import re, json

def product_from_page(store, handle, country="US"):
    target = f"{store}/products/{handle}"
    r = api_get(target, render=True, residential=True, country=country)
    for raw in re.findall(
            r'

',
            r.text, re.DOTALL):
        data = json.loads(raw)
        if data.get("@type") == "Product":
            return data
    return None

Residential IPs matter here because Cloudflare scores datacenter ranges more harshly than home connections. For the broader techniques, see How to Bypass Cloudflare When Web Scraping.

Put it together: paginate /products.json through the escalating fetcher, flatten to variant rows, and stream to JSONL so a crash never costs you the whole run. Jitter between requests keeps you polite and under the radar.

import json, time, random

def scrape_store(store, country="US", out="catalog.jsonl"):
    page, total = 1, 0
    with open(out, "w", encoding="utf-8") as f:
        while True:
            target = f"{store}/products.json?limit=250&page={page}"
            data = fetch_shopify_json(target, country=country)
            batch = data.get("products", [])
            if not batch:
                break
            for product in batch:
                for row in variant_rows(product):
                    f.write(json.dumps(row, default=str) + "\n")
                    total += 1
            page += 1
            time.sleep(random.uniform(1.0, 2.5))   # be a good guest
    return total

if __name__ == "__main__":
    n = scrape_store("https://store.sparkproxy.io", country="US")
    print(f"wrote {n} variant rows")

That's a whole Shopify store, every product and variant with prices and SKUs, in a format you can load into Postgres or a dataframe. Because the extraction rides a stable JSON contract rather than CSS selectors, it keeps working through theme changes, which is the maintenance cost that sinks DOM scrapers. If you'd rather weigh a managed API against running your own proxies for this, Web Scraping API vs Self-Managed Proxies lays out the tradeoff.