How to Scrape Costco Product Data: What Works

Everything below was run on 9 October 2026 against the US site, from an Indian residential IP on AS55836 (Reliance Jio). The egress matters, so I'm stating it: a US datacentre range may be treated differently, and you should repeat these probes from your own exit before trusting the numbers.

Costco.com is a Next.js App Router application. The response headers give it away immediately:

x-build-reference: prd-usbc-release-v1.0.79-v0.0.1294-80
x-nextjs-prerender: 1
x-nextjs-stale-time: 300
x-nextjs-cache: HIT
x-middleware-rewrite: /en-us/p/-/4201021105
x-costco-gdx-deployment: blue

There is no __NEXT_DATA__ script anywhere. Every hydration payload arrives as self.__next_f.push([1,"..."]) chunks, the React Server Components flight format. If you copied a Walmart-style __NEXT_DATA__ parser over to Costco, that's why it returned nothing.

The second header worth staring at is x-nextjs-stale-time: 300. Product pages are statically prerendered and served from an edge cache with a five-minute stale window, which is also the upper bound on how fresh your scraped price can be. No amount of browser automation improves that.

Here is the whole catalogue in two numbers. The product sitemap sitemap_lw_p_001.xml carried 17,698 URLs with a lastmod of the previous day. Median page weight was 280,676 bytes with Accept-Encoding: br, gzip, and 2.65 MB without. Send the header and a full-catalogue pass is roughly 5 GB; forget it and the same pass is roughly 47 GB for identical data.

Costco serves product data through three separate surfaces, and they behave nothing like each other. Pick the wrong one and you'll spend a week fighting a wall that you never needed to touch.

The listing page is not partially rendered or lazily rendered. It is empty. I decoded all 78 flight chunks from /gaming-monitors.html into 1.74 MB of payload and counted zero occurrences of onlinePrice and zero unique itemNumber values. The document even ships a Next.js tombstone announcing it:

<template data-dgst="BAILOUT_TO_CLIENT_SIDE_RENDERING"></template>

That marker is Next.js saying a Suspense boundary gave up on the server and deferred to the client. The grid is one of those boundaries. Adding ?currentPage=2&pageSize=24 changes nothing, because the server still returns the same cached shell: identical x-middleware-rewrite: /en-us/c/gaming-monitors, identical x-nextjs-cache: HIT. There is no server-rendered page two to fetch.

So the practical plan writes itself. Get your URLs from the sitemap, get your fields from the product page, and treat the listing API as off-limits.

Costco's robots.txt is unusual: most of it is commented out. Of the roughly 90 Disallow lines in the file, the large block covering ProductDisplay, CategoryDisplay, CatalogSearchResultView and friends sits behind # characters, leftovers from the WebSphere Commerce era. Read only the live lines.

The rules that are active and relevant:

Disallow: /CatalogSearch*
Disallow: /Browse/
Disallow: /Warehouse/
Disallow: /*?*pageSize=*
Allow:    /*?currentPage=*pageSize=*
Disallow: /*?*sortBy=*
Sitemap:  https://www.costco.com/sitemap_lw_index.xml

Three things follow. Search result pages are disallowed, and /CatalogSearch?keyword=... 301-redirects to /s?keyword=... anyway. Any URL carrying sortBy is disallowed, so sorted crawls are out. And pageSize is disallowed except in the specific currentPage=...&pageSize=... combination, which is explicitly allowed. Product pages under *.product..html and /p/-/ are not disallowed at all.

There is no Crawl-delay directive, which is not permission to hammer the origin. Pick your own ceiling. One request per second per IP kept everything at 200 for me across a few hundred requests.

Since search is disallowed and listings are empty, discovery comes from sitemap_lw_index.xml. It lists 15 shards. The product shards are sitemap_lw_p_001.xml and its _mod_ mirror, both carrying 17,698 URLs.

The sitemap publishes the legacy URL form, and that form redirects:

GET /msi-27%22-gaming...monitor.product.4201021105.html
  -> 302 Found
  -> x-powered-by: Servlet/3.0
  -> location: /p/-/msi-27-gaming...monitor/4201021105?DM_PersistentCookieCreated=true&langId=-1

The old WebSphere stack answers the legacy path and bounces you to the Next.js canonical. Two useful facts fall out. First, the number at the end is the only identifier that matters: x-middleware-rewrite shows the slug is discarded, and https://www.costco.com/p/-/4201021105 with no slug at all returns the identical 2,673,366-byte page. Second, you can skip the redirect entirely by rewriting the sitemap URLs yourself and saving a round trip per product.

import re
import httpx

SITEMAP = "https://www.costco.com/sitemap_lw_p_001.xml"
UA = ("Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
      "(KHTML, like Gecko) Chrome/140.0.0.0 Safari/537.36")

def catalog_item_ids(sitemap_url=SITEMAP):
    """Return every Costco item id in a product sitemap shard."""
    # Let httpx negotiate and decode compression itself. Setting
    # Accept-Encoding by hand is how you end up decoding brotli manually.
    xml = httpx.get(sitemap_url, headers={"User-Agent": UA}, timeout=90).text
    ids = re.findall(r"\.product\.(\d+)\.html", xml)
    return sorted(set(ids), key=int)

ids = catalog_item_ids()
print(len(ids), "items")                      # 17698 items
print(f"https://www.costco.com/p/-/{ids[0]}")  # canonical, no slug, no redirect

Hunting an undocumented endpoint is often the right move on a retail site, and we have a full method for that. Costco is the case where it isn't: every candidate endpoint sits behind a gateway, and the sitemap already hands you the complete catalogue for free.

Each product page carries two JSON-LD blocks: a BreadcrumbList giving you the full category path, and a Product block with the fields you came for. This is the real payload:

{
  "@context": "https://schema.org",
  "@type": "Product",
  "name": "MSI 27\" Gaming QHD QD-OLED 240Hz 0.03ms NVIDIA G-Sync & AMD FreeSync Monitor",
  "sku": "2056897",
  "brand": { "@type": "Brand", "name": "MSI" },
  "aggregateRating": { "@type": "AggregateRating", "ratingValue": 4.88, "ratingCount": 8 },
  "offers": {
    "@type": "Offer",
    "availability": "https://schema.org/OutOfStock",
    "priceCurrency": "USD",
    "price": 349.99
  }
}

Note the two different identifiers. sku is 2056897, the seven-digit Item # printed on the warehouse shelf tag and your receipt. The URL's 4201021105 is the internal catalogue id. In my 60-product sample, 54 SKUs were seven digits while the URL ids split 35 at ten digits and 24 at nine, a visible seam between catalogue generations. Key your database on both: the Item # is what a buyer recognises, the URL id is what fetches the page.

For the fetch itself, use a client that presents a real browser TLS fingerprint. curl_cffi is the shortest path in Python. Costco did not challenge my requests, so this is insurance rather than a workaround, and insurance that costs nothing. The impersonate argument also handles the compression negotiation for you, which is where the 2.65 MB to 280 KB saving comes from. Don't set Accept-Encoding by hand on top of it.

import json
import re
from curl_cffi import requests as cffi

LD = re.compile(
    r'<script[^>]*application/ld\+json[^>]*>(.*?)</script>', re.S)

def fetch_product(item_id):
    r = cffi.get(
        f"https://www.costco.com/p/-/{item_id}",
        impersonate="chrome",      # Chrome JA3/JA4 + Chrome's accept-encoding
        headers={"Accept-Language": "en-US,en;q=0.9"},
        timeout=60,
    )
    r.raise_for_status()
    for block in LD.findall(r.text):
        try:
            doc = json.loads(block)
        except json.JSONDecodeError:
            continue
        if doc.get("@type") == "Product":
            return doc
    raise LookupError(f"no Product JSON-LD on {item_id}")

One request, no browser, no token. On a retailer this size that's a pleasant surprise, and it's the opposite of what Home Depot's Akamai setup does to a plain client.

This is the part no other Costco guide mentions, and it will cost you a day if you miss it.

I pulled 60 product pages at random from the sitemap. All 60 returned HTTP 200. All 60 reported "availability": "https://schema.org/OutOfStock". Every single one, across monitors, groceries, furniture and jewellery, including items priced at $9.97 with 47 reviews. A catalogue where every random sample is out of stock isn't a catalogue, it's a default value baked into the prerender. The real stock state arrives client-side from ecom-api.costco.com/ebusiness/inventory/v1/inventorylevels/availability/batch, which needs a token the static page cannot have. So: ignore offers.availability entirely.

The second field is more subtle and genuinely useful. 58 of the 60 carried offers.price; two did not. Here's one of them:

{
  "name": "Interstate 8 Volt Golf Cart Battery",
  "sku": "850984",
  "description": "Interstate 8 Volt Golf Cart Battery, Costco Item Number: 850984, GC8 Group Size, 8 Volt, Not Available for Purchase Online,  May Be Available at Your Local Costco Warehouse,",
  "offers": { "@type": "Offer", "priceCurrency": "USD",
              "availability": "https://schema.org/OutOfStock" }
}

A missing price key, with priceCurrency still present, is Costco's marker for a warehouse-only item. Those products are in the sitemap and carry ratings, but they are not sold online. The description usually spells it out. Treat the absence as a signal, not an error:

WAREHOUSE_ONLY = "not available for purchase online"

def costco_fields(doc):
    offer = doc.get("offers") or {}
    price = offer.get("price")
    rating = doc.get("aggregateRating") or {}
    desc = (doc.get("description") or "").lower()
    return {
        "item_number": doc.get("sku"),
        "title": doc.get("name"),
        "brand": (doc.get("brand") or {}).get("name") or None,
        "price": float(price) if price not in (None, "", 0) else None,
        "currency": offer.get("priceCurrency"),
        "rating": rating.get("ratingValue"),
        "reviews": rating.get("ratingCount"),
        "sold_online": price not in (None, "", 0),
        "warehouse_only": WAREHOUSE_ONLY in desc,
        # availability is a constant in the prerender; never read it
        "stock_known": False,
    }

One more trap in the raw HTML. Grepping a page for Add to Cart or Out of Stock finds both on the same document, because the flight payload ships the entire i18n label dictionary: {"key":"instock","value":"In Stock"}, {"key":"lowstock","value":"Low Stock"}, {"key":"outofstock","value":"Out of Stock"}. Those are translation strings, not state. String-matching stock labels in Costco HTML produces noise in both directions.

Costco runs Kasada, and it tells you exactly where. Every page I fetched, homepage, category and product, contains this inline script:

document.addEventListener('kpsdk-load', async function () {
  var result = await window.KPSDK.configure({
    resourceAddress: window.location.origin +
      '/149e9513-01fa-4fb0-aad4-566afd725d1b/2d206a39-8ed7-437e-a3be-862e0f06eea3/',
    endpoints: [
      { domain: window.location.hostname,            path: '/*' },
      { domain: "gdx-api.costco.com",                path: '/*' },
      { domain: "gdx-api-mobileapps.costco.com",     path: '/*' },
      { domain: "ecom-api.costco.com",               path: '/*' }
    ]
  });
});

The SDK loads from /149e9513-.../2d206a39-.../p.js and posts its device fingerprint to /fp and /mfc under the same prefix. The 149e9513-01fa-4fb0-aad4-566afd725d1b prefix is Kasada's own constant and is how you identify the vendor on any site.

Read the endpoints array as a map of the enforcement perimeter. Kasada signs requests to four domains: the site itself plus the three API hosts. That is why the behaviour splits the way it does. A product page document is a cached static asset that the CDN answers before any token check, so an unsigned client gets it. An XHR to gdx-api.costco.com is signed, so an unsigned client gets 401 APIGATEWAY.CLIENT.FAULT from Apigee X with "description": "Missing credentials". The gateway credential is a static UUID that Costco ships to browsers in the page bundle. Replaying it makes your scraper an unauthorised API client, it rotates without notice, and it is the single most brittle thing you could build on. Don't.

If you need the background on how the token handshake works, we have a full write-up on Kasada. For Costco specifically, the point of this section is that you don't need to defeat it, because the field you want never leaves the cache.

Two more defences are wired in and currently quiet. Queue-it is loaded on every page: my headless run pulled queueclient.min.js, queueconfigloader.min.js, and Costco's own assets.queue-it.net/costco/integrationconfig/javascript/queueclientConfig.js. That waiting room is dormant until a hot drop turns it on, at which point your crawler sits in a FIFO queue with everyone else, so build for it before it happens and read how these waiting rooms behave. The CSP also permits h.online-metrix.net and mpsnare.iesnare.com, device-fingerprinting services that are checkout-scoped and irrelevant to public product data.

The obvious response to an empty grid is to render it. I tried, with a stock headless Chromium, a real Chrome user agent, a 1440x1000 viewport, America/Los_Angeles, six wheel scrolls and 21 seconds of settle time.

It did not work, and the failure mode is precise:

title: Gaming Monitors
body text len: 2749 | product links: 0 | skeletons: 156
prices:
--- visible text head ---
My Warehouse  Seattle   Delivery Location  98101
Filter Results  Clear All
Loading  Loading  Loading  Loading  Loading ...
--- failed requests ---
net::ERR_ABORTED https://www.costco.com/149e9513-.../2d206a39-.../fp
net::ERR_ABORTED https://www.costco.com/149e9513-.../2d206a39-.../mfc

The page chrome rendered correctly and even geolocated to a default Seattle warehouse with ZIP 98101. The grid stayed as 156 skeleton elements, forever. The cause is in the last two lines: the Kasada fingerprint posts were aborted, so no token was issued, so the search XHR never produced data. A headless browser that cannot complete the /fp handshake is no better off than curl, and it costs you five times as much.

That's the honest version of the escalation ladder. For Costco, rendering buys you nothing on listings and nothing on product pages either, since the price is already in the static HTML. Rendering is worth paying for only when a page's content genuinely materialises after hydration, which is the general case we cover in scraping single-page applications. Costco's product pages are not that case.

If you'd rather not run the proxy pool, the fingerprinting and the retry logic yourself, SparkProxy's Scraping API covers the same ground. Base URL is https://scrape.sparkproxy.io/api/v1, auth is the X-API-Key header, and signup includes 1,000 free credits.

The important parameter for Costco is render_js. It defaults to true and costs 5 credits. Everything above says you want it off, which is a plain HTTP fetch at 1 credit and roughly three times faster.

curl -G "https://scrape.sparkproxy.io/api/v1" \
  -H "X-API-Key: $SPARKPROXY_KEY" \
  --data-urlencode "url=https://www.costco.com/p/-/4201021105" \
  --data-urlencode "render_js=false" \
  --data-urlencode "proxy_type=premium" \
  --data-urlencode "country_code=US"

country_code=US adds 5 credits and exits through a US IP, which is what you want for US pricing and for consistent default-warehouse behaviour. Drop it when you're only collecting catalogue metadata.

For a full-catalogue pass, batch mode is the reason to use the API at all. Pass several URLs in url and the whole batch costs 1 credit, with the hard constraint that batch requires render_js=false:

import requests

API = "https://scrape.sparkproxy.io/api/v1"
KEY = "sk-your-key"

def scrape_batch(item_ids):
    """One credit per batch. render_js must be false."""
    urls = [f"https://www.costco.com/p/-/{i}" for i in item_ids]
    r = requests.post(API, headers={"X-API-Key": KEY}, json={
        "url": urls,
        "render_js": False,
        "proxy_type": "premium",
        "country_code": "US",
    }, timeout=300)
    r.raise_for_status()
    payload = r.json()
    print(payload["job_id"], payload["credits_used"], "credit(s)")
    return [row for row in payload["results"] if row["success"]]

def product_ld(html):
    for block in LD.findall(html):          # LD is the regex from Step 1
        try:
            doc = json.loads(block)
        except json.JSONDecodeError:
            continue
        if doc.get("@type") == "Product":
            return doc
    return None

for row in scrape_batch(ids[:20]):
    doc = product_ld(row["body"])
    if doc:
        print(costco_fields(doc))

Batch responses come back as {job_id, duration_ms, credits_used, results: [...]}, where each row is {url, success, httpStatus, body, error} and body is the page text, not base64. The job id is prefixed crawl_ so batch jobs are easy to pick out of your logs later.

Keep the batches modest anyway. The whole batch bills as one credit regardless of size, but the URLs are fetched in parallel, so a 500-URL batch is a 500-way parallel burst at one origin and a good way to earn the rate limiting you were avoiding. At 20 URLs per batch, the full 17,698-product catalogue is 885 batches, which is 885 credits plus the per-request country surcharge if you keep it. Bandwidth isn't a line item on a SparkProxy plan at all, which is the part that matters when the uncompressed version of this job is 47 GB.

Two parameters deserve a warning, because both have been used wrongly in the wild. json_response=true is the only thing that returns the metadata envelope, and the body inside it is base64. Decode before you match anything:

import base64

r = requests.get(API, headers={"X-API-Key": KEY}, params={
    "url": "https://www.costco.com/p/-/4201021105",
    "render_js": "false",
    "json_response": "true",
})
env = r.json()
print(env["status_code"], env["duration_ms"], "ms", env["credits_used"], "credits")
html = base64.b64decode(env["body"]).decode("utf-8", "replace")   # required
doc = product_ld(html)                   # only now will any matching work
print(doc["sku"], doc["offers"].get("price"))

format is a different thing: it converts the content to html, md, mdx, json, screenshot or pdf. format=json does not give you the envelope. And extract_rules replaces the response with an extracted object, so never send it together with json_response:

rules = {"jsonld": {"selector": 'script[type="application/ld+json"]',
                    "type": "list"}}

r = requests.get(API, headers={"X-API-Key": KEY}, params={
    "url": "https://www.costco.com/p/-/4201021105",
    "render_js": "false",
    "extract_rules": json.dumps(rules),
})
blocks = r.json()["extracted"]["jsonld"]       # no "body" key in this shape

If you'd rather see the target's real status code instead of the API's, add transparent_status_code=true. It applies only when render_js=false, which is exactly the mode you're in here, and it makes block detection a status-code check rather than a guessing game.

Four things on Costco are not reachable from a stateless, unauthenticated client, and you should plan around them rather than burn a sprint discovering them.

Real-time and warehouse-level stock. The inventory endpoint is Kasada-signed and gateway-authenticated. The prerendered availability field is a constant. A stateless client gets no stock data at all, only whether an item is sold online.

Member-only pricing and instant savings. Those render after sign-in, against a membership profile fetched from ecom-api.costco.com/ebusiness/customer/v1/loyalty/membershipprofile/status. Scraping an authenticated area is out of bounds here, so the public price is the ceiling of what you can collect.

Individual review text. The aggregateRating block gives you ratingValue and ratingCount for free, which covers most price-monitoring needs. Full review bodies come from Bazaarvoice via display.ugc.bazaarvoice.com, keyed by a display passkey, and that's a separate integration with its own terms.

Session continuity. Costco's warehouse and ZIP context lives in cookies, and the Scraping API's session_id is a log label, not a persistent cookie jar. If you need a pinned warehouse across requests, inject the cookies explicitly with the cookies parameter on every call and verify the ZIP echoed back in the page.

Short version: public pages only. Product pages, prices and aggregate ratings are published to anyone with a browser, and that's what this guide collects. Don't sign in and scrape member pricing, don't touch checkout or account paths, and don't collect reviewer names or any other personal data you have no use for.

Respect the live robots.txt rules rather than the commented-out ones: no /CatalogSearch*, no sortBy URLs, no /Browse/ or /Warehouse/. There's no Crawl-delay, so set your own and keep concurrency modest. Replaying the embedded API credential to reach gdx-api or ecom-api is unauthorised access to a private interface, not clever scraping. The sitemap plus the product page gives you the entire online catalogue without going anywhere near it.