How to Scrape Best Buy Product Data: SKUs & Stock

Settle the framing before you write any code, because "it's public" answers a narrower question than most people assume.

In the United States, the Ninth Circuit's decision in hiQ Labs v. LinkedIn (2022) held that scraping data that is publicly accessible, meaning no login and no authentication wall, generally does not violate the Computer Fraud and Abuse Act. That ruling is about unauthorized access, not a blanket license to do whatever you want with the data afterward. Best Buy's Terms and Conditions separately prohibit automated data collection, so a scraper can breach that contract even when it clears the CFAA bar. Two different legal questions, and public access only settles the first.

There is a Best Buy-specific wrinkle that changes the calculus, and it is worth knowing before you build anything. Best Buy publishes an official Best Buy Developer API (the products and stores API at api.bestbuy.com). It exposes SKU, name, salePrice, regularPrice, online availability, store inventory, and category data through a documented, key-authenticated interface. Access has been gated over the years and you have to apply, but where it covers your use case, the sanctioned API is the compliant path and it sidesteps the entire anti-bot problem. Reach for the scraper when the API does not expose a field you need, when your access request is pending, or when you are pulling public data the API does not surface.

Two housekeeping checks belong in any serious scraping project:

• Read bestbuy.com/robots.txt. It disallows a range of paths (cart, checkout, account, and internal search or API endpoints among them). Robots.txt is not law, but respecting it is the baseline for defensible, good-faith scraping, and ignoring it is the first thing a plaintiff points at.
• Collect public product data only. SKU, price, availability, ratings. Never anything behind a login, and never personal data about the customers who wrote reviews.

Guardrails that keep a project on the right side of the line: rate-limit yourself and back off on errors so you are not degrading the site for real shoppers, don't republish copyrighted assets (product photography, full review text) beyond fair use, and if the data feeds a commercial product, run it past a lawyer. This is engineering guidance, not legal advice. Price monitoring, catalog research, and competitive intelligence are common, legitimate uses of public retail data; for the business context, see how e-commerce companies use proxies for competitive intelligence.

A public Best Buy product page lives at /site//.p?skuId=, where is the numeric SKU that uniquely names the product. That SKU is your primary key. It appears in the URL, in the page's JSON-LD, and in Best Buy's own API, which makes it the one identifier that ties every data source together. Here is the reference set worth pulling, along with where each field lives in the JSON-LD Product object.

One point that catches people out: for Best Buy, the online price in offers.price is a single national number. Best Buy does not run the store-by-store base-price variance that grocery-heavy retailers do. The store-level dimension on Best Buy is availability, not price. Open-box and clearance items are the exception, since those are priced per unit and per condition, so treat an open-box listing as its own SKU-plus-condition record rather than a variant of the new item.

Best Buy's storefront is a React application, so scraping the rendered DOM with CSS selectors is the brittle path: class names are hashed and the layout shifts. Skip it. Every product page embeds a JSON-LD script tag that describes the product in schema.org's vocabulary:

<script type="application/ld+json">
{"@context":"https://schema.org","@type":"Product","sku":"6565837",
 "name":"Sony - WH-1000XM5 Wireless Headphones",
 "offers":{"@type":"Offer","price":399.99,"priceCurrency":"USD",
           "availability":"https://schema.org/InStock"},
 "aggregateRating":{"@type":"AggregateRating","ratingValue":4.7,"reviewCount":2841}}
</script>

That one detail changes the whole job. Instead of chasing selectors, you grab the JSON, parse it once, and read typed fields straight off the object. Price is a number, reviewCount is an integer, availability is a schema.org enum you can normalize to a boolean. No regex on "$399.99", no guessing which of five DOM containers holds the live price this session.

Two practical catches. First, a Best Buy page usually ships more than one JSON-LD block. You will often see a BreadcrumbList and sometimes an Organization block alongside the Product, so you cannot grab the first script tag and trust it. Filter by @type == "Product". Second, the JSON-LD carries the aggregate rating but not the individual reviews; those load from a separate widget, which we cover in the reviews section. For SKU, price, availability, and the rating summary, JSON-LD is the cleanest and most stable source on the page.

Getting the HTML is the hard part, and there are two reasons, one constant and one occasional.

The constant is Akamai Bot Manager. Best Buy sits behind Akamai's edge, and Akamai does not just check IP reputation. It fingerprints the TLS/JA3 handshake, inspects header order, and validates a _abck cookie that is set only after client-side sensor JavaScript runs and posts telemetry back. A plain requests.get with default headers fails that sensor check before the IP even matters, which is why bare HTTP clients get walled almost immediately. When Akamai decides you are a bot, the usual response is an "Access Denied" page carrying an Akamai reference number, often on a 403 but sometimes wrapped so the status looks benign. You have to inspect the body, not just the status code. The mechanics of _abck, bm_sz, and the sensor payload are covered in depth in how to bypass Akamai Bot Manager.

The occasional reason is the virtual waiting room. On high-demand launches (GPU restocks, new consoles, popular preorders), Best Buy puts shoppers into a queue: a "please wait" or "you're in line" holding page that throttles traffic to the product. For everyday catalog and price scraping you rarely see it, but if you target a product on launch day you will, and a scraper that treats the queue page as a product page stores garbage. Detect it and back off rather than hammering through.

A managed scraping API absorbs the fingerprint, the sensor execution, IP rotation, and rendering for you. For the proxy-side theory on why some IPs survive and others get burned, how to avoid getting your proxy blocked goes deep.

The SparkProxy Scraping API takes a target URL and handles the proxy, rotation, browser rendering, and anti-bot layer. You send one request and get the rendered HTML back. For Best Buy, four parameters carry the weight:

• render_js=true: runs the page in a real Chromium browser, which executes Akamai's sensor JavaScript, produces a legitimate fingerprint, and returns the hydrated HTML including the JSON-LD.
• premium_proxy=true: routes through residential IPs, which blend with real shopper traffic where datacenter ranges get challenged.
• stealth=true: adds extra anti-detection layers tuned for stacks like Akamai. It requires render_js=true.
• country_code=US: sets a US exit, which matters because bestbuy.com is a US storefront and geo-mismatched requests draw friction.

The base endpoint is https://scrape.sparkproxy.io/api/v1 and auth is a single header, X-API-Key. A minimal request in cURL:

curl -G "https://scrape.sparkproxy.io/api/v1" \
  -H "X-API-Key: YOUR_API_KEY" \
  --data-urlencode "url=https://www.bestbuy.com/site/6565837.p?skuId=6565837" \
  --data-urlencode "render_js=true" \
  --data-urlencode "premium_proxy=true" \
  --data-urlencode "stealth=true" \
  --data-urlencode "country_code=US"

The full parameter list and response fields are in the Scraping API docs. If you are weighing this against running your own proxy pool for a target this well defended, web scraping API vs self-managed proxies lays out the trade-off honestly.

Start with one product. Wrap the request so every call carries the Best Buy-specific parameters, and give it a generous timeout since a rendered request drives a real browser.

import requests

API = "https://scrape.sparkproxy.io/api/v1"
API_KEY = "YOUR_API_KEY"

def fetch_url(target: str, country: str = "US") -> str:
    resp = requests.get(
        API,
        headers={"X-API-Key": API_KEY},
        params={
            "url": target,
            "render_js": "true",       # runs Akamai's sensor JS + returns hydrated HTML
            "premium_proxy": "true",   # residential IPs blend with shopper traffic
            "stealth": "true",         # extra anti-detection (needs render_js)
            "country_code": country,   # US storefront expects US traffic
        },
        timeout=90,
    )
    resp.raise_for_status()
    return resp.text

def product_url(sku: str) -> str:
    # The slug is cosmetic; Best Buy resolves the page from the skuId alone.
    return f"https://www.bestbuy.com/site/{sku}.p?skuId={sku}"

def fetch_product(sku: str) -> str:
    return fetch_url(product_url(sku))

You do not need the human-readable slug. Best Buy resolves the page from the skuId, so /site/.p?skuId= is enough and the API follows any redirect to the canonical URL. Next, read the JSON out of that HTML.

Two moves make Best Buy parsing reliable: pull every JSON-LD script tag, then select the one whose @type is Product instead of trusting position.

import json
from selectolax.parser import HTMLParser

def extract_product_ldjson(html: str) -> dict | None:
    """Return the JSON-LD block whose @type is Product, or None."""
    tree = HTMLParser(html)
    for node in tree.css('script[type="application/ld+json"]'):
        try:
            data = json.loads(node.text())
        except json.JSONDecodeError:
            continue
        # A page can ship a list of blocks or a single object.
        for block in (data if isinstance(data, list) else [data]):
            if isinstance(block, dict) and block.get("@type") == "Product":
                return block
    return None

selectolax is a fast C-backed HTML parser; install it with pip install selectolax. Once you have the Product block, mapping fields is a matter of safe .get() chains, because Best Buy omits keys rather than nulling them (a preorder item may lack a numeric price, mpn is not always present):

def norm_availability(value: str | None) -> str | None:
    """Turn a schema.org URL into a short status string."""
    if not value:
        return None
    tail = value.rstrip("/").rsplit("/", 1)[-1]  # ".../InStock" -> "InStock"
    return tail or None

def parse_product(html: str) -> dict | None:
    block = extract_product_ldjson(html)
    if not block:
        return None

    offer = block.get("offers", {}) or {}
    if isinstance(offer, list):           # some pages nest a list of offers
        offer = offer[0] if offer else {}
    brand = block.get("brand", {}) or {}
    rating = block.get("aggregateRating", {}) or {}

    return {
        "sku":          block.get("sku"),
        "title":        block.get("name"),
        "brand":        brand.get("name") if isinstance(brand, dict) else brand,
        "model":        block.get("mpn") or block.get("model"),
        "price":        offer.get("price"),
        "currency":     offer.get("priceCurrency"),
        "availability": norm_availability(offer.get("availability")),
        "in_stock":     norm_availability(offer.get("availability")) == "InStock",
        "rating":       rating.get("ratingValue"),
        "review_count": rating.get("reviewCount"),
        "image":        block.get("image"),
        "gtin":         block.get("gtin13") or block.get("gtin"),
    }

Because the values are typed, you get a float price and an int review count with no post-processing. Two habits pay off. Keep both the raw availability string and a derived in_stock boolean, so a later Best Buy status you have not seen yet (BackOrder, LimitedAvailability) does not silently read as "in stock." And store the gtin when it is present, because it is the field that lets you match the same physical product across Best Buy, Amazon, and Walmart when you build a cross-retailer feed.

This is the check that separates a scraper you can trust from one that quietly stores denial pages. Because Akamai's "Access Denied" response and the waiting-room page can arrive without a clean error status, raise_for_status() will not always catch them, and a missing JSON-LD block does not tell you why it is missing. Scan the body for the markers, and treat a missing Product block as a soft block too:

def is_blocked(html: str) -> bool:
    """Akamai / the waiting room can serve a page that isn't the product."""
    markers = (
        "access denied",
        "reference #",           # Akamai edge error reference id
        "you don't have permission",
        "please wait",           # virtual waiting room
        "you're in line",
        "/akam/",                # Akamai sensor path leaking into markup
    )
    lowered = html.lower()
    if any(m in lowered for m in markers):
        return True
    # A real product page always carries a Product JSON-LD block.
    return extract_product_ldjson(html) is None

Now a fetch is honest: it either returns a real product page or tells you it was blocked so you can retry. Pair this with stealth=true and residential rotation, and Akamai challenges become an occasional retry rather than a wall. Keep the is_blocked marker list in one place, since Akamai reworks its denial copy from time to time and you want a single spot to update.

Here is the Best Buy-specific gotcha. The online price you read from JSON-LD is national, but pickup availability is per store. The same SKU can be "available today" at a store in one metro and "unavailable" fifty miles away, and that difference is often the whole point of the scrape for a fulfillment or inventory feed.

There are two clean ways to get store inventory, and they sit at different points on the compliance-versus-flexibility line.

The official Stores API. The Best Buy Developer API exposes store inventory: you query product availability for a SKU against a store, or list stores near a postal code, and get JSON back. When your access covers this, it is the sanctioned and least fragile route, no rendering and no anti-bot involved. Prefer it where it fits.

The site's fulfillment endpoint. The product page's "Check Stores" widget fires a background XHR to an internal fulfillment endpoint that returns availability keyed by store ID for a given location. You can reproduce that call, but the exact path and payload shape change, so capture the live request from your browser's network panel rather than hard-coding a URL you found in an old blog post. The stable idea is the shape of the answer: a JSON list of stores, each with a store ID and an in-stock flag for the SKU. Fetch it through the same rendered, residential-backed request so it inherits your anti-bot posture:

import json

def fetch_store_availability(endpoint: str) -> list[dict]:
    """
    `endpoint` is the fulfillment XHR you captured from the product page's
    'Check Stores' action (it encodes the skuId and a location). The response
    is JSON, so parse the body rather than the rendered HTML.
    """
    raw = fetch_url(endpoint)          # same render_js + residential request
    try:
        data = json.loads(raw)
    except json.JSONDecodeError:
        return []                      # blocked or non-JSON: treat as no data

    stores = []
    # Shape varies; walk to the per-store availability list you saw in devtools.
    for store in data.get("stores", data.get("locations", [])):
        stores.append({
            "store_id":   store.get("storeId") or store.get("id"),
            "city":       store.get("city"),
            "in_stock":   bool(store.get("inStoreAvailability")
                               or store.get("available")),
        })
    return stores

Whichever route you pick, standardize on a location per run (a fixed postal code or a fixed list of store IDs) and hold it constant, so a change in your data reflects a real inventory change and not a shift in which stores you asked about. If you are building a comparison or availability feed where that consistency is the entire game, datacenter proxies for price comparison websites covers the wider pattern.

The rating summary is easy: aggregateRating.ratingValue and reviewCount come straight out of the JSON-LD you already parsed, so a nightly job that only needs "how many stars, how many reviews" is done at that point.

Individual review text is a separate system. Best Buy's ratings and reviews are powered by Bazaarvoice, the same reviews platform many large retailers embed, and the review list loads from a Bazaarvoice-hosted widget rather than from the product HTML. That has two consequences. First, you will not find full review bodies in the JSON-LD, only the aggregate. Second, to pull individual reviews you request them from the Bazaarvoice display endpoint the page uses, paginated, keyed on the same Best Buy SKU. As with the fulfillment call, capture the live request from your network panel, because the client key and parameter names are specific to the deployment. The reliable pattern:

def parse_reviews(payload: dict) -> list[dict]:
    """
    `payload` is the JSON from the Bazaarvoice reviews request the page makes
    (it carries the SKU and a page offset). Map the fields you need.
    """
    rows = []
    for r in payload.get("Results", []):
        rows.append({
            "review_id":  r.get("Id"),
            "rating":     r.get("Rating"),          # 1 to 5
            "title":      r.get("Title"),
            "text":       r.get("ReviewText"),
            "submitted":  r.get("SubmissionTime"),
            "verified":   r.get("IsSyndicated") is False,
        })
    return rows

Two guardrails specific to reviews. Keep only what you need for analysis (rating distribution, recency, keyword signal) and do not republish full review text verbatim, since that is user-generated content with its own copyright and platform terms. And never attach reviewer names, profile links, or any other personal detail to your dataset. Aggregate sentiment is defensible; harvesting people is not.

Everything so far assumes you have a SKU. To build a catalog you need to discover them, and the cleanest source is Best Buy's own search and browse pages. Search URLs follow /site/searchpage.jsp?st=, and category/browse pages expose paginated product grids. Fetch a results page through the same rendered request, then pull the SKUs. Best Buy exposes the SKU on each product tile (commonly in a data-sku-id attribute), so a light DOM pass is enough here, and you do not need JSON-LD for the listing step:

from selectolax.parser import HTMLParser
from urllib.parse import quote_plus

def search_url(query: str, page: int) -> str:
    return f"https://www.bestbuy.com/site/searchpage.jsp?st={quote_plus(query)}&cp={page}"

def parse_search_skus(html: str) -> list[str]:
    tree = HTMLParser(html)
    skus = []
    for tile in tree.css("[data-sku-id]"):
        sku = tile.attributes.get("data-sku-id")
        if sku and sku.isdigit():
            skus.append(sku)
    # de-dupe while preserving order
    seen, unique = set(), []
    for s in skus:
        if s not in seen:
            seen.add(s)
            unique.append(s)
    return unique

Once you have SKUs, scrape their product pages at volume with three habits: retry on soft blocks, back off so you do not spike a single IP, and keep concurrency modest. With a scraping API the provider rotates the exit IP per request, so your ceiling is your plan's rate limit, not the number of proxies you own. Five to fifteen workers is plenty for a target this defended.

import time
import random
import csv
from concurrent.futures import ThreadPoolExecutor, as_completed

def fetch_with_retry(sku: str, attempts: int = 3) -> str | None:
    for i in range(attempts):
        html = fetch_product(sku)
        if not is_blocked(html):
            return html
        time.sleep(2 ** i + random.random())   # exponential backoff + jitter
    return None

def scrape_catalog(skus: list[str], workers: int = 8) -> list[dict]:
    out = []
    with ThreadPoolExecutor(max_workers=workers) as pool:
        futures = {pool.submit(fetch_with_retry, s): s for s in skus}
        for fut in as_completed(futures):
            html = fut.result()
            if html is None:
                continue
            record = parse_product(html)
            if record:
                out.append(record)
    return out

def save_csv(rows: list[dict], path: str = "bestbuy_products.csv") -> None:
    if not rows:
        return
    with open(path, "w", newline="", encoding="utf-8") as f:
        writer = csv.DictWriter(f, fieldnames=list(rows[0].keys()))
        writer.writeheader()
        writer.writerows(rows)

The jitter matters more than it looks: random.random() staggers retries so a batch of failures does not retry in lockstep and re-trigger the same rate limit. Persist as you go rather than holding everything in memory, so a crash at SKU 40,000 does not cost you the first 39,999. For a running price tracker, add a scraped_at timestamp to each row and write to a database keyed on (sku, scraped_at), which gives you a clean time series where every price point is comparable. The same pattern generalizes to sibling retailers; the extraction differs but the crawl scaffolding is identical to how to scrape Walmart product data.