How to Scrape Home Depot Product Data by Store

Settle the framing before you write any code, because "it's public" answers a narrower question than most people assume.

In the United States, the Ninth Circuit's decision in hiQ Labs v. LinkedIn (2022) held that scraping data that is publicly accessible, meaning no login and no authentication wall, generally does not violate the Computer Fraud and Abuse Act. That ruling is about unauthorized access, not a license to do whatever you want with the data afterward. Home Depot's Terms of Use separately prohibit automated data collection, so a scraper can breach that contract even when it clears the CFAA bar. Two different legal questions, and public access only settles the first.

There is a Home Depot wrinkle worth knowing before you build. Unlike Best Buy, which publishes an open, key-authenticated products API, Home Depot does not offer a self-serve public catalog API for general use. Its programmatic access is aimed at affiliates, suppliers, and Pro/enterprise partners, so for most price-monitoring or catalog-research use cases there is no sanctioned key-based shortcut. That raises the bar for doing public scraping responsibly rather than lowering it.

Two housekeeping checks belong in any serious project:

• Read homedepot.com/robots.txt. It disallows a range of paths (cart, checkout, account, and internal endpoints among them). Robots.txt is not law, but respecting it is the baseline for defensible, good-faith scraping, and ignoring it is the first thing a plaintiff points at.
• Collect public product data only. Price, SKU, availability, ratings. Never anything behind a login, and never personal data about the customers who wrote reviews.

Guardrails that keep a project on the right side of the line: rate-limit yourself and back off on errors so you are not degrading the site for real shoppers, don't republish copyrighted assets (product photography, full review text) beyond fair use, and if the data feeds a commercial product, run it past a lawyer. This is engineering guidance, not legal advice. Price monitoring, assortment research, and competitive intelligence are common, legitimate uses of public retail data; for the business context, see how e-commerce companies use proxies for competitive intelligence.

A public Home Depot product page lives at /p//, where is the numeric Internet # that uniquely names the product online. Home Depot actually gives you three identifiers, and knowing which is which saves you a lot of confusion later.

The Internet # is your anchor. Store it as the primary key and hang everything else off it. Here is the reference set worth pulling and where each field comes from.

The one nuance that trips people up: on Home Depot, price and inventory are both store-scoped. Best Buy runs a single national online price and varies only availability by store. Home Depot varies the price itself across stores and regions for many categories (appliances, lumber, seasonal, and clearance especially) and also exposes a per-store on-hand count. So the store you query is not a detail, it is part of the primary key.

Best Buy hands you a JSON-LD Product block. Walmart hands you a __NEXT_DATA__ blob. Home Depot hands you neither in full. Its storefront is a React application backed by a federated GraphQL architecture, so the product page loads a shell and then fetches the real data from a GraphQL gateway. The request goes to a /federation-gateway/graphql style endpoint as a POST, carrying an operation that takes the Internet # and a store id and returns typed JSON.

That changes your strategy. There are two clean ways in, and they sit at different points on the effort-versus-precision line.

Render and read the hydrated page. Load the product URL with a real browser, let the page make its own GraphQL call, and read what lands: a JSON-LD Product block (price and aggregate rating) plus the Internet #, Model #, and on-hand text in the hydrated DOM. This is the least fragile path for a handful of headline fields, and it reuses the same JSON-LD technique the Best Buy guide covers in depth.

Replay the GraphQL operation. For exact typed pricing and per-store inventory at scale, reproduce the call the page makes. Open your browser's network panel on a product page, filter to the graphql request, and copy three things: the operationName, the variables (you will see the item id and a store id), and the request headers the client sends (commonly x-experience-name, plus apollographql-client-name and apollographql-client-version). A representative query shape looks like this, though you should confirm the exact fields against the operation you capture, since Home Depot revises the schema:

query product($itemId: String!, $storeId: String) {
  product(itemId: $itemId) {
    itemId
    identifiers { brandName productLabel modelNumber storeSkuNumber upc }
    pricing(storeId: $storeId) { value original message }
    availabilityType { type buyable }
    fulfillment(storeId: $storeId) {
      fulfillmentOptions {
        type
        services {
          type
          locations { storeName storeNumber inventory { quantity isInStock } }
        }
      }
    }
    reviews { ratingsReviews { averageRating totalReviews } }
  }
}

This is the "hidden JSON API" pattern applied to a GraphQL backend, and it is worth reading the general playbook in how to scrape hidden JSON API endpoints and how to scrape GraphQL APIs. The catch specific to Home Depot is that the gateway is protected by the same anti-bot layer as the pages, so you cannot just fire the POST from a bare HTTP client. We handle that next.

Getting the data is the hard part, and the reason is Akamai Bot Manager. Home Depot sits behind Akamai's edge, the same family Best Buy uses, and Akamai does not just check IP reputation. It fingerprints the TLS/JA3 handshake, inspects header order, and validates an _abck cookie that is set only after client-side sensor JavaScript runs and posts telemetry back. A plain requests.get with default headers fails that sensor check before the IP even matters, which is why bare HTTP clients get walled almost immediately. When Akamai decides you are a bot, the usual response is an "Access Denied" page carrying a reference number, often on a 403 but sometimes wrapped so the status looks benign. You have to inspect the body, not just the status code. The mechanics of _abck, bm_sz, and the sensor payload are covered in depth in how to bypass Akamai Bot Manager.

There is a second-order problem unique to the GraphQL approach. The gateway expects the same valid _abck cookie and the client headers the page sends. Forge those by hand and you are back to reverse-engineering the sensor, which is a losing arms race. The clean move is to run the GraphQL call from inside a browser session that already passed the challenge, so it inherits the cookie for free. More on that in the inventory section.

A managed scraping API absorbs the fingerprint, the sensor execution, IP rotation, and rendering for you. For the proxy-side theory on why some IPs survive and others get burned, how to avoid getting your proxy blocked goes deep.

The SparkProxy Scraping API takes a target URL and handles the proxy, rotation, browser rendering, and anti-bot layer. You send one request and get the rendered result back. For Home Depot, four parameters carry the weight:

• render_js=true: runs the page in a real Chromium browser, which executes Akamai's sensor JavaScript, produces a legitimate fingerprint, and lets the storefront's own GraphQL call hydrate the page.
• premium_proxy=true: routes through residential IPs, which blend with real shopper traffic where datacenter ranges get challenged.
• stealth=true: adds extra anti-detection layers tuned for stacks like Akamai. It requires render_js=true.
• country_code=US: sets a US exit, which matters because homedepot.com is a US storefront and geo-mismatched requests draw friction.

The base endpoint is https://scrape.sparkproxy.io/api/v1 and auth is a single header, X-API-Key. A minimal request in cURL:

curl -G "https://scrape.sparkproxy.io/api/v1" \
  -H "X-API-Key: YOUR_API_KEY" \
  --data-urlencode "url=https://www.homedepot.com/p/RYOBI-ONE-18V-Drill-PCL206B/311452659" \
  --data-urlencode "render_js=true" \
  --data-urlencode "premium_proxy=true" \
  --data-urlencode "stealth=true" \
  --data-urlencode "country_code=US"

The full parameter list and response fields are in the Scraping API docs. If you are weighing this against running your own proxy pool for a target this well defended, web scraping API vs self-managed proxies lays out the trade-off honestly.

Start with one product. Wrap the request so every call carries the Home Depot parameters, and give it a generous timeout since a rendered request drives a real browser.

import requests

API = "https://scrape.sparkproxy.io/api/v1"
API_KEY = "YOUR_API_KEY"

def fetch_url(target: str, country: str = "US") -> str:
    resp = requests.get(
        API,
        headers={"X-API-Key": API_KEY},
        params={
            "url": target,
            "render_js": "true",       # runs Akamai's sensor JS + hydrates the page
            "premium_proxy": "true",   # residential IPs blend with shopper traffic
            "stealth": "true",         # extra anti-detection (needs render_js)
            "country_code": country,   # US storefront expects US traffic
        },
        timeout=90,
    )
    resp.raise_for_status()
    return resp.text

With the page HTML in hand, the fastest headline read is the JSON-LD Product block Home Depot embeds. Filter by @type == "Product", because the page ships more than one JSON-LD block:

import json
from selectolax.parser import HTMLParser

def extract_product_ldjson(html: str) -> dict | None:
    """Return the JSON-LD block whose @type is Product, or None."""
    tree = HTMLParser(html)
    for node in tree.css('script[type="application/ld+json"]'):
        try:
            data = json.loads(node.text())
        except json.JSONDecodeError:
            continue
        for block in (data if isinstance(data, list) else [data]):
            if isinstance(block, dict) and block.get("@type") == "Product":
                return block
    return None

selectolax is a fast C-backed HTML parser; install it with pip install selectolax. The JSON-LD gives you title, brand, an aggregate rating, and a price that reflects whichever store the render resolved to. That last part is the whole problem, and it is why the next section exists. Keep the Internet # as your key even here: you rarely need the human-readable slug, because the GraphQL query in the next section accepts the item id directly, no URL required.

Here is the Home Depot gotcha in one line: the price you read from a default render is the price for some store, and if you don't control which one, your data jitters for reasons that have nothing to do with the product. Pin the store, and both price and on-hand count become comparable across runs.

Two layers of control matter.

Pick a store and hold it. Home Depot persists the selected store in cookies, and the GraphQL product query takes a storeId variable. The cleanest way to stay consistent is to standardize on one store number per run (say 0121) and pass it every time. Pre-inject the store cookie with the cookies parameter so a rendered page resolves to that store:

import json

def fetch_product_pinned(internet_no: str, store_id: str = "0121") -> str:
    store_cookies = [
        # Cookie name to confirm from devtools; Home Depot keys localization
        # off a store/zip cookie. Hold the SAME store across every run.
        {"name": "THD_LOCALIZER", "value": store_id,
         "domain": ".homedepot.com", "path": "/"},
    ]
    resp = requests.get(
        API,
        headers={"X-API-Key": API_KEY},
        params={
            "url": f"https://www.homedepot.com/p/{internet_no}",
            "render_js": "true",
            "premium_proxy": "true",
            "stealth": "true",
            "country_code": "US",
            "cookies": json.dumps(store_cookies),
        },
        timeout=120,
    )
    resp.raise_for_status()
    return resp.text

Ask GraphQL for the exact numbers. The rendered page is fine for a price and a rating, but for a per-store on-hand quantity across many stores you want the typed GraphQL response. Rather than forging the Akamai cookie to POST from outside, run the query from inside the rendered session so the fetch inherits the sensor cookie and the client headers the browser already set. A js_scenario with an evaluate step does exactly that: the SparkProxy browser loads a Home Depot page (passing the challenge), then executes your fetch against the gateway.

def graphql_via_page(internet_no: str, store_id: str,
                     operation: dict) -> str:
    """
    Run the captured GraphQL operation from inside a rendered Home Depot
    session, so it inherits the _abck sensor cookie and client headers.
    `operation` is the {query, variables, operationName} you copied from
    the network panel, with variables set to this item id + store id.
    """
    body = json.dumps(operation).replace("\\", "\\\\").replace("'", "\\'")
    scenario = {
        "instructions": [
            {"wait": 2000},
            {"evaluate": (
                "fetch('/federation-gateway/graphql', {"
                "  method: 'POST',"
                "  headers: {'Content-Type': 'application/json',"
                "            'x-experience-name': 'general-merchandise'},"
                "  body: JSON.stringify(" + json.dumps(operation) + ")"
                "}).then(r => r.text())"
            )},
        ]
    }
    resp = requests.get(
        API,
        headers={"X-API-Key": API_KEY},
        params={
            "url": f"https://www.homedepot.com/p/{internet_no}",
            "render_js": "true",
            "premium_proxy": "true",
            "stealth": "true",
            "country_code": "US",
            "js_scenario": json.dumps(scenario),
        },
        timeout=120,
    )
    resp.raise_for_status()
    return resp.text

Confirm how your plan surfaces an evaluate result (it typically comes back in the response body or a scenario field) and adjust the read accordingly. Once you have the GraphQL JSON, parse the typed fields and walk the fulfillment tree to the store you pinned:

def parse_graphql_product(payload: dict, store_id: str) -> dict | None:
    product = (payload.get("data") or {}).get("product")
    if not product:
        return None
    ids     = product.get("identifiers", {}) or {}
    pricing = product.get("pricing", {}) or {}
    ratings = ((product.get("reviews") or {}).get("ratingsReviews") or {})

    on_hand, in_stock = None, None
    fulfillment = product.get("fulfillment", {}) or {}
    for opt in fulfillment.get("fulfillmentOptions", []) or []:
        for svc in opt.get("services", []) or []:
            for loc in svc.get("locations", []) or []:
                if str(loc.get("storeNumber")) == str(store_id):
                    inv = loc.get("inventory", {}) or {}
                    on_hand  = inv.get("quantity")
                    in_stock = inv.get("isInStock")

    return {
        "internet_number": product.get("itemId") or ids.get("itemId"),
        "title":           ids.get("productLabel"),
        "brand":           ids.get("brandName"),
        "model_number":    ids.get("modelNumber"),
        "store_sku":       ids.get("storeSkuNumber"),
        "upc":             ids.get("upc"),
        "store_id":        store_id,
        "price":           pricing.get("value"),
        "original_price":  pricing.get("original"),
        "on_hand_qty":     on_hand,
        "in_store_stock":  in_stock,
        "rating":          ratings.get("averageRating"),
        "review_count":    ratings.get("totalReviews"),
    }

Because the values are typed, you get a float price and an int on-hand count with no post-processing. Standardize on one store per run and stamp it onto every row, so a change in your data reflects a real price or inventory move and not a shift in which store you asked about. If you are building a comparison or availability feed where that consistency is the entire game, datacenter proxies for price comparison websites covers the wider pattern, and Walmart's version of this same store-pinning problem is in how to scrape Walmart product data.

This is the check that separates a scraper you can trust from one that quietly stores denial pages. Because Akamai's "Access Denied" response can arrive without a clean error status, raise_for_status() will not always catch it, and a missing product block does not tell you why it is missing. Scan the body for the markers, and treat a missing JSON-LD Product block as a soft block too:

def is_blocked(html: str) -> bool:
    """Akamai can serve a denial page that isn't the product."""
    markers = (
        "access denied",
        "reference #",           # Akamai edge error reference id
        "you don't have permission",
        "/akam/",                # Akamai sensor path leaking into markup
        "errors while accessing this page",
    )
    lowered = html.lower()
    if any(m in lowered for m in markers):
        return True
    # A real product page carries a Product JSON-LD block.
    return extract_product_ldjson(html) is None

Now a fetch is honest: it either returns a real product page or tells you it was blocked so you can retry. Pair this with stealth=true and residential rotation, and Akamai challenges become an occasional retry rather than a wall. Keep the marker list in one place, since Akamai reworks its denial copy from time to time and you want a single spot to update.

The rating summary is easy: averageRating and totalReviews come straight out of the GraphQL reviews.ratingsReviews node (or the JSON-LD aggregateRating), so a nightly job that only needs "how many stars, how many reviews" is done at that point.

Individual review text is a separate system. Home Depot's ratings and reviews are powered by Bazaarvoice, the same platform many large retailers embed, Best Buy included. The review list loads from a Bazaarvoice-hosted service rather than the product HTML, which has two consequences. You will not find full review bodies in the JSON-LD, only the aggregate. And to pull individual reviews you request them from the Bazaarvoice display endpoint the page uses, paginated and keyed on the Home Depot Internet #. As with the GraphQL call, capture the live request from your network panel, because the client key and parameter names are specific to the deployment.

def parse_reviews(payload: dict) -> list[dict]:
    """
    `payload` is the JSON from the Bazaarvoice reviews request the page makes
    (it carries the product id and a page offset). Map the fields you need.
    """
    rows = []
    for r in payload.get("Results", []):
        rows.append({
            "review_id": r.get("Id"),
            "rating":    r.get("Rating"),          # 1 to 5
            "title":     r.get("Title"),
            "text":      r.get("ReviewText"),
            "submitted": r.get("SubmissionTime"),
        })
    return rows

Two guardrails specific to reviews. Keep only what you need for analysis (rating distribution, recency, keyword signal) and do not republish full review text verbatim, since that is user-generated content with its own copyright and platform terms. And never attach reviewer names, profile links, or any other personal detail to your dataset. Aggregate sentiment is defensible; harvesting people is not. For the wider workflow, using proxies for review monitoring and sentiment analysis covers the pattern end to end.

Everything so far assumes you have an Internet #. To build a catalog you need to discover them, and the cleanest source is Home Depot's own search and category pages. Search lives at /s/ and category browse pages expose paginated product grids. Fetch a results page through the same rendered request, then pull the Internet # out of each product link, which always follows /p//:

import re
from selectolax.parser import HTMLParser

OMS_RE = re.compile(r"/p/(?:[^/]+/)*(\d{6,12})(?:/|$|\?)")

def parse_search_ids(html: str) -> list[str]:
    tree = HTMLParser(html)
    ids = []
    for a in tree.css("a[href*='/p/']"):
        href = a.attributes.get("href") or ""
        m = OMS_RE.search(href)
        if m:
            ids.append(m.group(1))
    # de-dupe while preserving order
    seen, unique = set(), []
    for i in ids:
        if i not in seen:
            seen.add(i)
            unique.append(i)
    return unique

The GraphQL backend also exposes a search operation that returns the same product ids as typed JSON, which you can replay the same way as the product query once you capture it. Either route works; the DOM pass is simpler to stand up, the GraphQL route is cleaner at volume.

Once you have ids, scrape their data with three habits: retry on soft blocks, back off so you don't spike a single IP, and keep concurrency modest. With a scraping API the provider rotates the exit IP per request, so your ceiling is your plan's rate limit, not the number of proxies you own. Five to fifteen workers is plenty for a target this defended.

import time
import random
import csv
from concurrent.futures import ThreadPoolExecutor, as_completed

def fetch_with_retry(internet_no: str, store_id: str = "0121",
                     attempts: int = 3) -> str | None:
    for i in range(attempts):
        html = fetch_product_pinned(internet_no, store_id)
        if not is_blocked(html):
            return html
        time.sleep(2 ** i + random.random())   # exponential backoff + jitter
    return None

def scrape_catalog(ids: list[str], store_id: str = "0121",
                   workers: int = 8) -> list[dict]:
    out = []
    with ThreadPoolExecutor(max_workers=workers) as pool:
        futures = {pool.submit(fetch_with_retry, i, store_id): i for i in ids}
        for fut in as_completed(futures):
            html = fut.result()
            if html is None:
                continue
            block = extract_product_ldjson(html)
            if block:
                out.append({"store_id": store_id, "ldjson": block})
    return out

def save_csv(rows: list[dict], path: str = "homedepot_products.csv") -> None:
    if not rows:
        return
    with open(path, "w", newline="", encoding="utf-8") as f:
        writer = csv.DictWriter(f, fieldnames=list(rows[0].keys()))
        writer.writeheader()
        writer.writerows(rows)

The jitter matters more than it looks: random.random() staggers retries so a batch of failures does not retry in lockstep and re-trigger the same rate limit. Persist as you go rather than holding everything in memory, so a crash at item 40,000 does not cost you the first 39,999. For a running price tracker, key each row on (internet_number, store_id, scraped_at), which gives you a clean time series where every price point is comparable because the store was held constant. The same crawl scaffolding generalizes to any retailer, and the wider price-tracking pattern is in how to scrape ecommerce prices.