How to Scrape Walmart Data: Prices, Stock, Ratings

Get the framing right before you write any code, because "it's public" answers a narrower question than people assume.

In the United States, the Ninth Circuit's ruling in hiQ Labs v. LinkedIn (2022) held that scraping data that is publicly accessible, meaning no login and no authentication wall, generally does not violate the Computer Fraud and Abuse Act. That decision is about unauthorized access, not a blanket license to do anything with the data. Walmart's Terms of Use separately prohibit automated data collection, so a scraper can breach that contract even where it clears the CFAA bar. Those are two different legal questions, and public access only settles the first one.

Two more housekeeping checks belong in any serious project:

• Read walmart.com/robots.txt. It disallows a set of paths (account, checkout, cart, and internal search endpoints among them). Robots.txt is not a law, but respecting it is the baseline for defensible, good-faith scraping, and ignoring it is the first thing a plaintiff points at.
• Collect public product data only. Title, price, availability, ratings, seller. Never anything behind a login, and never personal data about buyers or reviewers.

Practical guardrails that keep a project on the right side of the line:

• Rate-limit yourself and back off on errors so you aren't degrading Walmart's service for real shoppers.
• Don't republish copyrighted assets (product photography, full review text) beyond what fair use allows.
• If the data feeds a commercial product, run it past a lawyer. This is engineering guidance, not legal advice.

Price monitoring, catalog research, and competitive intelligence are common, legitimate uses of public retail data. For the business context around that, see How E-commerce Companies Use Proxies for Competitive Intelligence.

A public Walmart product page lives at /ip//, where is the numeric US Item ID that uniquely names the product. The visible fields are stable, and almost all of them come out of the page JSON rather than the rendered DOM. Here is the reference set worth pulling.

The usItemId is your anchor. Store it as the primary key and hang every other field off it. One nuance that trips people up: sellerName tells you whether you are looking at a first-party Walmart listing or a third-party marketplace offer, and marketplace offers price and ship differently, so keep that field even if you don't think you need it yet.

Walmart's storefront is a Next.js application. Next.js serializes the data used to render a page into a single JSON script tag so the client can hydrate without a second round trip:

<script id="__NEXT_DATA__" type="application/json">
{"props":{"pageProps":{"initialData":{"data":{"product":{ ... }}}}}, ...}
</script>

That one detail changes the whole job. Instead of writing brittle CSS selectors that break the next time Walmart ships a design tweak, you grab the JSON, parse it once, and read typed fields straight out of the object graph. Price is a number, numberOfReviews is an integer, availability is an enum. No regex on "$148.00", no guessing which of five price containers is populated this session.

The catch is that the object path is deep and it moves. On a product page the node usually sits at props.pageProps.initialData.data.product, but Walmart reorganizes the tree periodically, and search pages nest it differently again. So don't hard-code the full path. Load the blob, then walk it defensively looking for the node that has a usItemId. That approach survives reshuffles that would break a fixed path. We build exactly that walker in the parsing section.

Getting the HTML is the hard part, and the reason is PerimeterX (now part of HUMAN Security). It is one of the more capable bot-management stacks on the public web, and Walmart runs it aggressively. Three things break naive scrapers.

The "Robot or human?" challenge. When PerimeterX flags a request, Walmart serves a challenge page titled "Robot or human?" with a "Press & Hold" button. The trap is that this page does not always come back with an error status. You will see it on a 403 or a 429, but also sometimes on an HTTP 200. If your code trusts the status code, response.ok is True, you save the page, and you have stored a captcha shell instead of a product. You have to inspect the body.

Fingerprinting, not just IP reputation. PerimeterX scores TLS/JA3 fingerprints, header order, and JavaScript-execution signals, not only the IP. A plain requests.get with default headers is flagged before the IP even matters, which is why bare HTTP clients get walled almost immediately. You need a real browser fingerprint, which means real JavaScript execution.

Datacenter IP bias. Plain datacenter ranges carry poor reputation on Walmart and get challenged fast. Residential exit IPs blend in with normal shopper traffic and last far longer. Rotating the exit IP per request is the difference between a scraper that runs for ten minutes and one that runs for a week.

A managed scraping API absorbs the fingerprint, IP rotation, and rendering for you. For the proxy-side theory behind why some IPs survive and others don't, How to Avoid Getting Your Proxy Blocked goes deep.

The SparkProxy Scraping API takes a target URL and handles the proxy, rotation, browser rendering, and anti-bot layer. You send one request and get the rendered HTML back. For Walmart, four parameters carry the weight:

• render_js=true: runs the page in a real Chromium browser, which produces a legitimate JS fingerprint and returns the hydrated HTML (including a fully populated __NEXT_DATA__).
• premium_proxy=true: routes through residential IPs, which survive PerimeterX where datacenter IPs get challenged.
• stealth=true: adds extra anti-detection layers tuned for stacks like PerimeterX. It requires render_js=true.
• country_code=US: sets a US exit, which matters because Walmart's US storefront (walmart.com) expects US traffic and geo-mismatched requests draw friction.

The base endpoint is https://scrape.sparkproxy.io/api/v1 and auth is a single header, X-API-Key. A minimal request in cURL:

curl -G "https://scrape.sparkproxy.io/api/v1" \
  -H "X-API-Key: YOUR_API_KEY" \
  --data-urlencode "url=https://www.walmart.com/ip/5013218" \
  --data-urlencode "render_js=true" \
  --data-urlencode "premium_proxy=true" \
  --data-urlencode "stealth=true" \
  --data-urlencode "country_code=US"

The full parameter list and response fields are in the Scraping API docs. If you're deciding between this and running your own proxy pool for a target this well defended, Web Scraping API vs Self-Managed Proxies lays out the trade-off honestly.

Start with one product. Wrap the request so every call carries the Walmart-specific parameters, and give it a generous timeout since a rendered request drives a real browser.

import requests

API = "https://scrape.sparkproxy.io/api/v1"
API_KEY = "YOUR_API_KEY"

def fetch_url(target: str, country: str = "US") -> str:
    resp = requests.get(
        API,
        headers={"X-API-Key": API_KEY},
        params={
            "url": target,
            "render_js": "true",       # produces a real JS fingerprint + hydrated HTML
            "premium_proxy": "true",   # residential IPs survive PerimeterX
            "stealth": "true",         # extra anti-detection (needs render_js)
            "country_code": country,   # US storefront expects US traffic
        },
        timeout=90,
    )
    resp.raise_for_status()
    return resp.text

def fetch_product(item_id: str) -> str:
    return fetch_url(f"https://www.walmart.com/ip/{item_id}")

You can hit a product with just its item ID (/ip/); Walmart redirects the bare ID to the canonical slug URL, and the API follows it. The next step is reading the JSON out of that HTML.

Two moves make Walmart parsing reliable: pull the __NEXT_DATA__ script tag, then walk the parsed object to find the product node instead of trusting a fixed path.

import json
from selectolax.parser import HTMLParser

def extract_next_data(html: str) -> dict | None:
    """Pull and parse Walmart's __NEXT_DATA__ JSON blob."""
    tree = HTMLParser(html)
    node = tree.css_first("script#__NEXT_DATA__")
    if not node:
        return None
    try:
        return json.loads(node.text())
    except json.JSONDecodeError:
        return None

def find_node(obj, key: str):
    """Depth-first search for the first dict containing `key`."""
    if isinstance(obj, dict):
        if key in obj:
            return obj
        for v in obj.values():
            hit = find_node(v, key)
            if hit is not None:
                return hit
    elif isinstance(obj, list):
        for v in obj:
            hit = find_node(v, key)
            if hit is not None:
                return hit
    return None

selectolax is a C-backed HTML parser; install it with pip install selectolax. It finds the single script tag fast even on Walmart's large pages. Once you have the product node, reading fields is a matter of safe .get() chains, because Walmart omits keys rather than nulling them (wasPrice only exists on a rollback, for example):

def parse_product(html: str) -> dict | None:
    data = extract_next_data(html)
    if not data:
        return None
    product = find_node(data, "usItemId")
    if not product:
        return None

    price_info = product.get("priceInfo", {}) or {}
    current = price_info.get("currentPrice", {}) or {}
    was = price_info.get("wasPrice", {}) or {}
    image = product.get("imageInfo", {}) or {}

    return {
        "item_id":      product.get("usItemId"),
        "title":        product.get("name"),
        "brand":        product.get("brand"),
        "price":        current.get("price"),
        "price_string": current.get("priceString"),
        "was_price":    was.get("price"),
        "availability": product.get("availabilityStatus"),
        "rating":       product.get("averageRating"),
        "review_count": product.get("numberOfReviews"),
        "seller":       product.get("sellerName"),
        "seller_id":    product.get("sellerId"),
        "image":        image.get("thumbnailUrl"),
    }

Because everything is typed, you get a float price and an int review count with no post-processing. Normalize availabilityStatus into a boolean plus the raw string rather than an enum you'll keep extending as Walmart adds states like RETIRED. And keep both sellerName and sellerId: a change in seller on the same item ID is often the real story in a price feed, not the price itself.

This is the check that separates a scraper that quietly stores garbage from one you can trust. Because the "Robot or human?" page can return HTTP 200, raise_for_status() will not catch it, and neither will __NEXT_DATA__ being absent tell you why it's missing. Scan the body for PerimeterX markers, and treat a missing product node as a soft block too:

def is_blocked(html: str) -> bool:
    """Walmart/PerimeterX can serve a challenge on HTTP 200, so the status lies."""
    markers = (
        "Robot or human?",
        "px-captcha",
        "Press & Hold",
        "/_px/",
        "captcha",
    )
    lowered = html.lower()
    if any(m.lower() in lowered for m in markers):
        return True
    # A real product page always ships __NEXT_DATA__ with a usItemId.
    return "__NEXT_DATA__" not in html

Now a fetch is honest: it either returns a real product page or tells you it was blocked so you can retry. Pair this with stealth=true and residential rotation, and PerimeterX challenges become an occasional retry rather than a wall.

Here is the gotcha specific to Walmart. Walmart is not a single national catalog. Availability, fulfillment options (shipping vs pickup vs delivery), and some pricing (grocery and rollbacks especially) are resolved against a store, and the store is chosen from the shopper's location. Scrape the same item from rotating IPs with no fixed location and your "out of stock" and delivery signals will jitter for reasons that have nothing to do with the item actually changing.

There are two layers to control.

Country / storefront. country_code=US keeps you on a US exit for walmart.com. Walmart also runs separate storefronts like walmart.ca; match the country to the domain rather than mixing them.

Store, within the US. Walmart persists the selected store and location in cookies. The clean way to hold a store constant across a batch is to pre-inject that location with the cookies parameter, so every request in the run resolves to the same store:

import requests

location_cookies = [
    {"name": "assortmentStoreId", "value": "3081",
     "domain": ".walmart.com", "path": "/"},
    # Walmart also keys off a location cookie carrying the ZIP; set the
    # store you standardize on and hold it constant across every run.
]

def fetch_product_pinned(item_id: str) -> str:
    import json
    resp = requests.get(
        "https://scrape.sparkproxy.io/api/v1",
        headers={"X-API-Key": "YOUR_API_KEY"},
        params={
            "url": f"https://www.walmart.com/ip/{item_id}",
            "render_js": "true",
            "premium_proxy": "true",
            "stealth": "true",
            "country_code": "US",
            "cookies": json.dumps(location_cookies),
        },
        timeout=120,
    )
    resp.raise_for_status()
    return resp.text

If you would rather drive the UI, a js_scenario that opens the location modal, fills a ZIP, and submits works too, and it holds up better when Walmart changes its cookie names. Either way, pick one canonical store per run and hold it constant, so a change in your data reflects a real change and not a location swap. If you're building a comparison feed where this consistency is the whole game, Datacenter Proxies for Price Comparison Websites covers the wider pattern.

Search pages let you discover item IDs by keyword, and they carry the same __NEXT_DATA__ trick: the result list lives in the page JSON, not in scraped cards. The items sit under a search node (typically props.pageProps.initialData.searchResult.itemStacks[].items), so reuse the find_node walker to reach them.

from urllib.parse import quote_plus

def search_url(keyword: str, page: int) -> str:
    return f"https://www.walmart.com/search?q={quote_plus(keyword)}&page={page}"

def parse_search(html: str) -> list[dict]:
    data = extract_next_data(html)
    if not data:
        return []
    stacks = find_node(data, "itemStacks")
    if not stacks:
        return []
    rows = []
    for stack in stacks.get("itemStacks", []):
        for item in stack.get("items", []):
            if not item.get("usItemId"):        # skip ad / spacer tiles
                continue
            price = (item.get("priceInfo", {}) or {}).get("currentPrice", {}) or {}
            rows.append({
                "item_id":      item.get("usItemId"),
                "title":        item.get("name"),
                "price":        price.get("price"),
                "rating":       item.get("averageRating"),
                "review_count": item.get("numberOfReviews"),
                "seller":       item.get("sellerName"),
            })
    return rows

To walk every page, request &page=N and stop when a page returns no items:

def scrape_all_pages(keyword: str, max_pages: int = 25) -> list[dict]:
    results = []
    for page in range(1, max_pages + 1):
        html = fetch_url(search_url(keyword, page))
        if is_blocked(html):
            continue
        rows = parse_search(html)
        if not rows:
            break
        results.extend(rows)
    # dedupe on item_id, since results overlap near category edges
    seen, unique = set(), []
    for r in results:
        if r["item_id"] not in seen:
            seen.add(r["item_id"])
            unique.append(r)
    return unique

One hard limit to design around: Walmart caps organic search pagination, commonly around 25 pages (roughly 1,000 results) per query. You cannot page past it, so a broad query like "tv" leaves most of the catalog unreachable. The fix is to narrow, not to fight the cap: split by category, brand, or price band using Walmart's URL facets (&facet=..., &min_price=, &max_price=), run each narrow query to its cap, then dedupe item IDs. Ten targeted queries surface far more of the catalog than one broad one.

At volume, three habits keep the pipeline healthy: retry on soft blocks, back off so you don't spike a single IP, and keep concurrency modest. With a scraping API the provider rotates the exit IP per request, so your ceiling is your plan's rate limit, not the number of proxies you own. Five to fifteen workers is plenty.

import time
import random
from concurrent.futures import ThreadPoolExecutor, as_completed

def fetch_with_retry(item_id: str, attempts: int = 3) -> str | None:
    for i in range(attempts):
        html = fetch_product(item_id)
        if not is_blocked(html):
            return html
        time.sleep(2 ** i + random.random())   # exponential backoff + jitter
    return None

def scrape_catalog(item_ids: list[str], workers: int = 8) -> list[dict]:
    out = []
    with ThreadPoolExecutor(max_workers=workers) as pool:
        futures = {pool.submit(fetch_with_retry, i): i for i in item_ids}
        for fut in as_completed(futures):
            html = fut.result()
            if html is None:
                continue
            record = parse_product(html)
            if record:
                out.append(record)
    return out

The jitter matters more than it looks: random.random() staggers retries so a batch of failures doesn't retry in lockstep and re-trigger the same rate limit. Persist as you go rather than holding everything in memory, so a crash at item 40,000 doesn't cost you the first 39,999.

import csv

def save_csv(rows: list[dict], path: str = "walmart_products.csv") -> None:
    if not rows:
        return
    with open(path, "w", newline="", encoding="utf-8") as f:
        writer = csv.DictWriter(f, fieldnames=list(rows[0].keys()))
        writer.writeheader()
        writer.writerows(rows)

For a running price tracker, append a scraped_at timestamp and the store you pinned to each row, and write to a database keyed on (item_id, scraped_at). That gives you a clean time series where every price point is comparable because the location was held constant.