How to Scrape Lowes Product Data by Store

Fetch the Lowe's homepage with nothing but a browser User-Agent and look at what comes back in the Set-Cookie headers:

curl -s -o /dev/null -D - \
  -A "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 \
(KHTML, like Gecko) Chrome/140.0.0.0 Safari/537.36" \
  https://www.lowes.com/ | grep -i '^set-cookie'

Five of them describe a location, and four of those five arrive pre-expired:

Set-Cookie: region=central; expires=Sat, 10-Oct-2026 07:47:09 GMT; path=/
Set-Cookie: nearbyid=0595; expires=Wed, 01-Jan-2020 10:13:00 GMT; path=/; domain=.www.lowes.com
Set-Cookie: zipstate=NC;   expires=Wed, 01-Jan-2020 10:13:00 GMT; path=/; domain=.www.lowes.com
Set-Cookie: zipcode=12345; expires=Wed, 01-Jan-2020 10:13:00 GMT; path=/; domain=.www.lowes.com
Set-Cookie: sn=0595;       expires=Wed, 01-Jan-2020 10:13:00 GMT; path=/; domain=.www.lowes.com

An expiry of 1 January 2020 is a delete instruction. Lowe's names a store number, 0595, hands you a state and a placeholder ZIP, then tells your client to throw all four away in the same breath. Only region=central survives into the jar, and region is a coarse fulfilment bucket, not a store.

Store 0595 is /store/NC-Mooresville/0595 in Lowe's own store sitemap, the town of its corporate campus. What your scraper inherits when it sets no store is therefore neither a national list price nor the store your buyer walks into. It is an unresolved session rendered against whatever fallback the page has, and the output never tells you which.

That makes the scope of each field the first thing to get straight:

A store-scoped value saved without a store_id beside it is unfalsifiable: you can never prove later which store it described. That is the failure the rest of this guide is built to prevent.

How to scrape Home Depot product data covers the generic per-store problem plus Home Depot's federated GraphQL gateway, where a typed storeId travels inside the query. Lowe's gives you no such typed handle. Store context here lives entirely in cookie state on a plain product URL, a different problem with a different set of traps.

These are the location cookies lowes.com set on a successful homepage response on 2026-10-09, read straight off the wire:

Three details decide whether a hand-set store actually binds.

The domain is .www.lowes.com, not .lowes.com. Lowe's own EPID and dbidv2 cookies arrive on .lowes.com while the store cookies arrive on the dot-prefixed host. Set sn on .lowes.com and a strict jar may not send it back for the host that reads it. Mirror the domain Lowe's itself uses.

The value space is the store sitemap. Store URLs take the shape /store/-/, and that trailing number is the sn value. 0595 in the cookie and 0595 in /store/NC-Mooresville/0595 are the same identifier, so you can enumerate every legal sn without guessing.

Nothing persists it for you. SparkProxy's Scraping API documents session_id as a label for the browser profile in per-job logs and states plainly that it "does not persist cookies across requests." So the store cookies ride along on every single request, not once per session. If you have not built cookie handling that survives redirects and retries, handling cookies and sessions in web scraping is the groundwork.

This is the assumption that quietly ruins Lowe's datasets: that exiting through a Charlotte IP gets you Charlotte prices. It does not.

The probes in this guide ran from an egress IP in Bengaluru, India (AS55836, Reliance Jio, confirmed via ipinfo.io/json on 2026-10-09). Two cold fetches from that IP both came back with region=central and the same seeded-then-expired sn=0595 and zipstate=NC. An Indian IP gets handed the identical store seed a US one does, because the seed is not derived from your IP in any way you can steer.

A geo-targeted exit is still worth paying for, just for a different reason: a US exit looks domestic to the edge, keeps you on the US experience and avoids long-haul latency. On the SparkProxy Scraping API that is country_code=US at a documented +5 credits. An admission ticket, not a store selector.

So the division of labour is fixed:

• Proxy geography decides whether the edge lets you in and which locale you are served.
• Cookie state decides which store's price and on-hand count the page renders.

Mix them up and you get the worst kind of dataset, one that looks geographically diverse and is actually a single store's numbers repeated behind different IPs. Proxies for geo pricing audits covers keeping those two variables apart across markets.

Read Lowe's own rules before you write a crawler, and note that you need to pass the edge just to read them: a default-TLS curl request for /robots.txt returned 403 Access Denied on 2026-10-09, while a Chrome-fingerprinted client got the real 11,048-byte file.

What that file says, as read today:

• A header comment dated 09/17/2026, one User-agent: * block, 333 Disallow: rules, 8 Allow: rules.
• No Crawl-delay directive, so your request rate is your own judgement call.
• One sitemap line: Sitemap: https://www.lowes.com/sitemap.xml.

Now the part that lands on this guide's spine. The store-scoped sub-resources of a product page are explicitly disallowed, while the product page itself is not:

Disallow: /pd/*/*/pricing/
Disallow: /pd/*/*/pricing/*
Disallow: /pd/*/*/pricing/zipin
Disallow: /pd/*/*/availability/*
Disallow: /pd/*/*/zipin
Disallow: /pd/*/*/location/
Disallow: /pd/*/*/segment/
Disallow: /pd/*/*/displayonly/

Read that as a design instruction. Lowe's has drawn a line around every URL whose job is to resolve price and availability for a location, and left /pd// open. The compliant route to a store-scoped number is the slower one: establish store context, request the allowed product URL, read what the page renders. Calling /pd/.../pricing/zipin directly would be faster and is off the table.

Lowe's Terms and Conditions of Use go further than robots.txt. The prohibited-conduct list includes, verbatim:

Use any robot, spider, site search/retrieval application or other manual or automatic device or process to retrieve, index, data mine or in any way reproduce or circumvent the navigational structure or presentation of the Site or its contents

A separate reseller clause names scraping directly, barring "(iii) extracting, scraping, mining, copying or otherwise gathering information from the Site in connection with your sale or another party's sale of products ... through any channel to downstream customers." That targets resale arbitrage specifically, and it is stricter than the generic anti-bot language most retail terms carry. If your Lowe's data feeds a reseller listing, that sentence is about you. This is contract territory rather than criminal, and it is the document your counsel reads first.

Lowe's runs on Akamai, and the evidence sits in every layer:

• DNS. nslookup -type=NS lowes.com returns eight nameservers, all on akam.net (use1, use2, use4, usc5, eur2, asia1, ns1-42, ns1-216).
• Response headers. Akamai-GRN, X-Akamai-Transformed and Server-Timing: ak_p on every response.
• Load-balancer cookies. akaalb_prod_dual and akavpau_default.
• Bot Manager cookies. _abck, bm_sz, bm_s, bm_so, bm_mi. A fresh _abck arrives 511 characters long containing ~-1~, the unvalidated-sensor state.
• A behavioral interstitial. Product pages can answer with a 2,713-byte document holding
  , a /akam/13/pixel_... beacon and the line "Powered and protected by Akamai."

What three clients got on 2026-10-09. These are single-shot measurements from one Indian egress IP, not success rates, and your numbers will differ by IP reputation, time and path:

The ordering is the surprise. An HTTP client wearing a Chrome TLS fingerprint read the homepage, robots.txt and three sitemap files cleanly, while a genuine Chromium engine in its default automation configuration was denied outright. Transport-layer identity got further than browser realism, on this target, on this day. Reproduce the first two rows in ten seconds:

UA="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 \
(KHTML, like Gecko) Chrome/140.0.0.0 Safari/537.36"

# Row 1: default TLS stack
curl -s -o /dev/null -A "$UA" -w "curl      %{http_code} %{size_download}B\n" \
  https://www.lowes.com/robots.txt

# Row 2: Chrome TLS fingerprint (pip install curl_cffi)
python -c "from curl_cffi import requests as r; \
x=r.get('https://www.lowes.com/robots.txt', impersonate='chrome'); \
print('curl_cffi', x.status_code, len(x.content), 'B')"

Both libraries are current: curl_cffi 0.16.3 was published to PyPI on 2026-09-02 and playwright for Python 1.63.0 on 2026-09-15, both read from PyPI on 2026-10-09. The mechanics behind those rows are in what is TLS fingerprinting and headless browser detection, and the Akamai signal chain is in how to bypass Akamai Bot Manager.

One more measured detail, because it kills a popular shortcut: pre-warming on the homepage and then requesting the product page in the same session did not clear the challenge. The homepage returned 322,443 bytes of real HTML, _abck stayed ~-1~ unvalidated, and the product page still answered with the same 2,713-byte interstitial. Warming is necessary on Akamai targets and not sufficient here.

Count the 200s in that table. The default-curl homepage denial was HTTP 200 with an Access Denied body, three times out of three. The curl_cffi product-page challenge was HTTP 200 with a 2,713-byte interstitial. Yet /robots.txt, /sitemap.xml and /store/... returned clean 403s to the same client. Both shapes are in play depending on the path, so raise_for_status() passes denials into your database and status_code == 200 is not a success check.

This is where the bug usually lands with a managed API. Per SparkProxy's own documentation:

• Headless mode always returns 200 on success or 530 on failure, so the API's HTTP status describes the job, not Lowe's.
• transparent_status_code=true mirrors the target's status, and only applies to render_js=false.
• json_response=true is the only source of an envelope carrying the target's status_code, duration_ms and credits_used, and in it the page content is base64-encoded in body. Decode before you match anything. format=json is a content conversion, not the envelope, and reaching for it here is the most common mistake with this API.

So the detector has to work on decoded bytes and on markers, with the status as a weak hint:

import base64
import re

DENIAL_MARKERS = (
    "access denied",            # Akamai edge denial, arrives as 200 or 403
    "you don't have permission",
    "errors.edgesuite.net",     # the reference link in the denial body
    "sec-if-cpt-container",     # Akamai sec-cpt behavioral interstitial
    "/akam/",                   # sensor path leaking into markup
)

REFERENCE_RE = re.compile(r"reference\s*#\s*[\d.]+", re.I)


def decode_envelope(envelope: dict) -> str:
    """json_response=true returns the page base64-encoded in 'body'."""
    raw = envelope.get("body") or ""
    if envelope.get("encoding") == "base64":
        return base64.b64decode(raw).decode("utf-8", "replace")
    return raw


def classify(envelope: dict) -> str:
    html = decode_envelope(envelope)
    low = html.lower()
    if any(m in low for m in DENIAL_MARKERS) or REFERENCE_RE.search(low):
        return "blocked"
    if len(html) < 20_000:
        # A real Lowe's product page is hundreds of KB. A short 200 is a tell
        # even when no marker matches, so quarantine it instead of storing it.
        return "suspect"
    return "ok"

The length floor earns its place: both denial shapes measured today were under 6 KB while a real Lowe's page ran past 300 KB, so a size check catches a new interstitial variant before you have a marker for it. The general discipline is in how to detect when your scraper is blocked.

Start at the cheapest rung so you know what the expensive ones are buying. render_js=false is a plain HTTP fetch at 1 credit, and there is a parameter trap: the cookies array requires render_js=true. On a plain fetch you carry store context yourself, as a Cookie header inside forward_headers.

import json
import requests

API = "https://scrape.sparkproxy.io/api/v1"
API_KEY = "YOUR_API_KEY"


def store_cookie_header(store: str, zipcode: str, state: str) -> str:
    """Lowe's reads store context from these four names. Send them every time."""
    return "; ".join([
        f"sn={store}",
        f"nearbyid={store}",
        f"zipcode={zipcode}",
        f"zipstate={state}",
    ])


def fetch_plain(item_url: str, store: str, zipcode: str, state: str) -> dict:
    headers_out = {
        "Cookie": store_cookie_header(store, zipcode, state),
        "Accept-Language": "en-US,en;q=0.9",
        "Referer": "https://www.lowes.com/",
    }
    resp = requests.get(
        API,
        headers={"X-API-Key": API_KEY},
        params={
            "url": item_url,
            "render_js": "false",            # 1 credit, no browser
            "json_response": "true",         # the only source of status_code
            "transparent_status_code": "true",
            "forward_headers": json.dumps(headers_out),
            "tag": f"lowes-store-{store}",
        },
        timeout=90,
    )
    return resp.json()

Expect this rung to fail on product pages, and let it fail loudly. A default-TLS client got 403 on /pd/... today, so treat rung 1 as a probe: run it once per path shape, read status_code from the envelope, and record which Lowe's paths your account reaches without a browser. Sitemap and robots paths may answer where product pages do not, and that split is what lets you cost the rest of the job. On why two HTTP clients get different answers from one URL, curl vs Python requests has the detail.

The next rung puts a real browser behind the request and injects the store cookies before the first byte of the page is parsed, which is the only ordering that works. A cookie set after load does not re-render a price.

import json
import requests

API = "https://scrape.sparkproxy.io/api/v1"
API_KEY = "YOUR_API_KEY"

# Domain mirrors what Lowe's itself sends: .www.lowes.com, not .lowes.com
STORE_DOMAIN = ".www.lowes.com"


def store_cookies(store: str, zipcode: str, state: str) -> list:
    pairs = [("sn", store), ("nearbyid", store),
             ("zipcode", zipcode), ("zipstate", state)]
    return [
        {"name": n, "value": v, "domain": STORE_DOMAIN, "path": "/"}
        for n, v in pairs
    ]


def fetch_rendered(item_url: str, store: str, zipcode: str,
                   state: str) -> dict:
    """Rendered, premium-pool, US-exit fetch with the store pinned."""
    resp = requests.post(
        API,
        headers={"X-API-Key": API_KEY, "Content-Type": "application/json"},
        json={
            "url": item_url,
            "render_js": True,
            "proxy_type": "premium",        # file-based premium pool
            "country_code": "US",           # +5 credits, US exit
            "stealth": True,                # homepage pre-warm, +5 credits
            "cookies": store_cookies(store, zipcode, state),
            "wait": 5,                      # seconds after load, max 30
            "json_response": True,
            "tag": f"lowes-store-{store}",
        },
        timeout=180,
    )
    return resp.json()

Three parameter choices are deliberate. stealth is documented as adding a homepage pre-warm, a forced Google referrer and extended idle delays, the warming step my manual test showed to be necessary here. proxy_type=premium moves you onto residential-grade exits, where an Akamai sensor is least likely to see a datacentre ASN. wait buys render time without the extra credit of a scenario.

If your capture shows the store-scoped price only appears once the picker is driven, js_scenario is the escalation: up to 50 instructions using the keys click, wait_for, fill, wait, scroll, evaluate and screenshot, at +5 credits and requiring render_js=true. I print no selector for it, because I never reached a rendered Lowe's product page and an invented selector is worse than none. Capture one page by hand, read the real selector out of it, then script that.

The documented adders stack like this, and credits_used in the envelope is the number that actually bills:

That table is the reason store context is an architecture decision and not a flag. At 35 credits a row, a careless cross-join of SKUs and stores is a budget event.

The top rung stops writing parsers. extract_rules pulls fields out of the rendered DOM server-side and replaces the response with an extracted object, so you never stack it with json_response. Pick one shape per call.

import requests

API = "https://scrape.sparkproxy.io/api/v1"
API_KEY = "YOUR_API_KEY"

# PLACEHOLDERS. Lowe's markup is not public and this research never reached a
# rendered product page, so replace every selector below with one you read out
# of your own capture. Only "h1" and the two link/meta rules are safe as-is.
RULES = {
    "title": "h1",
    "price": "YOUR_PRICE_SELECTOR",
    "model": "YOUR_MODEL_SELECTOR",
    "store_label": "YOUR_STORE_LABEL_SELECTOR",
    "specs": {"selector": "table tbody tr", "type": "list"},
    "canonical": {"selector": "link[rel='canonical']", "type": "href"},
}


def extract_rendered(item_url: str, store_cookie_list: list) -> dict:
    """extract_rules replaces the normal response: no json_response here."""
    resp = requests.post(
        API,
        headers={"X-API-Key": API_KEY, "Content-Type": "application/json"},
        json={
            "url": item_url,
            "render_js": True,
            "proxy_type": "premium",
            "country_code": "US",
            "stealth": True,
            "cookies": store_cookie_list,
            "extract_rules": RULES,
        },
        timeout=180,
    )
    return resp.json().get("extracted", {})

The practical pattern is two-phase. While you are still learning the page, run json_response=true, decode the base64 body and keep the raw HTML so you can diff it when Lowe's changes markup. Once selectors are stable, switch the steady-state job to extract_rules and leave a small sample on the raw path as a canary. The store_label rule exists purely so the response carries back which store the page thought it was rendering, which is what makes the next section possible.

Two honest caveats. First, scrape.sparkproxy.io returned HTTP 521 from my network throughout this research, so every credit figure and parameter behaviour here comes from the documentation at https://www.sparkproxy.io/docs/scraping-api/ read on 2026-10-09, not from a live run. Second, I quote no success rate for Lowe's, from any provider including this one, because I measured none. Measure it yourself: the detector above plus a 200-request sample gives you a real number for your own IPs and your own paths, which is the only number worth planning against.

Here is the check almost nobody runs, and it is the one that catches a silently broken store pin. Fetch the same item against two deliberately distant stores. If the price fields come back identical, your cookies are not binding and you are writing one store's numbers under many store labels.

from dataclasses import dataclass


@dataclass
class Row:
    item: str
    store: str
    price: str | None
    store_label: str | None
    html_len: int


def accept(row: Row) -> tuple[bool, str]:
    """Reject anything we cannot prove is store-correct."""
    if row.price is None:
        return False, "no price parsed"
    if row.store_label and row.store not in row.store_label:
        return False, f"page rendered {row.store_label!r}, asked for {row.store}"
    if row.html_len < 20_000:
        return False, "body too short, likely a denial or interstitial"
    return True, "ok"


def pin_is_live(rows_a: list[Row], rows_b: list[Row]) -> list[str]:
    """
    rows_a and rows_b are the same items fetched against two distant stores.
    Any item whose price is identical in both is a candidate for a dead pin.
    Some items genuinely price the same nationwide, so this is a signal to
    investigate, not an automatic failure.
    """
    by_item_b = {r.item: r for r in rows_b}
    suspect = []
    for a in rows_a:
        b = by_item_b.get(a.item)
        if b is not None and a.price == b.price:
            suspect.append(a.item)
    return suspect

Wire it up as a pre-flight that runs before any scheduled crawl, against two stores as far apart as the chain goes:

# Store numbers come from store0.xml; use a ZIP inside each store's trade area.
NC_STORE = ("0595", "28115", "NC")   # /store/NC-Mooresville/0595
CA_STORE = ("2790", "95125", "CA")   # /store/CA-San-Jose/2790


def preflight(basket: list[str], fetch) -> None:
    """
    basket: product URLs. fetch(url, store, zip, state) -> Row.
    Fails the run rather than letting a dead store pin reach the warehouse.
    """
    rows_a = [fetch(u, *NC_STORE) for u in basket]
    rows_b = [fetch(u, *CA_STORE) for u in basket]

    for row in rows_a + rows_b:
        ok, why = accept(row)
        if not ok:
            raise SystemExit(f"{row.item} @ {row.store}: {why}")

    suspect = pin_is_live(rows_a, rows_b)
    ratio = len(suspect) / max(len(basket), 1)
    print(f"identical prices NC vs CA: {len(suspect)}/{len(basket)}")
    if ratio > 0.9:
        raise SystemExit("store pin is not binding: aborting the crawl")

Run it across a basket of 30 to 50 items, not one. A single matching price proves nothing, because plenty of national-brand items carry the same price everywhere. Ninety percent matching between North Carolina and California means the pin is dead. Pair that with three warehouse rules: store_id is NOT NULL on every price row, the natural key is (item_number, store_id, fetched_at), and no aggregate view averages price across stores. Those constraints make this class of bug impossible to commit.

You do not have to guess at store numbers, because Lowe's publishes them. The sitemap index listed 593 child sitemaps on 2026-10-09 with a lastmod of 2026-10-08: 481 detailN.xml product files, 71 refinementN.xml, 13 storeservicesN.xml, 10 reviewsN.xml, and a single store0.xml. That store file holds 1,766 store URLs across 51 state codes, and detail0.xml alone holds 7,921 product URLs, putting the full product set in the millions.

import re

STORE_RE = re.compile(
    r"<loc>https://www\.lowes\.com/store/"
    r"([A-Z]{2})-([^/]+)/(\d{4})</loc>"
)


def parse_store_sitemap(xml: str) -> list[dict]:
    """store0.xml -> the legal values for the sn cookie."""
    out = []
    for state, city, number in STORE_RE.findall(xml):
        out.append({
            "store": number,                    # goes straight into sn
            "state": state,
            "city": city.replace("-", " "),
            "url": f"https://www.lowes.com/store/{state}-{city}/{number}",
        })
    return out


def store_panel(stores: list[dict], per_state: int = 1) -> list[dict]:
    """
    Do not cross-join every SKU with 1,766 stores. Pick a panel: N stores per
    state. store0.xml carries 51 state codes (50 states plus DC), so
    per_state=1 is 51 rows instead of 1,766, a 35x cut for most of the signal.
    """
    seen: dict[str, int] = {}
    panel = []
    for s in sorted(stores, key=lambda x: (x["state"], x["store"])):
        n = seen.get(s["state"], 0)
        if n < per_state:
            panel.append(s)
            seen[s["state"]] = n + 1
    return panel

The arithmetic is unforgiving, so do it before you schedule anything. One thousand SKUs against the full store list is 1,766,000 rendered requests; at the documented 35 credits for a rendered, premium, US, stealth call that is 61.8 million credits. The same thousand SKUs against a one-store-per-state panel is 51,000 requests. Choose the panel.

Two further savings. Batch mode bills 1 credit for a whole comma-separated batch, but it requires render_js=false, and a default-TLS client got 403 on /sitemap.xml today, so probe it before budgeting on it. And refresh on change, not on a timer: sitemaps carry lastmod, prices move far less often than your cron fires, and incremental web scraping and change detection covers the diffing that turns a daily full crawl into a small delta. For the scheduling layer, see how to build an automated price monitoring system with proxy rotation.

SparkProxy's plumbing suits this shape of job. Plans start from $75/month with unlimited bandwidth, so a store fan-out that multiplies page weight does not multiply the bill, and the pool covers 1M+ premium IPs across 80+ countries with HTTP, HTTPS and SOCKS5, random or five-minute auto-rotation, and either IP whitelisting or user:pass auth. The Scraping API ships 1,000 free credits, and the 24-hour free trial needs no card, which is enough to run the two-store pin test on your own basket before you commit to a schedule.