How to Scrape SHEIN Product Data

Four requests, plain curl, no User-Agent header, no cookies, no proxy. This is the baseline every SHEIN guide should start from and almost none does.

Two of those deserve a second look.

The sitemap is the one robots.txt declares on its last line: Sitemap: https://us.shein.com/sitemap-index.xml. It returns a 403 carrying SHEIN's edge error page, whose body reads STATUS:403, The server's gone missing., EVENT ID:7657809567745158167. and Sorry, Please contact GSRM Security support for more information. The crawl entry point SHEIN publishes is closed to anyone not already trusted, so plan URL discovery around that.

The homepage returned 1.1 MB of HTML and zero product links. I grepped it for SHEIN's product URL pattern (-p-.html), got nothing, then rendered it in headless Chromium, waited six seconds and grepped the live DOM. Still nothing. It is a merchandising shell: banners, category rails, config. The response headers name who is serving it:

Server: APISIX/3.15.0
via-shein-gateway: c-shein-pc
x-anti: fDE3OTE1MjIyNTk2NTB8QzA4MzM1MjNfOUNGRl9FNTY0...
x-ad-flag-update: 8FkZa3NuCnoO+IBsqSDbVYIjAJ+rKVCv5/AvkV51i6yZ...
Set-Cookie: armorUuid=20261009130419fa9115d5...; domain=shein.com
Set-Cookie: AT=MDEwMDE.eyJiIjo3LCJnIjoxNzkxNTIyMjU5...

Apache APISIX is the gateway making the allow/deny call. armorUuid and x-anti are the first fingerprints of SHEIN's in-house anti-bot layer, which it calls Armor. Both matter later.

SHEIN's front end is not server-rendered the way an old PHP storefront was, and it is not a Next.js or Nuxt app either: there is no __NEXT_DATA__ and no window.__NUXT__ anywhere in the page. The globals it does set are window.gbCommonInfo and window.gbRawData, and on the homepage those carry configuration and layout, not catalogue.

The catalogue arrives over a backend-for-frontend layer. I captured every XHR and fetch response across two full page loads, 237 on the homepage and 89 on a listing attempt, and the shape is consistent:

So: no GraphQL. Anyone telling you to hunt for a SHEIN GraphQL endpoint is guessing. It is a versioned REST BFF whose product namespace is /bff-api/product/, and the live endpoints I confirmed by request include product/search/v3/get_keywords, product/get_goods_detail_realtime_data, order/get_cart_goods_sum_quantity and user/currency/list.

Here is the finding that changes your scraper's architecture. From the same IP already being 302'd on every HTML product route, a bare request with no cookies, no CSRF token and no signature headers returned real data:

from curl_cffi import requests

s = requests.Session(impersonate="chrome131")
r = s.get(
    "https://us.shein.com/bff-api/product/search/v3/get_keywords"
    "?_ver=1.1.8&_lang=en&word_type=2&scene=home",
    timeout=40,
)
print(r.status_code, len(r.content))
# 200 12497
print(r.json()["code"], r.json()["msg"])
# 0 ok

That response carried live goodsId values: 25870820, 53363429, 64851136, 508807931, 547070211. Real product keys, from a blocked IP, with an empty header bag.

The gate is per route. Ask for /bff-api/product/get_products_by_keywords and you get a 403 with the body {}, so the route exists and is protected. Ask for a path that does not exist and the upstream leaks its own error shape instead, {"timestamp":...,"status":404,"error":"Not Found","path":"/product/..."}, with the /bff-api prefix stripped: a Spring service behind APISIX. Some product routes are open, some are signed, and the HTML wrapping them is the most defended thing on the origin.

Our walkthrough on scraping JSON API endpoints behind a dynamic page covers the general technique. SHEIN is the aggressive version of it.

Named, with the signature your logs will show.

Two notes on that table.

The smCB_ callback with smdata and an organization key is the calling convention of the ShuMei device-intelligence SDK, and it fires on the challenge page. The challenge decision is therefore partly a device-fingerprint decision, not purely an IP one. SHEIN's robots.txt separately disallows /geetest/, a strong hint about the widget you will be asked to solve; our breakdown of the GeeTest challenge flow describes the mechanics.

captcha_type is not stable. The same URL, requested within a minute from the same IP with slightly different headers, returned 909 then 903. Do not branch on it. Branch on the path.

The standard advice for a target like this is to fix your TLS fingerprint, because Python requests presents a JA3 hash no browser has ever sent. Correct in general, insufficient here. I tested it with curl_cffi, which impersonates real browser TLS and HTTP/2 stacks:

from curl_cffi import requests
import re

URLS = {
    "home":    "https://us.shein.com/",
    "listing": "https://us.shein.com/Women-Dresses-c-12472.html",
}

for imp in ["chrome124", "chrome131", "chrome136", "safari18_0"]:
    s = requests.Session(impersonate=imp)
    for name, u in URLS.items():
        r = s.get(u, timeout=45, allow_redirects=False)
        pids = len(set(re.findall(r"-p-(\d{6,})", r.text or "")))
        print(f"{imp:12s} {name:8s} {r.status_code} "
              f"len={len(r.content):8d} pids={pids:4d} "
              f"loc={r.headers.get('location','')[:60]}")

Verbatim results:

The cookie-jar row is worth calling out. I warmed cookies on the homepage first, then sent the listing request with Referer, Accept-Language, the full sec-ch-ua trio, every Sec-Fetch-* header and Upgrade-Insecure-Requests. Still gated. Then the escalation: after roughly a dozen requests the challenge became /risk/action/limit, the harder state, and stayed there.

A correct TLS fingerprint is table stakes on SHEIN, not a bypass. It keeps you from being rejected as obviously synthetic and changes nothing about the IP reputation decision. Our primer on what TLS fingerprinting is explains why both halves get checked independently.

Next rung: a real browser. Real TLS, real JS engine, real navigator surface, so the Armor libs and the device SDK can run and post whatever they want.

import asyncio, re
from playwright.async_api import async_playwright

URL = "https://us.shein.com/Women-Dresses-c-12472.html"

async def main():
    async with async_playwright() as p:
        b = await p.chromium.launch(
            headless=True,
            args=["--disable-blink-features=AutomationControlled"],
        )
        ctx = await b.new_context(
            user_agent=("Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
                        "AppleWebKit/537.36 (KHTML, like Gecko) "
                        "Chrome/141.0.0.0 Safari/537.36"),
            viewport={"width": 1440, "height": 900},
            locale="en-US",
        )
        pg = await ctx.new_page()
        resp = await pg.goto(URL, wait_until="domcontentloaded", timeout=90000)
        await pg.wait_for_timeout(6000)
        html = await pg.content()
        print("final url:", pg.url)
        print("nav status:", resp.status)
        print("product ids:", len(set(re.findall(r"-p-(\d{6,})", html))))

asyncio.run(main())

Output:

final url:  https://us.shein.com/risk/action/limit?risk-id=E4926634595404893313
nav status: 200
product ids: 0

A real browser, from a flagged IP, lands on the same wall. The escalated state is keyed to the exit IP, not the client, so the fix is a different exit rather than a better browser. Nothing you do inside Chromium undoes an IP-level decision. This is the familiar residential versus datacenter split showing up as a hard yes/no instead of a success-rate curve.

Note the nav status: 200. Hold onto it, because it is the thing that silently ruins SHEIN pipelines.

The managed route moves the exit IP, the fingerprint and the pre-warm into request parameters. SparkProxy's Scraping API lives at https://scrape.sparkproxy.io/api/v1 and authenticates with an X-API-Key header. New accounts get 1,000 free credits, enough to characterise a target like this before you commit.

The parameters that matter for SHEIN, with the costs from the docs:

Reach for stealth=true first, for a reason specific to what I measured. The docs describe it as a homepage pre-warm plus a forced Google referrer. SHEIN's homepage is the one route that answers an unauthenticated datacenter request, so a pre-warm there is exactly the sequence a real visitor performs before touching a category page. The parameter happens to match the target's shape.

Start cheap, on the JSON surface:

import requests

API = "https://scrape.sparkproxy.io/api/v1"
KEY = "YOUR_API_KEY"

r = requests.get(API, headers={"X-API-Key": KEY}, params={
    "url": ("https://us.shein.com/bff-api/product/search/v3/get_keywords"
            "?_ver=1.1.8&_lang=en&word_type=2&scene=home"),
    "render_js": "false",                 # 1 credit, roughly 3x faster
    "transparent_status_code": "true",    # do not hide SHEIN's 302
    "country_code": "US",
    "proxy_type": "premium",
})
print(r.status_code)   # mirrors what SHEIN returned
print(r.text[:200])

transparent_status_code is not optional here. Without it you cannot tell a real payload from a redirect, and SHEIN redirects constantly. It only applies when render_js=false, which is another reason to stay on the JSON surface.

When you do need the rendered storefront, use the envelope for the metadata. One correction, because it trips people up: format selects content conversion only (html, md, mdx, json, screenshot, pdf). The envelope comes from json_response=true, and the body inside it is base64-encoded, so decode before you match on anything:

import base64, requests

r = requests.get(API, headers={"X-API-Key": KEY}, params={
    "url": "https://us.shein.com/Women-Dresses-c-12472.html",
    "render_js": "true",
    "stealth": "true",
    "country_code": "US",
    "proxy_type": "premium",
    "json_response": "true",
    "wait_for": "[class*='product-card']",
})
env = r.json()
print(env["status_code"], env["duration_ms"], env["credits_used"])

html = base64.b64decode(env["body"]).decode("utf-8", "ignore")   # decode FIRST
if "/risk/challenge" in html or "/risk/action/limit" in html:
    raise RuntimeError("risk gate, rotate the exit IP")

Decoding first is not a style point. Marker-match against the raw base64 and every block check you write returns false, so your pipeline cheerfully stores the challenge page as a product.

For the flow a human performs, chain it in one browser session rather than hoping a warm cookie survives between calls:

scenario = {"instructions": [
    {"wait": 1200},
    {"click": "a[href*='Women-Dresses-c-12472']"},
    {"wait_for": "[class*='product-card']"},
    {"scroll": 1600},
    {"wait": 1500},
]}

r = requests.post(API,
    headers={"X-API-Key": KEY, "Content-Type": "application/json"},
    json={
        "url": "https://us.shein.com/",      # start on the route SHEIN leaves open
        "render_js": True,
        "stealth": True,
        "country_code": "US",
        "proxy_type": "premium",
        "js_scenario": scenario,
        "json_response": True,
    })

Once you know the product calls you need, batch the cheap ones. Batch mode works only with render_js=false and the whole batch costs 1 credit, which makes a wide sweep of realtime price and stock calls close to free:

ids = ["53363429", "64851136", "508807931", "547070211"]
base = ("https://us.shein.com/bff-api/product/get_goods_detail_realtime_data"
        "?_ver=1.1.8&_lang=en&goods_id=")

r = requests.get(API, headers={"X-API-Key": KEY}, params={
    "url": ",".join(base + i for i in ids),   # comma-separated = batch
    "render_js": "false",
    "transparent_status_code": "true",
    "country_code": "US",
})

If you want fields rather than markup, extract_rules replaces the response with an extracted object. Do not combine it with json_response; it is one or the other:

rules = {
    "title":  "h1",
    "price":  "[class*='productPrice'] [class*='value']",
    "images": {"selector": "[class*='crop-image-container'] img", "type": "src"},
}
r = requests.post(API,
    headers={"X-API-Key": KEY, "Content-Type": "application/json"},
    json={"url": PDP_URL, "render_js": True, "stealth": True,
          "country_code": "US", "extract_rules": rules})
print(r.json()["extracted"])

SHEIN's class names are hashed per build, so treat those selectors as a shape to confirm against a live page rather than as constants. The JSON surface is the stable one.

I am not going to hand you a reverse-engineered listing path, for two reasons. I could not retrieve a listing from a datacenter IP, so I would be guessing. And SHEIN versions these routes (search/v3), so a path published today is a broken path in your repo next quarter.

Capture it from your own browser instead.

1. Open the category you care about in normal Chrome, logged out.
2. DevTools, Network tab, Fetch/XHR filter.
3. Clear the log, then scroll the grid or click page 2.
4. Sort by Size descending. The listing payload is the large application/json response from /bff-api/product/....
5. Right-click it, Copy, then Copy as cURL.
6. Read the params off the copied command. Page number, page size, category id, sort and filter keys are all there under their real names.

Store those real names, not a normalised guess, and keep the request params next to every row you write. When SHEIN changes a page-size default, the params column is the only way you will notice that your historical rows stopped being comparable.

Step 4 is where the pagination answer lives. Paginate by replaying that call with the page parameter incremented, not by driving a browser through a grid: same data, 1 credit instead of 5, no DOM to break. The one case that still needs a browser is a grid that loads only on scroll, which the API handles with scroll and js_scenario. For that pattern, see scraping infinite-scroll pages.

Go back to that Playwright output: nav status: 200. And to this, the most expensive five minutes of the exercise.

Late in the session I called the realtime product endpoint again, the one that had answered cleanly earlier, and got HTTP 200 with 272,938 bytes of text/html. A JSON endpoint, status 200, serving a quarter of a megabyte of HTML. With allow_redirects=False the real answer appeared:

# allow_redirects=False
302   (no content-type, no body)
location: /risk/challenge?captcha_type=909
          &redirection=https%3A%2F%2Fus.shein.com%2Fbff-api%2Fproduct
          %2Fget_goods_detail_realtime_data%3F...
          &risk-id=E4926638101381671811

# allow_redirects=True
200   content-type: text/html; charset=utf-8

Status codes are not a block signal on SHEIN. Follow the redirect and you get a 200; do not follow it and you get a 302 with no body. Either way r.status_code == 200 tells you nothing. Check the content, in this order:

LIMITED   = ("/risk/action/limit",)                   # hard, this IP is done for a while
CHALLENGE = ("/risk/challenge",)                      # soft, retry on a new exit
WAF       = ("GSRM Security", "EVENT ID:")            # edge 403 page, route is forbidden

def classify(resp):
    """Return 'ok', 'challenge', 'limited' or 'waf' for a SHEIN response."""
    loc   = resp.headers.get("location", "")
    ctype = resp.headers.get("content-type", "")
    body  = resp.text or ""

    for hay in (loc, body):
        if any(m in hay for m in LIMITED):
            return "limited"
    if any(m in body for m in WAF):
        return "waf"
    for hay in (loc, body):
        if any(m in hay for m in CHALLENGE):
            return "challenge"
    # a JSON route answering with HTML is blocked, whatever the status says
    if "/bff-api/" in resp.url and "html" in ctype:
        return "challenge"
    return "ok"

That last clause earns its place: a content-type mismatch on a JSON route is the cheapest reliable block signal SHEIN gives you, and it fires before any marker string. The broader patterns are in our guide on detecting when your scraper is blocked.

Act on the three block states differently. challenge means rotate the exit and retry once. limited means stop using that exit for a while, because retrying from it deepens the state. waf means the route itself is forbidden and rotation will not help, as with /sitemap-index.xml.

The useful part of an honest write-up is the list of things that do not work.

You cannot forge the gateway signature. The browser sends protected BFF calls with x-gw-auth: a=@2.0.13&b=&d=&e=, alongside x-cs-random (136 hex characters, produced by csrandom.1.1.0.min.js), x-oest, x-csrf-token fetched from /bff-api/user-api/csrf/update, and webversion: 15.4.6. The b field is a millisecond timestamp, so the value is time-bound as well as derived. Those headers are minted in-page by SHEIN's own bundle, and a stateless HTTP client cannot compute them. If a route demands them you need a browser that ran the bundle, or a different route.

You cannot pre-fill the device tokens. In the browser's own BFF requests, armortoken and smdeviceid were both sent empty, because the Armor libs and the device SDK populate them asynchronously after boot. A headless browser that navigates immediately sends blanks too. If a route cares about those values you have to let the page sit, which is what wait and the leading {"wait": 1200} in the scenario above are for.

You cannot carry a cookie jar across API calls with session_id. The docs are explicit: it is a label for the browser profile in per-job logs and does not persist cookies between requests. Warming a jar means capturing the cookies and passing them back with the cookies parameter on the next call (name, value and domain are all required, and domain should be shein.com so armorUuid scopes correctly), or doing the whole sequence inside one js_scenario.

You cannot solve the challenge statelessly. The challenge page is a JS application that posts to /risk/verify/identity/validation/token, .../publish/sign/rule and .../resources, then asks for an interactive gesture. Treat /risk/challenge as a routing instruction, not a puzzle.

I could not retrieve a listing page at all. Not with curl, not with four TLS fingerprints, not with headless Chromium, not from five country hosts including ones I had never touched: it.shein.com and www.shein.se both challenged on first contact. So I cannot tell you from observation whether the product grid markup sits in SHEIN's server HTML, and I will not assert it. The question is answerable in sixty seconds from your own browser, and the JSON path makes it mostly irrelevant.

Three structural facts that will cost you a migration if you get them wrong.

The host is part of the key. www.shein.com is a geo dispatcher, not a storefront. From my IP it returned a 403 and bounced to https://www.sheinindia.in/?cdn_rsite=ak&ref=www&rep=dir&ret=in. Storefronts are us.shein.com, de.shein.com, www.shein.co.uk, www.shein.se, it.shein.com, plus mobile web at m.shein.com/us/.... Request the country host explicitly and store it on the row. Treating country as a filter rather than a key is the mistake that makes multi-market price data useless, and it is the same one we unpacked for Shopee and Lazada.

URL shapes carry the type. Categories are /-c-.html, confirmed live on /Women-Dresses-c-12472.html, /Women-Clothing-c-2030.html and /Kids-c-2031.html. Curated rails use -sc- instead, as in /recommend/US-New-sc-1004675159.html. Product pages are /-p-.html, where the slug is decorative and only -p- is load-bearing.

goodsId is not eight digits. One live BFF response gave me 25870820, 53363429, 64851136, 97777548, 508807931 and 547070211. Eight and nine digits in the same payload, no padding. Store it as text, or at minimum a 64-bit integer. A goods_id INT column and a regex expecting a fixed width both break on real data, quietly, on a subset of rows.

A table that holds up:

CREATE TABLE shein_observation (
  captured_at    TIMESTAMPTZ  NOT NULL,   -- when YOU fetched, not a site field
  host           TEXT         NOT NULL,   -- 'us.shein.com', part of the key
  goods_id       TEXT         NOT NULL,   -- text: width varies
  cat_id         TEXT,                    -- from the -c- segment
  price_amount   NUMERIC(12,4),           -- numeric value
  price_currency CHAR(3),                 -- ISO 4217, stored separately
  in_stock       BOOLEAN,
  request_params JSONB,                   -- the exact params you sent
  exit_country   CHAR(2),                 -- country_code you asked for
  PRIMARY KEY (captured_at, host, goods_id)
);

Two of those columns are the ones people leave out and regret. request_params is how you detect that a default page size changed under you. exit_country is how you prove a price series came from the market it claims to describe, which matters the moment someone puts two markets on one chart.

Short version, because this is an engineering guide. Public product data only: title, price, currency, availability, images, aggregate rating. Nothing behind a login, nothing from /user/, /cart/ or any authenticated area, and no personal data about reviewers or buyers. SHEIN's robots.txt disallows /user/, /cart/, /atomic/, /geetest/, /abt/userinfo and /datasecurity; honour all of it, and remember that a crawler-directive file under RFC 9309 is not a licence for the paths it leaves out.

In the United States, the Ninth Circuit's decision in hiQ Labs v. LinkedIn (2022) held that scraping genuinely public data generally does not violate the Computer Fraud and Abuse Act. That is a ruling about unauthorised access, not blanket permission. SHEIN's terms of use separately prohibit automated collection, so a scrape can be a contract problem where it is not a CFAA problem. If the data feeds a commercial product, ask a lawyer rather than a blog.

On pacing, the measurements above are the argument. A dozen requests from one datacenter IP moved me from a soft challenge to a hard limit a real browser could not get through. Keep concurrency per exit in low single digits, add real jitter rather than a fixed sleep, and back off on challenge before you reach limited, because the second state costs you the IP. Our notes on ethical scraping and rate limiting give the numbers to start from.

SHEIN is a cousin of the Alibaba-era marketplaces in how it serves data, a thin shell over a private JSON layer, and the parsing lessons transfer even though the anti-bot stack does not. For the same pattern on a target with a softer gate, scraping AliExpress product data is the gentler version of this article.