How to Scrape Shopee and Lazada Marketplace Data

Shopee and Lazada are both operated as a set of national marketplaces sharing a codebase. The HTML looks identical across markets. Everything underneath is different.

Shopee also runs shopee.tw, shopee.com.br and shopee.com.mx outside the region, sharing the same API shapes. Lazada stays inside these six markets.

Three consequences shape the whole pipeline.

Product IDs are per-market. A Shopee listing is keyed by the pair (shop_id, item_id) visible in the URL as -i.{shop_id}.{item_id}. Lazada keys on i{itemId}-s{skuId}. The same physical product sold by the same brand on shopee.sg and shopee.co.id carries completely unrelated IDs, because the seller created two listings in two seller accounts. There is no cross-market product key. You have to build one.

Currency minor units differ. Vietnamese dong has zero decimal places under ISO 4217, and Indonesian rupiah is priced in whole units in practice despite a nominal two. If your ingest stores everything as integer cents, a 250,000 VND item becomes 2,500 VND and nobody notices for a month. Store the numeric amount and the currency code, not "cents".

Geo is enforced at the edge. Requesting shopee.co.th from a US datacenter IP typically gets you a region-selection redirect, a thinner catalogue, or a block, and requesting a Lazada catalogue endpoint from outside the market often returns an empty result set rather than an error. Country targeting is structural here, not an optimisation. For the routing theory behind it, what geo-targeting means in proxies covers the mechanics.

The usual framing applies with a regional wrinkle. In the United States, the Ninth Circuit's decision in hiQ Labs v. LinkedIn (2022) held that scraping publicly accessible data does not by itself violate the Computer Fraud and Abuse Act. That addresses unauthorised access, not contract. Both platforms prohibit automated collection in the Terms of Use linked from every country site's footer, so a scrape can breach contract even where no computer-misuse statute is engaged.

The wrinkle is data protection. Singapore's Personal Data Protection Act, Indonesia's PDP Law 27/2022, and Vietnam's Decree 13/2023 all cover personal data of individuals in those countries, and none of them offers a broad "publicly available" carve-out you can lean on.

That drives one hard rule: collect product, price and aggregate rating data only. A product's average star rating and review count are facts about the product. A reviewer's username, avatar and review text are facts about a person. Do not build reviewer profiles, do not store reviewer identifiers, and do not design a schema with a column that could hold one. The schema later in this guide has no reviewer identity field, and that is deliberate.

Respect robots.txt as specified in RFC 9309, stay off anything behind a login, and get counsel before shipping a commercial product on this data.

Neither site needs HTML parsing for the fields that matter. Both are single-page applications that fetch their own JSON, which is far more stable than any CSS selector. The general technique appears in how to scrape hidden JSON API endpoints. Here are the specific endpoints.

Shopee

Shopee wraps every response in {"error": 0, "error_msg": null, "data": {...}}. A missing or delisted item returns HTTP 200 with error: 4 and data: null, so the status code tells you nothing. Check the envelope.

The endpoints are picky about headers. A bare request usually fails. One carrying Referer set to the product page URL, plus X-API-SOURCE: pc, X-Requested-With: XMLHttpRequest and Accept: application/json, usually succeeds.

Lazada

Lazada's trick is simpler and less well known: append ajax=true to almost any catalogue, search or category URL and it returns the page's JSON payload instead of HTML.

Results land in mods.listItems, each item carrying itemId, skuId, name, price, originalPrice, discount, ratingScore, review, sellerName, sellerId, brandName, location and itemSoldCntShow. Product detail pages do not accept ajax=true. Their state is serialised into a window.__moduleData__ assignment, extracted with a brace-matching scan.

Lazada is owned by Alibaba Group and inherits the anti-bot stack you meet on AliExpress, including the _____tmd_____/punish interstitial and x5secdata challenge pages that return HTTP 200 with no payload. Scraping AliExpress product data covers those block signatures in depth.

There is also a mobile gateway at acs-m.lazada.sg running Alibaba's mtop protocol, signed with a rotating _m_h5_tk token cookie. Reproducing that signing scheme is a maintenance treadmill and the web endpoints return the same catalogue. Skip it.

The SparkProxy Scraping API takes a target URL and returns the response, handling proxy rotation, geo-routing and the anti-bot layer. Base endpoint: https://scrape.sparkproxy.io/api/v1, authenticated with an X-API-Key header.

Four parameters carry a Shopee or Lazada scrape:

• country_code: ISO 3166-1 alpha-2 exit country. This is the one that matters most here. SG, MY, ID, TH, VN, PH.
• premium_proxy=true: residential exits. Both platforms flag datacenter ranges quickly on repeated catalogue hits.
• render_js: set false for the JSON endpoints, which need no browser and cost far fewer credits, and true for HTML product pages where state is injected during render.
• forward_headers: a JSON object of headers merged into the request. This is how you satisfy Shopee's Referer and X-API-SOURCE requirements.

A JSON-endpoint call for a Thai Shopee product:

curl -G "https://scrape.sparkproxy.io/api/v1" \
  -H "X-API-Key: YOUR_API_KEY" \
  --data-urlencode "url=https://shopee.co.th/api/v4/pdp/get_pc?item_id=12345678901&shop_id=234567890" \
  --data-urlencode "render_js=false" \
  --data-urlencode "premium_proxy=true" \
  --data-urlencode "country_code=TH" \
  --data-urlencode 'forward_headers={"Referer":"https://shopee.co.th/product-i.234567890.12345678901","X-API-SOURCE":"pc","X-Requested-With":"XMLHttpRequest","Accept":"application/json"}'

Full parameter reference is in the Scraping API docs. Note the credit asymmetry: a no-JS request with a premium proxy and a country code costs a fraction of the same request with rendering on. On a six-market hourly crawl, that difference decides whether the project is viable.

Start by making the market a first-class argument, never a hardcoded host.

import json
import requests

API = "https://scrape.sparkproxy.io/api/v1"
API_KEY = "YOUR_API_KEY"

SHOPEE_MARKETS = {
    "SG": ("shopee.sg",     "SGD"),
    "MY": ("shopee.com.my", "MYR"),
    "ID": ("shopee.co.id",  "IDR"),
    "TH": ("shopee.co.th",  "THB"),
    "VN": ("shopee.vn",     "VND"),
    "PH": ("shopee.ph",     "PHP"),
}

def shopee_pdp(country: str, shop_id: int, item_id: int) -> dict:
    host, _currency = SHOPEE_MARKETS[country]
    target = f"https://{host}/api/v4/pdp/get_pc?item_id={item_id}&shop_id={shop_id}"
    referer = f"https://{host}/product-i.{shop_id}.{item_id}"
    resp = requests.get(
        API,
        headers={"X-API-Key": API_KEY},
        params={
            "url": target,
            "render_js": "false",
            "premium_proxy": "true",
            "country_code": country,
            "forward_headers": json.dumps({
                "Referer": referer,
                "X-API-SOURCE": "pc",
                "X-Requested-With": "XMLHttpRequest",
                "Accept": "application/json",
            }),
        },
        timeout=90,
    )
    resp.raise_for_status()
    return resp.json()

Then treat the payload as suspect until the envelope proves otherwise.

def shopee_data(payload: dict) -> dict | None:
    """Shopee returns HTTP 200 for blocks and dead listings alike."""
    if not isinstance(payload, dict):
        return None
    if payload.get("error"):        # 4 = not found/delisted, others = throttled
        return None
    data = payload.get("data") or {}
    return data.get("item") or data or None

Now the field that trips up every first-time Shopee scraper. Prices in the API are integers scaled by 100,000, not by 100.

from decimal import Decimal

SHOPEE_PRICE_SCALE = Decimal(100_000)

def shopee_price(raw):
    """998000000 -> 9980.00 THB. Verify against one displayed price per market."""
    if raw in (None, 0):
        return None
    return (Decimal(raw) / SHOPEE_PRICE_SCALE).quantize(Decimal("0.01"))

def core_fields(item: dict, country: str) -> dict:
    rating = item.get("item_rating") or {}
    return {
        "country": country,
        "item_id": item.get("itemid") or item.get("item_id"),
        "shop_id": item.get("shopid") or item.get("shop_id"),
        "title": item.get("name"),
        "currency": item.get("currency"),
        "price": shopee_price(item.get("price")),
        "price_min": shopee_price(item.get("price_min")),
        "price_max": shopee_price(item.get("price_max")),
        "price_before_discount": shopee_price(item.get("price_before_discount")),
        "discount_pct": item.get("raw_discount"),
        "stock": item.get("stock"),
        "units_sold_total": item.get("historical_sold"),
        "units_sold_recent": item.get("sold"),
        "rating_avg": rating.get("rating_star"),
        # rating_count is [total, 1-star, 2-star, 3-star, 4-star, 5-star]
        "rating_count": (rating.get("rating_count") or [None])[0],
        "seller_location": item.get("shop_location"),
    }

That divisor is consistent across markets, but confirm it once per market against a displayed price before trusting a whole crawl to it. On a VND listing the raw integer runs to eleven digits and looks broken even when it is right.

Lazada's search JSON needs no headers beyond a normal browser set, which makes it the cheaper of the two to crawl.

import re
from urllib.parse import quote

LAZADA_MARKETS = {
    "SG": ("www.lazada.sg",     "SGD"),
    "MY": ("www.lazada.com.my", "MYR"),
    "ID": ("www.lazada.co.id",  "IDR"),
    "TH": ("www.lazada.co.th",  "THB"),
    "VN": ("www.lazada.vn",     "VND"),
    "PH": ("www.lazada.com.ph", "PHP"),
}

class BlockedError(RuntimeError):
    pass

def lazada_search(country: str, keyword: str, page: int = 1) -> list:
    host, _ = LAZADA_MARKETS[country]
    target = f"https://{host}/catalog/?ajax=true&q={quote(keyword)}&page={page}"
    resp = requests.get(
        API,
        headers={"X-API-Key": API_KEY},
        params={
            "url": target,
            "render_js": "false",
            "premium_proxy": "true",
            "country_code": country,
        },
        timeout=90,
    )
    body = resp.text
    if "_____tmd_____/punish" in body or "x5secdata" in body:
        raise BlockedError(f"lazada punish page for {country}/{keyword}")
    payload = json.loads(body)
    return (payload.get("mods") or {}).get("listItems") or []

Lazada prices arrive as display strings, currency symbol and thousands separators included.

MONEY = re.compile(r"[\d.,]+")

def lazada_price(text, currency: str):
    """'RM1,299.00' -> 1299.00 ; '2.450.000' -> 2450000"""
    if not text:
        return None
    found = MONEY.search(text)
    if not found:
        return None
    s = found.group(0)
    if currency in ("VND", "IDR"):
        # dot is the thousands separator, no minor units in practice
        s = s.replace(".", "").replace(",", "")
    else:
        s = s.replace(",", "")
    return Decimal(s)

For product detail pages, the state sits in a script assignment rather than an endpoint:

def lazada_module_data(html: str):
    marker = "window.__moduleData__"
    i = html.find(marker)
    if i == -1:
        return None
    start = html.index("{", i)
    depth, in_str, esc = 0, False, False
    for j in range(start, len(html)):
        ch = html[j]
        if in_str:
            if esc:          esc = False
            elif ch == "\\": esc = True
            elif ch == '"':  in_str = False
            continue
        if ch == '"':   in_str = True
        elif ch == "{": depth += 1
        elif ch == "}":
            depth -= 1
            if depth == 0:
                try:
                    return json.loads(html[start:j + 1])
                except json.JSONDecodeError:
                    return None
    return None

This is the part generic e-commerce guides get wrong for Southeast Asia. On a Shopify store or Amazon, a scraped price is usually valid for hours or days. On Shopee and Lazada it is often valid for ninety minutes.

Both platforms run three overlapping layers of time-boxed pricing:

• Flash sale slots. Shopee Flash Deals and LazFlash run in fixed daily blocks (commonly 00:00, 09:00, 12:00, 15:00, 18:00 and 21:00 local), each with capped stock. Price and stock both reset at slot boundaries.
• Double-date campaigns. The 9.9, 10.10, 11.11 and 12.12 sales, plus monthly X.X events and Lazada's March birthday sale. Campaign prices flip at 00:00 local time on the campaign day.
• Payday sales. Late-month promotions timed to salary cycles, which differ per market.

All of those clocks are local. Singapore, Malaysia and the Philippines run UTC+8; Indonesia spans UTC+7 to UTC+9 with Jakarta on UTC+7; Thailand and Vietnam are UTC+7. Resolve them against the IANA time zone database rather than hardcoding offsets, and store both the UTC instant and the local wall-clock time. A price series that only knows UTC cannot answer "what did this cost at noon on 11.11 in Jakarta", which is the question the business will actually ask.

from datetime import datetime, timezone
from zoneinfo import ZoneInfo

MARKET_TZ = {
    "SG": "Asia/Singapore", "MY": "Asia/Kuala_Lumpur", "ID": "Asia/Jakarta",
    "TH": "Asia/Bangkok",   "VN": "Asia/Ho_Chi_Minh",  "PH": "Asia/Manila",
}

def stamp(country: str) -> dict:
    now = datetime.now(timezone.utc)
    local = now.astimezone(ZoneInfo(MARKET_TZ[country]))
    return {
        "captured_at_utc": now.isoformat(timespec="seconds"),
        "captured_at_local": local.isoformat(timespec="seconds"),
        "local_date": local.date().isoformat(),
        "local_hour": local.hour,
    }

def flash_window(item: dict) -> dict:
    fs = item.get("flash_sale") or item.get("upcoming_flash_sale") or {}
    return {
        "promo_type": "flash" if item.get("flash_sale")
                      else ("upcoming_flash" if item.get("upcoming_flash_sale") else "none"),
        "promo_id": fs.get("promotionid"),
        "promo_start_utc": fs.get("start_time"),   # unix seconds
        "promo_end_utc": fs.get("end_time"),
        "promo_stock": fs.get("stock"),
    }

The practical rule: never store a price without a capture timestamp and a promo state, and never average prices across a campaign boundary. During 11.11 week, sample the tracked SKUs at least once per flash-sale slot. Outside campaign season, twice a day is usually enough.

The number on the product card is not what anyone pays. Both platforms stack discounts that never appear in the price field:

A competitor whose card price is 5% higher than yours can be 18% cheaper at checkout after a shop voucher and a two-piece bundle. If your dashboard compares card prices, it will confidently tell you the wrong thing.

The fix is to store the components separately and compute the comparable figure in the analysis layer, never at ingest time.

def effective_unit_price(row: dict):
    """Best realistic per-unit price for a single-SKU purchase.
    Deliberately excludes coins (deferred) and platform vouchers (basket-level)."""
    base = row.get("price")
    if base is None:
        return None
    best = base

    for v in row.get("shop_vouchers") or []:
        if v.get("min_spend") and Decimal(v["min_spend"]) > base:
            continue                    # a single unit cannot reach the threshold
        if v.get("discount_percentage"):
            best = min(best, base * (1 - Decimal(v["discount_percentage"]) / 100))
        elif v.get("discount_value"):
            best = min(best, base - Decimal(v["discount_value"]))

    bundle = row.get("bundle_deal") or {}
    if bundle.get("min_qty") == 1 and bundle.get("unit_price"):
        best = min(best, Decimal(bundle["unit_price"]))

    return best.quantize(Decimal("0.01"))

Keep price, price_before_discount, effective_unit_price and shipping_fee as four separate columns. Collapsing them into one "price" is the most common way these datasets become unusable six months in. The same pattern applies to any marketplace: see scraping e-commerce prices.

Both platforms run a tiered seller system, and the tier predicts price, authenticity risk and delivery time well enough to belong in your model as a feature rather than a footnote.

Two things follow. First, a Mall listing and a regular listing of the same product are not comparable observations. Mall prices commonly sit above marketplace prices for the identical SKU because the buyer is paying for return rights. Blending them into one average produces a number that describes nothing.

Second, the cross-border flag is the most underused field on both platforms. An overseas seller in a Thai or Filipino result set has a materially different delivery window and often a different tax position, so it deserves its own boolean rather than being folded into a free-text location.

def seller_tier(platform: str, row: dict) -> str:
    if platform == "shopee":
        if row.get("is_official_shop"):          return "mall"
        if row.get("is_preferred_plus_seller"):  return "preferred_plus"
        if row.get("is_preferred_seller"):       return "preferred"
        if row.get("shopee_verified"):           return "verified"
        return "regular"
    # lazada
    if row.get("inLazMall") or row.get("isLazMall"):  return "mall"
    if row.get("isTopSeller"):                        return "top_seller"
    return "regular"

def is_cross_border(country: str, row: dict) -> bool:
    loc = (row.get("shop_location") or row.get("location") or "").strip().lower()
    if loc in ("overseas", "international", "china"):
        return True
    return bool(loc) and country.lower() not in loc and loc != "-"

Here is where most cross-market projects quietly fail. You have a Vietnamese title on shopee.vn, a Thai title on lazada.co.th and an Indonesian title on shopee.co.id, and you want to know they are the same product. The obvious move is to machine-translate everything and fuzzy-match. That does not work, for four specific reasons.

Translation is lossy in exactly the wrong places. Marketplace titles in the region are keyword-stuffed strings, not sentences: brand, model, spec, colour, then a tail of search terms in mixed languages. A translator normalises "Tai nghe Bluetooth Sony WH-1000XM5 chính hãng" and the Thai equivalent into similar English, but it also normalises three different Sony models into similar English. Recall goes up, precision collapses.

Thai has no spaces between words. Whitespace tokenisation produces one giant token. You need dictionary-based segmentation following the text-boundary rules in UAX #29, and even then results vary by library.

Vietnamese has two valid Unicode encodings of the same text. Precomposed (NFC) and decomposed (NFD) forms of "chính hãng" are different byte sequences that render identically. String equality fails silently. Normalise per UAX #15 before comparing anything.

Indonesian and Filipino titles are code-mixed. Sellers write half in English, so a language detector labels titles inconsistently and any language-conditional logic branches unpredictably.

The approach that works ignores the prose entirely and matches on identifiers that are language-neutral:

import unicodedata

MODEL = re.compile(r"\b([A-Z]{2,5}[-\s]?\d{2,6}[A-Z]{0,3})\b")
GTIN  = re.compile(r"\b(\d{12,14})\b")

def match_key(title: str, brand):
    """Language-neutral key: GTIN, else brand + model code. No translation involved."""
    t = unicodedata.normalize("NFC", title or "").upper()

    gtin = GTIN.search(t)
    if gtin:
        return f"gtin:{gtin.group(1)}"

    model = MODEL.search(t)
    if model and brand:
        code = re.sub(r"[-\s]", "", model.group(1))
        return f"{brand.strip().lower()}:{code.lower()}"
    return None

Rank your join keys and fall back down the list rather than sideways into translation:

1. GTIN or EAN in the title or attributes, per the GS1 GTIN standard. Uncommon in the region but decisive when present.
2. Brand plus manufacturer model code extracted from the title, normalised to NFC and case-folded.
3. Perceptual hash of the primary product image. Sellers reuse the manufacturer's press photo across markets constantly, which makes image hashing surprisingly strong here.
4. A manual mapping table for your tracked SKU set.

For a competitor panel of a few hundred SKUs, option 4 plus option 2 reaches production faster than any translation pipeline, and stays correct.

Both platforms are app-first. The web catalogue is a subset of what the app shows, and the differences are systematic rather than random:

• App-exclusive vouchers and prices. Both run "app-only" promotions that never render on the web PDP. Your scraped price can be genuinely correct for web and genuinely wrong for the majority of buyers.
• In-app games and coin balances. Shopee Coins earned through daily check-ins discount checkout in ways no web field exposes.
• Live commerce. Shopee Live and LazLive sell at prices set during a stream. Those transactions never touch a PDP price field.
• Different flash-sale inventory. Slot stock is sometimes allocated separately per channel.
• Signed API access. The apps talk to a signed mtop-style gateway, which is why app data is not casually reachable.

Record this rather than pretending it away. Add a channel column set to web and treat your series as a web-price index, not a market-price index. When a stakeholder asks why a competitor's app screenshot disagrees with your dashboard, that column is the answer.

Here is a table that survives all six markets, both platforms, campaign season, and a privacy review. Note the absence of any reviewer identity column, by design.

CREATE TABLE marketplace_observation (
  id                    BIGSERIAL PRIMARY KEY,
  platform              TEXT NOT NULL,        -- 'shopee' | 'lazada'
  country_code          CHAR(2) NOT NULL,     -- ISO 3166-1 alpha-2
  channel               TEXT NOT NULL DEFAULT 'web',
  -- identity (per-market, never assume global uniqueness)
  item_id               TEXT NOT NULL,
  shop_id               TEXT,
  sku_id                TEXT,
  product_url           TEXT NOT NULL,
  match_key             TEXT,                 -- 'gtin:...' or 'brand:model'
  -- listing
  title_raw             TEXT NOT NULL,        -- original language, NFC normalised
  title_lang            TEXT,
  brand                 TEXT,
  seller_tier           TEXT NOT NULL,        -- mall|preferred_plus|preferred|verified|top_seller|regular
  seller_location       TEXT,
  is_cross_border       BOOLEAN NOT NULL DEFAULT FALSE,
  -- money (never store as integer cents: VND has no minor units)
  currency              CHAR(3) NOT NULL,     -- ISO 4217
  price                 NUMERIC(14,2),
  price_before_discount NUMERIC(14,2),
  discount_pct          SMALLINT,
  effective_unit_price  NUMERIC(14,2),
  shipping_fee          NUMERIC(14,2),
  -- promotion state
  promo_type            TEXT NOT NULL DEFAULT 'none',
  promo_id              TEXT,
  promo_start_utc       TIMESTAMPTZ,
  promo_end_utc         TIMESTAMPTZ,
  has_shop_voucher      BOOLEAN NOT NULL DEFAULT FALSE,
  has_bundle_deal       BOOLEAN NOT NULL DEFAULT FALSE,
  -- availability and aggregate signals (NO reviewer identity, by design)
  stock                 INTEGER,
  units_sold_total      INTEGER,
  rating_avg            NUMERIC(3,2),
  rating_count          INTEGER,
  -- time
  captured_at_utc       TIMESTAMPTZ NOT NULL,
  captured_at_local     TIMESTAMP   NOT NULL,
  local_hour            SMALLINT    NOT NULL,
  UNIQUE (platform, country_code, item_id, sku_id, captured_at_utc)
);

CREATE INDEX ON marketplace_observation (match_key, country_code, captured_at_utc DESC);

Every row is an observation, not a product. Never update a row in place, because the time series is the whole point of the dataset. Storage patterns for that live in how to store scraped data.

Both platforms are aggressive with automated traffic, and no amount of header craft outruns a bad request rate. What keeps a crawl alive:

• Per-market concurrency, not global. Two to four concurrent requests per country host is a sane starting point. Six markets at three each is eighteen in flight, which is plenty.
• Jitter every interval. Fixed 500 ms gaps are a fingerprint. Randomise between roughly 400 ms and 1,400 ms.
• Back off on the envelope, not the status code. Shopee throttling shows up as a non-zero error field with HTTP 200. Lazada throttling shows up as a punish page with HTTP 200. Both need exponential backoff and a fresh exit IP.
• Crawl campaign days hard and ordinary days lightly. A flat hourly schedule wastes budget for 340 days and undersamples on the 25 that matter.
• Search endpoints cost more goodwill than product endpoints. Resolve your SKU list once, then poll product endpoints. Re-running broad keyword searches hourly is what gets a project shut down.

This guide to ethical scraping and rate limiting has concrete numbers.