How to Scrape IndiaMART Supplier Data: B2B Market Intel
Scrape IndiaMART supplier data for B2B market research: parse lakh and crore prices, MOQ and units, map supplier geography, and stay inside DPDP Act limits.
Teams scrape IndiaMART supplier data for one of two reasons, and only one of them holds up. The first is market intelligence: how deep a category actually is, what a stainless steel flange costs per kilogram in Rajkot versus Ludhiana, how many verified suppliers sit behind a product term. The second is harvesting phone numbers for cold calls, which walks straight into India's Digital Personal Data Protection Act, 2023 and, if any of those contacts sit in the EU, GDPR. This guide builds the first thing properly, and marks exactly where the second one starts so you do not wander into it by accident.
Short version: collect commercial facts, not contact facts. Render the category page through an Indian residential exit, pull cards by URL scheme rather than CSS class, expand lakh and crore before any number reaches your database, treat "Ask Price" as a coverage rate you publish rather than rows you delete, and give the warehouse table no phone column at all.
What you can legitimately collect
IndiaMART is a B2B marketplace where Indian manufacturers, traders, and exporters publish product listings and company storefronts. A listing carries commercial facts (product name, specification, price, minimum order quantity, city, verification badge) and, on the storefront, contact facts (a phone number, sometimes an address, sometimes a named person). Those two groups carry completely different legal weight, and the most useful thing you can do before writing code is decide which columns your pipeline is allowed to have.
| Field group | Examples | Verdict |
|---|---|---|
| Product commercial data | Title, specification bullets, price, price unit, MOQ, brand, packaging | Collect freely |
| Category structure | Category name, sub-category, product term, listing count | Collect freely |
| Company identity | Company name, storefront URL, member-since year, TrustSEAL badge, star rating | Collect freely |
| Coarse geography | City, state, GSTIN state code | Collect freely |
| Named individuals | Proprietor or contact person's name, designation | Careful, this is personal data |
| Direct contact details | Phone number, mobile, email, WhatsApp link | Careful, and usually login-gated |
| Anything behind a login | Revealed numbers, buyer leads, inquiry data | Do not collect |
The first four rows answer almost every real business question: category depth, price bands, spec distributions, supplier density by city, assortment overlap with a competitor, how much of a category carries a TrustSEAL badge. None of that needs a single phone number. Build the pipeline so the contact columns do not exist, rather than collecting them and promising to be careful later. Absence is a control you can audit; intent is not.
The privacy boundary: DPDP Act, GDPR, and cold outreach
"Publicly visible" and "free to process" are not the same statement, and IndiaMART is where a lot of teams learn that expensively. Three separate regimes bite here.
India's DPDP Act, 2023. The Digital Personal Data Protection Act received presidential assent on 11 August 2023 and governs digital personal data about identifiable individuals. Two clauses matter for scraping. Section 3(b) extends the Act to processing done outside India where it relates to offering goods or services to data principals inside India, so a US or EU team scraping Indian suppliers is inside its scope, not outside it. Section 3(c)(ii) then carves out personal data that the data principal has made publicly available themselves. That carve-out reads like a green light and is much narrower than it looks. It covers data the individual published about themselves. A proprietor's mobile number on their own storefront may qualify. The name and number of an employee, uploaded by the company they work for, was not made public by that individual. And the carve-out only limits the Act. It does nothing about IndiaMART's own terms of use, and nothing about the rules governing what you then do with the number.
Pin the timing rather than assuming it. Assent came in August 2023, the draft DPDP Rules went out for public consultation on 3 January 2025, and the final Rules were notified in November 2025 with a phased commencement instead of a single switch-on date. That phase-in was still running as of August 2026, so check the commencement schedule in the Gazette notification against your own go-live date before you plan around it.
The outreach itself. In India, unsolicited commercial calls and SMS fall under TRAI's Telecom Commercial Communications Customer Preference Regulations, 2018, with a preference registry (still widely called DND) and registration duties on senders. The exposure from a scraped contact list usually arrives through this door rather than through data protection law. Buying a list someone else scraped does not launder it either.
GDPR, if any contact sits in the EU. GDPR has no public-availability exemption. If you process personal data of people in the EU, Article 14 requires you to tell each person that you obtained their data indirectly, generally within one month, including what you hold and where it came from. Poland's data protection authority fined Bisnode in March 2019 (PLN 943,000) precisely for skipping that notice on data pulled from public registers. Public sourcing was never the defence.
The practical line for an engineer is sharper than any of this. On IndiaMART a supplier's number is typically revealed only after you log in. The moment your scraper authenticates to reveal contact details you are no longer collecting public data: you have accepted the terms of use, you are acting as a logged-in user, and every framework that treats contrary-to-terms access as an aggravating factor now applies to you. Keep the crawler logged out. If your team needs contact data for outreach, source it through a route with consent and an audit trail, and read Proxies for Lead Generation for how that side is structured. This article is engineering guidance, not legal advice, and a commercial project deserves counsel who knows Indian data law.
Scraping at scale? Skip the blocks.
Fast, unblockable datacentre proxies with unlimited bandwidth.
How IndiaMART is organised: dir, impcat, and city facets
IndiaMART splits across two hostnames, and knowing which is which saves you from crawling the wrong shape of page.
| URL pattern | What it is | Use it for |
|---|---|---|
| `dir.indiamart.com/impcat/ | Product category listing ("impcat") | The main harvest surface for a category |
| `dir.indiamart.com/ | The same category filtered to one city | Geographic distribution, no personal data involved |
| `dir.indiamart.com/search.mp?ss= | Free-text search across listings | Discovery when you do not know the category slug |
| `www.indiamart.com/proddetail/ | One product listing detail page | Full specification tables |
| `www.indiamart.com/ | A supplier storefront | Company identity, badges, GST number |
| `m.indiamart.com/...` | Mobile rendering of the same content | Sometimes lighter HTML, worth testing |
The impcat path is IndiaMART's product-category namespace, and the term inside it is the slugified product name (industrial-valves, stainless-steel-flange, cotton-knitted-fabric). The city segment is the quietly valuable part: the same category rendered per city gives supplier density by location as plain listing counts, with nothing personal in it. Indian manufacturing clusters hard. Castings around Rajkot, bicycle and auto components around Ludhiana, knitwear around Tiruppur, ceramic tiles around Morbi. A per-city sweep of one category surfaces those clusters in an afternoon.
Before you crawl, read https://www.indiamart.com/robots.txt and https://dir.indiamart.com/robots.txt yourself and honour what they say for your crawler's user agent. They change, so treat any blog's summary of them (including this one) as stale. Our guide on ethical scraping and rate limiting covers how to turn that into concurrency settings you can defend.
Field reference: what a listing actually exposes
A category listing card is denser than most directory results. Here is the set worth modelling, with where each value lives.
| Field | Where it appears | Notes |
|---|---|---|
| Product title | Card heading, links to `/proddetail/` | Your listing-level label |
| Price value | Card price line, prefixed with a rupee sign | Often absent, see the Ask Price section |
| Price unit | After the price: "/ Piece", "/ Kg", "/ Metric Ton" | Meaningless without normalisation |
| MOQ | "Minimum Order Quantity" line | A quantity plus its own unit |
| Company name | Card footer link to the storefront | Not a reliable key, see modelling |
| Storefront URL | The company link's `href` | The best stable key you get without logging in |
| City / location | Card location line | Often the sales office, not the plant |
| Verification badges | "TrustSEAL Verified", "Verified Supplier", "Leading Supplier" | Useful as a market-maturity signal |
| Member since | Storefront header, a year | Rough proxy for supplier tenure |
| Rating and response rate | Storefront header | Directional only, coverage is patchy |
| GST number | Storefront company details, when published | The strongest join key available, see geography |
| Specification table | Product detail page, label and value rows | The richest structured block on the site |
Two of these are traps. City is where the seller chose to be listed, frequently a sales office rather than a factory, so it is not a manufacturing location on its own. Company name is a string, and one Indian firm arrives in a dozen renderings of itself, which the modelling section deals with.
Set up the SparkProxy Scraping API
IndiaMART is not fortified like a major retail marketplace, but it rate limits per IP, lazy-loads listing cards with JavaScript, and serves a degraded or differently routed experience to some foreign traffic. Three parameters carry the work:
render_js=true: category pages hydrate their cards after load, so a real Chromium render gets you the populated DOM instead of a skeleton.premium_proxy=true: routes through residential IPs, which survive far longer than a datacenter range on a directory that watches per-IP request rates.country_code=in: places the exit in India. Prices are quoted in INR to Indian visitors, and you want the market as an Indian buyer sees it, not whatever a foreign IP gets. If that reasoning is new, what geo-targeting means in proxies explains the mechanics.
The endpoint is https://scrape.sparkproxy.io/api/v1 and auth is one header, X-API-Key:
curl -G "https://scrape.sparkproxy.io/api/v1" \
-H "X-API-Key: YOUR_API_KEY" \
--data-urlencode "url=https://dir.indiamart.com/impcat/industrial-valves.html" \
--data-urlencode "render_js=true" \
--data-urlencode "premium_proxy=true" \
--data-urlencode "country_code=in"
The Python wrapper every later snippet uses:
import requests
API = "https://scrape.sparkproxy.io/api/v1"
API_KEY = "YOUR_API_KEY"
def fetch(url: str, **extra) -> str:
params = {
"url": url,
"render_js": "true", # cards hydrate client-side
"premium_proxy": "true", # residential exit
"country_code": "in", # see INR pricing as an Indian buyer does
}
params.update(extra)
resp = requests.get(API, headers={"X-API-Key": API_KEY}, params=params, timeout=90)
resp.raise_for_status()
return resp.text
Full parameter and response details are in the Scraping API docs. Note what the setup omits: no cookie jar, no session, no login. That is deliberate, and it keeps the crawler on the public side of the line drawn earlier.
Scrape IndiaMART supplier data from a category page
Every IndiaMART tutorial you will find hardcodes a CSS class for the listing card, and every one of them breaks, because directory markup churns. Anchor on structure instead. A listing card is, definitionally, the smallest element containing both a link to /proddetail/ and a price or an "Ask Price" label. Find the product anchors, then walk up until the container satisfies that definition:
import re
from selectolax.parser import HTMLParser
RUPEE_RE = re.compile(r"(?:₹|Rs\.?|INR)\s*[\d,]", re.I)
ASK_RE = re.compile(r"ask\s*price", re.I)
def card_of(anchor, max_levels: int = 5):
"""Walk up from a /proddetail/ link to the smallest container that looks like a card."""
node = anchor
for _ in range(max_levels):
node = node.parent
if node is None:
return None
text = node.text()
if RUPEE_RE.search(text) or ASK_RE.search(text):
return node
return None
def listing_cards(html: str):
tree = HTMLParser(html)
seen, cards = set(), []
for a in tree.css("a[href*='/proddetail/']"):
href = (a.attributes.get("href") or "").split("?")[0]
if not href or href in seen:
continue
card = card_of(a)
if card is not None:
seen.add(href)
cards.append({"listing_url": href,
"title": a.text(strip=True),
"text": card.text(separator=" ", strip=True)})
return cards
This survives a redesign because it depends on IndiaMART's URL scheme, which is stable, rather than on class names, which are not. Install the parser with pip install selectolax; it is C-backed and chews through a 400-card page far faster than the pure-Python default.
Before trusting any of it, check that you got a page rather than a wall. Directories often answer a rate limit with HTTP 200 and an interstitial, so the status code lies:
def looks_blocked(html: str) -> bool:
markers = ("captcha", "unusual traffic", "access denied",
"request could not be processed", "are you a human")
low = html.lower()
return any(m in low for m in markers) or len(html) < 3000
The length floor catches what marker matching misses: a rendered shell that returned successfully and contains no cards at all.
Verify the extractor before you trust a run
Structural extraction removes the class-name fragility but moves the failure mode. Instead of breaking loudly when a class is renamed, it quietly returns fewer cards when the nesting depth changes. Measure the yield on every new category and store it beside the run.
def extraction_report(html: str) -> dict:
"""Yield near 1.0 is healthy. Under ~0.85, IndiaMART changed the card
nesting: raise max_levels and re-measure. Anchors > 0 with cards == 0
means the price line left the card subtree, which is a parser fix and
not a block. Anchors == 0 means you were served a shell."""
tree = HTMLParser(html)
anchors = {(a.attributes.get("href") or "").split("?")[0]
for a in tree.css("a[href*='/proddetail/']")
if a.attributes.get("href")}
cards = listing_cards(html)
return {"product_anchors": len(anchors),
"cards_built": len(cards),
"card_yield": round(len(cards) / len(anchors), 3) if anchors else 0.0}
Computing both numbers tells you which of those four cases you are in. Without them a weekly job degrades quietly for a month before anyone notices the category thinned.
Parse Indian price formats: lakh, crore, units, MOQ
This is where most IndiaMART datasets quietly go wrong. Indian price strings use two conventions that Western parsers mangle.
The first is Indian digit grouping. 1,50,000 is one hundred fifty thousand, grouped as 1,50,000 rather than 150,000. Stripping commas handles it fine, but a locale-aware parser configured for en_US will either error or, worse, silently truncate.
The second is the killer. IndiaMART sellers routinely quote in words: ₹ 1.5 Lakh / Unit, ₹ 2.4 Crore / Piece. Strip the commas, cast to float, and you have recorded a ₹150,000 machine as ₹1.50. That error does not look like an error. It looks like a suspiciously cheap supplier, and it will sit in your median for months. One lakh is 100,000. One crore is 10,000,000.
import re
MULTIPLIERS = {
"lakh": 100_000, "lakhs": 100_000, "lac": 100_000, "lacs": 100_000,
"crore": 10_000_000, "crores": 10_000_000, "cr": 10_000_000,
"k": 1_000, "thousand": 1_000,
}
PRICE_RE = re.compile(
r"(?:₹|Rs\.?|INR)\s*"
r"(?P<amount>\d[\d,]*(?:\.\d+)?)\s*"
r"(?P<scale>lakhs?|lacs?|crores?|cr|k|thousand)?\s*"
r"(?:/|per\s+)?\s*"
r"(?P<unit>[A-Za-z][A-Za-z .]{0,24})?",
re.I,
)
def parse_price(text: str) -> dict | None:
m = PRICE_RE.search(text or "")
if not m:
return None
amount = float(m.group("amount").replace(",", ""))
scale = (m.group("scale") or "").lower().rstrip("s")
if scale:
amount *= MULTIPLIERS.get(scale, MULTIPLIERS.get(scale + "s", 1))
unit = (m.group("unit") or "").strip(" .").lower() or None
return {"inr": round(amount, 2), "unit": unit, "raw": m.group(0).strip()}
Always keep the raw string. When a number looks wrong six weeks later, that raw field is the difference between a two-minute check and a full re-crawl.
Worked example: what the scale words do to a median
Six listings, a fixture rather than a market sample, put through both parses:
| Raw price string | Naive parse (strip commas, cast) | Correct parse |
|---|---|---|
| ₹ 450 / Piece | 450 | 450 |
| ₹ 1,250 / Piece | 1,250 | 1,250 |
| ₹ 1.5 Lakh / Unit | **1.5** | 150,000 |
| ₹ 85,000 / Piece | 85,000 | 85,000 |
| ₹ 2.4 Crore / Piece | **2.4** | 24,000,000 |
| ₹ 12,00,000 / Unit | 1,200,000 | 1,200,000 |
The naive median is ₹850. The correct median is ₹117,500, a factor of 138 apart on six rows. The damage comes from the two rows a human skimming the CSV would read straight past, because ₹1.50 and ₹2.40 are plausible prices for a washer. Nothing throws and nothing logs. The category simply reads cheap. Lock the behaviour in a test so a later regex tweak cannot quietly undo it:
FIXTURE = ["₹ 450 / Piece", "₹ 1,250 / Piece", "₹ 1.5 Lakh / Unit",
"₹ 85,000 / Piece", "₹ 2.4 Crore / Piece", "₹ 12,00,000 / Unit"]
assert [parse_price(s)["inr"] for s in FIXTURE] == [
450.0, 1250.0, 150_000.0, 85_000.0, 24_000_000.0, 1_200_000.0]
The last row is Indian digit grouping on its own: comma-stripping handles 12,00,000 fine. Only the spelled-out scale words break things.
Now the units. A price is not a number, it is a number attached to a dimension, and IndiaMART quotes across at least five of them. ₹450 per Piece and ₹52,000 per Metric Ton are not comparable and must never land in the same average. Normalise to a base unit per dimension, and refuse to compare across dimensions:
# unit -> (dimension, factor to base unit)
UNITS = {
"piece": ("count", 1), "pieces": ("count", 1), "pcs": ("count", 1),
"unit": ("count", 1), "number": ("count", 1), "nos": ("count", 1),
"set": ("count", 1), "pack": ("count", 1),
"dozen": ("count", 12), "gross": ("count", 144),
"kilogram": ("mass_kg", 1), "kg": ("mass_kg", 1),
"gram": ("mass_kg", 0.001), "g": ("mass_kg", 0.001),
"quintal": ("mass_kg", 100),
"ton": ("mass_kg", 1000), "tonne": ("mass_kg", 1000),
"metric ton": ("mass_kg", 1000), "mt": ("mass_kg", 1000),
"meter": ("length_m", 1), "metre": ("length_m", 1), "mtr": ("length_m", 1),
"foot": ("length_m", 0.3048), "feet": ("length_m", 0.3048), "ft": ("length_m", 0.3048),
"square feet": ("area_sqm", 0.092903), "sq ft": ("area_sqm", 0.092903),
"square meter": ("area_sqm", 1), "sq mtr": ("area_sqm", 1),
"litre": ("volume_l", 1), "liter": ("volume_l", 1), "ltr": ("volume_l", 1),
}
def normalise(price: dict) -> dict | None:
"""₹52,000 / Metric Ton becomes 52.0 INR per kg, dimension mass_kg."""
if not price or not price["unit"]:
return None
entry = UNITS.get(price["unit"])
if not entry:
return None # unknown unit: keep the raw row, exclude from stats
dimension, factor = entry
return {"dimension": dimension, "inr_per_base": price["inr"] / factor}
MOQ deserves the same treatment and its own column, because it changes what a price means. ₹12 per piece at an MOQ of 10,000 pieces is a different market position than ₹18 per piece at an MOQ of 50, and a benchmark that ignores MOQ blends wholesale and retail into one meaningless band.
MOQ_RE = re.compile(r"minimum\s*order\s*quantity\s*:?\s*(\d[\d,]*)\s*([A-Za-z ]+)?", re.I)
def parse_moq(text: str) -> dict | None:
m = MOQ_RE.search(text or "")
if not m:
return None
return {"moq_qty": int(m.group(1).replace(",", "")),
"moq_unit": (m.group(2) or "").strip().lower() or None}
Handle "Ask Price" without poisoning your benchmarks
A large share of IndiaMART listings show "Ask Price" instead of a number. Sellers of engineered, made-to-order, or high-value goods deliberately withhold price to force an inquiry. That is a product decision on their side and a statistics problem on yours.
Here is the trap, and it is the most common error in published IndiaMART analyses. Drop the blank-price rows, average what remains, and you have not measured the category. You have measured the subset of suppliers willing to publish a number, which skews commodity, standardised and cheap. Your "average price of industrial valves" is really the average price of its catalogue end, and it reads low every time.
Treat price availability as a measurement in its own right:
from statistics import median
def category_stats(rows: list[dict]) -> dict:
total = len(rows)
priced = [r for r in rows if r.get("inr_per_base") is not None]
coverage = len(priced) / total if total else 0.0
values = sorted(r["inr_per_base"] for r in priced)
return {
"listings": total,
"priced_listings": len(priced),
"price_coverage": round(coverage, 3), # report this beside every price
"median_inr_per_base": round(median(values), 2) if values else None,
"p10": values[int(len(values) * 0.10)] if len(values) >= 10 else None,
"p90": values[int(len(values) * 0.90)] if len(values) >= 10 else None,
"confidence": "low" if coverage < 0.4 else "medium" if coverage < 0.7 else "good",
}
Three rules follow. Publish price_coverage next to every price figure, so nobody downstream reads a 22% sample as a market rate. Use the median and the p10 to p90 band rather than the mean, because a single mis-parsed crore listing drags a mean into fiction while barely moving a median. And track coverage over time: a category where published prices are becoming more common is usually a category commoditising, which is a finding on its own. Once these tables exist, the habits in how to clean scraped data apply directly.
Map supplier geography with city facets and GSTIN
Supplier distribution is the highest-value output here, and it needs no personal data at all. Sweep one category across a city list and record counts:
CITIES = ["mumbai", "delhi", "ahmedabad", "rajkot", "ludhiana", "coimbatore",
"chennai", "pune", "surat", "bengaluru", "jaipur", "kolkata"]
def city_url(city: str, term: str) -> str:
return f"https://dir.indiamart.com/{city}/{term}.html"
def category_by_city(term: str) -> list[dict]:
out = []
for city in CITIES:
html = fetch(city_url(city, term))
if looks_blocked(html):
continue
out.append({"city": city, "term": term,
"listings_sampled": len(listing_cards(html))})
return out
Page-one counts are a sample, not a census, so label the column that way and keep the sampling depth identical across cities or the comparison is worthless.
SAMPLE_DEPTH = 1 # pages fetched per city; identical everywhere, stored as a column
def density_index(rows: list[dict]) -> list[dict]:
"""Share of the sampled supply per city. Comparable only at a fixed depth."""
total = sum(r["listings_sampled"] for r in rows) or 1
return sorted(({**r, "sample_depth": SAMPLE_DEPTH,
"share": round(r["listings_sampled"] / total, 4)}
for r in rows), key=lambda r: -r["share"])
Report the share, not the count. A share taken at a fixed depth stays comparable across categories and across months, while a raw count moves whenever the site changes its page size. If a question needs an absolute supplier population, page one cannot answer it.
The stronger geographic signal is the GSTIN, published on many storefronts under company details. A GSTIN is 15 characters with real structure: characters 1 and 2 are the state code, 3 to 12 are the entity's PAN, 13 is the registration number within that state, 14 is normally Z, and 15 is a checksum. Because GST registration is state-wise, those leading two digits give you a verified state that does not depend on whatever city the seller typed into their listing.
CODES = "0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ"
GSTIN_RE = re.compile(r"\b\d{2}[A-Z]{5}\d{4}[A-Z][A-Z\d][A-Z\d][A-Z\d]\b")
STATE = {"07": "Delhi", "24": "Gujarat", "27": "Maharashtra", "29": "Karnataka",
"33": "Tamil Nadu", "09": "Uttar Pradesh", "06": "Haryana", "08": "Rajasthan",
"19": "West Bengal", "23": "Madhya Pradesh", "03": "Punjab", "32": "Kerala",
"36": "Telangana"}
def gstin_valid(gstin: str) -> bool:
"""Mod-36 checksum: rejects typos and OCR noise before they enter the dataset."""
if len(gstin) != 15:
return False
total = 0
for i, ch in enumerate(gstin[:14]):
if ch not in CODES:
return False
product = CODES.index(ch) * (2 if i % 2 else 1)
total += product // 36 + product % 36
return CODES[(36 - total % 36) % 36] == gstin[14]
def gstin_from(html: str) -> dict | None:
for candidate in GSTIN_RE.findall(html):
if gstin_valid(candidate):
return {"gstin": candidate,
"state_code": candidate[:2],
"state": STATE.get(candidate[:2]),
"pan": candidate[2:12]}
return None
assert gstin_valid("24ABCDE1234F1Z6") # synthetic, checksum-correct
assert not gstin_valid("24ABCDE1234F1Z7") # one character off, rejected
Running the checksum is worth the ten lines: it rejects mistyped and truncated numbers before they reach a warehouse where the GSTIN is doing work as a join key. It proves the string is well formed and nothing more, so when a registration carries weight in a report, confirm it against the official taxpayer search on the GST portal at services.gst.gov.in.
One caution belongs with this code. A GSTIN embeds a PAN, and for a sole proprietorship that PAN belongs to a named individual, not to a separate legal entity. A business identifier is not the same risk as a mobile number, but a proprietor's PAN is close enough to personal data that you should store the GSTIN, use the state code, and leave the PAN segment alone.
Paginate and scale without hammering the site
IndiaMART category pages load an initial batch of cards and fetch more as you scroll. Two approaches work, and one is more durable than the other.
Scroll the rendered page. Ask the API to render and scroll before returning HTML. This survives changes to the internal endpoint because it does what a browser does:
import json
scroll = {"instructions": [
{"scroll_y": 2500}, {"wait": 1200},
{"scroll_y": 5000}, {"wait": 1200},
{"scroll_y": 7500}, {"wait": 1500},
]}
html = fetch("https://dir.indiamart.com/impcat/industrial-valves.html",
js_scenario=json.dumps(scroll))
Call the underlying XHR. Open the category page in Chrome DevTools, filter the Network tab to XHR, scroll once, and read the request that returns the next batch. It is faster and lighter than rendering a browser. It is also undocumented and can change without notice, so treat it as an optimisation you re-verify, never the foundation of a pipeline that runs unattended for a year. If you do use it, record the verification date in a comment and assert on the response shape, so a change fails the job instead of silently emptying a category.
Whichever route you take, category listings run deep and your patience should be finite. Consistent sampling at the fixed depth from the previous section beats exhaustive crawling for every comparative question, and it is far kinder to the site.
Concurrency and retries, with the API rotating exits for you:
import time, random
from concurrent.futures import ThreadPoolExecutor, as_completed
def fetch_with_retry(url: str, attempts: int = 3, **extra) -> str | None:
for i in range(attempts):
try:
html = fetch(url, **extra)
if not looks_blocked(html):
return html
except requests.RequestException:
pass
time.sleep(2 ** i + random.random()) # exponential backoff with jitter
return None
def crawl(urls: list[str], workers: int = 6) -> dict[str, str]:
results = {}
with ThreadPoolExecutor(max_workers=workers) as pool:
futures = {pool.submit(fetch_with_retry, u): u for u in urls}
for fut in as_completed(futures):
html = fut.result()
if html:
results[futures[fut]] = html
return results
Six workers is plenty for a directory. Push harder and you will spend the afternoon tuning backoff instead of analysing suppliers.
Dedupe and model the dataset
Two modelling decisions decide whether the dataset is usable.
Never key on company name. The same firm appears as "Shree Industries", "Shree Industries Pvt. Ltd.", "M/s Shree Industries", and "Shree Industries Private Limited", frequently across several categories at once. Key on the storefront path, which is stable and unique, and use the GSTIN when published to merge storefronts belonging to one entity:
from urllib.parse import urlparse
SUFFIXES = re.compile(
r"\b(m/s|pvt\.?|private|ltd\.?|limited|llp|inc\.?|co\.?|company|and|&)\b", re.I)
def supplier_key(storefront_url: str) -> str:
return urlparse(storefront_url).path.strip("/").lower()
def name_fingerprint(name: str) -> str:
"""For fuzzy grouping only, never as a primary key."""
cleaned = SUFFIXES.sub(" ", (name or "").lower())
return re.sub(r"[^a-z0-9]+", "", cleaned)
Make the contact columns structurally absent. This is the part that turns the earlier legal section into engineering. If the table has no phone column, a careless downstream join cannot leak one, a new analyst cannot quietly start populating it, and an audit takes thirty seconds:
CREATE TABLE supplier (
supplier_key TEXT PRIMARY KEY, -- storefront path, stable
company_name TEXT NOT NULL,
name_fp TEXT, -- fingerprint, grouping only
city TEXT, -- listed location, may be a sales office
gstin TEXT, -- checksum-validated, nullable
gst_state TEXT, -- derived from GSTIN characters 1-2
trustseal INTEGER DEFAULT 0,
member_since INTEGER,
first_seen TEXT NOT NULL,
last_seen TEXT NOT NULL
-- deliberately no phone, email, or contact_person column
);
CREATE TABLE listing (
listing_id TEXT PRIMARY KEY, -- proddetail path
supplier_key TEXT REFERENCES supplier(supplier_key),
title TEXT NOT NULL,
category_term TEXT NOT NULL,
price_raw TEXT, -- the original string, always
price_inr REAL, -- parsed, lakh and crore expanded
price_unit TEXT,
dimension TEXT, -- count | mass_kg | length_m | area_sqm | volume_l
inr_per_base REAL,
ask_price INTEGER DEFAULT 0, -- 1 when no number was published
moq_qty INTEGER,
moq_unit TEXT,
city TEXT,
scraped_at TEXT NOT NULL
);
Add first_seen and last_seen from day one. Directory listings go stale silently, and a supplier who has not reappeared in ninety days is worth knowing about before you quote their price to a client. How to store scraped data covers where all this lands, and the approach generalises to other business directories, including Yellow Pages.
Frequently asked questions
FAQ
Collecting public commercial data (product titles, prices, MOQ, categories, company names, cities) is normal market research, but IndiaMART's terms of use restrict automated collection, so a project can breach contract even where no data protection law is triggered. Personal data is the sharper risk: India's DPDP Act, 2023 applies to processing outside India when it relates to offering goods or services to people in India, so a foreign team sits inside its scope. Get counsel before any commercial deployment.
Technically you would usually have to log in to reveal them, and that is exactly why you should not. Authenticating turns public collection into logged-in access under the platform's terms, and the resulting contact list carries DPDP Act exposure in India, GDPR Article 14 notification duties for any EU contacts, and TRAI's unsolicited-communication rules for the outreach itself. Build the market-intelligence dataset with no contact columns and source outreach data through a consented channel.
Sellers of made-to-order, engineered, or high-value goods withhold price to force an inquiry, so "Ask Price" is a business choice rather than missing data. Dropping those rows and averaging the rest biases your benchmark toward cheap commodity listings, so record price availability as its own field and publish the coverage percentage next to every price figure.
Match the numeric amount and any scale word separately, strip the commas from Indian digit grouping (1,50,000 becomes 150000), then multiply by 100,000 for lakh or 10,000,000 for crore. Skipping the scale word records a ₹1.5 lakh machine as ₹1.50, which looks like a cheap supplier rather than a parsing bug and can survive months in a dataset undetected.
In practice, yes. IndiaMART serves the Indian market, prices are quoted in INR, and some foreign traffic gets a degraded or differently routed experience, so country_code=in with premium_proxy=true on the SparkProxy Scraping API gives you an Indian residential exit and the view a domestic buyer would see. Residential exits also outlast a datacenter range against the site's per-IP rate limiting.
Key each supplier on their storefront path, then store the company name, listed city, verification badges, member-since year, and a checksum-validated GSTIN, deriving the state from the GSTIN's first two characters. That gives you supplier counts, geographic clusters, and verification-tier distribution per category with no phone numbers, no emails, and no named individuals anywhere in the schema.
Get 50% off your first month
Premium datacentre proxies with unlimited bandwidth. Use the code at checkout.
Offer ends soon — claim it before it's gone
Related articles

How to Scrape Baidu Search Results Accurately
How to scrape Baidu search results: the pn parameter, GBK encoding traps, resolving baidu.com/link redirects, and parsing Baijiahao and Zhidao blocks.

How to Scrape Alibaba Product Data
Scrape Alibaba product data at B2B depth: parse ladder pricing, normalize MOQ units, read supplier trust badges, and pivot rows from products to suppliers.

How to Detect When Your Scraper Is Blocked
Detect when your scraper is blocked, including silent HTTP 200 soft blocks: baselines, selector contracts, canary URLs, and per-IP success rates.
