How to Collect Product and Pricing Data From Retail Sites

There are three things you can buy for product and pricing data, and they are not variations of each other. They put the work in different places and bill on different meters.

Most teams that end up unhappy bought on the wrong axis. They compared a per-GB residential price against a per-request API price without normalising to cost per usable row, the only figure that survives retries, blocked responses, and pages that return HTTP 200 with an empty price field. We laid that framing out in web scraping API vs self-managed proxies: buy the API when target difficulty is uneven, buy raw proxies when volume is high and targets are boring.

A useful filter. If you can name the engineer who will fix the parser at 2am when a retailer ships a redesign, raw proxies belong on your shortlist. If you cannot name that person, an API is cheaper even when its sticker price is higher.

Every proxy sales page asks how many IPs you want. That is the wrong question for retail price scraping. The number that maps to a plan is peak concurrency, and you get it with arithmetic rather than guesswork.

requests_per_day = SKUs x refreshes_per_day x pages_per_SKU x (1 + retry_rate)
peak_rps         = requests_per_day / collection_window_seconds
concurrency      = peak_rps x average_round_trip_seconds

Worked through with plausible inputs for a mid-size catalogue:

Those figures are arithmetic from stated assumptions, not measurements from a test. The shape of the answer is the point: a catalogue of 120,000 SKUs refreshed twice a day needs roughly 34 concurrent connections, not 34,000 IPs. That is why thread-based plans and IP-count plans are so hard to compare directly, and why a buyer who anchors on pool size overbuys. If the threads-versus-IPs distinction is new to your team, understanding concurrent connections in proxies covers it properly.

Two adjustments before you shop. Widen the collection window if the data does not have to be simultaneous, because a 4-hour window costs roughly three times the concurrency of a 12-hour one. And measure round trip against your real targets, since a JavaScript-rendered page can take 4 to 8 seconds where a plain HTML fetch finishes in under one.

Sort your target list into three tiers before you choose an IP type. This one step prevents the most expensive mistake in the category, which is buying residential bandwidth for a catalogue that is 80 percent tier 1.

Tiering is not a ranking of retailers. It describes what their edge does to automated traffic. A tier 3 target is not better protected because it is bigger, it runs a vendor product that scores datacenter ASNs down by default, the kind of stack described in how to bypass DataDome. Plan for those separately instead of degrading the whole pipeline to their level. Running the checks from how to detect when your scraper is blocked against a sample of each retailer assigns tiers fast, and it costs nothing before you sign.

One shortcut applies across all three tiers. Many retail sites populate product pages from a JSON endpoint the browser calls after load. Reading that endpoint directly returns clean fields instead of markup and cuts transfer size to a fraction of the rendered page, which lowers both parse fragility and bandwidth. Look for it first every time, using the method in how to scrape hidden JSON API endpoints.

Match the plan to your tier mix, then to the concurrency number you calculated.

SparkProxy's published datacenter plans all carry unlimited bandwidth on 30-day terms and differ on threads and whitelist slots:

Two larger tiers, Pro at 1500 threads and Pro+ at 2000 threads, appear in the Fair Usage Policy at 200 and 250 Mbps but carry no public price, and custom arrangements go up to 1 Gbps. The network covers more than 1 million datacenter IPs across 80 or more countries, including over 50,000 US datacenter IPs, which matters when you need many exits inside one market rather than one exit in many.

The 34-thread example above fits Starter with room to grow. That is the usual outcome for a first retail pipeline, and it is worth knowing before a salesperson sizes you at 1,000 threads. For the reasoning behind how these plans are structured, understanding datacenter proxy pricing models compares thread pricing against per-GB pricing directly.

This is the part buyers underestimate. Retail product pages are among the heaviest documents on the public web once images, scripts and personalisation payloads are counted. When your proxy bills per GB, page weight is your invoice.

Run the same 268,800 requests per day through both meters:

Residential per-GB list rates vary widely by vendor and commitment, and vendors revise them, so price your own volume against each vendor's current published rate rather than a number lifted from a blog post. The structural point holds regardless of the rate: 4.6 TB a month on any per-GB meter is a large recurring bill for data you can often collect from datacenter IPs at a fixed price.

The honest counterweight is that unlimited bandwidth is not unlimited throughput. A speed cap is a ceiling, not a promised rate, so sanity-check it against your byte volume. At 25 Mbps sustained the arithmetic ceiling is about 3.1 MB per second, near 270 GB per day. The 154 GB per day above fits under Starter's ceiling on paper with no headroom for bursts, so a buyer at that volume should look at Core rather than assume the cap never binds. Stripping images out of your fetches, or hitting a JSON route instead of the rendered page, moves that number further than any plan upgrade will.

For a fuller treatment of when datacenter economics stop working, see residential vs datacenter proxies and the build described in datacenter proxies for price comparison websites.

Scraping API pricing is per credit, and credits are consumed by request type: a plain fetch costs 1 credit, a JavaScript render costs 5, a screenshot or PDF costs 10. That makes the comparison arithmetic rather than opinion. SparkProxy's published plans are Starter at $49 for 250,000 credits and 50 concurrent requests, Growth at $99 for 1,000,000 credits and 100 concurrent, Pro at $249 for 3,000,000 credits and 200 concurrent, and Scale at $599 for 8,000,000 credits and 400 concurrent. There are 1,000 free credits with no card, which is enough to tier your target list before you pay anyone anything.

The fourth row is the one to stare at. Turning rendering on for every request multiplies credit burn by five and pushes a mid-size catalogue past the largest published plan. Rendering is a per-target decision, never a global default. In a typical retail list only a minority of targets genuinely need a browser, and routing just those through rendering keeps your bill in the second row rather than the fourth.

Now compare the raw-proxy path at the same 3,000,000 monthly requests. Core at $140 carries that concurrency and bandwidth is not metered, so the proxy line is cheaper on paper. What it excludes is engineering time on retries, header and TLS realism, challenge handling and parser repair. Price that time honestly. A few hours a week of maintenance closes the gap between $140 and $249 fast, and the comparison flips the moment a tier 3 retailer joins your list.

The base endpoint is https://scrape.sparkproxy.io/api/v1 and every request authenticates with the X-API-Key header. Target URLs below use sparkproxy.io demo paths, so swap in the retailer URLs you track. Full parameters live in the Scraping API documentation.

Start with the cheapest request available, a plain fetch with structured extraction:

import requests, json

API = "https://scrape.sparkproxy.io/api/v1"
API_KEY = "sk-xxxxxxxxxxxxxxxx"

def fetch_product(url, country="US"):
    params = {
        "url": url,
        "country_code": country,
        "format": "json",
        "extract_rules": json.dumps({
            "price":      ".pdp .price-current",
            "list_price": ".pdp .price-was",
            "stock":      ".pdp .availability",
            "sku":        ".pdp .sku-code",
        }),
        "tag": "retail/plain",
    }
    r = requests.get(API, headers={"X-API-Key": API_KEY}, params=params, timeout=60)
    return r.status_code, r.json()

print(fetch_product("https://www.sparkproxy.io/demo-retail/pdp/sku-44821"))

Escalate to rendering only when the plain fetch returns without a price. Keep the escalation explicit so credit burn stays visible in code review rather than showing up on an invoice:

RENDER_DOMAINS = {"demo-market.sparkproxy.io"}   # the short list that truly needs a browser

def fetch_with_policy(url, country="US"):
    host = url.split("/")[2]
    params = {"url": url, "country_code": country, "format": "json", "tag": "retail/policy"}
    if host in RENDER_DOMAINS:
        params["render_js"] = True          # 5 credits instead of 1
        params["wait_for"] = ".pdp .price-current"
    return requests.get(API, headers={"X-API-Key": API_KEY}, params=params, timeout=90).json()

For retailers that price by store or region, pin the exit country and hold a session so consecutive requests keep the same store context:

params = {
    "url": "https://www.sparkproxy.io/demo-retail/pdp/sku-44821",
    "country_code": "DE",           # price and tax as the German market sees them
    "premium_proxy": True,          # residential exit where the target demands one
    "session_id": "de-store-1044",  # hold the store selection across requests
    "format": "json",
}
r = requests.get(API, headers={"X-API-Key": API_KEY}, params=params, timeout=90)

If you run your own scrapers instead, point them at the proxy gateway. HTTP and HTTPS use gateway.sparkproxy.io on port 11000, sticky sessions use port 11002, and SOCKS5 uses port 13000:

curl -x http://user-xxxxxxxx:pass@gateway.sparkproxy.io:11000 \
  "https://www.sparkproxy.io/demo-retail/pdp/sku-44821"

curl -x socks5h://user-xxxxxxxx:pass@gateway.sparkproxy.io:13000 \
  "https://www.sparkproxy.io/demo-retail/pdp/sku-44821"

Sticky sessions on port 11002 are the right choice for multi-step flows such as selecting a store and then reading its price, because the price you read has to belong to the store you selected. Rotating exits on 11000 are correct for wide catalogue sweeps where every request stands alone.

A price scrape can return HTTP 200 on every request and still produce a table nobody should act on. These are the fields that cause it.

That last row is the one teams skip and later wish they had not. Storing collection context with every observation turns a disputed number into a five-minute audit, and it is what lets you show a price was live at a moment instead of arguing about it. The cleanup patterns in how to clean scraped data apply directly, and if your use case is buy-side sourcing rather than monitoring, the margin logic in proxies for retail arbitrage picks up where this table ends.

Never buy an annual plan on a vendor's demo URL. Run your own test, on your own targets, in this order:

1. Pull 200 real product URLs from your live target list, weighted the way your catalogue is weighted, and include your three hardest retailers.
2. Run those same 200 URLs through each candidate using the exact IP type and plan tier you intend to buy, not a sales-engineer sandbox.
3. Score success by field completeness, not by HTTP status. A 200 with a null price is a failure, and vendors who report on status codes will look better than they are.
4. Repeat the run at three different times of day. Bot management posture shifts with traffic load, and a clean 3am result tells you very little about noon.
5. Divide total cost by usable rows. Cost per usable row is the number you compare across vendors, and it is often two or three times the headline rate.
6. Get the trial and refund terms in writing before you start, so a failed test does not turn into a sunk twelve-month commitment.

Two extra checks earn their time. Confirm geo accuracy on the exits you will actually use, because a proxy labelled for one country that egresses somewhere else hands you the wrong prices without ever erroring, a point covered in what geo-targeting means in proxies. Then test concurrency at your peak rather than your average, since plans are sold on a ceiling you only meet in the busiest hour.

Collecting public product and pricing data is ordinary commercial activity, and it is also an area where details decide your exposure. Keep the scope to publicly accessible pages, do not create accounts or bypass authentication, and pace requests so collection does not degrade the target. The practical framework sits in ethical scraping and rate limiting, and the wider programme view is in how ecommerce companies use proxies for competitive intelligence.

On the procurement side, get three things in writing before you sign: how the provider sources its IP supply, what the acceptable use policy forbids, and what happens to your service if a target complains. Ask about KYC and use-case review too. A provider that asks who you are and what you collect is applying the same scrutiny to the accounts sharing your subnets, which is a feature rather than friction. None of this is legal advice, and a regulated or cross-border programme deserves review by counsel in your jurisdiction.