Proxies for Gaming Item Price Data and Leaderboards

Four distinct datasets get lumped together under "game data," and they behave nothing alike.

Storefront prices. What a title costs in each region, in local currency, including sale state and edition. Valve, Epic, Xbox, PlayStation, and Nintendo all price regionally.

In-game item and currency prices. The World of Warcraft auction house, the EVE Online regional markets, the Old School RuneScape Grand Exchange, Path of Exile currency ratios, Counter-Strike skins. These are real economies with liquidity, spreads, and thin tails.

Third-party marketplace prices. Sites that sit outside the publisher and list items with attributes the publisher's own API does not expose, including wear float and pattern index for CS2 skins.

Competitive and leaderboard data. Ranked ladders, seasonal standings, Mythic+ scores, tournament results, and the player counts that correlate with all of the above.

The first two usually have official, documented APIs. The last two usually do not, or expose them only through HTML and internal JSON. That split determines your entire architecture, and it determines whether a proxy earns its cost.

Skipping the documented API to scrape HTML is the most common mistake in this space. It is slower, more brittle, and more likely to violate terms than the endpoint the publisher built for you.

Read that table twice, because the last column is the whole story.

The auction house APIs are better than people expect

Blizzard's connected-realm auctions endpoint returns the full snapshot of live auctions for a realm, and the region-wide commodities endpoint covers stackable goods like herbs and ore. Both refresh roughly hourly and carry a Last-Modified header. Polling them every five minutes does not get you fresher data, it just burns 12x the quota and writes eleven duplicate rows per hour that look like real observations later.

EVE's ESI is the most complete public game economy API in existence: live orders and historical daily aggregates per region and per type, no key required. Its throttle is unusual. ESI does not cap successful requests, it caps errors, using a budget exposed in X-ESI-Error-Limit-Remain and X-ESI-Error-Limit-Reset. Blow through it and you are blocked for the rest of the window regardless of how polite your success rate was.

Here is the honest part that most vendor content skips.

A proxy does not raise a rate limit counted against your API key. If Riot allows your production key 500 requests per 10 seconds, that number is identical from one IP or from four hundred. Worse, sending one credential from many unrelated ASNs looks exactly like a leaked key being resold, which is the pattern abuse detection is tuned for. The realistic outcome is a revoked key, not more data. The same holds for Blizzard's client ID, Bungie's key, and Steam's Web API key. Want headroom? Apply for a production key tier, cache aggressively, and cut redundant calls.

A proxy does raise an IP-keyed ceiling, and gaming data has plenty of those:

• Steam's store and market JSON endpoints require no key and throttle by source IP.
• Path of Exile's trade API returns its budget in X-Rate-Limit-Ip and X-Rate-Limit-Account headers, two separate counters that you can only relieve one of by changing IP.
• Third-party skin and item marketplaces almost universally throttle by IP, often behind a WAF.
• Publisher web pages that render leaderboards, and the internal JSON behind them, are IP-limited.
• Regional storefront pricing is gated by the geography of the requesting IP.

Build your collector around that split. Route key-authenticated traffic from a small set of stable, well-behaved IPs, and reserve the proxy pool for the anonymous surfaces. Mixing them is how teams get an API key banned while trying to solve an unrelated problem. If you need the mechanics of per-IP budgeting, we covered them in rotating proxies and per-IP request limits.

Storefronts price by region, and the region is resolved from the requesting IP more often than from a URL parameter. Steam accepts a cc country parameter on appdetails, but it is not authoritative alone: ask for Turkish pricing from a German IP and you will often get a normalized or refused response. Console storefronts are stricter and route you to a country-specific catalog before showing a price. That makes regional collection a genuine geo-targeting problem. To see Argentina's price you need an Argentine exit IP. No header substitutes for it.

Three details cause most of the bad data here:

Currency is not a formatting concern. Store price_minor as an integer in the smallest unit plus an ISO 4217 currency code. Never store a converted USD figure as the primary value, because the conversion rate you used in March silently rewrites your March data when you re-run the report in June.

Regional prices move on their own schedule. Publishers adjust regional pricing tiers in bulk, sometimes with weeks of drift between markets. A price change in one region is not evidence of a global change.

Sale state is a separate field. initial, final, and discount_percent need three columns. Collapsing them into one "price" destroys your ability to compute realized discount depth later. For the mechanics of pulling Steam's regional pricing specifically, see how to scrape Steam game data.

This is where proxies do the most work. Third-party item marketplaces publish listings with per-instance attributes the publisher's API omits, and they defend those pages hard because scraped inventory is commercially valuable to a competitor. Expect strict per-IP budgets, a WAF, listings rendered client-side from an internal JSON call, and cursor pagination that expires. That internal JSON call is usually the right target rather than the rendered DOM, using the technique in scraping hidden JSON API endpoints.

One thing worth stating plainly: Valve's own Market priceoverview endpoint returns three fields, lowest_price, median_price, and volume, and most pipelines store lowest_price as "the price." That is an ask, not a trade. It is the cheapest thing anyone is currently willing to sell for, which can sit far above or below where the item actually changes hands, and on a thin item it can be a single optimistic seller. median_price is derived from completed sales in the trailing 24 hours and volume tells you how many of those there were. If you record only lowest_price you have built a listings dataset while calling it a price dataset.

Leaderboards look like the easy part and are not, because the dataset moves while you read it.

Walk a ladder of 400,000 ranked players at 100 rows per page and that is 4,000 requests. From one IP with a polite delay, the run takes long enough that players near the top of the list have already gained or lost rating by the time you reach the bottom. Page 39 and page 40 will show you the same account twice, or drop one entirely, because rank is not a stable key.

Three rules fix it:

Key on the account identifier, never the rank. Rank is an attribute of the observation. The primary key is (snapshot_id, account_id), and rank is just another column.

Stamp one snapshot_id per run. Every row from a single ladder walk carries the same snapshot identifier and the same nominal snapshot time, even though the individual fetches span minutes. That is what makes two runs comparable.

Shorten the wall-clock window. This is the real argument for a proxy pool on leaderboards. If a single IP allows 1 request per second, 4,000 pages take over an hour and your snapshot is smeared across it. Spread the same 4,000 requests across a pool at the same per-IP rate and the window collapses to minutes. You are not exceeding any single IP's budget, you are shrinking the time distortion inside one snapshot.

Then store the smear. Record fetch_started_at and fetch_completed_at on the snapshot row, so when someone asks why a player appears twice the answer sits in that interval. Some ladders are served per shard or per region and are only reachable from that region, so the pool has to cover the geographies you track. Viewership context for the same titles usually comes from streaming platforms, covered in how to scrape Twitch data.

Most gaming economy datasets are broken at the schema level before a single row is collected, because they assume an item name maps to a price. In CS2 and similar markets it does not, and the reason is instance-level attributes. A skin's float value is a number between 0 and 1 describing wear, mapped to a named exterior bucket:

Within a bucket, price is not flat. The lowest floats in Factory New carry a premium, and so do specific values near a bucket boundary. Separately, the paint seed (pattern index) determines the visual pattern, and for certain finishes a rare seed multiplies the price many times over relative to an identical name and float. So two rows with the same market_hash_name and the same exterior can legitimately differ by an order of magnitude. Any schema with one price per name has already lost that information.

A workable shape splits identity from observation:

CREATE TABLE item (
  item_id          BIGSERIAL PRIMARY KEY,
  game_id          TEXT NOT NULL,
  market_hash_name TEXT NOT NULL,
  rarity           TEXT,
  is_stackable     BOOLEAN NOT NULL DEFAULT FALSE,
  UNIQUE (game_id, market_hash_name)
);

CREATE TABLE price_observation (
  observation_id BIGSERIAL PRIMARY KEY,
  item_id        BIGINT NOT NULL REFERENCES item(item_id),
  snapshot_id    UUID NOT NULL,
  source         TEXT NOT NULL,          -- steam_market | third_party | storefront | auction_house
  region         TEXT,                   -- realm, market hub, or ISO country
  price_type     TEXT NOT NULL,          -- ask | bid | sale
  price_minor    BIGINT NOT NULL,        -- integer minor units, never a float
  currency       CHAR(3) NOT NULL,       -- ISO 4217, stored as observed
  sample_size    INT,                    -- completed sales behind a median
  float_value    NUMERIC(17,15),         -- NULL for stackable goods
  paint_seed     SMALLINT,
  observed_at    TIMESTAMPTZ NOT NULL,   -- the SOURCE's clock
  fetched_at     TIMESTAMPTZ NOT NULL    -- your clock, for debugging only
);

Four decisions in there carry their weight:

price_type is mandatory and has no default. An ask and a completed sale are different measurements, and a pipeline that cannot distinguish them will produce a spread of zero and a chart nobody can defend.

price_minor is an integer. Money in binary floating point accumulates error across millions of aggregations, and item prices span from fractions of a cent to five figures.

sample_size gates every downstream statistic. An item with 3 sales in 24 hours has a median that means almost nothing, and one wash trade moves it entirely. Set a floor (10 or 20 trades in the window is a reasonable start), exclude thin items from any index, and use a trimmed median or a median-absolute-deviation filter rather than a raw mean on anything below that floor. Do not delete the thin rows. Flag them and keep them, because thin-market behaviour is itself interesting.

float_value is stored with real precision. Truncating to four decimals destroys exactly the low-float signal that explains the price premium you are trying to model.

An item economy can move meaningfully in an hour on a patch note. The integrity rules that follow are cheap to apply on day one and painful to retrofit.

Take the timestamp from the source. Blizzard's auction endpoints give you Last-Modified. ESI gives you Last-Modified and Expires. The OSRS Wiki price API returns a per-item highTime and lowTime. Use those for observed_at. If you stamp your own fetch time instead, an hourly-refreshing source polled at 09:05 and 09:55 produces two rows that look 50 minutes apart and are actually the same observation.

Append only. Never UPDATE a price row. If a source corrects itself, insert a new row. A time series you can overwrite is a time series you cannot audit.

Record the misses. A 429, a 503, or a delisted item is data. Write a row with a null price and a status code. Gaps that are silently skipped become straight lines the moment someone interpolates, and a straight line through a crash is a lie.

Freeze FX per snapshot. If you compare regions, store the exchange rate you used alongside the snapshot. Otherwise re-running last quarter's analysis today produces different numbers from the same raw data, and you will spend a day finding out why.

Do not smooth at write time. Store raw observations. Compute rolling medians, indices, and outlier flags in a view or a downstream table. The moment smoothing is baked into storage you can no longer change the window.

For the IP-keyed surfaces, the SparkProxy Scraping API handles the exit IP, the geography, and the browser layer from a single endpoint at https://scrape.sparkproxy.io/api/v1.

Steam's market and store JSON endpoints need no browser, so turn rendering off and keep the request cheap:

curl -G "https://scrape.sparkproxy.io/api/v1" \
  -H "X-API-Key: YOUR_API_KEY" \
  --data-urlencode "url=https://steamcommunity.com/market/priceoverview/?appid=730&currency=1&market_hash_name=AK-47%20%7C%20Redline%20(Field-Tested)" \
  --data-urlencode "render_js=false" \
  --data-urlencode "country_code=US"

Regional storefront pricing needs the exit IP to match the region you are asking about. Loop the country codes and let the API place each request:

import requests

API = "https://scrape.sparkproxy.io/api/v1"
HEADERS = {"X-API-Key": "YOUR_API_KEY"}

def regional_price(appid: int, cc: str) -> dict:
    target = (
        "https://store.steampowered.com/api/appdetails"
        f"?appids={appid}&cc={cc}&filters=price_overview"
    )
    r = requests.get(
        API,
        headers=HEADERS,
        params={
            "url": target,
            "country_code": cc.upper(),   # exit IP in the pricing region
            "render_js": "false",
            "tag": f"storefront-{cc}",    # shows up in your usage reporting
        },
        timeout=60,
    )
    r.raise_for_status()
    return r.json()

for cc in ("us", "de", "br", "tr", "jp"):
    print(cc, regional_price(730, cc))

Leaderboard pages usually do render client-side. Give the renderer a selector to wait for so you never parse a half-built table, and pin one snapshot per run:

import uuid
from concurrent.futures import ThreadPoolExecutor

snapshot_id = uuid.uuid4()

def ladder_page(page: int) -> str:
    r = requests.get(
        API,
        headers=HEADERS,
        params={
            "url": f"https://www.sparkproxy.io/example-ladder?page={page}",
            "render_js": "true",
            "wait_for": "table.ladder tbody tr",
            "block_resources": "true",     # skip images and fonts, cuts render time
            "session_id": f"ladder-{page % 20}",
        },
        timeout=90,
    )
    return r.text

with ThreadPoolExecutor(max_workers=20) as pool:
    pages = list(pool.map(ladder_page, range(1, 4001)))

Two parameters there matter more than they look. block_resources=true drops images, fonts, and media before render, which on an image-heavy marketplace listing page is most of the payload. session_id pins a label to an exit IP so a paginated cursor stays valid across sequential pages instead of resetting when the IP changes mid-walk. Use format=json with extract_rules if you would rather have the API return parsed fields than raw HTML. Every parameter is documented in the Scraping API reference.

This entire article is about read-only observation of published prices and public rankings. It is not about botting, account automation, trade execution, or anything touching a duplication bug or an exploit. Those are different activities with different consequences, and no proxy makes them acceptable.

A few boundaries worth holding:

Do not put a proxy in front of the game client. A proxy adds a hop and latency, which degrades live gameplay traffic and, in a competitive title, is indistinguishable from the kind of connection manipulation anti-cheat systems flag. Web endpoints only.

Many titles restrict automated access in their terms. The Steam Subscriber Agreement, the Riot API policies, and Blizzard's API terms all place conditions on automated collection and on redistributing what you collect. Read the ones that apply to your title before you build, particularly the redistribution clauses if you plan to publish the data.

Respect the limit even when you can exceed it. The point of a proxy pool on IP-keyed surfaces is to run many polite streams, not one rude one. Cache, use conditional requests where the source supports them, and back off on 429s. Our ethical scraping and rate limiting guide covers the mechanics.

Skin markets carry legal complications beyond scraping law. Tradeable items with real-money value sit close to gambling regulation in several jurisdictions, and regulators and platform holders have both acted against third-party sites that used game items as casino chips. If your project touches wagering, case-opening, or anything resembling a betting product, the question is licensing and consumer-protection law in each market you serve, not scraping policy. Get it reviewed before you ship.