Proxies for Auction and Liquidation Marketplace Data

Work out how many lots close inside your busiest fifteen-minute window, divide by the polling interval you need in those final minutes, multiply by your average response time. That product is the number of concurrent connections you have to buy. Then check it against a plan priced by concurrency, because a per-gigabyte plan will not stop you at that peak but a per-address plan absolutely will, and a plan sized on your monthly average will fail at the only moment anyone cares about. For most single-platform operations the answer lands between 100 and 500 threads, which is $75 to $240 a month on SparkProxy's ladder, and datacenter exits are usually sufficient because these are public listing pages rather than consumer login surfaces.

Five structural differences, each with a direct consequence for what you buy.

The data has an expiry. A catalogue price is true until it changes. A live bid is true for seconds. If your collection lags the close by two minutes on a lot that received four bids in its final ninety seconds, you recorded a number that was never the outcome. Retry-later is not a recovery strategy here.

Demand is scheduled, not smooth. Platforms close lots in batches, often on the hour, often on the same weekday. Your traffic profile is a sawtooth with predictable teeth. That predictability is useful, because you can plan for it, and dangerous, because so can the platform's rate limiter.

One lot is one entity with a life cycle. Unlike a product SKU, a lot is created, bid on, and destroyed within days. Your store needs versioned observations with timestamps, not a current-value row, and your crawler needs to track state transitions rather than diffs. The mechanics in incremental scraping and change detection apply, with the twist that a lot's disappearance is itself a data point.

Closing time is not yours to define. Extended bidding rules move it. More on that below, because it is the single most common reason auction datasets are quietly wrong.

The interesting lots are a small fraction. On a liquidation platform, most pallets sell near reserve with little movement. The lots that carry information are the contested ones, and you cannot tell which those are until bidding starts. So the polling strategy has to be adaptive rather than uniform, which is where the cost control lives.

The arithmetic here is Little's Law, which says the number of requests in flight equals the request rate multiplied by the time each request takes. Applied to a closing window:

concurrency  =  (lots closing in the window / polling interval in seconds) × average response seconds

Three worked cases, using a 1.2 second average response time as a working assumption you should replace with your own measurement:

Now compare that to the monthly average. A job doing 2.4 million requests a month averages about one request per second. Size a plan on that number and you buy four threads. The peak in the third row needs 160. The peak is roughly 150 times the average, and the average is the number most buyers quote to a sales team.

This is why concurrency is the unit that matters for auction work, and why plans priced by concurrent thread fit it well: you are buying the peak and paying nothing extra for the troughs. SparkProxy sells that shape, from Starter at $75 a month for 100 threads through Core at $140 for 250, Boost at $240 for 500 and Plus at $440 for 1,000, all with unlimited bandwidth and 30 days validity across 1M+ datacenter IPs in 80+ countries. If the term is unfamiliar, understanding concurrent connections covers how thread counts translate into throughput.

One caveat to state plainly: the fair usage policy attaches a speed ceiling per tier, 25 Mbps on Starter up to 150 Mbps on Plus. That is a ceiling, not a guaranteed rate, and on a closing spike of small HTML responses it is rarely the binding constraint. On image-heavy lot galleries it can be.

Uniform polling is the most expensive mistake on this workload. An adaptive ladder cuts request volume by an order of magnitude with almost no loss of signal.

Two rules make the ladder work. Promote a lot to the contested track the moment it receives a second distinct bidder or its watch count crosses a threshold you tune per category. And demote nothing: a lot that becomes contested stays on the fast track until it terminates.

The cost effect is large. Polling 3,000 open lots uniformly every ten minutes is 432,000 requests a day. The ladder above, on the same 3,000 lots with roughly 400 in their final day and 60 genuinely contested, lands near 80,000 a day. Same dataset, a fifth of the requests, and the difference is entirely in lots that were never going to move.

Most timed auction platforms run some form of anti-sniping extension, variously called soft close, extended bidding, popcorn bidding or dynamic close. A bid landing inside the final two to five minutes pushes the closing time out by a similar interval, and it keeps doing so until the bidding stops.

The consequence for data collection is blunt: a lot's scheduled close time tells you when to start watching, not when to stop. A crawler that polls until the published timestamp and then moves on will systematically record the wrong final price on exactly the contested lots that carry the most information. That is not random error. It is bias concentrated in your most valuable rows.

Three practical implications:

1. Terminate on state, not on time. Keep polling until the lot's own status field reads sold, closed, passed or withdrawn. Store the terminal status alongside the price so you can distinguish a sale from a no-sale later.
2. Budget for the tail. A batch scheduled to close at 18:00 can still be going at 18:40. Your concurrency peak lasts longer than the published window, and your job scheduler needs to know that.
3. Record both timestamps. Scheduled close and actual close, and the delta between them. That delta is one of the cleanest contested-lot signals available, and it costs nothing to capture.

Clock handling matters here too. Use the platform's own countdown or server timestamp rather than your machine's clock, and store everything in UTC with the platform's stated local timezone as a separate field. Auction houses publish close times in local time, daylight saving transitions happen mid-season, and a one-hour error looks exactly like a missing dataset.

Auction and liquidation platforms fall into groups with genuinely different requirements.

Retail returns and wholesale liquidation (B-Stock marketplaces, Liquidation.com, BULQ and similar) publish pallet manifests and close on schedules. Listing pages are public, detail pages often require a free account, and the commercially useful fields are manifest contents and closing price per unit. Datacenter exits handle the public side well.

Salvage and insurance vehicle auctions (Copart, IAAI) are heavily regionalised, with lot availability and visible fields varying by location and by whether you appear to be in-market. US datacenter addresses matter here, and SparkProxy's 50,000+ US IPs sit in the right place for it. Related work on automotive listings aggregation covers the vehicle data side.

Government and institutional surplus (GovDeals, Public Surplus, GSA Auctions) is public by design and among the least hostile data on the open web. Light pacing, US exits, no need for expensive address types. If you are already collecting public procurement data, proxies for government tender data is the adjacent workload.

Industrial and heavy equipment (Ritchie Bros., IronPlanet, BidSpotter) runs both timed and live webcast formats. Webcast events stream state over websockets and are a different engineering problem from polling, though the proxy requirement is milder because you hold one connection rather than making thousands.

Aggregators and auction house networks (HiBid, Proxibid, Invaluable and similar) front hundreds of independent houses on shared infrastructure, so rate limits are usually applied at the platform level and geography matters less than pacing.

Consumer marketplace auctions on general platforms behave differently again, because they sit behind the same anti-automation infrastructure as the rest of the site. Our notes on eBay listing collection cover that surface.

Across all of these, the common answer is that public lot pages are served to anyone and datacenter exits are appropriate. If a specific platform's edge blocks hosting ASNs outright, that is a target-level fact you measure rather than assume, and the fallback is residential exits, which SparkProxy does not sell.

An illustrative model, not a measurement. Assume 3,000 open lots tracked on the ladder above, roughly 80,000 requests a day, so about 2.4 million a month, at an assumed 250 KB average HTML response. Substitute your own numbers before deciding anything.

The gap between the first and third rows is the entire argument for measuring before you buy. Most auction and liquidation listing pages are public marketing surfaces that want to be indexed, and paying residential rates to read them is a large avoidable cost.

One nuance that moves the numbers hard in the other direction: if the platform serves lot state from a JSON endpoint rather than a rendered page, your per-request payload can drop by a factor of ten, which shrinks the per-gigabyte column proportionally and makes it competitive again. Find the JSON before you price anything.

If you take the managed route, a single call looks like this, and the response carries the credits and duration it used so you can cost the job before scaling it:

curl -G "https://scrape.sparkproxy.io/api/v1" \
  -H "X-API-Key: YOUR_API_KEY" \
  --data-urlencode "url=https://www.sparkproxy.io/example-lot/1234" \
  --data-urlencode "render_js=false" \
  --data-urlencode "country_code=US"

render_js=false keeps the call at 1 credit. Turning rendering on costs 5, which on 2.4 million requests is the difference between the Pro tier and a much larger conversation, so render only the pages that genuinely need it. The credit arithmetic is laid out in scraping API credit pricing. If you need residential exits for a particular target, premium_proxy=true is the parameter that selects them.

A large share of liquidation inventory sits behind a free business registration. Manifests, unit counts and sometimes closing prices are visible only to signed-in users.

That changes the proxy requirement in one specific way: a session has to stay on one exit. An account that appears from a different address on every request will be challenged, and correctly so. Use sticky sessions, which on SparkProxy's gateway means port 11002 on gateway.sparkproxy.io rather than the per-request rotation on 11000, and bind one session to one account for its lifetime. SOCKS5 sits on 13000 if you need it. Our explainer on sticky session proxies covers how long to hold one.

Two constraints worth stating clearly. Register accounts you are entitled to hold, under terms you have read, and one account per legitimate business identity. And expect gated pages to carry stricter rate limits than public ones, because the platform knows exactly who is asking.

Run these before you commit to a plan. Two hours of work, and each one changes what you should buy.

Check the close-time behaviour. Watch ten contested lots through their scheduled close. Count how many extended past it and by how long. If extensions are common, your concurrency peak is wider than the published schedule implies.

Check whether lot state needs JavaScript. Fetch a lot page with rendering off and search the raw HTML for the current bid. If it is there, you never need to render, and your cost per request stays at the floor. If it is not, look for the JSON endpoint the page calls before you accept a 5x cost increase.

Check the per-address tolerance. Poll one lot from one address at your intended final-minutes interval and see how long before the responses change. That number sets how many addresses the rotation pool has to supply during a peak, and it is the input nobody measures. Detecting when your scraper is blocked covers what to watch for beyond status codes, because auction platforms frequently degrade quietly by serving a cached page rather than returning an error.

Check the timezone and daylight saving handling. Pull one lot's published close time and compare it against the countdown the page renders. Any disagreement is a bug you want to find now.

Check for a soft ban on the spike. Run a dress rehearsal at full concurrency against a low-value closing batch. The failure mode you are hunting is not a 403. It is stale data returned with a 200, arriving a few seconds late, on precisely the requests that mattered.

Collect public listing and result data. Do not automate bidding, do not create accounts you are not entitled to, and read the platform terms before you register anything.

Pace deliberately even where the platform tolerates more. The polling ladder above is not only a cost control, it is also the difference between a collector that reads like a market analyst and one that reads like a sniping tool, and platforms treat those two very differently. Our ethical scraping and rate limiting guide has the general principles.

Two specifics for this sector. Respect reserve and identity fields: bidder identities, where visible, are personal data and generally should not be retained. And treat manifest contents as commercial information whose redistribution may be restricted even where collection is not.