Digital Shelf Analytics Proxies: What CPG Brands Buy

There are three ways a CPG brand gets digital shelf data, and they are not competing on the same axis.

Most brands with any in-house data capability end up on the third option for the daily bulk, with a scraping API layered onto the handful of retailers that fight back. That hybrid is cheaper than either extreme, and it is the same pattern behind most ecommerce competitive intelligence stacks.

If you are buying proxies, the short answer is this: start on unlimited-bandwidth datacenter proxies, sized by thread count, and add a residential or premium tier only for the specific retailer surfaces that block datacenter ranges. Buying residential first is the most common overspend in this category.

"Digital shelf" is a bundle of six distinct measurements with very different collection costs. Pricing a proxy plan means knowing which ones you are actually running.

Two things fall out of that table. Search result pages carry the highest blocking pressure of anything you collect, because that is where marketplaces concentrate their anti-bot effort. And content compliance is cheap in requests but heavy in bytes, because verifying that a SKU carries six images at the retailer's minimum resolution means actually retrieving those images.

Here is the arithmetic vendors rarely walk you through. Take a mid-size CPG program: 250 tracked SKUs, 6 retailers, 150 tracked keywords, daily price and availability checks, weekly content and review audits.

Product pages   = 250 SKUs x 6 retailers x 1/day        = 1,500 /day
Search pages    = 150 keywords x 6 retailers x 2 pages  = 1,800 /day
Review pages    = 250 x 6 x 3 pages, weekly, / 7        =   643 /day
Product images  = 250 x 6 x 7 images, weekly, / 7       = 1,500 /day
                                                 TOTAL  ~ 5,443 /day
                                                        ~ 163,000 /month

Look at what that reveals. Search pages outnumber product pages. Image retrieval matches product pages almost one for one. The SKU count you priced your plan against accounts for barely a quarter of your traffic.

Now the bytes. A rendered retailer product page with assets typically runs 1 to 3 MB, and product imagery at retailer minimum resolutions averages a few hundred KB per file. For the program above:

Rendered pages : 3,943 /day x ~1.5 MB   ~ 5.9 GB /day
Images         : 1,500 /day x ~0.25 MB  ~ 0.4 GB /day
                                 TOTAL  ~ 6.3 GB /day  ~ 190 GB /month

Those page-weight figures are planning estimates drawn from public page-weight norms, not measurements taken from any specific retailer. Measure your own targets during a trial. The shape holds regardless: a modest CPG program moves well over 100 GB a month once rendering is involved, and that number decides your billing model more than anything else on this page.

Do not buy one proxy type for the whole program. Match the type to the surface.

The practical split for most brands lands around 85 percent datacenter and 15 percent residential by request count. If your split is inverted, you are almost certainly overpaying. The same escalation pattern drives MAP monitoring, where a cheap broad sweep flags candidates and an expensive narrow pass verifies them.

This is the trade-off nobody surfaces on a sales call. Residential proxies are almost universally sold per gigabyte. Datacenter proxies are commonly sold per thread with unlimited bandwidth. For digital shelf work, that difference decides the bill.

Across the major residential vendors' own published rate cards, mid-tier commitments sit in a low-single-digit dollars-per-GB range as of September 2026, with entry tiers considerably higher. Check each vendor's pricing page for current rates, because these numbers move. Apply anything in that range to the 190 GB per month from the sizing section and monthly bandwidth alone lands somewhere between a few hundred and well over a thousand dollars, for one mid-size brand, before you add a second country.

Now the unlimited-bandwidth alternative. SparkProxy datacenter plans are priced by concurrent threads with unlimited bandwidth on a 30-day term:

At those terms, 190 GB costs the same as 19 GB or 1.9 TB, because bandwidth is not the metered unit. That is why image-heavy content compliance, which is genuinely valuable and genuinely byte-expensive, is the strongest single argument for buying datacenter capacity for the bulk of your shelf.

The honest counterweight: unlimited bandwidth does not mean unconstrained throughput. The speed figure is a ceiling, not a guaranteed rate, and it constrains how fast you can drain a sweep. Pulling 6 GB inside a 30-minute window needs roughly 27 Mbps sustained, which pushes you past Starter's ceiling and into Core. Widen the window or buy the higher tier, but decide that deliberately rather than discovering it in week three.

Brands routinely buy expensive geo-targeted residential IPs to solve something that is not an IP problem.

For grocery, mass and pharmacy retailers, price and availability are set by the selected fulfilment store, not by your IP address. The site picks a default store from your IP on first visit, then stores that choice in a cookie or a URL parameter. Once you set the store explicitly, a datacenter IP in the wrong state returns the same data as a residential IP on the right block. Establish the store, persist the session, and most of the case for residential disappears.

Where IP geography genuinely matters:

1. Country-level storefront routing. Requesting a UK domain from a US IP can redirect you or serve a different currency. Country targeting solves this, and country targeting is cheap.
2. First-visit defaults you cannot override, on the few retailers that expose no store selector to unauthenticated sessions.
3. Marketplaces that price by detected region with no user-visible control.

Everything else is session management. For the mechanics of country targeting itself, see what geo-targeting means in proxies. The practical setup is a sticky session port so the store cookie survives the whole sweep for that retailer:

# Sticky session: store context survives across paginated requests
curl -x http://USER:PASS@gateway.sparkproxy.io:11002 \
  -b cookies.txt -c cookies.txt \
  "https://www.sparkproxy.io/demo-grocer/search?q=oat+milk&store=1042"

# Rotating pool: a fresh IP per request, for independent product page fetches
curl -x http://USER:PASS@gateway.sparkproxy.io:11000 \
  "https://www.sparkproxy.io/demo-grocer/p/oat-milk-1l"

SOCKS5 sits on port 13000 if your collector needs it.

Concurrency, not bandwidth, sets your tier. The formula:

threads = ceil( requests x avg_seconds_per_request / sweep_window_seconds ) x 1.4

The 1.4 covers retries and slow tails. Use 2 to 4 seconds for a static fetch and 6 to 10 seconds for a rendered page. Applying it across realistic program shapes:

The mid-size row is the point of the whole table. A 250-SKU program across six retailers needs roughly 35 concurrent threads. Starter's 100 covers it with room to double the catalogue. Brands routinely buy four times the capacity they will ever use because they sized on catalogue size instead of on the sweep window.

Whitelist slots are the quiet constraint. If your collectors run from three cloud regions plus a staging box plus an analyst's workstation, Starter's five IP-authentication slots are already spent. Teams with more egress points than that should price Core for the slots rather than for the threads.

For the retailer surfaces that fight back, the SparkProxy Scraping API bundles the proxy, the browser and the unblocking into one request. Credits are the unit: a plain fetch costs 1, a JavaScript render costs 5, a screenshot or PDF costs 10. You get 1,000 free credits with no card, which is enough to run the tests in the next section for real.

A rendered search page from a US IP, the standard share-of-search request:

curl -X GET "https://scrape.sparkproxy.io/api/v1?url=https://www.sparkproxy.io/demo-grocer/search%3Fq%3Doat%2Bmilk&country_code=US&render_js=true&format=json" \
  -H "X-API-Key: sk-xxxxxxxxxxxxxxxx"

A daily shelf sweep in Python, keeping rendering off for pages whose price already sits in the raw HTML:

import requests

API = "https://scrape.sparkproxy.io/api/v1"
KEY = "sk-xxxxxxxxxxxxxxxx"

def fetch(url, country="US", render=False):
    r = requests.get(
        API,
        params={
            "url": url,
            "country_code": country,        # ISO alpha-2: US, GB, DE
            "render_js": str(render).lower(),
            "format": "json",
        },
        headers={"X-API-Key": KEY},
        timeout=90,
    )
    d = r.json()
    return d["status_code"], d["body"], d["credits_used"]

# Static product page: 1 credit. Marketplace search: 5 credits.
status, html, cost = fetch("https://www.sparkproxy.io/demo-grocer/p/oat-milk-1l")

Budgeting credits is straightforward once you have the request math. The mid-size program's 163,000 monthly requests cost 163,000 credits if every fetch is static, and 815,000 if every fetch is rendered. Against the published tiers, that is Starter at $49 for 250,000 credits and 50 concurrent, or Growth at $99 for 1,000,000 credits and 100 concurrent. Pro is $249 for 3,000,000 credits at 200 concurrent, and Scale is $599 for 8,000,000 at 400 concurrent.

The lever that matters here: rendering costs five times a static fetch. Audit which of your targets genuinely need a browser. On most retailer product pages the price sits in the server-rendered HTML or in an embedded JSON blob, and turning rendering off cuts that page's cost by 80 percent. Marketplace pages such as Amazon product data and Walmart product data are where rendering earns its price.

Run this against any provider's trial before committing to an annual term. It takes an afternoon.

1. Success rate per retailer surface, separately. A blended 97 percent hides a 60 percent failure rate on the one marketplace search page you care most about. Log success by target, never in aggregate.
2. Measure real page weight on your own targets. Fetch 100 product pages and 100 images, record the bytes, project the monthly total. That single number decides metered against unlimited.
3. Verify store context survives a sticky session. Set a fulfilment store, paginate 20 pages, confirm the price does not drift back to a national default.
4. Time a full sweep at your intended concurrency. If 5,000 requests do not finish inside your window at the thread count you are buying, your prices are not comparable across retailers, because they were captured hours apart.
5. Check whitelist and authentication slots against your actual egress IPs, including staging and CI runners.
6. Confirm what happens at a hard block. Does the provider return a clean error you can retry, or a soft 200 carrying a challenge page your parser will silently ingest as real data? Silent challenge pages are the most damaging failure mode in shelf analytics, because they look exactly like valid records in your warehouse.
7. Price the second country before you sign. Multi-market expansion is where flat thread pricing and per-GB pricing diverge hardest.

Ratings and review collection carries its own cadence and parsing considerations, covered in review monitoring and sentiment analysis.