How to Build an Automated Price Monitoring System

Three routes exist, and the right one depends almost entirely on how many targets you track and how much engineering time you can spend keeping parsers alive.

The honest split: SaaS wins below roughly 500 tracked products, because per-SKU pricing is still cheap there and you skip all the plumbing. Between 500 and about 50,000 products, a scraping API plus a small parsing service is usually the lowest total cost once salary is included. Above that, or wherever the same pages get pulled several times a day, raw proxies with unlimited bandwidth pull decisively ahead, because your marginal cost per check drops to nearly zero while every per-request meter keeps ticking.

This is not a permanent decision, which buyers often miss. Plenty of teams start on a scraping API to prove the business case, then move the high-frequency, easy-to-parse targets onto raw proxies once volume justifies the engineering. Keep the fetch layer behind an interface and switching later costs an afternoon.

Concurrency is the number every proxy plan is sold on, and almost nobody calculates it. The formula is short:

threads = ceil( requests_per_run x avg_seconds_per_request / window_seconds )

Then provision about 3x that, because retries, slow targets, and uneven scheduling all eat headroom. Run the numbers for a few realistic catalogues:

Look at the fourth row. Four hundred thousand requests a day, a catalogue most teams would call large, and the theoretical requirement is 47 concurrent connections. This is why "how many proxies do I need" is the wrong opening question for price monitoring. The binding constraint is rarely your thread count. It is per-domain politeness, because hammering one retailer with 200 parallel requests gets you blocked no matter how many IPs sit behind them.

SparkProxy's published tiers map onto that table directly: Starter at $75/mo for 100 threads and 5 whitelist slots, Core at $140/mo for 250 threads and 10 slots, Boost at $240/mo for 500 threads and 15 slots, and Plus at $440/mo for 1000 threads and 25 slots. All four carry unlimited bandwidth and 30 days validity. Pro and Pro+ tiers exist for higher thread counts under the Fair Usage Policy but are quoted rather than listed publicly.

Product listing pages are public, cacheable, and mostly served to anonymous visitors. That profile favours datacenter IPs far more than the residential-by-default advice you will read elsewhere.

Start datacenter-first and escalate only the domains that actually fail. In most catalogues that escalation set is a handful of aggressive marketplaces, not the whole target list. The reason is arithmetic, not preference. If you check 20,000 products four times a day and each page transfers roughly 300 KB, you move about 24 GB a day, or 720 GB a month. On a flat unlimited plan that number is free. On a per-GB meter it is the entire budget.

Competitor per-GB rates are published on each vendor's own pricing pages and change frequently. As of September 2026, the major residential networks (Bright Data, Oxylabs, Decodo, IPRoyal) publish entry residential rates in the single-digit dollars per GB, falling with volume commitment. Those are each vendor's own published figures, including their pool-size claims, so check the current pages before you model anything.

For a deeper split of the three families and where each earns its price, see the comparison of residential, datacenter, and mobile proxy types.

"Unlimited bandwidth" still has a ceiling, and it is the speed cap. This is the calculation that should decide your tier, and virtually no buyer runs it. A Mbps cap converts to a daily transfer ceiling at full saturation:

Two caveats, both important. The cap is a ceiling, not a guaranteed rate, and no real workload saturates a link continuously, so plan against 30 to 50 percent of these numbers. Even so, the point holds: a 20,000 SKU catalogue checked four times daily uses roughly 9 percent of a Starter plan's theoretical ceiling. Bandwidth is almost never the reason to upgrade in this use case. Threads and whitelist slots usually are.

Two behaviours, two ports, and the difference decides whether your prices are correct.

Rotating per request is right for the bulk of the work. Where it goes wrong is any page whose price depends on state you established earlier. If you set a delivery postcode, then rotate to a new IP for the product page, you get a price for the wrong region and silently poison your dataset. Keep those flows on a sticky session for their whole lifetime.

The rotation mechanics themselves, including retry handling and thread-safe pools, are covered in what proxy rotation is and how it works and in the Python patterns in how to rotate proxies in Python.

Keep these as separate components. Teams that fuse fetching and parsing into one script end up re-scraping every time a selector changes, which is both slow and rude to the target.

1. Scheduler. Decides what gets checked and when. Cron is fine to start; a queue such as Celery or BullMQ scales better once you need per-domain rate control.
2. Fetcher. The only component that touches proxies. Returns raw HTML plus status code, timing, and the exit IP used.
3. Raw store. Persist the fetched HTML, compressed, keyed by URL and timestamp. This is the single highest-value habit in the whole system: when a parser breaks you re-parse history instead of re-scraping it.
4. Parser. Extracts price, currency, availability, seller, and shipping into a normalised record.
5. Change detector and alerter. Compares against the previous observation and emits only real movements.

Tier your check frequency instead of sweeping everything at one cadence. Roughly: hourly for the top 1 to 2 percent of revenue SKUs, four times daily for the competitive core, daily for the long tail, weekly for discontinued or low-velocity items. That single decision often cuts total request volume by 60 to 80 percent against a flat hourly sweep, with no loss of usable signal. If you need the scheduling layer in more depth, see how to schedule and automate web scrapers.

A minimal fetcher with per-domain politeness, retries, and rotation. Rotation happens server-side, so the client simply points at the rotating port.

import time, random, threading, requests
from collections import defaultdict

PROXY = "http://USER:PASS@gateway.sparkproxy.io:11000"
PROXIES = {"http": PROXY, "https": PROXY}

HEADERS = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
                  "AppleWebKit/537.36 (KHTML, like Gecko) Chrome/141.0 Safari/537.36",
    "Accept-Language": "en-US,en;q=0.9",
}

_last_hit = defaultdict(float)
_lock = threading.Lock()
MIN_GAP = 1.2  # seconds between requests to the same host

def _throttle(host):
    with _lock:
        wait = MIN_GAP - (time.time() - _last_hit[host])
        if wait > 0:
            time.sleep(wait)
        _last_hit[host] = time.time()

def fetch(url, host, attempts=3):
    for i in range(attempts):
        _throttle(host)
        try:
            r = requests.get(url, proxies=PROXIES, headers=HEADERS, timeout=30)
            if r.status_code == 200:
                return r.text
            if r.status_code in (403, 429, 503):
                time.sleep((2 ** i) + random.random())
                continue
            return None
        except requests.RequestException:
            time.sleep((2 ** i) + random.random())
    return None

Verify rotation is live before you trust any of it. Two calls, two different exits:

curl -s -x http://USER:PASS@gateway.sparkproxy.io:11000 https://ipinfo.io/json
curl -s -x http://USER:PASS@gateway.sparkproxy.io:11000 https://ipinfo.io/json

For a postcode-bound or logged-in flow, switch to the sticky port and hold one session for the whole sequence:

curl -s -x http://USER:PASS@gateway.sparkproxy.io:11002 https://ipinfo.io/json

Success rate per domain is the metric to watch here, not raw throughput. If a given retailer drops below roughly 90 percent on datacenter IPs, that domain is a candidate for escalation, not the whole system. Detecting when your scraper is blocked covers the soft-block signals that never show up as a non-200 status.

When targets render prices in JavaScript and you would rather not run a browser fleet, the managed route removes an entire category of work.

import requests

API = "https://scrape.sparkproxy.io/api/v1"
API_KEY = "sk-xxxxxxxxxxxxxxxx"

def fetch_price_page(url, country="US", render=False):
    r = requests.get(
        API,
        params={
            "url": url,
            "country_code": country,     # ISO alpha-2
            "render_js": str(render).lower(),
            "format": "json",
        },
        headers={"X-API-Key": API_KEY},
        timeout=90,
    )
    d = r.json()
    return d["status_code"], d["body"], d.get("credits_used")

The billing model is what decides this, so do the arithmetic openly. A plain fetch costs 1 credit, JS rendering costs 5, and a screenshot or PDF costs 10. Published tiers: Starter $49 for 250,000 credits at 50 concurrent, Growth $99 for 1,000,000 at 100 concurrent, Pro $249 for 3,000,000 at 200 concurrent, Scale $599 for 8,000,000 at 400 concurrent. There are 1,000 free credits with no card, which is enough to validate parsers against your real target list before spending anything.

Apply it to the 20,000 SKU example: 80,000 requests a day is 2.4 million a month. As plain fetches that lands inside Pro at $249. Turn on rendering for every request and the same workload needs 12 million credits, which is past the largest published tier. That crossover is the most useful number in this article. Rendering everything on a per-credit meter does not scale; rendering selectively does. Most retail product pages ship the price in the initial HTML or in an embedded JSON blob, so check first and render only the domains that genuinely need it. Practical recipes for that extraction step are in how to scrape ecommerce prices.

Store observations, not current state. An append-only table lets you answer "when did they drop it" months later, which is the question that actually drives pricing decisions.

CREATE TABLE price_observation (
  id           BIGSERIAL PRIMARY KEY,
  sku          TEXT NOT NULL,
  competitor   TEXT NOT NULL,
  url          TEXT NOT NULL,
  price_cents  INTEGER,
  currency     CHAR(3),
  in_stock     BOOLEAN,
  seller       TEXT,
  observed_at  TIMESTAMPTZ NOT NULL DEFAULT now(),
  exit_ip      INET,
  http_status  SMALLINT
);
CREATE INDEX ON price_observation (sku, competitor, observed_at DESC);

Then gate alerts so the channel stays credible:

def should_alert(prev, curr, floor_cents=100, pct=3.0):
    if prev is None or curr is None:
        return False
    delta = abs(curr - prev)
    if delta < floor_cents:
        return False
    return (delta / prev) * 100 >= pct

Two guards worth adding on day one. First, treat a parse returning None as a fetch failure, never as a price of zero, because a single null written as 0 fires a "competitor dropped 100 percent" alert at 3 a.m. and burns trust in the system. Second, require two consecutive observations to agree before alerting on a large move. Retailers do briefly serve broken or test prices, and a confirm-on-repeat rule removes nearly all of that noise at the cost of one extra check cycle.

Take the 20,000 SKU, four-checks-daily workload: 2.4 million requests and roughly 720 GB a month.

The residential rows are arithmetic, not a price quote. They show what any per-GB meter does to a re-fetch-heavy workload. That gap, roughly $75 against a four-figure bill for identical data, is why proxy type matters more than any other decision in this build. Residential IPs remain the correct answer for the specific domains that reject datacenter ranges. They are the wrong default for the other 90 percent of your target list.

Add engineering honestly when you compare. A working system is a week or two of initial build plus a few hours a month of parser maintenance. If that time is scarce, the $249 API line is cheaper than the $75 proxy line once salary is in the model. Teams already running competitive intelligence programmes will recognise the pattern from how ecommerce companies use proxies for competitive intelligence.

• Flat cadence for everything. Hourly checks across an entire catalogue multiply cost by 6 to 24x for signal nobody uses. Tier by revenue.
• No raw HTML archive. Parsers break often. Without stored HTML, every break costs a full re-scrape and leaves a permanent hole in the history.
• Rotating IPs mid-session. Region and cart flows need sticky exits. Rotating through them yields prices from the wrong locale, and nothing in the data will tell you.
• Ignoring currency and tax presentation. The same SKU shows ex-VAT in one market and inclusive in another. Store currency and a tax flag or your comparisons are meaningless.
• Alerting on every movement. Untuned alerts get muted within a fortnight, and the mute is permanent.
• Buying threads instead of measuring them. Run the concurrency formula. Most teams need a fraction of what they were sold.
• Scraping behind a login you agreed not to automate. Public listing pages are one thing; an account whose terms you accepted is a different risk profile. Get that reviewed before you build it, not after.

Related reading if your targets are marketplaces rather than direct retailers: how to scrape Amazon product data. If the job is enforcing your own pricing policy rather than watching competitors, proxies for MAP monitoring covers that variant.