Proxies for Appointment and Slot Availability Monitoring

Everything else in this workload is downstream of one question: when a slot opens, how long until your system knows.

If you poll an endpoint every P seconds, and a slot can appear at any moment within that interval, your average detection delay is P divided by 2 and your worst case is P. That is the whole model. There is no clever architecture that beats it, because you cannot know about a change you have not looked for.

What this means in practice is that the interval is the product. A customer who needs to know within a minute and a customer who needs to know within a day are buying two completely different systems at two completely different prices, and the difference between them is a single number in a config file. Get that number agreed in writing before you build anything, because halving it doubles your request volume and every cost in this post scales with request volume.

Second-order point that teams miss: detection delay and capture probability are different things. Detection delay is how stale your data is. Capture probability is whether you see a slot at all, and that depends on how long the slot survives before somebody takes it.

Let S be the lifetime of a slot, the time between it appearing and it disappearing. Let P be your polling interval.

• If P is less than S, you will always see the slot, because at least one poll lands inside its lifetime.
• If P is greater than S, you see it with probability roughly S divided by P, because the slot exists for only part of each polling window.

Work it through. Suppose slots in your market are typically claimed within 90 seconds.

• Poll every 300 seconds and you catch roughly 90 divided by 300, which is 30% of them. Seven out of ten never appear in your data at all, and your dataset is not just late, it is wrong about what was released.
• Poll every 60 seconds and you catch all of them, because 60 is under 90 and every slot's lifetime spans at least one poll.

That gives the governing rule: the poll interval must be shorter than the shortest lifetime you need to catch, not shorter than the freshness you promised. Those are different numbers and the first is usually much smaller. A dashboard that promises five-minute freshness while polling every five minutes silently loses most of the short-lived inventory, and nobody notices because the missing rows do not look missing.

Measure S before you pick P. Poll one endpoint aggressively for a day, record every appearance and disappearance, and take the tenth percentile of the lifetimes rather than the median. You are sizing for the short ones.

Availability monitoring gets lumped in with ticketing because both involve watching for inventory. The operational shapes are opposites.

The failure modes are the important row. In a drop you know immediately that you lost. In availability monitoring the failure is silent: the endpoint keeps returning 200, the page keeps rendering, and the availability array is simply always empty because you have been shadow-limited. Weeks of clean-looking data can be worthless. Our guide to ticketing and event registration covers the burst shape and queue systems in depth, and almost none of it transfers here.

The other structural difference is that a drop is a single endpoint and monitoring is many. Your request volume is the product of the number of endpoints and the polling frequency, which is the multiplication that gets people into trouble.

Take a concrete workload: 120 locations to watch, and a 60 second interval because measured slot lifetimes came in around 90 seconds.

Flat polling: 86,400 seconds in a day divided by 60 is 1,440 polls per location per day. Times 120 locations is 172,800 requests a day, which is 5,184,000 a month. That is a serious scraping operation to watch what is, in information terms, a handful of small arrays.

Now use what you know about the target. Releases in most booking systems are not uniform across the day. They cluster: a batch job runs at a fixed time, or staff release capacity when the clinic opens, or cancellations concentrate in the hour before a cutoff. Measure when yours happen, then poll fast inside that window and slowly outside it.

Say releases cluster in a two hour window:

• Hot window, 2 hours at 30 second intervals: 7,200 seconds divided by 30 is 240 polls.
• Cold window, 22 hours at 900 second intervals: 79,200 seconds divided by 900 is 88 polls.
• Total per location per day: 240 plus 88, which is 328 polls.
• Across 120 locations: 39,360 requests a day, or 1,180,800 a month.

Against the flat schedule's 5,184,000, that is a 77% reduction in request volume while polling twice as fast during the window that matters. Nothing about the data quality got worse. The cold-window polls exist only to catch irregular releases, and if you miss one by ten minutes at three in the morning, nobody cares.

Two refinements worth building in from the start. Back off automatically on endpoints that have not changed in days, and accelerate on ones that just did, since activity clusters. And treat any 429 or Retry-After as a hard instruction rather than a suggestion, because on a months-long watch the relationship with the source is the asset. Our notes on scheduling and automating web scrapers cover the orchestration side.

Here is where the workload inverts everyone's intuition.

Threads. The hot window is the peak: 120 locations times 240 polls is 28,800 requests, spread across 7,200 seconds, which is 4 requests a second. At 1.5 seconds per request that is 6 requests in flight. Six. A 100 thread Starter plan is more than an order of magnitude more concurrency than this workload can use, and buying a bigger plan for it would be pure waste.

Addresses. This is the real constraint. At 39,360 requests a day, if a target starts limiting an address at roughly 300 requests a day, you need those requests spread across 39,360 divided by 300, which is 132 distinct addresses. On a rotating pool that is automatic and costs nothing extra: SparkProxy's plans rotate randomly per request on port 11000 of gateway.sparkproxy.io, drawing from 1M+ addresses across 80+ countries, with 5 minute auto rotation as well. On static addresses you would be buying 132 of them, which is the expensive way to solve a problem rotation solves for free. Our explainer on proxy rotation interval covers how to pick between per-request and timed rotation.

One caveat that matters specifically here. If the booking system issues a session, or the availability endpoint requires a token obtained from a page load, per-request rotation will break it. Use the sticky session port, 11002, for the token-fetch plus availability-read pair, then let the next pair get a fresh address. You want rotation between observations, not inside one.

Bandwidth. Negligible, and worth noting because it is the axis people budget for. A JSON availability payload is often under 10 KB. At 1,180,800 requests a month that is under 12 GB, which on a per-gigabyte plan is cheap and on an unlimited plan is free. If you are pulling full rendered pages instead, you are paying 50 to 100 times that for the same handful of fields, which brings us to the cost section.

Two sane ways to run this, and for once the arithmetic points clearly at one of them.

Option 1: the managed Scraping API. At 1,180,800 requests a month, all plain fetches at 1 credit each, you need 1,180,800 credits. That overshoots the Growth tier's 1,000,000 credits at $99, so you land on Pro at $249 a month for 3,000,000 credits. Workable, with headroom to nearly triple the watch list.

Two things make this worse in a hurry. If the availability data only appears after JavaScript runs, every poll costs 5 credits instead of 1, so 1,180,800 requests becomes 5,904,000 credits and you are on the Scale tier at $599. And if you need country targeting, country_code adds 5 credits per request, which on this volume is another 5,904,000 credits on top. High-frequency polling is exactly the workload where per-request parameters compound into real money.

Option 2: a flat concurrency proxy plan. The same job needs 6 threads in the peak window and under 12 GB a month. SparkProxy Starter is $75 a month for 100 threads with unlimited bandwidth and 30 day validity, and country selection is part of the plan rather than a per-request surcharge. You write and run the polling loop yourself, which for a fixed set of known endpoints is a modest amount of code.

For a long-lived, high-frequency, low-payload watch, the flat plan wins by a wide margin: $75 against $249, and the gap grows if rendering or geo-targeting is involved. That is the opposite of what we find for low-volume, high-complexity collection, where credit pricing is cheaper. The deciding variable is requests per month, and the crossover for this shape sits somewhere under a million.

Buy the API instead when the endpoints keep changing, when you need rendering on a minority of them, or when you do not want to own retry and rotation logic. Buy the flat plan when the endpoint list is stable and the volume is high, which describes most availability monitoring after the first month.

On a workload measured in months, the practices that are polite and the practices that keep you working are the same list. This is unusual and worth exploiting.

Send conditional requests. If the endpoint returns an ETag or Last-Modified, send If-None-Match or If-Modified-Since and take the 304. You get the same information, the origin does almost no work, and your transfer drops to nothing. Surprisingly few monitoring systems bother.

Jitter every interval. One hundred and twenty locations polling on exact 60 second boundaries produces a visible spike on the origin every minute, which is both rude and the easiest possible signature to filter. Multiply each interval by a random factor between 0.8 and 1.2 and the aggregate flattens.

Cap concurrency per origin, not globally. One or two in flight per origin is plenty for this. Your global thread pool can be large and mostly idle; that is the correct shape.

Honour backoff signals literally. A 429 with a Retry-After is the source telling you the price of continuing. Pay it. Exponential backoff with a ceiling, plus a circuit breaker that stops an endpoint entirely after repeated failures, keeps a bad hour from becoming a permanent block. Our guide to retry and backoff strategies has the patterns.

Find the JSON. Most booking front ends call an availability endpoint that returns a small structured payload. Polling that instead of the rendered page cuts your bytes by an order of magnitude, removes the rendering cost, and makes change detection trivial. How to scrape hidden JSON API endpoints covers finding them. Treat any documented public API as strictly better than both.

Alert on silence. An endpoint that has returned zero availability for a week is either genuinely full or quietly blocking you, and those need different responses. Alert on the absence of change, not just on errors. How to avoid getting your proxy blocked covers the soft-block signatures to watch for.

The detector itself is small. Poll, hash, compare, record transitions. The discipline is in what you store: the observation series, not just the latest state, because the value of availability data is almost entirely in the history.

import hashlib, os, random, time
import requests

API = "https://scrape.sparkproxy.io/api/v1"
KEY = os.environ["SPARKPROXY_API_KEY"]

def poll(slot_url):
    r = requests.get(API, headers={"X-API-Key": KEY}, params={
        "url": slot_url,
        "render_js": "false",
        "tag": "slots-clinic-network",
    }, timeout=45)
    r.raise_for_status()
    return r.text, r.headers.get("X-Job-Id"), int(r.headers.get("X-Credits-Used", 0))

def watch(slot_url, interval, store):
    previous = None
    while True:
        started = time.time()
        body, job_id, credits = poll(slot_url)
        digest = hashlib.sha256(body.encode("utf-8")).hexdigest()
        store.append_observation(slot_url, started, digest, body, job_id, credits)
        if previous is not None and digest != previous:
            store.record_transition(slot_url, started, previous, digest)
        previous = digest
        time.sleep(interval * random.uniform(0.8, 1.2))

Three notes on that loop. The tag parameter groups the spend so you can attribute credits per watch list later. X-Credits-Used lets you reconcile your own request count against the bill, which catches the case where a parameter you forgot about is silently costing 5 credits a poll. And the hash comparison is deliberately over the raw body rather than a parsed structure, so a change in the payload shape shows up as a transition rather than vanishing into a parser that no longer matches. Compare the parsed availability sets separately, once you trust the parser.

For the diffing and storage side, incremental web scraping and change detection covers the patterns, and proxies for uptime and synthetic monitoring covers the adjacent case where you are watching your own endpoints rather than someone else's.

This use case has a sharper ethical edge than most, and pretending otherwise would be dishonest.

There is a real and legitimate market here. Healthcare networks measuring their own appointment lead times across sites. Service businesses benchmarking how far out competitors are booked. Aggregators operating with the provider's agreement. Researchers measuring access to public services, which is a genuine public good. Operations teams watching their own booking funnel from outside. All of these are ordinary capacity research, and the technical guidance above applies directly.

There is also a version of this that causes harm, and it tends to involve public services with scarce capacity: visa appointments, driving tests, clinic slots in a stretched system. Systems that detect and claim those slots faster than a person can, particularly when the access is then resold, take capacity from people who need it and push the operator into defensive measures that make the service worse for everyone. We will not help with that, and neither will most providers' acceptable use terms once the use case is described accurately.

The practical line we apply: monitor to understand availability, do not automate the claiming of scarce public capacity, and never resell access to a queue you did not create. If a use case needs to be described vaguely to sound acceptable, that is the answer. The general principles are in our guide on ethical scraping and rate limiting.