Proxies for Streaming Catalog Research

Catalog research is the study of a service's shelf, not its content. The shelf is published: title pages, browse grids, "new this month" rails, "leaving soon" rails, and the plan and pricing page. All of it is served to anonymous visitors, all of it is indexed by search engines, and much of it carries structured data the platform published deliberately so it would be indexed.

The audience for this is narrower than the search volume suggests, and it is entirely legitimate: content acquisition and distribution teams pricing a rights package, licensors auditing where their titles actually surfaced, competitive-intelligence analysts at streamers, media investors modelling library value, and compliance teams with a quota to report against.

That last one is a real regulatory driver, not a hypothetical. The EU Audiovisual Media Services Directive as amended by Directive (EU) 2018/1808 requires on-demand services to hold at least a 30 percent share of European works in their catalogue and to give those works prominence. Canada's Online Streaming Act (Bill C-11, royal assent April 2023) pushed similar obligations into the Broadcasting Act. You cannot report a catalogue share you have not measured, and the only way to measure it per territory is to look at the catalogue as it is presented in that territory.

The out-of-scope column is not a formality. Playback restriction and catalog listing are separate systems on separate infrastructure. Reading a title page does not require touching the player, so a well-built pipeline never gets near it. If a design decision starts pulling you toward the player, the design is wrong.

The catalog you see is resolved at the edge from your client IP and baked into the page payload before it reaches you. Rights are territorial, so the territory is the input. A language toggle in the interface is a presentation setting layered on top.

This produces the single most common data-quality failure in catalog projects. An analyst sets Accept-Language: de-DE, sees German labels, and records the result as the German catalog. It is the US catalog rendered in German. Every availability row from that run is wrong, and nothing in the output looks wrong.

The rule follows directly: your territory coverage equals your proxy territory coverage. Thirty markets means exits in thirty markets. And because a mis-geolocated IP produces a plausible record rather than an error, you need a positive assertion on every fetch.

Assert the territory from the target page, not from an IP geolocation database. The page knows which catalog it served. Useful markers, in rough order of reliability:

1. A territory or locale code inside the page's own JSON payload or JSON-LD block.
2. The currency symbol and format on any pricing or rental element.
3. The pre-selected value of the site's own country or region switcher.
4. Territory-specific legal or regulator text in the footer.

An IP database tells you where the proxy claims to be. The page tells you which catalog you actually received. Only the second one belongs in your data. If you are new to how country selection works at the proxy layer, our explainer on what geo-targeting means in proxies covers the mechanics.

Catalog research produces three primary signals plus one derived one.

Availability is the atom: title T is listed on service S in territory C at observation time t. Everything else is built from a series of these.

Licensing window churn is derived, mostly. Services publish arrival dates readily and departure dates unevenly. Licensed content often gets an explicit "leaving soon" rail, which is high precision and low recall. Originals rarely announce departures at all. So you get end dates two ways: harvest the leaving-soon rails where they exist, and derive the rest from your own observation series. The derived path is the one that needs discipline, and it gets its own section below.

Regional price tiers come from the plan page in each territory: tier names, monthly amount, currency, whether an ad-supported tier exists, annual billing options, and per-title rental or purchase prices where the service lists them.

Catalog overlap is the derived signal that most analyses actually want. Once availability is normalized across services within a territory, overlap is set arithmetic: shared titles, service-exclusive titles, and the churn rate of each.

Store all of it as immutable observations. Do not maintain a mutable "current catalog" table that you update in place. Every row carries observed_at and the job identifier that produced it, and current state is a query over the series. When you later find a parser bug (you will), you can reprocess history instead of discovering that state was overwritten months ago.

The tax_inclusive flag looks pedantic until the first cross-region price chart goes into a board deck. More on that below.

A platform's title ID is not stable across territories. The same series frequently carries a different internal ID per market, and the displayed title is often localized: La Casa de Papel in Spain, Money Heist in English-language markets. Join naively on the platform ID and you get one row per territory with no way to compute overlap. Join on title text alone and you silently merge distinct works.

The workable approach is layered:

1. External identifier first. Many title pages emit JSON-LD or embedded metadata containing a sameAs link or an external database reference. Where that exists, it is the cleanest join key available and it costs nothing extra to capture.
2. Fingerprint second. Build normalized_title + release_year + content_type, with normalization that casefolds, strips diacritics, removes a leading article per the page's language, collapses punctuation and whitespace, and drops trailing edition markers such as season or year suffixes. For series, add the listed season count as a tiebreaker.
3. Alias table third. Whatever survives goes to a reviewed alias table mapping platform IDs to your title_key.

Budget for that third layer. Across a genuinely multilingual set of territories, expect somewhere in the low single-digit percent of titles to need manual aliasing, concentrated in localized-title films and re-releases where the streaming listing year differs from the theatrical year. Fuzzy string matching will get you close and will also confidently merge a 1998 film with its 2019 remake. Keep a human in that loop and keep the decisions in a table you can audit.

One more trap: release year disagreements are systematic, not random. A film listed as 2019 in one territory and 2020 in another usually reflects a delayed local release. Allow a one-year tolerance in the fingerprint, and only a one-year tolerance.

This is where catalog projects live or die, and where most published guidance stops short. A title missing from today's crawl is not a removal. It is a non-observation, and the causes are mundane.

Three mechanisms handle nearly all of it.

The two-miss rule. Never write a removal event on a single miss. Require two consecutive misses at least 24 hours apart, and prefer that the second check hit the title's own page rather than a browse listing.

The positive control set. Pick 20 to 50 titles per territory that are stable, long-tenured, and unlikely to churn. Fetch them on every run. If fewer than, say, 95 percent of them are present, the crawl is broken and its output is discarded rather than diffed. This single check is the highest-value engineering in the whole pipeline, because it converts silent corruption into a loud failure.

The completeness metric. Track item count per territory per run against its own trailing 7-day median. A large unexplained drop quarantines the run for review. Catalogs do shrink, but they rarely shrink by a third overnight, and when they do you want a human to look before the number reaches a slide.

Browse rails are personalized and reordered constantly. Title pages are comparatively deterministic. Use rails for discovery of new titles and title pages for confirmation of state, and the false-positive rate drops sharply.

The SparkProxy Scraping API handles the exit country, rendering, and retries in one call, which keeps the catalog pipeline free of proxy plumbing. Base endpoint is https://scrape.sparkproxy.io/api/v1 and the key goes in the X-API-Key header.

Start with a single public title page through a specific territory:

import requests

API_KEY = "YOUR_API_KEY"

r = requests.get(
    "https://scrape.sparkproxy.io/api/v1",
    headers={"X-API-Key": API_KEY},
    params={
        "url": "https://example-streamer.test/title/12345",
        "country_code": "DE",       # ISO 3166-1 alpha-2, +5 credits
        "render_js": "false",       # 1 credit, about 3x faster
        "tag": "catalog/de/title-page",
    },
)
print(r.status_code, r.headers.get("X-Credits-Used"))

Test render_js=false first on every service you add. If the availability fields are present in the server-rendered HTML or in a JSON island inside it, you never need the browser, and the cost difference is large. Diff the extracted field set both ways once, record the answer per service, and move on.

Pull structured fields directly instead of writing a parser per service:

import requests

rules = {
    "title":       "h1",
    "year":        "[data-testid='release-year']",
    "type":        "[data-testid='content-type']",
    "seasons":     {"selector": "section.seasons li", "type": "list"},
    "price":       ".plan-price",
    "canonical":   {"selector": "link[rel='canonical']", "type": "href"},
    "region_hint": "[data-region]",
}

r = requests.post(
    "https://scrape.sparkproxy.io/api/v1",
    headers={"X-API-Key": API_KEY, "Content-Type": "application/json"},
    json={
        "url": "https://example-streamer.test/title/12345",
        "country_code": "FR",
        "extract_rules": rules,
        "tag": "catalog/fr/extract",
    },
)
print(r.json()["extracted"])

Catalog pages are one of the better cases for format=json, which returns headings, links, images, page meta, and any JSON-LD the page publishes. That JSON-LD is where external identifiers and availability metadata usually live:

r = requests.get(
    "https://scrape.sparkproxy.io/api/v1",
    headers={"X-API-Key": API_KEY},
    params={
        "url": "https://example-streamer.test/title/12345",
        "country_code": "GB",
        "render_js": "true",     # json format needs rendering
        "format": "json",
        "tag": "catalog/gb/jsonld",
    },
)
payload = r.json()

The daily watchlist sweep is where batch mode pays for itself. Pass comma-separated URLs with render_js=false and the whole batch bills as one base request:

watchlist = [f"https://example-streamer.test/title/{i}" for i in range(1000, 1050)]

r = requests.get(
    "https://scrape.sparkproxy.io/api/v1",
    headers={"X-API-Key": API_KEY},
    params={
        "url": ",".join(watchlist),   # batch requires render_js=false
        "render_js": "false",
        "country_code": "ES",
        "tag": "catalog/es/watchlist",
    },
)
for item in r.json()["results"]:
    print(item["url"], item["httpStatus"], item["success"])

European territories add a wrinkle that nobody warns you about: a consent management platform wall renders before the catalog does. That is one of the few cases where the browser is genuinely required, and js_scenario clears it in one call:

scenario = {
    "instructions": [
        {"click": "#onetrust-accept-btn-handler"},
        {"wait_for": "[data-testid='browse-grid']"},
        {"scroll": 2400},
        {"wait": 1200},
        {"scroll": 2400},
    ]
}

r = requests.post(
    "https://scrape.sparkproxy.io/api/v1",
    headers={"X-API-Key": API_KEY, "Content-Type": "application/json"},
    json={
        "url": "https://example-streamer.test/browse/movies",
        "country_code": "IT",
        "render_js": True,
        "js_scenario": scenario,
        "tag": "catalog/it/browse",
    },
)

For a monthly full census across many territories, do not hold connections open. Fire jobs with callback_url and take the results at your endpoint:

for territory in ["US", "GB", "DE", "FR", "ES", "IT", "BR", "JP"]:
    requests.get(
        "https://scrape.sparkproxy.io/api/v1",
        headers={"X-API-Key": API_KEY},
        params={
            "url": "https://example-streamer.test/browse/all",
            "country_code": territory,
            "render_js": "true",
            "callback_url": "https://collector.sparkproxy.io/webhook/catalog",
            "tag": f"census/{territory.lower()}",
        },
    )  # returns 202 with a job_id immediately

The geo assertion belongs in the parser, not in a separate check. Fail closed:

class GeoMismatch(Exception):
    pass

CURRENCY_BY_TERRITORY = {"DE": "EUR", "GB": "GBP", "US": "USD", "JP": "JPY"}

def assert_territory(extracted, expected):
    seen = (extracted.get("region_hint") or "").upper()
    currency = (extracted.get("price_currency") or "").upper()
    if seen and seen != expected:
        raise GeoMismatch(f"page reports {seen}, requested {expected}")
    if currency and currency != CURRENCY_BY_TERRITORY.get(expected, currency):
        raise GeoMismatch(f"currency {currency} wrong for {expected}")
    return True

And the two-miss confirmation, which is the piece that keeps bad removals out of the warehouse:

def confirm_removals(prev_seen, today_seen, pending, control_pass_rate):
    """Return (removals, pending) for one service and territory."""
    if control_pass_rate < 0.95:
        return [], pending          # crawl is untrustworthy, change nothing

    missing = prev_seen - today_seen
    removals = sorted(missing & pending)     # missed twice in a row
    pending = missing - set(removals)        # first miss, hold for tomorrow
    return removals, pending

Credits are deducted before the run and refunded automatically on failure, and a failed scrape returns 530 with a reason field rather than a silent empty body, so a quarantine rule can key off the response instead of guessing from a short page.

Crawling every title in every territory every day is the default instinct and the wrong one. Split the work by how fast each signal actually moves.

The cost gap between tiers is the whole reason to split them. Take a daily watchlist of 3,000 title pages across 30 territories. Batched 50 URLs per request, that is 60 requests per territory. Each request costs 1 credit for the plain fetch plus 5 for the geo add-on, so 6 credits per request, 360 per territory, 10,800 credits per day across all thirty.

Run the same 3,000 pages one at a time with rendering and you pay 5 credits for the render plus 5 for geo on every page: 30,000 credits per territory, 900,000 per day. Same data, roughly eighty times the spend. Confirm the arithmetic against your own account with one small batch before you size a budget on it, then build the watchlist tier around plain fetches and reserve rendering for the census and the consent-walled browse pages.

Two scheduling details matter more than they look. Run each territory at a consistent local hour so day-over-day diffs are not contaminated by catalog updates that land mid-window. And stagger territories rather than firing all thirty at once, which keeps you inside concurrency limits and avoids the 429 responses that turn into phantom removals. Our guide to ethical scraping and rate limiting covers the pacing side in more detail.

Regional pricing is the part of the dataset most likely to end up in a chart, and most likely to be wrong in a way nobody notices.

The tax point deserves the emphasis. Convert a VAT-inclusive European price to dollars, put it next to a pre-tax US price, and the European market looks more expensive than it is by roughly the local VAT rate, which across the EU commonly sits somewhere between 17 and 27 percent. That is more than large enough to reverse a ranking. If the page does not state whether tax is included, record tax_inclusive as null rather than guessing, and exclude those rows from cross-region comparisons instead of quietly averaging them in.

Price observations need the same provenance discipline as availability. Plans change without announcement, and being able to point at a job ID and a timestamp is the difference between a defensible finding and a screenshot.

Catalog pages are public marketing surfaces. They are built to be crawled by search engines, so their defenses are usually far lighter than a checkout flow or a search API.

There is one service-specific behavior worth testing for deliberately, because it produces bad data rather than errors. Some sites serve a reduced or stale page to datacenter ranges: fewer rails, a truncated grid, or a cached payload from another region. Nothing fails. You just record a smaller catalog.

Test for it once per service, per territory. Fetch the same browse page and the same three title pages through a datacenter exit and a residential exit in the same country on the same day, then diff the extracted field sets and the item counts. If they disagree beyond noise, that service goes on the residential list permanently. Re-test quarterly, since the behavior changes when a site changes its bot policy. The comparison of residential, datacenter, and mobile proxy types covers the underlying trade-offs, and if your team is also testing localized front ends rather than only reading them, proxies for localization testing is the adjacent workflow.

The boundary from the top of this article is the operating rule, and a few more sit alongside it.

Public pages only. No accounts, no shared credentials, no authenticated surfaces, no paywalled catalogs. If a page requires a login, it is out of scope for this pipeline.

Never touch playback. No player, no licence server, no DRM, no Widevine, no device spoofing to obtain a stream. Beyond the platform terms, circumvention of technical protection measures is separately actionable under 17 U.S.C. 1201 and Article 6 of Directive 2001/29/EC. Catalog metadata does not live behind any of that, so there is no research reason to go there.

Respect robots.txt and pace requests. Catalog crawling has no reason to be aggressive. The signals move on a daily-to-monthly cadence, not a per-second one.

Skip personal data. Reviewer names, profile pages, and user-generated content attached to identities are personal data under the GDPR. Availability and pricing are not. Collect the second, leave the first.

Consider buying the data. Several platforms and specialist vendors license catalog metadata commercially. For some teams that is cheaper and cleaner than building this, and the two approaches combine well: license the spine, crawl the territories and services the licence does not cover.

Keep provenance. Every claim in a deck should trace to a job_id and an observed_at. The teams that get burned are the ones that cannot reconstruct where a number came from six months later. If this feeds a wider intelligence programme, the patterns in using proxies for market research and data collection and proxies for news and media monitoring apply to storage and refresh policy too.