Proxies for Telecom Plan and Roaming Price Checks

Buy on country coverage first, request volume last. A tariff feed covering a few hundred operators generates a trivial number of requests and needs single-digit concurrency, so the plan that wins is the one that can exit from every market you track, reliably, on the same schedule.

Two shapes work. A flat datacenter plan gives you unmetered collection from a fixed set of countries: SparkProxy's plans run $75 a month for 100 threads up to $440 for 1000, unlimited bandwidth, across 1M+ IPs in 80+ countries. A managed endpoint gives you rendering and per-request country selection without running anything: the Scraping API starts at 1,000 free credits with no card, then $49 a month for 250,000 credits. The deciding question is whether the carriers you track serve their tariff pages to hosting address space at all, which the last section turns into a checklist.

Telecom pricing is the most heavily segmented retail category on the public web, and each axis of segmentation is a way for a naive scraper to record a number that is true and useless.

1. Country. Multinational groups run separate sites per market with unrelated portfolios, and some route a single domain to a country variant at the edge. Arriving from the wrong place gets you another market's catalogue, or a redirect you may not notice because it returns 200.
2. Region within a country. Coverage-dependent products, regional operators and store-specific offers all vary below national level, usually behind a postcode field rather than IP detection.
3. Contract shape. The same plan name exists as prepaid, SIM-only, 12 month, 24 month and handset-bundled, at five different prices. A price without its contract shape is not a data point.
4. Promotional window. First-N-months discounts, limited-time offers with a countdown, and win-back pricing that appears only on certain entry paths.
5. Eligibility. Student, senior, family, business, existing-customer and port-in prices, sometimes shown to everyone and sometimes gated behind a selector.
6. Device subsidy. A bundled handset moves the monthly figure by more than any tariff difference, so the monthly price on a bundled page tells you about the phone, not the plan.

The consequence for your schema is immediate: a telecom price record needs at minimum country, region granularity, contract length, upfront cost, promotional structure, eligibility class and whether a device is included. Six of those seven are not the price. A single price column is how telecom datasets become unusable within a quarter.

Carriers advertise the lowest number that is defensibly true. Comparing those numbers across operators compares marketing decisions rather than prices. The only comparable figure is the effective monthly cost across the full committed term, and you have to compute it because nobody publishes it.

Take a plan advertised at "from $15 a month". The page's small print, which is where the real terms live, says the $15 rate applies for the first 6 months, the standard rate of $40 is charged for the remaining 18 months of a 24 month agreement, and there is a $129 upfront charge.

• Promotional period: 6 months at $15 is $90.
• Standard period: 18 months at $40 is $720.
• Upfront: $129.
• Total cost of ownership across the term: $939.
• Effective monthly cost: $939 divided by 24 is $39.13.

The advertised figure understates the real one by a factor of 2.6. Those inputs are an illustrative construction, not a real operator's terms, but the shape is entirely ordinary in this category, and it is why a feed that stores only the headline number produces confident nonsense.

Three rules follow:

Store the components, compute the comparison. Keep promotional rate, promotional duration, standard rate, term length and upfront charge as separate fields. Derive effective monthly cost at query time so that a change in your normalisation does not require a re-scrape.

Treat an unknown term as a null, not a zero. If you cannot extract the contract length, the effective cost is unknown. Recording the headline price with a missing term silently pollutes every average that touches it.

Capture the small print, not just the price block. The qualifying conditions are usually in a footnote or an expandable panel rendered after page load, and they are the difference between a data point and a guess.

Roaming pricing behaves nothing like tariff pricing and should be collected by a separate pipeline.

It is usually published as a zone table: a list of countries mapped to zones, and a price per zone for calls, texts and data, often with a daily-pass alternative. Two properties make it awkward:

It frequently lives in a PDF. Carrier roaming rates are commonly published as a downloadable document that gets replaced in place, at the same URL, with no visible version marker. A byte-level hash of the file is your change signal, and you must keep every version you download, because the previous one is gone the moment they upload the new one.

The zone map matters as much as the prices. A carrier can leave every zone price unchanged and move twenty countries between zones, which changes the price of roaming in those countries without changing a single published number. If you only extract the price table and not the country-to-zone mapping, you will miss the most commercially significant changes this dataset contains.

Regional regulation adds a third wrinkle. Where a regulator requires domestic-equivalent pricing within a bloc, the published roaming rate for those destinations is not a price so much as a statement of a fair use allowance, and the number that actually matters is the volume threshold at which surcharges begin. Collect the allowance, not just the rate, and record the regulatory regime alongside it so cross-bloc comparisons do not silently mix two different kinds of number.

Fetching the PDF itself is a plain request. There is no need to render anything:

curl -G "https://scrape.sparkproxy.io/api/v1" \
  -H "X-API-Key: YOUR_API_KEY" \
  --data-urlencode "url=https://example.com/roaming/zone-rates.pdf" \
  --data-urlencode "render_js=false" \
  --data-urlencode "country_code=DE"

That is 6 credits: 1 for the plain fetch plus 5 for the country-targeted exit. Archive the bytes as received, because a re-download next quarter will not give you the same file. The same discipline as web archiving and compliance capture.

In most price-monitoring categories, geo-targeting is a cost you apply selectively to the pages that genuinely vary. Telecom inverts that. Almost every page you want varies by country, because the product is sold per market by an operator licensed in that market, so the geo add-on applies to essentially the whole workload and the arithmetic has to be planned around it rather than trimmed.

On SparkProxy's Scraping API, country_code adds 5 credits to whatever the base request costs. A plain fetch is 1 credit and a JavaScript-rendered fetch is 5, so:

The third row is the one that catches people. Many carrier tariff pages only reveal a price after the visitor selects a term, a data allowance or a device, which means a scenario step rather than a single load. That is another 5 credits on top, and it applies per request, not per session. Where the price is reachable by URL parameter instead of by clicking, use the parameter: it is the difference between 15 credits and 10.

Region below country level is usually not an IP problem. A postcode field, a store selector or a region cookie is the mechanism most carriers use, and those you set directly. Validating that your exit really is where the provider claims is worth doing once per market, using the method in validating IP geolocation accuracy, because a mislabelled exit produces a market's worth of wrong data before anyone notices. The general mechanics are in what geo-targeting means in proxies.

Assumptions below are chosen to be plausible for a multi-market tariff feed. They are model inputs, not measurements from any test we ran.

Coverage. 25 markets, averaging 10 operators each including MVNOs, so 250 operators. Each operator contributes 20 tariff pages and 4 roaming documents, so 6,000 URLs in total: 5,000 tariff pages and 1,000 documents.

Refresh. Tariff pages daily, because promotions appear and expire on short notice. Roaming documents weekly, because they do not.

• Tariff requests: 5,000 a day, 150,000 across a 30 day month.
• Roaming requests: 1,000 a week, about 4,300 a month.
• Total: roughly 154,300 requests a month. That is a small feed by any measure.

Concurrency, which is almost nothing. Run the daily tariff sweep inside a four hour window and you need 5,000 requests across 14,400 seconds, about 0.35 a second. At 6 seconds for a rendered configurator page, that is 0.35 times 6, roughly 2 connections in flight. Compress the window to one hour and it is 1.39 a second, about 8 connections. Either way you are nowhere near the 100 threads on the smallest flat plan. Sizing method in how many proxies you need for scraping.

Credits, where the money actually goes. Assume every tariff page needs rendering and a country-targeted exit, at 10 credits, and every roaming document is a plain country-targeted fetch at 6 credits.

• Tariff: 150,000 times 10 is 1,500,000 credits.
• Roaming: 4,300 times 6 is 25,800 credits.
• Total: 1,525,800 credits a month, which needs the Pro tier at $249 for 3,000,000 credits and 200 concurrent.

Now price the same feed with the country targeting removed, purely to see what the geo requirement costs: 150,000 rendered fetches at 5 credits is 750,000, plus 4,300 plain fetches at 1 credit, about 754,300 in total, which fits the Growth tier at $99 for 1,000,000.

Geo-targeting therefore just over doubles the credit consumption and moves the plan from $99 to $249. That is not a saving you can take, because the cheaper version collects the wrong country's prices. It is the honest price of correctness in this category, and it is worth knowing before you build a business case on the $99 figure. Full mechanics in scraping API credit pricing.

The flat alternative changes the calculus if your markets are covered: at $75 a month for 100 threads and unlimited bandwidth across 80+ countries, a datacenter plan costs less than either credit tier for this volume. The catch is in the next section but one.

This is the modelling error that quietly ruins telecom datasets, and it has nothing to do with scraping.

Mobile virtual network operators resell capacity on somebody else's radio network. A price comparison that treats an MVNO and its host as independent competitors will report a competitive market that does not exist, and will miss the fact that a host network's wholesale change propagates to a dozen retail brands at once. Some groups run several brands on one network deliberately, priced to occupy different segments.

Three fields fix it, and all three have to be maintained by hand because no carrier publishes them in machine-readable form:

• Host network for every retail brand in your set.
• Ownership of the brand, which is often the same group as the host and sometimes not.
• Network access class, since some MVNOs get full-speed access and others are contractually de-prioritised, which makes a headline data allowance mean different things on two brands at the same price.

Budget for that table being wrong. Brands get sold, wholesale agreements get renegotiated, and the only reliable signal is periodic manual review. If your comparison output ever ranks operators, this table decides whether the ranking means anything.

The same applies to the fair use footnotes on plans sold as unlimited. A plan with a de-prioritisation threshold and a plan without one are different products at the same advertised price, and the threshold is almost always in the footnote rather than in the price block. Extract it as a field.

Telecom pricing is a category where somebody eventually disputes your figure, usually the operator. Build for that from the first row.

Store, per observation:

1. Identity fields. Operator, brand, host network, market, and the URL as requested.
2. Exit country actually used. Not the country you intended. The one the request went out through.
3. Contract shape. Term length, prepaid or postpaid, SIM-only or device-bundled, eligibility class.
4. Price components. Promotional rate, promotional months, standard rate, upfront charge, any recurring add-ons.
5. Allowance fields. Data, calls, texts, plus any fair use or de-prioritisation threshold in the footnote.
6. Evidence. The raw response, and a rendered capture where the offer only makes sense in layout. A screenshot or PDF costs 5 credits on top of a rendered fetch, so attach it to confirmed changes rather than to every poll.
7. Timestamp in UTC and the parser version that produced the extraction.

Change detection on these pages needs unusually careful normalisation, because carrier pages are full of things that change on every load: countdown timers on promotions, rotating hero banners, session identifiers and personalisation slots. Hash the extracted fields rather than the page body, or strip aggressively before hashing, or your alerting is noise by the end of the first week. The techniques are in incremental scraping and change detection, and the wider pattern in building an automated price monitoring system.

One more detail specific to this category. Where a price only appears after selecting options, record the selection path alongside the price. Two analysts reproducing your number by hand will otherwise reach different figures and neither will be able to tell who is wrong.

Run these four checks against your actual target markets before spending anything. They take an afternoon and they decide the purchase.

Check one: does the carrier serve hosting address space? Fetch three tariff pages from a datacenter exit in the right country and compare against the same pages from a normal connection. If they match, a flat datacenter plan is the cheapest correct answer for this workload. If the datacenter request gets a challenge or a stripped page, read residential vs datacenter proxies before you buy anything, because no amount of concurrency fixes it.

Check two: is every market you need actually available? SparkProxy covers 80+ countries including 50,000+ US addresses, which is broad, and it is still not every market. Get the list, compare it against your target markets, and identify the gaps before you build the pipeline around a provider. For the markets nobody covers from a hosting network, your options narrow to residential or carrier-network exits, the latter being a category SparkProxy does not sell at all.

Check three: does the price need a click? Count how many of your tariff pages reveal the price only after a selection. That fraction decides whether you are paying 10 credits a page or 15, and it is often reducible by finding the URL parameter or the underlying JSON endpoint the configurator calls. Hidden JSON API endpoints has the method.

Check four: pace it like a shopper. One sweep a day per tariff page is invisible. Anything approaching per-minute polling on a retail carrier site is both rude and self-defeating, since it is the fastest way to get your address range classified. The reasoning is in ethical scraping and rate limiting.

For the adjacent problem of verifying that a carrier's own site renders correctly for customers in each market, which is a testing job rather than a competitive intelligence one, localisation testing covers the setup, and the broader market-research framing is in using proxies for market research.