Proxies for Pharmaceutical and Healthcare Price Data

Before writing a scraper, check whether a regulator already made someone publish the number. In this vertical they usually did, and the file will be cleaner, dated, versioned and defensible in a way scraped HTML never is.

United States

The federal government publishes more drug and care pricing than any other regulator, and almost all of it is bulk-downloadable.

• NADAC, the National Average Drug Acquisition Cost, is a weekly survey of what retail pharmacies actually pay to acquire drugs, published on data.medicaid.gov with an Open Data API. It is the closest public proxy for real acquisition cost, and it is free.
• The FDA NDC Directory is the product spine: labeler, product, package, dosage form, route, marketing category, marketing start and end dates. Query it programmatically through openFDA.
• Hospital price transparency under 45 CFR Part 180 requires every US hospital to publish a machine-readable file of standard charges: gross charges, discounted cash prices, payer-specific negotiated charges, and de-identified minimum and maximum negotiated charges. The CY 2026 OPPS/ASC final rule moved all hospitals onto the v3.0 CMS schema from 1 January 2026, with enforcement from 1 April 2026, and it closed the biggest weakness in earlier versions: negotiated rates expressed as "120 percent of Medicare" must now be encoded as actual dollar amounts derived from EDI 835 remittance data.
• Transparency in Coverage obliges health plans to post monthly in-network rate files against a published JSON schema. Payer index files routinely point at hundreds of thousands of individual rate files, so plan for selective retrieval rather than a full mirror.
• The Medicare Drug Price Negotiation Program publishes maximum fair prices outright. CMS announced the 2027 MFPs for 15 additional drugs on 25 November 2025, taking the negotiated total to 25 products, with reductions running from 38 percent to 85 percent against 2024 list prices. Nobody needs to scrape those numbers.
• Medicare Part B ASP files land quarterly, and the Part B and Part D spending dashboards land annually.

Europe and other national systems

• The UK publishes the NHS Drug Tariff monthly, the English Prescribing Dataset as NHSBSA open data, and dm+d as the SNOMED-coded product dictionary.
• Australia publishes the full PBS Schedule monthly in structured downloads, including dispensed prices and item codes.
• France publishes the Base de données publique des médicaments with CIP13 codes and reimbursement rates.
• Germany's reference prices, the Festbeträge, come from GKV-Spitzenverband, though the underlying Lauer-Taxe article file remains a commercial licence.
• The EU-level EURIPID price collaboration exists but is restricted to public authorities, which is worth knowing before you budget for access you cannot get.

Global health and lower-income markets

• The WHO Model List of Essential Medicines, 24th list, was published on 5 September 2025 with 523 medicines for adults and 374 for children. It defines the basket most national procurement is measured against.
• The Global Fund Price and Quality Reporting tool publishes actual transaction prices paid by grant recipients, and UNICEF Supply Division publishes vaccine and commodity prices.

If a row on that table says no and you built a scraper for it anyway, you built a worse copy of a free file.

Three lines. They are not stylistic preferences and they do not have exceptions worth discussing.

No patient data of any kind. This work is product and price data. Data concerning health is a special category under Article 9 of the GDPR, which prohibits processing unless a narrow condition applies, and ordinary legitimate interest is not one of them. In the US, protected health information sits under HIPAA. A data collection vendor is usually not a covered entity, which is exactly why ingesting PHI is dangerous: you acquire the risk without the compliance apparatus that normally surrounds it. Design the pipeline so patient identifiers cannot enter, and drop any record that looks like one.

Nothing behind authentication on a prescription or telehealth portal. Patient portals, e-prescribing systems, pharmacy account areas and telehealth platforms are off limits, credentials or not. That is not a price dataset, it is a clinical record system.

Price comparison is legitimate. Facilitating unlicensed supply is not. Comparing publicly displayed pharmacy prices is ordinary market research and consumers benefit from it. Tooling that routes people toward unlicensed sellers, or toward cross-border prescription purchases they are not entitled to make, is a different activity with criminal exposure in many jurisdictions. Some markets also restrict how medicine prices may be advertised at all: German price competition on prescription medicines was constrained by the Arzneimittelpreisverordnung until the Court of Justice ruling in Deutsche Parkinson Vereinigung, C-148/15, and direct-to-consumer prescription advertising remains broadly prohibited outside the US and New Zealand. Collecting a price and republishing it as a consumer-facing offer are separate decisions with separate legal tests.

With the published set taken, four questions remain that no regulator answers.

Retail pharmacy pricing by region

Cash and out-of-pocket prices at pharmacy chains and online pharmacies are commercial decisions, displayed on public web pages and unavailable in bulk anywhere. They also vary by region inside a single chain, which is what surprises analysts arriving from a reimbursement background. A national average tells you little when the same pack is priced differently in three postcodes.

Generic versus branded spreads

The gap between an originator and its generics is the most useful competitive number in the vertical, and it only exists once you hold both sides at the same pack size, strength and channel. Reimbursement files give you one side, retail collection the other, and normalisation joins them. That is why the next section is the longest one here.

Formulary and availability differences by market

Whether a molecule is listed, at what tier and under what restriction differs sharply across markets, even inside the EU. Public formularies cover the listing decision. They rarely cover whether the product exists on a wholesaler's site this month.

Shortage tracking

Shortage registers are public and worth reading first: the FDA Drug Shortages Database, the EMA shortages catalogue, and the TGA Medicine Shortage Reports Database all publish declared shortages. What they do not do is move fast. The leading indicator is usually a wholesaler or pharmacy page quietly switching a presentation to unavailable, days or weeks before a formal notification appears, and that signal only exists if you sample those pages on a schedule.

Everything above is easy compared with joining it. Budget accordingly: on most pharma price projects, identifier work consumes more engineering time than collection does.

NDC, and the mistake almost everyone makes

The US National Drug Code is assigned as ten digits in three segments, and the segment lengths vary by labeler: 4-4-2, 5-3-2 or 5-4-1. Billing systems use an eleven-digit form in 5-4-2. Converting between them is not a hyphen strip. You have to zero-pad the deficient segment, and which segment is deficient depends on the original configuration.

def ndc10_to_ndc11(ndc: str) -> str:
    """Convert a hyphenated 10-digit NDC to the 11-digit 5-4-2 billing form."""
    labeler, product, package = ndc.split("-")
    fmt = (len(labeler), len(product), len(package))
    if fmt == (4, 4, 2):
        labeler = "0" + labeler
    elif fmt == (5, 3, 2):
        product = "0" + product
    elif fmt == (5, 4, 1):
        package = "0" + package
    else:
        raise ValueError(f"unrecognised NDC configuration {fmt} in {ndc}")
    return labeler + product + package

assert ndc10_to_ndc11("0002-8215-01") == "00028215010"   # 4-4-2
assert ndc10_to_ndc11("50242-051-21") == "50242005121"   # 5-3-2
assert ndc10_to_ndc11("00069-0090-1") == "00069009001"   # 5-4-1

Strip the hyphens instead and 0002-8215-01 becomes 0002821501, which either fails to join or, worse, joins to the wrong product. That failure is silent. It raises nothing, it just quietly degrades a spread analysis. Treat any NDC arriving without hyphens as ambiguous until you resolve its configuration against the NDC Directory.

GTIN and national pack codes

Outside the US, packs carry a GS1 GTIN plus a national code. Under the EU Falsified Medicines Directive, Delegated Regulation (EU) 2016/161, applicable since 9 February 2019, prescription packs carry a 2D Data Matrix encoding product code, serial number, batch and expiry, plus a national reimbursement number in several countries. Those national codes are what local price files key on: PZN in Germany, CIP13 in France, AIC in Italy, Código Nacional in Spain, dm+d AMPP codes in the UK. None derive from one another. A GTIN bridges many of them, but coverage is imperfect, so measure your match rate rather than assuming it.

ATC and DDD for cross-market comparison

To compare across markets you eventually stop comparing packs and start comparing molecules. The ATC classification from the WHO Collaborating Centre for Drug Statistics Methodology gives a five-level code, and the Defined Daily Dose gives a unit that makes a 30-pack of one strength comparable with a 90-pack of another. Price per DDD is the cross-market price metric that survives contact with different pack conventions. Price per pack is not.

One practical rule: store every identifier you receive, never overwrite one with a normalised form, and keep a separate crosswalk table with a confidence score per link. Analysts will ask how a join was made, and "we normalised it" is not an answer that survives a market access review.

For the published files, often they do not. Download NADAC directly and be done. Proxies earn their place in four specific situations.

Geography decides the price and the catalogue. A pharmacy chain operating in six countries serves a different catalogue, currency, reimbursement note and price per country, resolved from your IP before you click anything. Read the Spanish site from a German IP and you get the German answer. Same country-level resolution as geo-targeting in proxies, applied where the wrong answer looks perfectly valid and is therefore hard to catch.

National portals restrict foreign traffic. Several national reimbursement systems and pharmacy locators either block or heavily throttle requests from outside the country, and some serve a reduced dataset to foreign IPs instead of an error, which is the worse failure mode.

Volume against one origin gets throttled. Even friendly public APIs have ceilings: openFDA allows 240 requests per minute and 1,000 per day per IP without a key, rising to 120,000 per day with one. File discovery across thousands of hospital domains, or shortage polling across several registers on a tight cadence, meets per-origin limits a single office IP cannot clear politely.

Longitudinal integrity. A price series is only comparable if the observation conditions stay constant. If your exit country drifts between runs, a movement in your chart may just be a different storefront. Pin the country and record it as a column on every row.

A corollary worth saying plainly: aggressive high-volume crawling of a national health portal is the wrong instinct here. Rate-limit conservatively, cache hard, and read ethical scraping and rate limiting before scaling any job pointed at public health infrastructure.

The base endpoint is https://scrape.sparkproxy.io/api/v1 and every request authenticates with the X-API-Key header. Target URLs below use sparkproxy.io demo paths. Swap in the real public source you are collecting from.

A country-pinned fetch of a public pharmacy price page, rendered and returned as JSON:

curl -X GET "https://scrape.sparkproxy.io/api/v1?url=https://www.sparkproxy.io/demo-pharmacy/es/producto/ibuprofeno-600&country_code=ES&premium_proxy=true&render_js=true&format=json" \
  -H "X-API-Key: sk-xxxxxxxxxxxxxxxx"

Pull structured fields instead of raw HTML with extract_rules, so the output lands in your warehouse with identifiers already separated from the price:

import requests, json

API = "https://scrape.sparkproxy.io/api/v1"
API_KEY = "sk-xxxxxxxxxxxxxxxx"

payload = {
    "url": "https://www.sparkproxy.io/demo-pharmacy/es/producto/ibuprofeno-600",
    "country_code": "ES",
    "premium_proxy": True,
    "render_js": True,
    "format": "json",
    "extract_rules": json.dumps({
        "product_name":  ".pdp h1",
        "national_code": ".pdp .codigo-nacional",
        "pack_size":     ".pdp .presentacion",
        "price":         ".pdp .precio",
        "availability":  ".pdp .disponibilidad",
        "rx_status":     ".pdp .receta",
    }),
}
r = requests.get(API, headers={"X-API-Key": API_KEY}, params=payload)
print(r.json())

Some formulary and reimbursement lookups only render after a region or insurer is selected. Drive that with js_scenario and confirm the result loaded with wait_for rather than guessing at a sleep:

scenario = {
    "steps": [
        {"click": "#region-select"},
        {"fill": ["#region-input", "Cataluña"]},
        {"click": "#apply-region"},
        {"wait": 1200},
    ]
}

payload = {
    "url": "https://www.sparkproxy.io/demo-formulary/lookup",
    "country_code": "ES",
    "premium_proxy": True,
    "render_js": True,
    "js_scenario": json.dumps(scenario),
    "wait_for": ".formulary-results",
    "format": "json",
}
r = requests.get(API, headers={"X-API-Key": API_KEY}, params=payload)

Shortage notices are usually prose, and Markdown is a better input to a diff or a classifier than a wall of HTML:

curl -X GET "https://scrape.sparkproxy.io/api/v1?url=https://www.sparkproxy.io/demo-registry/shortages/current&country_code=GB&format=md" \
  -H "X-API-Key: sk-xxxxxxxxxxxxxxxx"

Long-running bulk pulls should not block a worker. Hand SparkProxy a callback_url and take the 202, and set tag so every row can be traced back to the run that produced it:

payload = {
    "url": "https://www.sparkproxy.io/demo-registry/reimbursement-list-2026.csv",
    "country_code": "FR",
    "premium_proxy": True,
    "render_js": False,
    "tag": "reimbursement-fr-2026-08-18",
    "callback_url": "https://www.sparkproxy.io/webhooks/price-ingest",
}
r = requests.get(API, headers={"X-API-Key": API_KEY}, params=payload)
print(r.status_code)  # 202, result POSTed to the callback on completion

Where a site loads its prices from an internal JSON endpoint, calling that endpoint directly is cheaper and far more stable than parsing a rendered page. The technique is covered in scraping hidden JSON API endpoints, and it applies unusually well here because pharmacy catalogues are heavily templated.

This is the part most teams reinvent badly, so it is worth spelling out.

CMS did not only mandate that hospitals publish a price file. It mandated how you find it. Since 1 January 2024, every hospital must place a plain-text file named cms-hpt.txt in the root of its website containing the hospital location name, the URL of the page hosting the file, the direct link to the machine-readable file itself, and a contact email. The file follows a naming convention of __standardcharges.[json|csv].

That turns discovery from a crawling problem into one cheap request per domain:

import requests

API = "https://scrape.sparkproxy.io/api/v1"
API_KEY = "sk-xxxxxxxxxxxxxxxx"

def find_mrf(domain: str):
    r = requests.get(
        API,
        headers={"X-API-Key": API_KEY},
        params={
            "url": f"https://{domain}/cms-hpt.txt",
            "country_code": "US",
            "render_js": False,
            "format": "json",
            "transparent_status_code": True,
            "tag": "hpt-discovery",
        },
        timeout=60,
    )
    if r.status_code != 200:
        return {"domain": domain, "status": "no-hpt-file"}
    return {"domain": domain, "status": "found", "body": r.json()}

Set render_js to false, because a text file needs no browser and rendering it wastes time and money. Set transparent_status_code so a 404 reaches you as a 404 rather than masked by a success wrapper. Then fetch only the files you need: one large health system's MRF can run to several gigabytes, and there is no reason to mirror the country.

The same discipline applies to Transparency in Coverage. Read the payer index first, filter to the plans and geographies in scope, then pull only those rate files. Teams that skip the index step end up with a storage bill and no analysis.

Two design notes carry most of the value. Compute spreads at ATC and DDD level, not pack level, or your cross-market numbers turn into pack-size artefacts. And treat the audit log as non-negotiable: in a regulated function an unexplainable number is a wrong number, so every price should carry the URL, exit country, timestamp and run tag that produced it.

The staged cadence keeps cost sane the same way it does in other price verticals. Cheap published loads run wide and often, and the expensive geo-pinned retail requests run only where they add what the published files cannot, a pattern that also underpins proxies for grocery and delivery prices and general price comparison site builds.

Collecting publicly displayed product and price information is generally treated as distinct from unauthorised system access. In the US, hiQ Labs v. LinkedIn held that scraping publicly available data does not violate the Computer Fraud and Abuse Act, and a price displayed to any visitor is public in that sense. That is a footing, not a blanket permission.

Four traps specific to this vertical:

• Publication is regulated even where collection is not. In several jurisdictions you may lawfully collect a medicine price and still be barred from advertising it to consumers. Treat the publication decision as a separate legal review per market.
• Reimbursement data carries licence terms. Public does not mean unlicensed. Some national files impose attribution or redistribution conditions, and commercial article files such as Lauer-Taxe are licensed products regardless of how the data reaches you.
• Reviews and clinician pages are people data. A pharmacy review with a name attached is personal data, and if it describes a condition it is Article 9 health data. Exclude the field at parse time rather than planning to filter it later.
• Do not transact. Read public catalogue and price pages. Never place, modify or cancel an order, and never complete a prescription workflow.

Academic and health economics work adds one more expectation: reproducibility. Archive the raw response alongside the parsed row so a finding can be re-derived years later, a discipline covered in proxies for academic research data collection.