Proxies for Government Tender and Procurement Data

In commercial scraping, the site owner's interest runs against yours. In procurement, the publishing body's legal interest runs with yours. Transparency is the whole point of the register.

The EU rule is Directive 2014/24/EU, which requires contract notices above the financial thresholds to be sent to the Publications Office and published in the Supplement to the Official Journal. Since 25 October 2023 those notices must use the eForms schema set by Commission Implementing Regulation (EU) 2019/1780, which replaced the 2015 standard forms. In the US, FAR Part 5 requires agencies to publicise proposed contract actions on the governmentwide point of entry, and the Digital Accountability and Transparency Act of 2014 pushed award and spending data into a standardised public feed. The UK's Procurement Act 2023 went live on 24 February 2025 and made Find a Tender the central digital platform for notices.

Three practical consequences follow, and each one saves work:

• The schema is stable because a regulation defines it. eForms fields do not get renamed in a Tuesday redesign. Compare that to a retail site where a CSS class change breaks your extractor overnight.
• Completeness is auditable. If a notice above threshold is missing from TED, that is a compliance failure somebody can be asked about. You are not guessing whether you got everything.
• Blocking you is off-mission. Central registers rarely run bot defences of the kind you meet on airline or ecommerce sites, because turning away automated readers defeats the register's purpose.

None of this makes the job trivial. It relocates the difficulty from getting past the door to finding the doors that exist, which is a research problem more than an engineering one. The same shift shows up in other public-record work, and if you have built a filings pipeline before, the shape will feel familiar. Our walkthrough of how to scrape SEC EDGAR filings applies the same "official bulk feed first" instinct to corporate disclosure.

Start with this map. Most teams reinvent it badly.

Read that table top to bottom and the pattern is obvious. Coverage quality falls off a cliff the moment you drop below the national threshold. TED is a solved data problem. A county utility board publishing a bid opportunity as a PDF link on a page that has not been redesigned since 2011 is not.

What each source is actually good for

TED is the reference set for EU cross-border opportunities and for comparative research on award patterns. SAM.gov is the notice side of US federal buying, including sources-sought and presolicitation stages that hand you weeks of lead time before a formal solicitation appears. USAspending is the money side, better when your question is "who won" rather than "what is open". Find a Tender is unusually pleasant to work with because the UK committed to a standard format rather than a bespoke one, so its output drops straight into an open contracting pipeline.

UNGM is the awkward one. The UN Global Marketplace aggregates opportunities across agencies, but much of the tender detail sits behind supplier registration and there is no comprehensive open bulk export. Expect a session-based collection path rather than a file download.

Say it plainly: where an official bulk download or API exists, use it. It is faster, it is authorised, it gives you fields the HTML never rendered, and it does not break on a redesign. Scraping is the fallback for portals that publish no such thing.

Pulling a day of TED notices takes one request and no credentials:

# TED publishes daily XML packages keyed by the OJ S number (year + issue).
# No account, no key, no rate-limit games.
curl -L -o ted-2026-155.tar.gz \
  "https://ted.europa.eu/packages/notice/daily/202600155"

tar -tzf ted-2026-155.tar.gz | head

USAspending is similarly open. No key at all:

import requests

# US federal award data. No API key, no auth header.
r = requests.post(
    "https://api.usaspending.gov/api/v2/search/spending_by_award/",
    json={
        "filters": {
            "award_type_codes": ["A", "B", "C", "D"],
            "time_period": [{"start_date": "2026-07-01", "end_date": "2026-08-18"}],
        },
        "fields": ["Award ID", "Recipient Name", "Award Amount", "Awarding Agency"],
        "page": 1,
        "limit": 100,
    },
    timeout=60,
)
for row in r.json()["results"]:
    print(row["Award ID"], row["Recipient Name"], row["Award Amount"])

The SAM.gov caveat nobody warns you about

Here is the one that catches teams out, and it is the strongest argument for reading rate-limit docs before you design an ingestion schedule. The SAM.gov Get Opportunities Public API is keyed through api.data.gov, and the daily call budget depends entirely on what kind of account issued the key. A personal, non-federal key gets a daily allowance in the single or low double digits. A system account gets roughly a thousand calls a day. Government accounts on a .gov or .mil address get ten thousand.

That is a per-day budget, not a per-second one. Paginate naively at 100 records a page across a wide date range and a personal key is exhausted before breakfast, with every later call returning 429 until the daily reset. The design response is not more IP addresses, because the limit is bound to the key rather than the address. It is: request a system account, pull the bulk CSV extracts for backfill, and spend your small API budget only on the recent window where freshness actually matters.

Work down that table in order. Most projects answer 80 percent of their question at the first two rows.

Once you pull from more than one country you have a schema problem: eForms XML from TED, OCDS JSON from the UK, a bespoke JSON envelope from SAM.gov, and hand-parsed HTML from everywhere else. Normalise all of it into one shape, and the shape you should pick already exists.

The Open Contracting Data Standard is maintained by the Open Contracting Partnership and sits at version 1.1 (1.1.5 as the current patch). Its core ideas are worth internalising even if you never publish OCDS yourself:

• The ocid is a globally unique identifier for a whole contracting process, not for a single document. Every notice, amendment, award and contract belonging to the same procurement carries the same ocid. This is the field that turns a pile of disconnected notices into a lifecycle.
• Releases and records. A release is an immutable event ("this award happened"). A record is the compiled current state of everything filed under one ocid. Store both. The release stream is your audit trail; the record is what you query.
• Stage tags. planning, tender, award, contract, implementation. Tagging incoming data by stage is what lets you ask "how many tenders never reached award" without special-casing every source.

Even a rough mapping pays for itself. Give each source an adapter that emits an OCDS-shaped release and everything downstream gets simpler:

def to_ocds_release(source, notice):
    """Minimal OCDS 1.1 release. Adapters differ; the output shape does not."""
    return {
        "ocid": f"ocds-{source.prefix}-{notice['reference']}",
        "id": notice["notice_id"],
        "date": notice["published_at"],          # ISO 8601, UTC
        "tag": ["tender"],
        "initiationType": "tender",
        "language": notice.get("language", "en"),
        "buyer": {"name": notice["authority"]},
        "tender": {
            "id": notice["reference"],
            "title": notice["title"],
            "status": "active",
            "value": {"amount": notice.get("value"), "currency": notice.get("currency")},
            "tenderPeriod": {"endDate": notice["deadline"]},
            "items": [{"classification": {
                "scheme": notice.get("scheme", "CPV"),
                "id": code,
            }} for code in notice.get("cpv_codes", [])],
        },
    }

The deliberate choice there is that deadline is a required field in your adapter contract rather than an optional one. A tender record without a submission deadline is close to useless to the people who need it most.

Everything above is the easy half. The hard half is that the world's procurement spend does not live in six central registers.

Below the EU thresholds and below national publication rules, buying moves to a long tail: municipalities, school districts, transit authorities, water boards, regional health bodies, port authorities, state universities. Each publishes somewhere, usually a page on the authority's own website or a seat on a regional e-procurement platform serving a few dozen bodies. There are thousands of these across Europe and North America alone, and the honest description of their data maturity is a table of links, sometimes an RSS feed if you are lucky, and the actual specification as a PDF attachment.

This is precisely where an SME bidder loses. The contracts they can realistically win are exactly the ones too small to hit a central register.

What breaks at the municipal layer

• No identifiers. No ocid, often no stable notice reference. Deduplication has to key on a hash of buyer plus title plus deadline, because the URL changes whenever the page is regenerated.
• PDF-first publication. The web page carries a title and a date. The scope, the value, the evaluation criteria and the contact all sit in an attached PDF or Word document. Text extraction is not optional here, and our guide on how to extract data from PDFs covers the layout-preservation problem you will hit with tabular bid schedules.
• Session-driven search. Many regional platforms will not render a notice list at a plain URL. You submit a search form, accept a cookie banner, then page through results with state held server side.
• Silent removal. Notices disappear the day after the deadline with no archive. If you did not capture it, it is gone. Snapshot the HTML and the attachments at fetch time.
• Fragile hosts. These are small servers. A polite crawl is not just courtesy; hammering a county web server is how you get an address range banned and, deservedly, a complaint.

Aggregate scale is the reason this is worth automating at all. One municipal portal is a manual bookmark. Twelve hundred of them, polled twice a day, is a data product. That is the part where distributed collection stops being decoration and becomes the actual mechanism.

Tender data has an expiry date printed on it. A notice with a Friday submission deadline is a live business opportunity on Monday, a scramble on Thursday, and landfill on Saturday. Compare that to price monitoring, where yesterday's number still tells you something.

This reorders the usual priorities. Throughput matters less than latency. A pipeline that ingests every notice in Europe with a 36-hour lag is worse, for a bidder, than one covering a hundred relevant municipalities within the hour.

Two implementation notes matter more than they sound. First, treat the amendment stream as a first-class object rather than an update to a row. Clarification rounds routinely move a deadline or change a requirement, and a bidder needs the diff, not just the current state. Second, compute a days_to_deadline field at ingest and let alerting key on it. Almost every useful notification in this domain is a function of that number crossing a boundary.

You cannot match a supplier to opportunities by keyword. "Groundworks" in one notice is "earthmoving" in another and "Erdarbeiten" in a third. Classification codes are the join key, and there are two you will meet.

Working rules for using them:

• Truncate before you match. CPV is hierarchical, so 45233000 (construction work for roads) rolls up to 45230000 and then to 45000000. Store the full code, index the 3, 5 and 8 digit prefixes, and let users subscribe at whatever level they want. Exact-code matching alone produces terrible recall, because authorities pick codes inconsistently.
• Expect several codes per notice. Notices carry one main CPV plus additional ones. Match on any of them.
• Do not trust the code blindly. Miscoding is common, usually accidental and occasionally deliberate. Keep a text-similarity fallback on the title and scope so a badly coded notice still surfaces.
• Cross-walking CPV to UNSPSC is lossy. There is no clean one-to-one mapping. Build a mapping table for the categories your users actually care about, and accept that a general mapping will misfire.

TED publishes notices in the language of the contracting authority, with structured fields available across EU languages. That sounds like it solves multilingualism. It does not, for one reason: the structured fields are translated and the free text usually is not.

Title, CPV code, buyer country, value and deadline are reliable across languages because they are typed fields. The description of what is being bought, the eligibility criteria and the specification attachment stay in the original language. So a cross-border matching system should:

1. Filter and rank on typed fields, which are language-independent by construction. CPV plus country plus value band plus deadline gets you most of the way.
2. Use original-language text only for secondary scoring, and record which language it was in.
3. Machine-translate for human display, never for the matching key. Translated text as a join key introduces drift you cannot debug later.
4. Keep the original document. If a bid gets challenged, the original-language version is authoritative.

At the municipal layer none of this is handled for you. A Spanish provincial council publishes in Spanish, and possibly in Catalan or Galician too. Detect the language at ingest and store it as a field rather than inferring it from the country code.

Here is the honest version of the proxy argument for this use case, without the overselling.

For the central registers you do not need proxies at all. TED serves anyone, USAspending serves anyone, the UK feeds serve anyone. Adding a proxy to those requests buys you nothing.

Three situations at the long-tail layer are different.

Portals that only serve their own country. A meaningful number of national and regional procurement systems geo-filter at the edge, sometimes as deliberate policy and more often as a blunt anti-abuse measure applied by a hosting provider or a national CDN. Requests from outside the country get a block page, a redirect to a generic landing page, or a timeout. There is no header you can set to fix that. The request has to originate from an address in the country. This is the one genuinely structural reason a geo-distributed exit matters here, and it is worth understanding what geo-targeting means in proxies before assuming a country parameter is cosmetic.

Localised content behind the same URL. Some regional platforms serve a different notice list, or a different default language, based on the requester's location. Fetching from inside the region gives you what a local supplier sees.

Polite distribution across a long tail. Twelve hundred small municipal servers polled several times a day from one address looks like exactly one thing to a network operator. Spreading that across a pool is as much about not concentrating load as about avoiding blocks. Rate limiting yourself per host is the more important half of that, and our guide on ethical scraping and rate limiting sets out cadences that keep a crawler welcome.

That is the whole case. Anyone telling you that you need a large residential pool to collect TED notices is selling you something you do not need.

For the portals with no feed, here is what the fetch layer looks like using the SparkProxy Scraping API. Markdown output is a good default for notice pages, because these are text documents and you want prose plus links rather than a wall of layout div elements.

curl -X GET "https://scrape.sparkproxy.io/api/v1?url=https%3A%2F%2Fexample-municipality.gov%2Ftenders&render_js=true&format=md" \
  -H "X-API-Key: YOUR_API_KEY"

For a geo-locked national portal, add country_code and route through the residential pool:

import requests

API = "https://scrape.sparkproxy.io/api/v1"
KEY = "YOUR_API_KEY"

def fetch_notice_list(url, country=None, premium=False):
    params = {
        "url": url,
        "render_js": "true",
        "format": "md",
        "block_resources": "true",   # notices are text; skip images and fonts
    }
    if country:
        params["country_code"] = country      # ISO 3166-1 alpha-2
    if premium:
        params["premium_proxy"] = "true"
    r = requests.get(API, headers={"X-API-Key": KEY}, params=params, timeout=120)
    r.raise_for_status()
    return r.text

md = fetch_notice_list(
    "https://portal-de-contratacion.example/anuncios",
    country="es",
    premium=True,
)

Structured extraction is worth setting up per portal, because a notice board is a repeating list and extract_rules gets you rows instead of prose:

rules = {
    "notices": {
        "selector": "table.tender-list tbody tr",
        "type": "list",
        "output": {
            "reference": "td.ref",
            "title": "td.title a",
            "url": {"selector": "td.title a", "type": "href"},
            "buyer": "td.authority",
            "deadline": "td.closing-date",
        },
    }
}

r = requests.post(
    API,
    headers={"X-API-Key": KEY, "Content-Type": "application/json"},
    json={
        "url": "https://example-municipality.gov/tenders",
        "render_js": True,
        "extract_rules": rules,
    },
    timeout=120,
)
rows = r.json()["extracted"]["notices"]

Platforms that hide the list behind a search form need a scenario. This is the common shape: dismiss the cookie banner, set a date filter, submit, wait for results.

scenario = {
    "instructions": [
        {"click": "#cookie-accept"},
        {"wait": 500},
        {"fill": {"selector": "#dateFrom", "value": "2026-08-11"}},
        {"click": "#btnSearch"},
        {"wait_for": "table.tender-list tbody tr"},
        {"scroll": 1200},
    ]
}

r = requests.post(
    API,
    headers={"X-API-Key": KEY, "Content-Type": "application/json"},
    json={
        "url": "https://regional-eprocurement.example/search",
        "render_js": True,
        "js_scenario": scenario,
        "extract_rules": rules,
        "tag": "regional-eproc-daily",
    },
    timeout=180,
)

For a wide municipal sweep, run it asynchronously so one slow host does not stall the batch. Pass callback_url and take the 202:

r = requests.post(
    API,
    headers={"X-API-Key": KEY, "Content-Type": "application/json"},
    json={
        "url": target,
        "render_js": True,
        "format": "md",
        "callback_url": "https://ingest.sparkproxy.io/hooks/tender-fetch",
        "tag": f"muni:{authority_id}",
    },
    timeout=30,
)
assert r.status_code == 202       # {"job_id": "...", "status": "queued"}

The tag field is doing quiet work in those last two examples. With a thousand-plus sources you need to know which authority a callback belongs to without parsing the URL back out, and the tag is echoed in the response.

Downstream, the rest of the pipeline is unremarkable and should stay that way: hash-based deduplication, attachment download and text extraction, CPV assignment, mapping to the OCDS shape from earlier, and a deadline-driven alerting job. If you are folding tender signals into a wider competitive picture, our guide on using proxies for market research and data collection covers the sampling and storage side.

This is public-interest data with an unusually clean footing, and it is worth being precise about why.

The information exists because a law compelled its publication for the express purpose of public scrutiny. Transparency organisations use it to track where money goes. Journalists use it to find single-bidder awards, repeat winners and suspiciously specific specifications. Small suppliers use it because the alternative, an expensive aggregator subscription, is exactly the barrier the publication mandate was meant to remove. That is the intended audience of a public register.

The operating rules are still the ordinary ones:

• Prefer the official channel. If a bulk file or API exists and you scrape the HTML instead, you are generating load for no reason and getting worse data. This is the most-ignored rule in the field.
• Treat robots.txt as intent. The Robots Exclusion Protocol is a published standard. It is not a statute, but ignoring it on a government host is a bad look and often breaches the site's terms.
• Do not defeat access controls. Supplier registration portals and tender document rooms behind a login are out of scope unless you hold a legitimate account whose terms permit automated access.
• Watch personal data. Notices carry named contact officers. Those are personal data under GDPR even inside a public document. Keep collection proportionate, and do not build a marketing list out of procurement officers.
• Attribute and republish carefully. Many registers carry an explicit open licence. Check it. TED data and UK OCDS feeds are broadly reusable; some national portals attach conditions.
• Rate limit per host, not globally. A ten-request-per-second global budget spread over a thousand hosts is polite. The same budget aimed at one county server is not.

This is not legal advice. For a programme operating across jurisdictions, particularly one that republishes or resells, run the licensing and data-protection questions past counsel. The methodology overlaps closely with proxies for academic research data collection, which goes deeper on consent, retention and reproducibility.