Proxies for KYC and Sanctions Screening Vendors

Start by removing work you do not have to do.

The consolidated sanctions lists published by the major authorities are intended for exactly this use. The US Treasury's OFAC programme, the EU's consolidated list, the UK's OFSI list and the UN Security Council's consolidated list are all distributed as structured files that a scheduled job can pull directly, and the publishing bodies want screening systems to have them. Download them from the source, over a plain connection, on a schedule, and record the file hash. There is no rate limiting to route around and no reason to add a proxy hop to a compliance-critical download.

The proxy need starts one layer out, and it comes from four distinct pressures:

• Volume against rate-limited portals. A company registry that tolerates a few lookups an hour from one address will not serve the several thousand a day a screening vendor generates during onboarding peaks.
• Geographic variance. Plenty of registries, licence registers and news sites serve different content, or serve nothing at all, depending on the requesting country. If your screening result depends on which exit you used, you need that to be a deliberate choice you recorded rather than an accident.
• Availability verification. A screening vendor sometimes has to demonstrate that a source was reachable and what it returned. Checking from more than one network is how you distinguish "the registry was down" from "the registry blocked us".
• Adverse media at scale. Open-source media screening is the highest-volume component of most screening products by an order of magnitude, and it is the part that looks most like ordinary web collection.

None of those four is about the sanctions lists. Read that as good news: the highest-stakes data in your product is the part with the simplest pipeline.

Sorting sources by how they behave, rather than by what they contain, gives you five tiers with genuinely different requirements.

Tiers 3 to 5 are your proxy budget. Tier 4 overlaps heavily with the collection patterns in our guide to proxies for legal and court records data, which covers the case-record side in depth. Tier 5 behaves like any other media monitoring workload and is covered in proxies for news and media monitoring.

Two practical notes on tier 3. Many registries publish bulk extracts alongside their lookup interfaces, and a weekly bulk ingest is cheaper, faster and far better evidence than ten thousand individual lookups. And where a registry sells an official API, buy it. Arguing with a regulator about why you scraped a registry that sold you a feed is not a position you want to be in.

This is the requirement that separates a screening pipeline from a price-monitoring pipeline. You are not collecting data to make a decision today, you are collecting evidence that will be re-examined later, possibly by someone hostile.

Five things have to be stored for every fetch, and they have to be stored together:

1. The raw response bytes, before any parsing. Parsed output is your interpretation; the bytes are the evidence. Store the hash alongside them so tampering is detectable.
2. The exact request, including the full URL with query string, the method, and any headers you set. "We searched the registry" is not reproducible. The URL is.
3. A timestamp from a source you control, recorded at request time and at response time, so latency is visible.
4. The exit country the request left from, because for a geo-variant source this is part of the answer, not part of the plumbing.
5. The HTTP status and any redirect chain, because a 200 at the end of three redirects is a different fact from a direct 200.

Three architectural consequences follow.

Do not let anything cache or rewrite. A caching layer that quietly serves yesterday's page will eventually serve it on the day that matters. Disable transparent caching on the collection path, and prefer a proxy configuration that forwards bytes rather than one that transforms them.

Prefer the plainest fetch that works. A rendered browser session introduces variability: scripts run, content loads conditionally, timing changes the output. When a registry serves usable HTML without JavaScript, take the HTML. Render only where you must, and record that you rendered, because a rendered capture is a weaker piece of evidence than a static one and a reviewer may care about the difference.

Keep collection and interpretation in separate systems. When your matching logic changes, and it will, you want to re-run it against stored bytes rather than re-fetch a source that has since been updated. Our notes on proxies for web archiving and compliance capture cover the capture-fidelity side of this in more detail.

Most proxy buyers treat exit country as a targeting feature. In a screening vendor it is a control, and it belongs in your policy documents rather than in a config file somebody edits on a Friday.

Never egress from an embargoed or comprehensively sanctioned jurisdiction. This should be obvious and it is regularly violated by accident, because a global rotating pool includes countries your compliance team has a list about. Pin your exits to an explicit allowlist of countries rather than excluding a denylist, because a denylist fails open when a provider adds coverage. This is a genuine advantage of a provider with clear country targeting: you can enumerate what you are using.

Record the exit country per request, not per job. If the job is configured for Germany and one request fell back to a neighbouring country, the evidence for that request is different from the evidence for the rest. Requests, not jobs, are the unit a reviewer will ask about.

Match the exit to the legal basis you are relying on. If you collect from a source under an access regime that is country-specific, collecting from elsewhere may put you outside it. This is a question for your counsel rather than your infrastructure team, but the infrastructure has to be able to honour whatever answer they give, which means country targeting has to be reliable and auditable. Our explainer on what geo-targeting means in proxies covers how targeting is actually implemented and where it degrades.

Expect to be asked, and expect to answer. Proxy providers themselves increasingly require customer KYC and use-case approval, which for a screening vendor is straightforward and worth doing early, because it also produces paperwork your own auditors will want. Our piece on why proxy providers require KYC and use-case approval covers that process, and are proxies legal for business use covers the general position.

Screening workloads have an unusual shape. Monthly volume is modest by scraping standards, but it arrives in two spikes that have nothing to do with each other.

The first spike is the list and registry refresh, which is schedule-driven. It runs overnight, it hits a fixed set of sources, and its size is predictable to within a few percent. This is the easy half.

The second spike is onboarding demand, which is customer-driven and bursty. A corporate client pushing a batch of 4,000 subjects through your product on a Monday morning generates registry lookups and adverse media searches at a rate that has nothing to do with your monthly average. Size your concurrency for the burst, not the average, or your queue depth becomes a service-level problem on exactly the days your customers are paying attention.

There is a counterweight, and it is the reason concurrency rarely becomes expensive here. Per-source politeness usually binds before your thread count does. A registry that tolerates two requests a second does not care that you have 500 threads available, and hammering it is both a data risk and a relationship risk in a sector where relationships with data sources matter. Configure per-source rate limits explicitly, run them well under the observed tolerance, and let the thread pool sit idle. The correct queue design here is one bucket per source with its own rate, not one global pool.

Every input below is an assumption, chosen to be plausible for a mid-size vendor. Replace them with your own before you use the totals for anything. Nothing here is a measurement from a test we ran.

Assume 3,000 onboarding subjects a month, and per subject:

• 12 registry and portal lookups, giving 36,000 requests.
• 20 adverse media result pages, giving 60,000 requests.

Add a scheduled refresh of 60 tier 2 to tier 4 sources every day, which is 60 times 30, or 1,800 requests.

Monthly total: 36,000 plus 60,000 plus 1,800, which is 97,800 requests.

Now price it through SparkProxy's Scraping API, where a plain fetch is 1 credit and a JavaScript-rendered fetch is 5. Assume 80% of the sources serve usable HTML without rendering:

• Plain: 97,800 times 0.8 is 78,240 requests at 1 credit, so 78,240 credits.
• Rendered: 97,800 times 0.2 is 19,560 requests at 5 credits, so 97,800 credits.
• Total: 176,040 credits a month, which fits inside the Starter tier at $49 a month for 250,000 credits.

Now add the compliance requirement, and watch what it costs. Country targeting via country_code adds 5 credits per request. Apply it to the 30% of sources that are geo-variant:

• 97,800 times 0.3 is 29,340 requests, at 5 extra credits each, so 146,700 extra credits.
• New total: 176,040 plus 146,700, which is 322,740 credits.

That crosses the 250,000 Starter allowance and moves you to Growth at $99 a month for 1,000,000 credits, with room to double your onboarding volume before the next step. The lesson generalises: in a screening pipeline the compliance parameters, not the request count, tend to decide the plan.

Check whether concurrency is ever the constraint. 97,800 requests over 30 days is 3,260 a day. Spread over a six hour collection window that is 3,260 divided by 21,600 seconds, or 0.15 requests a second. Even at three seconds per request, a slow registry included, that is under one request in flight on average. Burst capacity for a 4,000-subject Monday is the only reason to buy threads, and per-source politeness will cap you long before a 100 thread Starter proxy plan at $75 a month does.

The crossover to a flat concurrency plan arrives when volume grows into the millions of requests a month. Below a few hundred thousand, the credit-priced API is both cheaper and, as the next section shows, better paperwork. Our breakdown of financial data collection with proxies covers the adjacent case where volume is far higher and the evidence bar is lower.

The Scraping API returns three response headers on every plain HTTP call: X-Job-Id, X-Credits-Used and X-Duration-Ms. The first is the one your compliance team cares about, because it gives every capture a provider-side identifier you can quote in an audit response. The tag parameter lets you stamp the request with your own case reference so the two systems reconcile.

curl -sS -D headers.txt -o capture.html -G "https://scrape.sparkproxy.io/api/v1" \
  -H "X-API-Key: YOUR_API_KEY" \
  --data-urlencode "url=https://example.gov/registry/company/12345678" \
  --data-urlencode "render_js=false" \
  --data-urlencode "country_code=GB" \
  --data-urlencode "tag=case-2026-09-04417"

sha256sum capture.html
grep -Ei '^(X-Job-Id|X-Credits-Used|X-Duration-Ms):' headers.txt

Write five things into your evidence store from that one call: the response bytes in capture.html, their SHA-256, the X-Job-Id, the resolved request URL, and the country_code you asked for. Keep the raw headers.txt too, because the redirect chain and content type live there.

Where a source genuinely needs a browser, or where a datacenter exit is refused, add residential routing with premium_proxy=true rather than any other parameter name, and record in your evidence store that the capture used a residential exit. It is a different collection condition and a reviewer may treat it differently.

curl -sS -G "https://scrape.sparkproxy.io/api/v1" \
  -H "X-API-Key: YOUR_API_KEY" \
  --data-urlencode "url=https://example.com/licence-register/search?q=acme" \
  --data-urlencode "render_js=true" \
  --data-urlencode "premium_proxy=true" \
  --data-urlencode "wait_for=.results-table" \
  --data-urlencode "tag=case-2026-09-04417"

Budget for that second shape deliberately. A rendered request through a residential exit is materially more expensive per call than a plain one, so keep it for the sources that require it rather than making it the default, and let the tag tell you afterwards which cases consumed it.

In most scraping projects a failure costs you a row of data. Here it can cost you an assertion you have already made to a customer.

The silent soft block. A registry that returns HTTP 200 with an empty result set when it has decided you are automated is indistinguishable from a genuine "no match" unless you check for it. A screening system that records "no adverse findings" because it was blocked has produced a false negative in the one product category where false negatives are the whole risk. Assert on content shape, not status code, and fail the check loudly rather than recording a clean result.

Partial pagination. Search interfaces that silently truncate at page three will make a subject look clean. Always verify that the last page you fetched really is the last page, by checking a total count or the absence of a next link, and treat a truncated crawl as a failed screen rather than a completed one.

Stale exits presenting as fresh data. If your collector retries through a cached or proxied intermediary, you may record a fetch timestamp that does not match the content's actual age. Disable caching on the path and, where the source provides one, record the source's own last-modified or publication date separately from your fetch time.

Undetected geographic drift. A country target that degrades to a neighbouring country will change results on a geo-variant source without erroring. Verify the observed exit country per request against what you asked for, and alert on mismatches.

Screening on a changed schema. Registries redesign. A parser that silently stops finding the officers table will produce clean screens forever. Add a canary assertion per source, running daily against a known subject with a known result, and page somebody when it fails. Our guide to detecting when your scraper is blocked covers the detection patterns; the difference here is that the alert has to be treated as a control failure rather than a bug ticket.

You will be asked about your sub-processors. These are the questions worth answers in writing before you sign, over and above the usual commercial ones.

• Which countries can traffic exit from, and can exits be restricted to an explicit allowlist rather than a denylist?
• Is the exit country reported back per request, or only configured per job?
• Is any response content cached, rewritten, compressed or otherwise transformed in transit?
• What logs are retained about our requests, for how long, and in which jurisdiction?
• What is the notification process if an exit range is implicated in abuse, and what is the remediation time?
• Is there a request-level identifier we can quote in an audit response, and how long is it retained on your side?
• What is the contractual position on service credits and incident disclosure? Our guide to reading a proxy provider SLA covers what the answers usually leave out.
• Does the acceptable use policy permit the collection you intend, in writing? See proxy acceptable use policies explained.

The last one matters more than teams expect. A screening vendor whose collection quietly breaches its own proxy provider's terms has a single point of failure with a 30 day notice period attached to it.