Proxies for AI Training Data Collection at Scale

Start with the numbers, because they set the requirements. The Chinchilla work from DeepMind (Hoffmann et al., 2022) put the compute-optimal ratio near 20 training tokens per model parameter, and production models now train far past that point: Llama 3 was trained on roughly 15 trillion tokens. Hugging Face's FineWeb dataset, released in 2024, derives about 15 trillion tokens from 96 Common Crawl snapshots. Whatever your target size, the raw input is a very large slice of the public web, fetched and refetched over time.

That workload runs into three walls the moment it leaves a demo notebook:

Per-IP rate limits. Content sites cap how many requests one address can make in a short window. Sweep a large domain from a single IP and you hit a throttle, a CAPTCHA, or a ban long before the crawl finishes. The issue is not that your total volume is unreasonable for the site. It is that the volume arrives from one address.

Bot detection on the high-value sites. The pages worth training on, long-form articles, documentation, forums, reference material, tend to sit behind the same anti-bot stacks (Cloudflare, Akamai, DataDome) that guard commercial content. A datacenter IP making thousands of requests gets fingerprinted and blocked.

Regional serving. Many sites return different content by visitor location through hreflang tags, CDN edge routing, or outright geo-redirects. Collect everything from one country and you see one country's version of the web.

Proxies answer all three. A rotating pool spreads requests so each IP stays polite while total throughput scales. Residential IPs pass the detection that stops datacenter ranges on the harder sources. Geo-targeting lets one pipeline read each region as a local user would. This is the same distributed-collection pattern behind building a distributed web scraper and scraping high-volume data without rate limiting; training-data collection is that pattern taken to its largest scale.

A proxy layer sits between your collection code and the public web, distributing requests across many IP addresses so no single source sees enough traffic from one address to block you. For training-data work it does three concrete jobs:

Rate distribution. Requests spread across the pool so each IP's rate stays inside a site's tolerance while aggregate throughput covers the domain. Fifty IPs at one request per second give you fifty requests per second of coverage without any single IP looking abusive.

Geo diversity. Requests route through exit IPs in chosen countries, so each site serves that region's content, language, and page variant. This is what makes a multilingual or globally representative corpus possible from one pipeline.

Resilience. When an IP gets flagged, rotation moves the next request to a clean address, so one block never halts a crawl that has to run for days. Datacenter IPs handle tolerant sources cheaply; residential IPs carry the block-prone majority of high-value content.

The mental model matters, because it also keeps you on the right side of the ethics line covered below. You are not defeating a policy. You are distributing a reasonable request rate across enough addresses that each one stays within what the site allows.

Corpus collection is not a single burst. It is a continuous crawl that can run for weeks to reach billions of pages, then repeats to refresh the set. That duration and breadth is exactly what defeats a small pool.

Pool size tracks the rate limit, not the data volume. If a site tolerates about one request per second per IP and you want a hundred requests per second of coverage, you need roughly a hundred IPs rotating in round-robin. Sizing to the per-IP limit is what keeps each address polite while you still finish the crawl this month instead of next year.

Residential is the default at this breadth, not the exception. For a narrow scrape of tolerant targets, datacenter proxies are cheaper and fine. A broad corpus crawl is different: it deliberately reaches the long-form, content-dense sites that run aggressive detection, and those are the sites a residential pool passes because the requests look like ordinary home users. A large rotating residential pool is what sustains coverage across the block-prone majority of the web. For the tradeoff in depth, see web scraping API versus self-managed proxies.

Geo diversity is a data-quality control, not just a block-avoidance trick. This is the part most collection guides miss.

What we've found: A single-country IP pool quietly biases your corpus. Sites localize by IP through hreflang, CDN edge caching, and geo-redirects, so a US-only pool collects the US variant of every multi-region page and under-samples everything else. For a multilingual or globally representative model, that shows up later as language and register skew that no amount of filtering fully repairs, because the underrepresented text was never fetched in the first place. Spreading exit IPs across the regions and languages you care about is the cheapest way to fix representation at the source, upstream of every filter.

Concretely, that means assigning country_code values that match your target language and region mix, not just defaulting every request to one exit. A corpus meant to cover German, Japanese, and Brazilian Portuguese content needs DE, JP, and BR exits pulling their local page variants, not a US pool guessing at translations.

Models train on text, not HTML. A raw page is mostly boilerplate: navigation, sidebars, cookie banners, ad slots, footers, and scripts. The training signal is the main content, and getting to it cleanly is half the battle. The standard open recipes (C4, RefinedWeb, FineWeb) all spend heavy engineering on boilerplate removal with tools like trafilatura, jusText, and resiliparse before a single quality filter runs.

There is a real efficiency argument for doing that extraction at fetch time rather than storing raw HTML and cleaning later:

What we've found: Boilerplate is near-identical across every page of a site, so if you store raw HTML, that repeated markup dominates your deduplication step and inflates storage several times over before you have kept a single useful token. Returning clean Markdown at the edge flips the order: you deduplicate and quality-filter on the actual article text, which makes near-duplicate detection far more precise (two pages that share a header but differ in body are correctly kept) and cuts the bytes you store and reprocess. Extract first, then filter, and the whole pipeline gets cheaper and more accurate.

Markdown specifically, rather than a flat text dump, keeps the document structure a model benefits from: headings, lists, code blocks, and tables survive as lightweight syntax. That structure also makes downstream chunking for a RAG data pipeline cleaner, since you can split on headings instead of guessing at paragraph boundaries in an undifferentiated blob.

The SparkProxy Scraping API handles proxy rotation, geo-targeting, headless rendering, and clean-text extraction in one request, so your collection code stays focused on the corpus rather than on infrastructure. Send a GET to https://scrape.sparkproxy.io/api/v1 with your key in the X-API-Key header.

The function below fetches one page as boilerplate-stripped Markdown, routed through a residential exit in a chosen country. Set format=md for clean text, render_js=true for pages that hydrate client-side, country_code for the region variant you want, and premium_proxy=true to use the residential pool on block-prone sources.

import requests

API = "https://scrape.sparkproxy.io/api/v1"
API_KEY = "YOUR_API_KEY"  # copy from your SparkProxy dashboard


def fetch_clean_text(page_url: str, country: str = "US") -> str | None:
    """Fetch one page as clean Markdown, ready for a training corpus."""
    resp = requests.get(
        API,
        headers={"X-API-Key": API_KEY},
        params={
            "url": page_url,
            "format": "md",          # boilerplate-stripped Markdown, not raw HTML
            "render_js": "true",     # many content pages hydrate client-side
            "country_code": country, # rotate geo to capture regional page variants
            "premium_proxy": "true", # residential pool for large, block-prone sweeps
        },
        timeout=90,
    )
    if resp.status_code == 530:      # scrape failed upstream, credits refunded
        return None
    resp.raise_for_status()
    return resp.text


# Spread a seed list across the regions your corpus needs to represent.
seeds = [
    ("https://www.sparkproxy.io/blog", "US"),
    ("https://www.sparkproxy.io/blog", "DE"),
    ("https://www.sparkproxy.io/blog", "JP"),
]
corpus = []
for url, region in seeds:
    text = fetch_clean_text(url, region)
    if text:
        corpus.append({"url": url, "region": region, "text": text})

A quick curl check confirms your key, the Markdown format, and geo-targeting before you build the crawl loop:

curl -H "X-API-Key: YOUR_API_KEY" \
  "https://scrape.sparkproxy.io/api/v1?url=https://www.sparkproxy.io/blog&format=md&render_js=true&country_code=DE"

Because the API returns clean Markdown directly, the output of this stage is already close to training-ready text. The stages that follow are filtering and deduplication, not HTML parsing. For pages where the content lives inside a PDF rather than HTML, the same request pattern pairs with the approach in extracting data from PDFs.

Raw web text is full of duplicates and junk. Two processing steps turn it into a corpus.

Deduplication. Removing duplicate and near-duplicate documents is one of the highest-value steps in the whole pipeline. Lee et al. (2021), "Deduplicating Training Data Makes Language Models Better," showed that dedup cuts memorization, reduces the number of training steps needed, and improves held-out accuracy. Exact duplicates are easy; near-duplicates need a similarity method. MinHash with Locality-Sensitive Hashing (LSH) is the standard, and it scales because LSH only compares documents likely to be similar instead of every pair.

from datasketch import MinHash, MinHashLSH


def shingles(text: str, k: int = 5):
    tokens = text.split()
    return {" ".join(tokens[i:i + k]) for i in range(len(tokens) - k + 1)}


def signature(text: str, num_perm: int = 128) -> MinHash:
    m = MinHash(num_perm=num_perm)
    for sh in shingles(text):
        m.update(sh.encode("utf-8"))
    return m


lsh = MinHashLSH(threshold=0.8, num_perm=128)
unique = []
for doc in corpus:                       # corpus = clean-text docs from the fetch stage
    sig = signature(doc["text"])
    if not lsh.query(sig):               # nothing already near-duplicate at 0.8 Jaccard
        lsh.insert(doc["url"], sig)
        unique.append(doc)

Quality filtering. After dedup, you drop low-quality pages. The open datasets combine two approaches. Heuristic filters, drawn from the C4 and Gopher recipes (Rae et al., 2021), remove pages by simple rules: too few words, an abnormal symbol-to-word ratio, too many boilerplate lines, missing terminal punctuation, or a failed language check. On top of that, a trained classifier scores each page. FineWeb-Edu, for example, filtered FineWeb with a classifier trained on LLM-rated educational quality to produce a smaller, higher-signal subset of roughly 1.3 trillion tokens. Language identification usually runs first, with fastText's lid.176 model tagging each document so you can route it to the right language track.

Order matters, and it ties back to extraction: run all of this on clean Markdown, not raw HTML, or your filters spend their budget judging navigation menus. The same collection discipline shows up in the large-scale market research data collection workflow, which handles sharded storage of web data the same way.

A production corpus pipeline runs in repeatable stages. Proxies do their work in the fetch stage; the rest is extraction, filtering, and storage.

The stage that separates a strong corpus from a mediocre one is the pairing of stages 2 and 3: extract clean text at fetch time, then filter on that text. Teams that skip clean extraction and store raw HTML pay for it three times over, in storage, in slower dedup, and in filters that waste their budget on boilerplate.

Training-data collection is where legal exposure concentrates, and the rules tightened fast over the past two years. Treat the following as the baseline.

Collect public data only. Restrict fetching to pages any browser can reach without logging in. Do not bypass paywalls, login walls, or other access controls. In the US, circumventing an access restriction can implicate the Computer Fraud and Abuse Act, and it breaches nearly every site's terms. Public accessibility is not the same as unrestricted permission, but it is the floor you never go below.

Honor robots.txt and AI-crawler opt-outs. Major crawlers now present dedicated user agents, including CCBot (Common Crawl), GPTBot (OpenAI), Google-Extended, ClaudeBot, and PerplexityBot, and a growing share of sites disallow these by name. Respect Disallow rules and crawl-delay for the identity you present, and do not switch user agents to slip past an opt-out.

Respect TDM reservations. The EU's Copyright in the Digital Single Market Directive (Directive (EU) 2019/790) created a text-and-data-mining exception in Article 4, but it lets rightsholders reserve their rights by machine-readable means. When a site expresses a TDM opt-out through robots.txt, response headers, or the TDM Reservation Protocol, the Article 4 exception stops covering that content. Article 3, the separate exception for scientific research organizations, cannot be contractually overridden, which is one reason academic and commercial collection follow different rules. Our academic research data collection guide covers the research side in more depth.

Assume the copyright question is unsettled. Whether training on copyrighted web text is fair use in the US is being litigated right now, including The New York Times v. OpenAI (filed December 2023), Getty Images v. Stability AI, and several authors' class actions, none fully resolved as of early 2026. Do not assume a favorable outcome. Prefer permissively licensed sources (Creative Commons, public domain, open-data portals), record provenance for every document, and track the license of any dataset you build on, including Common Crawl's own terms of use.

Meet transparency duties. The EU AI Act's obligations for general-purpose AI models began applying on 2 August 2025. They include publishing a sufficiently detailed summary of training content and maintaining a policy to comply with EU copyright law, Article 4 opt-outs included. If your model may reach EU users, your collection pipeline has to produce that paper trail, which means logging what you fetched, from where, and under what permission.

Strip PII and keep it out. Web pages carry names, emails, and other personal data. Under the GDPR, processing personal data needs a lawful basis, and regulators have already acted on AI training practices (Italy's Garante temporarily blocked ChatGPT in 2023 over data-protection concerns). Run PII detection and removal before text enters the corpus, and drop anything that looks private or sensitive. Aggregate, non-personal text is the safe core of a training set.

Be a polite collector. Reasonable per-site rates, backing off on 429 and 403, respecting Retry-After, and identifying honestly where a source asks are the norms that keep large-scale collection sustainable. Distributing load across a proxy pool to stay under each site's limit is polite access at scale, not circumvention.

Corpus-Scale Proxy Infrastructure

SparkProxy residential and datacenter pools support AI data pipelines with per-IP rate control, pools that scale to hundreds of IPs, and country-level geo-targeting for representative, multilingual corpora. The Scraping API returns clean Markdown with format=md, render_js, and country_code in a single call.

Start building your AI training data pipeline