Proxies for Job Board Aggregation: Operator's Guide

A single-source scraper has one schema, one rate limit, one locale, and one notion of "this listing exists". Every one of those becomes plural the moment you aggregate. Rate limits turn into a scheduling constraint: sweep 40 boards for 60 role-and-location queries each and that is 2,400 discovery requests before you touch a detail page, against sources that cap requests per IP per window. From one IP the run dies in minutes.

Then the schemas disagree. Board A gives you "London, United Kingdom", Board B "Greater London", the company's own Workday page "GBR-London-1 Finsbury Ave". One requisition, three jobs in your database, inflated coverage numbers and broken geo filters. Freshness stops being free too: with forty expiry conventions and no shared identifier, "is this job still open?" costs money to answer, every day, forever. Proxies solve neither identity nor freshness, but without distributed collection you cannot gather enough of the graph, often enough, to solve either.

Four concrete jobs, roughly in order of how much they matter:

Distribute request volume under per-IP limits. A rotating pool spreads a 250,000-request day across enough exits that no single IP looks abnormal to any source.

Serve the correct regional variant. Boards route by visitor IP. The same search from a US exit and a German exit returns different listings, different salary currency, sometimes different mandatory fields. For the German market you need a DE exit, not a US one with a language header.

Keep collection isolated per source. Distinct exit ranges keep a bad afternoon on one board from taking out the whole refresh cycle.

Make cheap liveness checks possible at all. Verifying 400,000 listings means 400,000 small requests. From a handful of addresses that reads as an attack. Spread across a pool it reads as background traffic.

Most aggregators start with fuzzy text matching, then spend two years tuning thresholds. Start somewhere else.

A large share of board listings are syndicated copies of one applicant tracking system posting, and the copy carries the original's requisition ID in its apply link. That is an exact identifier, not a similarity score. Resolve the apply URL, extract the ID, and the copies collapse deterministically. Fuzzy matching then handles only the residue.

Duplicate classes you will actually see

That last row is why similarity-only dedupe caps out. Two real openings for "Software Engineer II, Payments" in the same city have near-identical descriptions. They are two jobs, and only an identifier separates them.

Tier 1: identity from the apply link

import re

ATS_PATTERNS = [
    # Greenhouse: job-boards.greenhouse.io/acme/jobs/4172233007
    (r"greenhouse\.io/(?P<tenant>[^/]+)/jobs/(?P<req>\d+)", "greenhouse"),
    # Lever: jobs.lever.co/acme/8f3c2a9e-1d44-4c2b-9f77-2b0e5a1d9c31
    (r"lever\.co/(?P<tenant>[^/]+)/(?P<req>[0-9a-f-]{36})", "lever"),
    # Workday: acme.wd3.myworkdayjobs.com/en-US/External/job/London/..._R-12345
    (r"(?P<tenant>[\w-]+)\.wd\d+\.myworkdayjobs\.com/.*?(?P<req>R-\d+)", "workday"),
    # Ashby: jobs.ashbyhq.com/acme/9c1e2f0a-...
    (r"ashbyhq\.com/(?P<tenant>[^/]+)/(?P<req>[0-9a-f-]{36})", "ashby"),
    # SmartRecruiters: jobs.smartrecruiters.com/Acme/743999812345
    (r"smartrecruiters\.com/(?P<tenant>[^/]+)/(?P<req>\d+)", "smartrecruiters"),
]

def ats_identity(apply_url: str):
    for pattern, vendor in ATS_PATTERNS:
        m = re.search(pattern, apply_url, re.I)
        if m:
            return f"{vendor}:{m.group('tenant').lower()}:{m.group('req')}"
    return None

Board apply links are usually redirects, so resolve the chain first. A plain fetch at render_js=false costs 1 credit:

import requests

API = "https://scrape.sparkproxy.io/api/v1"
HEADERS = {"X-API-Key": "YOUR_API_KEY"}

def resolve_apply_url(tracking_url: str) -> str:
    r = requests.get(API, headers=HEADERS, params={
        "url": tracking_url,
        "render_js": "false",
        "json_response": "true",
    }, timeout=60)
    body = r.json()
    return body.get("meta", {}).get("url", tracking_url)

Tier 2: blocking plus SimHash on the residue

Do not compare every listing to every other. Block first, then compare inside blocks.

import hashlib, re

STOP = {"senior", "junior", "lead", "staff", "i", "ii", "iii", "sr", "jr"}

def title_tokens(title: str) -> frozenset:
    words = re.findall(r"[a-z0-9+#]+", title.lower())
    return frozenset(w for w in words if w not in STOP)

def blocking_key(employer_id: str, title: str, country: str) -> str:
    toks = "|".join(sorted(title_tokens(title))[:4])
    raw = f"{employer_id}::{toks}::{country}"
    return hashlib.blake2b(raw.encode(), digest_size=8).hexdigest()

Inside a block, compare SimHash fingerprints of the normalised description. Manku, Jain and Das Sarma's WWW 2007 paper on near-duplicate detection for web crawling used 64-bit fingerprints with a Hamming distance threshold of 3, still a sane starting point for job descriptions.

def near_duplicate(fp_a: int, fp_b: int, k: int = 3) -> bool:
    return bin(fp_a ^ fp_b).count("1") <= k

Strip the boilerplate first. Boards append their own apply instructions, equal-opportunity blocks, and cookie notices, and that shared text inflates similarity between different roles from the same board. Fingerprint the requirements and responsibilities sections only. The same normalise-then-compare discipline applies downstream; the guide to cleaning scraped data covers it in depth.

Record merges as edges with a confidence and a method, never as destructive overwrites: when a threshold turns out wrong you want to unmerge without re-crawling.

Employer names are the least reliable field in the record. One company appears as "Acme Ltd", "ACME LIMITED", "Acme Group (UK)", "Acme Technologies GmbH", and, via an agency, "Confidential".

String normalisation is table stakes:

import re, unicodedata

SUFFIXES = r"\b(ltd|limited|inc|incorporated|llc|plc|gmbh|ag|sa|sas|bv|nv|oy|ab|as|pty|pte|srl|spa|kk|co)\b\.?"

def normalise_employer(name: str) -> str:
    s = unicodedata.normalize("NFKC", name).casefold()
    s = s.replace("&", " and ")
    s = re.sub(SUFFIXES, " ", s)
    s = re.sub(r"[^\w\s]", " ", s)
    return re.sub(r"\s+", " ", s).strip()

That stalls quickly. "Acme Group" and "Acme Technologies" normalise to different strings while being one employer, and two unrelated "Apex Consulting" entries normalise to the same string while being two.

The stronger key is structural. The apply link almost always reveals the employer's ATS tenant or careers domain, and those are stable identifiers the employer controls:

Build the registry on the structural signals, attach every observed name string as an alias, and use the name only to match new listings to an entity you already know. For the display name, pick the most frequent alias seen on the employer's own domain, not overall: board listings skew toward whatever the recruiter typed that morning.

Two cases need explicit handling. Agency listings that mask the employer should stay unresolved rather than guessed: a wrong attribution is worse than a missing one. ATS migrations break tenant continuity, so keep a merged_into pointer on the old entity instead of rewriting history.

Location strings arrive in every shape a human can type.

Resolve against a gazetteer such as GeoNames, store ISO 3166-1 alpha-2 for country and ISO 3166-2 for subdivisions, and keep the raw string beside the resolved values so you can re-resolve without re-crawling.

The failure that quietly ruins aggregator data is treating "Remote" as a place. Stamp a remote role with the employer's headquarters city and every city-level count you publish is wrong, and users get results they cannot apply for. Model it as two fields: a workplace_type of onsite, hybrid, or remote, and an eligibility_scope holding the countries where the employer will hire. Schema.org's JobPosting vocabulary draws that line with jobLocationType: TELECOMMUTE and applicantLocationRequirements, so prefer the markup over the visible text where a source emits it.

Parsing structured data is cheaper and more accurate than scraping the rendered page:

import json
from selectolax.parser import HTMLParser

def job_posting_jsonld(html: str):
    tree = HTMLParser(html)
    for node in tree.css('script[type="application/ld+json"]'):
        try:
            data = json.loads(node.text())
        except ValueError:
            continue
        for obj in (data if isinstance(data, list) else [data]):
            if obj.get("@type") == "JobPosting":
                return obj
    return None

A well-formed JobPosting block hands you datePosted, validThrough, hiringOrganization, jobLocation, employmentType, baseSalary, and often an identifier carrying the ATS requisition number. Identity, employer, location, and expiry, all in one parse.

Listings leave your dataset three ways, and only one of them announces itself.

Removed. The detail URL returns 404 or 410, or the listing disappears from the source's index or sitemap. Cheap and unambiguous.

Silently filled. The page still returns 200 and the role is closed. Sometimes there is a DOM marker: "no longer accepting applications", a disabled apply button, a validThrough date in the past. Often there is nothing, and the listing simply stops appearing in search results while staying reachable by direct URL. Index membership, not HTTP status, is the reliable liveness test on boards that never delete pages.

Re-posted. The employer or board refreshes the posting date to push the role back up the sort order, sometimes under a new URL. Treat that as a new listing and your "new jobs today" metric becomes noise. Treat it as the same listing and you lose the signal that the role has been open ninety days, which is itself valuable labour-market data.

Handle all three by separating a listing (one URL on one source) from a posting (one requisition's continuous life) from a posting event (a single observation). A re-post creates a new listing and a new event, both attached to the existing posting.

Detection rules, ordered by what they cost you:

1. Absence from the source index or ATS feed since the last discovery pass. Free, it falls out of the crawl you already ran.
2. validThrough in the past, from JSON-LD you already parsed. Free.
3. HTTP status on the canonical URL. One credit per batch, not per URL.
4. DOM markers for closed roles. Needs a fetch, sometimes a full render.

Escalate to step 4 only for listings that survived steps 1 to 3 and are suspiciously old.

Nightly full re-crawls are the default and they are wrong. Fresh listings change often because someone is actively managing them; very old listings are mostly dead but expensive to keep checking. Schedule by age band and source volatility instead:

Layer a per-source volatility multiplier on top: track how fast each source's listings change state per day and scale its intervals by that. A board where five percent of listings expire daily deserves several times the attention of one at half a percent, and you learn both from your own logs inside two weeks.

def next_check_days(age_days: int, source_volatility: float) -> float:
    base = 1 if age_days <= 3 else 2 if age_days <= 14 else 4 if age_days <= 45 else 7
    # volatility is the observed daily state-change rate, e.g. 0.05
    multiplier = max(0.5, min(2.0, 0.02 / max(source_volatility, 0.001)))
    return round(base * multiplier, 1)

Deduplication and scheduling interact, which is easy to miss. Once five listings merge into one posting they no longer need the same cadence: check the authoritative copy, usually the employer's own ATS page, on the tight schedule, and check the syndicated copies rarely, only to confirm they still exist. For the orchestration side, see the guide on scheduling and automating web scrapers.

Listings differ by country, and not only in language.

Pay disclosure keeps changing. Colorado's Equal Pay for Equal Work Act (in force 2021) started the US wave, California and New York followed in 2023, and EU Directive (EU) 2023/970 requires member states to transpose pay-transparency rules by 7 June 2026. Salary coverage therefore varies by jurisdiction and year, so any cross-country salary statistic must control for that or it will show trends that are really just disclosure law arriving.

One rule saves a lot of pain: pin the collection country per source and keep it stable. Fetch the same listing from a US exit on Monday and a UK exit on Tuesday and the currency, formatting, and appended notices all change, your content hash flips, and change detection reports an edit that never happened. Every false positive costs a re-parse and a re-index, so normalise before hashing and pin the locale anyway. What geo-targeting means in proxies covers the mechanics.

Three patterns cover almost all aggregator traffic, against the documented endpoint https://scrape.sparkproxy.io/api/v1 with the key in an X-API-Key header.

Discovery pass, geo-targeted and rendered. Search and index pages are usually client-rendered, so this is the expensive tier: 5 credits for rendering plus 5 for country_code.

import requests

API = "https://scrape.sparkproxy.io/api/v1"
HEADERS = {"X-API-Key": "YOUR_API_KEY"}

def discover(board_search_url: str, country: str) -> str:
    r = requests.get(API, headers=HEADERS, params={
        "url": board_search_url,
        "render_js": "true",
        "country_code": country,        # ISO 3166-1 alpha-2, +5 credits
        "wait_for": ".job-results",     # gives up after 30 seconds
        "block_resources": "true",      # skip images, fonts, CSS
        "tag": f"discovery-{country}",  # echoed back for grouping
    }, timeout=120)
    return r.text

Detail extraction with extract_rules. Pull typed fields instead of shipping raw HTML around:

import json, requests

rules = {
    "title":       "h1.job-title",
    "employer":    ".company-name",
    "location":    ".job-location",
    "posted":      "time.posted-date",
    "apply_url":   {"selector": "a.apply-button", "type": "attribute", "attribute": "href"},
    "description": ".job-description",
}

r = requests.get(API, headers=HEADERS, params={
    "url": "https://example-board.io/jobs/48291",
    "render_js": "true",
    "extract_rules": json.dumps(rules),
    "tag": "detail",
}, timeout=120)
record = r.json()

Batch liveness checks. The pattern that changes aggregator economics. Batch mode takes comma-separated URLs, requires render_js=false, and the whole batch costs 1 credit however many URLs it holds:

import requests

def liveness_batch(urls: list) -> list:
    r = requests.get(API, headers=HEADERS, params={
        "url": ",".join(urls),      # batch mode
        "render_js": "false",       # required for batch
        "tag": "liveness",
    }, timeout=180)
    return r.json()["results"]      # url, success, httpStatus, body, error

dead = [x["url"] for x in liveness_batch(batch) if x["httpStatus"] in (404, 410)]

For long discovery jobs, callback_url returns HTTP 202 immediately and POSTs the result to your webhook, so your scheduler never holds thousands of open connections. When a source fights back, premium_proxy=true moves it to residential exits at 10 credits without JS or 25 with, and stealth=true adds a homepage pre-warm, a forced Google referrer, and longer idle delays for 5 more. Reserve both for the sources that need them; failed scrapes surface as HTTP 530 and their credits are refunded.

Work an illustrative model: 400,000 live listings across 40 sources, 12,000 new per day, at the documented rates of 1 credit for a plain fetch, 5 with rendering, 5 more for country_code.

The tiered breakdown, with liveness batched 50 URLs to a call:

Roughly a 23x reduction against the naive nightly re-render, and the dataset comes out fresher, because the savings get spent checking new and volatile listings more often instead of polling ninety-day-old ones daily.

Three levers do the work: batching (50 URLs per 1-credit call, a 50x cut on the largest volume line), rendering discipline (5x cheaper without a browser, and most JobPosting JSON-LD is emitted server-side anyway), and dedupe-aware scheduling (check one authoritative copy, not five syndicated ones).

The KPI worth tracking is cost per accurate listing-day: collection spend divided by the listing-days where your record matched reality. It penalises under-crawling (stale records shown to users) and over-crawling (money spent confirming nothing changed) at once, which cost per request does not.

Scraping public job listings has been litigated, and outcomes turn on three variables: whether authentication was involved, whether personal data was collected, and what the contract said. Nothing here is settled in either direction, and none of this is legal advice.

Authentication is the sharpest line. In hiQ Labs v. LinkedIn, the Ninth Circuit held in 2019, and again in April 2022 after the Supreme Court sent the case back in light of Van Buren v. United States (2021), that scraping public profile data was unlikely to violate the Computer Fraud and Abuse Act, because no authorisation gate stood in the way. Van Buren narrowed the CFAA's "exceeds authorized access" clause to a gates-up-or-down question. That was not the end: in November 2022 the district court found hiQ had breached LinkedIn's User Agreement, and the case resolved with a judgment and injunction against hiQ. On the other side of that line, a Delaware jury found for Ryanair on a CFAA claim against Booking.com in 2024, over automated access to authenticated accounts.

Contract terms bite where the CFAA does not. In Meta Platforms v. Bright Data (N.D. Cal., January 2024) the court granted Bright Data summary judgment on Meta's contract claims covering logged-out scraping of public data, partly because the terms did not clearly bind users who were never logged in. Next to hiQ, the pattern is clear: what you agreed to, and whether you agreed at all, matters more than whether the data was visible.

In the EU two extra layers apply. The sui generis database right under Directive 96/9/EC can protect substantial extraction even where the individual facts are not protected. And the CJEU held in Ryanair v. PR Aviation (C-30/14, January 2015) that where a database is not protected by copyright or the sui generis right, the Directive's user exceptions do not apply either, leaving contractual restrictions enforceable. GDPR attaches separately, the moment a listing carries personal data such as a named hiring manager or a recruiter's direct email.

Copyright sits underneath all of it. Job descriptions are authored text. Republishing them verbatim and in full is a different exposure from indexing title, employer, location, and a snippet with a link back.

A defensible operating posture:

• Collect only what is reachable without creating an account or logging in.
• Prefer official feeds: Greenhouse and Lever expose public JSON endpoints for a company's own board, and the employer's ATS feed is the authoritative copy anyway.
• Strip personal data at ingestion rather than at publication.
• Snippet and link rather than republishing full descriptions.
• Keep provenance on every field (source URL, fetch timestamp, method) and run a documented takedown and opt-out process. Together they turn a takedown request into a five-minute job.
• Get jurisdiction-specific advice before launching in a new market.

If your pipeline reaches into profile data rather than listings, the risk profile changes substantially: see scraping LinkedIn public data and the broader proxies for recruitment data overview.