Proxies for Automotive Listings Aggregation at Scale

A single-source scraper has one schema, one rate limit, one currency, one locale, one definition of "this listing exists". Aggregation makes all of them plural.

Coverage becomes a sharding question. Most portals cap paginated results near 1,000 items, so a national search never returns national inventory. You shard by postcode radius, then price band, then make, until every shard comes back under the cap. Twelve sources, 400 postcode seeds, eight price bands: tens of thousands of discovery requests per sweep, against sources that rate-limit per IP.

Counts stop being trustworthy. Merge nothing and a market with 180,000 real cars reads as 340,000. Merge too aggressively and two genuinely different silver Golfs at the same dealer become one. Both errors survive a spot check and both wreck a price index.

Freshness stops being free. "Is this still for sale, and at what price" is a per-listing question that costs money every day, forever, across every source. The same shape appears in datacenter proxies for real estate data aggregation, but cars are worse in one way. A house has an address. A car has a VIN that half the market declines to publish.

Proxies do not solve identity. They make identity solvable, by letting you gather enough of the graph, often enough, from the right places.

Three jobs. They spread the discovery sweep across enough IPs that a 40,000-request shard plan finishes inside a crawl window instead of tripping a per-IP cap in ten minutes. They pin geography, so a German source is always fetched from a German exit, which keeps currency, VAT presentation and date format stable across runs. And they let you re-fetch the same listing daily for months without one IP looking like a monitoring bot.

Everything downstream depends on collection being complete and repeatable: the VIN join, the dedupe, the trim map, the price history. Incomplete collection looks exactly like inventory churn, and you cannot tell the difference after the fact.

The Vehicle Identification Number is a 17-character code standardised as ISO 3779 and mandated in the United States by 49 CFR Part 565. It is the closest thing the used-car market has to a primary key. Structure worth knowing before you write matching code:

The letters I, O and Q never appear in a VIN. A candidate string containing one of them is a transcription or OCR error, and you should reject it before it poisons a join.

Validate the check digit, then know when not to

TRANSLIT = {**{str(d): d for d in range(10)},
            "A":1,"B":2,"C":3,"D":4,"E":5,"F":6,"G":7,"H":8,
            "J":1,"K":2,"L":3,"M":4,"N":5,"P":7,"R":9,
            "S":2,"T":3,"U":4,"V":5,"W":6,"X":7,"Y":8,"Z":9}
WEIGHTS = [8,7,6,5,4,3,2,10,0,9,8,7,6,5,4,3,2]

def vin_check_digit_ok(vin: str) -> bool:
    vin = vin.strip().upper()
    if len(vin) != 17 or any(c in "IOQ" for c in vin):
        return False
    total = sum(TRANSLIT[c] * w for c, w in zip(vin, WEIGHTS))
    expected = total % 11
    return vin[8] == ("X" if expected == 10 else str(expected))

Here is the part most pipelines get wrong. The check digit is required by US federal regulation, so a North American VIN that fails it is bad data. European manufacturers are under no such obligation, and plenty of genuine European VINs fail the calculation. Rejecting on the check digit alone silently deletes real cars from a German or Italian dataset. Gate the rule on the World Manufacturer Identifier region.

Two more limits. Position 10 encodes model year on a 30-year cycle, so a 1997 and a 2027 vehicle from the same plant can share a full 17-character VIN. That rarely bites in a live dataset, but pin the year to a plausible range rather than trusting the code alone. And the VIN describes the car as built, not as equipped: no reliable trim level outside North America, and no options anywhere.

For US-market vehicles, the NHTSA vPIC API decodes make, model, year, body class, engine and plant for free, in batches:

import requests

def decode_vins(vins):
    r = requests.post(
        "https://vpic.nhtsa.dot.gov/api/vehicles/DecodeVINValuesBatch/",
        data={"format": "json", "data": ";".join(f"{v}," for v in vins)},
        timeout=30,
    )
    return r.json()["Results"]

Treat vPIC output as layer-1 truth for US VINs and as a partial hint for European ones, where it typically resolves the manufacturer and little else.

Franchise dealers publish VINs, independents publish them inconsistently, private sellers almost never do. Depending on the market you get a VIN on somewhere between 35% and 70% of listings. The rest still has to merge.

The instinct is fuzzy attribute matching. That is the weakest signal available and it belongs last, not first.

The strongest non-VIN signal is the photograph. When a dealer syndicates stock to three portals, the same JPEG goes to all three. Portals re-encode and watermark, which breaks a byte hash but not a perceptual one.

from PIL import Image
import numpy as np

def dhash(img_file, size=32):
    im = Image.open(img_file).convert("L")
    w, h = im.size
    im = im.crop((0, 0, w, int(h * 0.88)))       # drop the watermark strip
    im = im.resize((size + 1, size), Image.LANCZOS)
    a = np.asarray(im, dtype=np.int16)
    bits = a[:, 1:] > a[:, :-1]
    return int("".join("1" if b else "0" for b in bits.flatten()), 2)

def hamming(a, b):
    return bin(a ^ b).count("1")

Cropping the bottom 12% before hashing is the detail that makes this work: watermark bars live there, and a 32x32 difference hash is otherwise stable across the re-encoding portals apply. Hamming distance of 6 or less on the primary photo, plus agreeing make, model and year, is a merge. Above 12 it is noise.

Second signal: the free-text description. Syndication feeds ship it verbatim, so a SimHash in the style of Manku, Jain and Das Sarma, WWW 2007, threshold 3 over 64-bit fingerprints, catches copies that photo hashing missed because one portal reordered the gallery.

Attribute matching is the fallback. Block first:

def blocking_key(listing):
    return (
        listing["make_id"],
        listing["model_id"],
        listing["model_year"],
        listing["mileage_km"] // 1000,          # 1,000 km buckets
        listing["geohash5"],                    # ~4.9 km cell
    )

Mileage is the discriminating field: high-cardinality, monotonic over time, and two genuinely different cars of the same model, year and colour within five kilometres of each other rarely share a 1,000 km bucket.

Dedupe is tractable. Trim normalisation is where these projects actually stall, because the same car is described differently on every site and no source agrees on what a trim even is.

One BMW, four sources:

Three complications sit under that. Trim names change mid-generation, so a 2019 "Sport" and a 2022 "Sport" are different equipment sets. They are market-specific, so a US "Sport" and a UK "Sport" share a badge and nothing else. And equipment migrates between standard and optional across model years, so the trim label never fully determines the spec.

Stop trying to resolve everything to one canonical trim string. Resolve to a layered spec with provenance:

One rule matters more than the rest: never overwrite a layer-1 field with a layer-3 inference, and never discard the raw string. Mapping tables go stale every model year, and if you have destroyed the source text you cannot re-run the mapping when the table improves. The discipline behind cleaning scraped data applies double here.

Tokenising the title gets you most of the way:

BODY  = {"saloon","estate","hatchback","coupe","convertible","suv","4dr","5dr","3dr"}
TRANS = {"auto","automatic","manual","step","dsg","s-tronic","tiptronic","cvt"}
FUEL  = {"diesel","petrol","hybrid","phev","electric","mht","mhev","tdi","tfsi"}
DRIVE = {"xdrive","quattro","4matic","awd","4wd","fwd","rwd"}

def trim_residue(title, make, model):
    toks = title.lower().replace("-", " ").split()
    drop = BODY | TRANS | FUEL | DRIVE | set(make.lower().split()) | set(model.lower().split())
    return [t for t in toks if t not in drop and not t.isdigit()]

Run that on the four titles and each collapses toward m sport, with 320d surviving as the badge and xdrive captured as a drivetrain flag rather than trim noise. Resolve the badge to power through the generation, not globally: a 320d is 135 kW in one generation and 140 kW in the next, and one global lookup table quietly mislabels a decade of cars.

The insight worth stealing

The VIN join produces the labelled training data for the listings that have no VIN, for free. Every time one VIN appears on two portals with two different trim strings, you have a confirmed synonym pair that nobody had to label.

-- VIN-matched listings are a self-generating labelled corpus for the trim normaliser.
SELECT v.vin, a.source_id, a.raw_trim, b.source_id, b.raw_trim
FROM trim_observation a
JOIN trim_observation b
  ON a.vehicle_id = b.vehicle_id AND a.source_id < b.source_id
JOIN vehicle v ON v.vehicle_id = a.vehicle_id
WHERE v.vin IS NOT NULL AND a.raw_trim <> b.raw_trim;

Run that weekly and the synonym table grows on its own, keyed by generation so a 2019 mapping never contaminates a 2023 one. After a few months it starts resolving the VIN-less listings too, because the vocabulary is the same vocabulary. Most teams treat dedupe and normalisation as separate projects. They feed each other.

Both are derived facts that require stable repeat identity. If the entity ID changes between crawls, the price history forks and the day counter resets, and nothing in the data tells you it happened.

Store observations and derive everything else. A price column you update in place destroys the only signal you were collecting.

Two traps are specific to cars. Price is per source, not global: a dealer cuts the price on their own site first and the portal feeds lag a day to three days. So there is no single current price, only a per-source price with a freshness stamp plus an aggregate you compute with a rule you can defend.

The second is tax treatment. UK trade listings quote +VAT, German listings distinguish MwSt. ausweisbar from margin-scheme cars, and commercial vehicles are quoted net across most of Europe. Store it as an enum alongside the amount or your price index is wrong by roughly a fifth on an unpredictable subset of rows.

Days on lot belongs to the pair (vehicle, dealer), not to a listing ID and not to a portal. The clock starts when a dealer first offers a car anywhere, including their own website, and it does not stop because a portal ID changed.

Dealers delete a stale listing and post it again to reset the "new" badge and the visible age counter. Treat the new listing ID as a new car and your inventory count inflates, days-on-lot collapses, and the signal buyers care about disappears.

Model three entities rather than two:

• Vehicle: the physical car, keyed by VIN or a high-confidence composite.
• Listing: one offer of that vehicle by one seller on one source, carrying source_listing_id.
• Observation: one crawl result against one listing.

A relist is then a new Listing under an existing Vehicle at the same seller, opened close to when the previous one disappeared.

RELIST_GAP_DAYS = 30      # gap below this = continuous tenure
RESALE_GAP_DAYS = 90
MILEAGE_JUMP_KM = 1500

def classify_reappearance(prev, new):
    gap = (new.first_seen - prev.last_seen).days
    delta = new.mileage_km - prev.mileage_km
    if gap <= RELIST_GAP_DAYS and delta < MILEAGE_JUMP_KM:
        return "relist"          # carry days-on-lot forward
    if gap >= RESALE_GAP_DAYS and delta >= MILEAGE_JUMP_KM:
        return "returned_stock"  # new tenure, keep vehicle history
    return "ambiguous"

Guard against the false positive that otherwise wrecks a week of data. When a portal migrates its URL scheme, every listing ID changes overnight and the pipeline reports a hundred thousand relists. The check is cheap: if more than 20% of a source's listing IDs are unrecognised in one crawl, freeze relist inference for that source and re-key on the canonical URL or the VIN until it stabilises.

Done properly, relist rate becomes a product of its own. A car relisted twice with two price cuts is a car that is not selling, which is a useful input to a valuation model, a dealer tool, or a buyer-facing "this has been sitting" badge.

Automotive inventory and pricing are regional in ways that break naive collection.

Search results are localised by IP before you touch a single parameter. Default radius, currency, whether prices show with or without VAT, which dealers surface first: all of it shifts with the exit country, and some markets filter or 404 listings for out-of-country visitors entirely.

Pin the exit country per source and never let it float. This is not only about access, it is about diff stability. Fetch the same listing from two countries and the currency symbol, thousands separator and date format all change, the content hash flips, and your change detector fires a price-change event for a price that never moved. Phantom events are worse than missed ones because they look like signal. The mechanics are covered in what geo-targeting means in proxies.

Sub-national geography matters too, because portals return results by radius from a postcode. Covering a country means overlapping radius searches across enough postcodes that every dealer falls inside at least one, then deduplicating on vehicle identity.

A private-seller listing carries a name, a phone number, often a postcode, sometimes photos taken on a driveway with a house number and a plate in frame. All of that is personal data under Article 4(1) of the GDPR, and the seller having published it does not remove the protection. The CJEU in Breyer, C-582/14 confirmed that data counts as personal where identification is reasonably likely by any means, including indirect ones. A plate plus a national vehicle register is exactly that.

There is also Article 14, which obliges you to inform people when you collect their personal data from somewhere other than them. The disproportionate-effort exemption in Article 14(5)(b) exists, but building a consumer-scale dataset on it is a bad bet, and the UK ICO's guidance on what counts as personal data is worth reading first.

The clean answer is architectural rather than legal. You do not need to know who the seller is. You need to know whether two listings come from the same seller, a different and much smaller question. So collect vehicle and price attributes, and design the schema with no seller identity column at all:

CREATE TABLE vehicle (
  vehicle_id     BIGSERIAL PRIMARY KEY,
  vin            CHAR(17) UNIQUE,        -- NULL where the source omits it
  vin_verified   BOOLEAN NOT NULL,       -- check digit valid for the WMI region
  make_id        INT NOT NULL,
  model_id       INT NOT NULL,
  generation_id  INT,
  model_year     SMALLINT,
  body_style     TEXT, fuel TEXT, transmission TEXT, drivetrain TEXT,
  engine_kw      SMALLINT,
  first_seen     TIMESTAMPTZ NOT NULL,
  last_seen      TIMESTAMPTZ NOT NULL
);

CREATE TABLE listing (
  listing_id        BIGSERIAL PRIMARY KEY,
  vehicle_id        BIGINT REFERENCES vehicle,
  source_id         SMALLINT NOT NULL,
  source_listing_id TEXT NOT NULL,
  seller_type       TEXT CHECK (seller_type IN ('trade','private')),
  seller_key        BYTEA NOT NULL,   -- trade: dealer domain. private: HMAC(source||id, pepper)
  location_geohash5 CHAR(5),          -- ~4.9 km cell, never a full postcode for private
  first_seen        TIMESTAMPTZ NOT NULL,
  last_seen         TIMESTAMPTZ NOT NULL,
  status            TEXT NOT NULL
);

CREATE TABLE price_observation (
  listing_id    BIGINT REFERENCES listing,
  observed_at   TIMESTAMPTZ NOT NULL,
  price_minor   BIGINT NOT NULL,
  currency      CHAR(3) NOT NULL,
  tax_treatment TEXT NOT NULL,        -- gross | net_plus_vat | margin_scheme
  mileage_km    INT
);

-- Deliberately absent, and not an oversight:
--   seller_name, seller_phone, seller_email, seller_address, registration_plate

seller_key is a keyed hash, not a name. It supports dedupe and relist detection and nothing else. Be honest about what that buys: pseudonymised data is still personal data under Recital 26, so this is data minimisation rather than an exit from the regulation. It is still a large cut in what you hold and what a breach could expose.

Three practical follow-ons. Strip EXIF from any image you keep, because phone photos carry GPS coordinates of a private address. Store a perceptual hash rather than the original image where you can, since a dHash is not reversible into a legible plate. And truncate private-seller location to a geohash-5 cell instead of an exact postcode: precise enough for regional pricing, not precise enough to find a house. Trade listings differ, because a dealer's address is a business address. Sole traders sit in a grey area, so treat them as private. Marketplaces heavy with private sellers, such as the one covered in how to scrape Facebook Marketplace, deserve the strictest version of this policy.

None of this is legal advice, and the analysis shifts with jurisdiction and with what you do with the output.

The SparkProxy Scraping API covers the two phases an aggregator runs on different cadences: a cheap wide discovery sweep and a targeted detail fetch.

Discovery pages are usually server-rendered, so skip the browser. Batch mode takes comma-separated URLs, requires render_js=false, and bills the batch as one credit:

import requests

shard_urls = [
    f"https://cars.sparkproxy.io/search?postcode={pc}&radius=50&price_max={band}"
    for pc in POSTCODES for band in PRICE_BANDS
]

for i in range(0, len(shard_urls), 50):
    r = requests.get(
        "https://scrape.sparkproxy.io/api/v1",
        headers={"X-API-Key": API_KEY},
        params={
            "url": ",".join(shard_urls[i:i + 50]),
            "render_js": "false",          # required for batch mode
            "tag": "discovery-uk-nightly",
        },
        timeout=120,
    )
    for result in r.json()["results"]:
        if result["success"]:
            index_page(result["url"], result["body"])

Detail pages hold the VIN, the spec table and the seller type, and those usually need rendering. extract_rules pulls fields out without a parser, and country_code takes an ISO 3166-1 alpha-2 value:

import json, requests

rules = {
    "vin":      "[data-spec='vin']",
    "price":    ".listing-price",
    "mileage":  "[data-spec='odometer']",
    "title":    "h1.listing-title",
    "trim_raw": "[data-spec='trim']",
    "photos":   {"selector": ".gallery img", "type": "src"},
}

r = requests.get(
    "https://scrape.sparkproxy.io/api/v1",
    headers={"X-API-Key": API_KEY},
    params={
        "url": "https://cars.sparkproxy.io/listing/8814023",
        "render_js": "true",
        "country_code": "GB",            # pinned per source, never floating
        "block_resources": "true",
        "extract_rules": json.dumps(rules),
        "tag": "detail-uk",
    },
    timeout=180,
)
record = r.json()["extracted"]

For a full nightly sweep, callback_url returns HTTP 202 immediately and POSTs each result to your endpoint on completion, keeping a long crawl off your scheduler's critical path. Failed jobs return 530 and refund the credits. Pair it with how to schedule and automate web scrapers.

The refresh economics

Take 250,000 live listings across twelve sources, refreshed daily. Fetching every detail page with rendering costs 5 credits each: 1,250,000 credits a day. Beat it by reading price and status off the discovery pages you already fetch.

A result page carries roughly 30 cars with ID, price and mileage, so 250,000 listings is about 8,300 result pages. Assume half the sources render server-side:

A twelvefold reduction, and the gap widens as the dataset grows, because discovery scales with result pages rather than listings. Track cost per confirmed price point rather than cost per request: it penalises stale records and wasted re-fetches in one number, which is what you actually want to optimise.