Proxies for Freight and Logistics Rate Monitoring

The distinction sounds pedantic until it corrupts a dataset. Here is what actually differs:

Every freight mode expresses this differently. Full container load ocean is priced per container on a port pair. Less than container load and air are priced per chargeable unit, where air bills the greater of gross weight and volumetric weight at the IATA 1:6000 ratio, meaning one cubic metre equals 166.67 kg. Less than truckload in North America is priced by freight class and density. United States domestic parcel bills dimensional weight at a divisor of 139 cubic inches per pound. Full truckload is priced per mile with lane imbalance baked in, and rail intermodal blends the two.

The collection patterns that work for datacenter proxies for price comparison websites assume a listed price with a stable identity. Almost none of that carries over here. The closest adjacent problem on this blog is travel fare aggregation, which is also quote shaped, but a fare at least resolves to a bookable, publicly visible number. Freight quotes frequently do not.

Treat this as a composite key. If any field is missing or different, the two rows do not belong in the same comparison. Discard the pair rather than fuzzy matching it.

That density row deserves attention because it silently broke a lot of historical series. The National Motor Freight Traffic Association restructured the NMFC classification system with the first phase effective 19 July 2025, moving the majority of commodities from fixed item classes onto a density based scale. An LTL rate quoted for the same commodity in June 2025 and in August 2025 may sit in a different class entirely. Unless you stored dimensions and computed density on both sides, a year over year comparison across that date is comparing two different products.

A rate row needs at least three dates, and usually four. Observation time is when you collected it. Validity is the window in which the carrier will honour it. Effective date is when an announced change starts to apply, which is frequently weeks after you read the announcement.

CREATE TABLE freight_quote (
  quote_id        BIGSERIAL PRIMARY KEY,
  observed_at     TIMESTAMPTZ NOT NULL,
  source          TEXT NOT NULL,        -- index | carrier_tariff | calculator | marketplace
  lane_origin     TEXT NOT NULL,        -- UN/LOCODE
  lane_dest       TEXT NOT NULL,
  basis           TEXT NOT NULL,        -- cy_cy | door_door | port_door
  equipment       TEXT NOT NULL,        -- 40HC | 20GP | 40RF | LTL_CLASS_70 | AIR_GEN
  service_level   TEXT NOT NULL,
  chargeable_qty  NUMERIC NOT NULL,
  chargeable_unit TEXT NOT NULL,        -- container | kg | cwt | shipment
  valid_from      DATE,
  valid_until     DATE,
  validity_source TEXT NOT NULL,        -- stated | inferred
  effective_from  DATE,                 -- for announced future changes
  currency        CHAR(3) NOT NULL,
  fx_rate_date    DATE,
  base_amount     NUMERIC NOT NULL,
  all_in_amount   NUMERIC,              -- derived, never the source of truth
  account_tier    TEXT NOT NULL,        -- public_tariff | registered | contracted
  raw_ref         TEXT NOT NULL         -- pointer to the stored HTML, JSON or PDF
);

Two rules make this table trustworthy. First, all_in_amount is always derived from the base plus the surcharge rows, never written straight from a headline figure on a page, because a page's own "all in" rarely includes the same set of lines twice in a row. Second, a quote with no stated validity gets an inferred expiry per source, flagged as inferred, and it is never carried forward past that date as though it were still live.

This is ordinary bitemporal modelling, the same discipline that keeps a financial time series honest. If that pattern is unfamiliar, the reasoning in using proxies for financial data collection transfers directly: you need to answer both "what was the rate for 3 September" and "what did we believe the rate for 3 September was, as of 20 August".

Store every line item as its own row with its own time series. Folding them into a single number destroys the only signal that matters.

Two things fall out of keeping these separate.

The base to surcharge shuffle

Carriers routinely hold an all-in figure roughly flat while moving money between the base rate and the surcharge lines, or announce a base reduction alongside a bunker increase that more than cancels it. A procurement team watching only the all-in sees stability. A team watching the base series sees the base falling while the recoverable surcharges climb, which is the actual negotiating position, because base rate is what you can argue about and index linked surcharges usually are not.

New line items poison year over year comparisons

The EU emissions trading extension to maritime transport began applying on 1 January 2024, with a phase-in of 40 percent of verified emissions for 2024, 70 percent for 2025 and 100 percent from 2026. FuelEU Maritime added a second new line from 1 January 2025. On any lane touching the European Economic Area, an all-in figure from 2023 and one from 2026 differ partly because two surcharge categories now exist that did not exist before. Year over year analysis on those lanes has to either exclude new codes or backfill them at zero, and say which it did.

Detention and demurrage deserve separate treatment because they are billed per event rather than per shipment, and because the United States Federal Maritime Commission's demurrage and detention billing requirements rule took effect on 28 May 2024, changing what has to appear on an invoice. Invoice-derived cost data from before and after that date is not structurally identical.

The mistake worth avoiding: teams build a scraping programme to answer "are we paying too much", when their own contracted rates already sit in their TMS or ERP and the market side is a licensed product. The comparison you want is internal contract against market benchmark, which needs one internal source and one purchased source. Public collection is a third leg, and its job is leading indicators rather than levels.

Read that table honestly. A large share of freight spend is priced inside authenticated portals, under agreements that both restrict automated access and make the numbers commercially sensitive. Proxy infrastructure does not change that, and no amount of it produces contract rates. What collection gives you is the public tariff layer and the forward looking announcements, which is genuinely valuable and genuinely underused.

Buy when you need a normalised lane level series with several years of history, when the number has to survive an audit or sit in front of a carrier during a tender, when the source's terms are a contract your company signed, or when collection cost plus engineering time exceeds the subscription price. That last case is more common than the industry admits.

Collect when the data is public and forward dated, which is exactly what tariff and surcharge announcements are. Collect when no vendor covers your specific lane, mode or accessorial, which is normal for LTL, regional parcel and inland moves. Collect when you need the underlying evidence, the actual notice with its effective date, rather than a number in a spreadsheet. Collect when your required frequency is higher than the vendor's publication cadence.

Run the arithmetic before choosing. A matrix of 250 lanes across 4 equipment types is 1,000 quotes a day. Priced through a rendered, geo-targeted, form-driven quote flow at 15 credits each, that is 15,000 credits a day and roughly 450,000 a month. Priced through a batched JSON endpoint at 1 credit per batch, with escalation only when something changed, the same coverage costs a small fraction of that. If your design lands closer to the first number than the second, a subscription is probably cheaper than your pipeline. The general trade-off is covered in web scraping versus using an API, and it applies with unusual force here, because freight has mature commercial data vendors that retail pricing does not.

Carriers run country specific sites with country specific tariffs, currencies and tax display rules. The same parcel calculator queried from a United States exit and a German exit returns a different currency, different tax inclusion, and sometimes a different published tariff for an identical shipment. The exit IP is what decides which tariff you get served, ahead of any language header, so keep the two aligned. Our explainer on what geo-targeting means in proxies covers how country selection is enforced at the exit node.

Account tier is the second axis, and it is the one that misleads people. A public calculator returns the published tariff. That is a ceiling, not a market rate, and treating it as a market rate produces a series that is wrong in level while still being useful in shape.

The useful application is exposure modelling. Major parcel carriers have announced average annual general rate increases of 5.9 percent for several consecutive years, including the published 2026 rate changes. That average tells you very little about your own bill, because the increase varies widely across zones, weight bands and accessorials, and a shipper concentrated in short zones with heavy accessorial use can face something far from the headline. Price your own actual shipment profile against the published tariff before and after the change date and you get your number instead of the press release's number. That is a job public calculators plus geo-targeted collection do well.

All examples use the HTML API at https://scrape.sparkproxy.io/api/v1 with the key in the X-API-Key header, per the SparkProxy API documentation.

Start with the highest value target, a carrier rate announcement page. These are static, so skip rendering entirely and pay 1 credit:

curl -G "https://scrape.sparkproxy.io/api/v1" \
  -H "X-API-Key: YOUR_API_KEY" \
  --data-urlencode "url=https://carrier.sparkproxy.io/tariffs/rate-announcements" \
  --data-urlencode "render_js=false" \
  --data-urlencode "json_response=true" \
  --data-urlencode "tag=tariff/announcements/weekly"

Pull the fields out with extract_rules instead of writing a parser. The effective date is the field that matters most, and it is the one most pipelines drop:

import json
import requests

API = "https://scrape.sparkproxy.io/api/v1"
KEY = {"X-API-Key": "YOUR_API_KEY"}

rules = {
    "notice_title":   {"selector": ".notice-card h3", "type": "list"},
    "trade_lane":     {"selector": ".notice-card .lane", "type": "list"},
    "surcharge_code": {"selector": ".notice-card .code", "type": "list"},
    "amount":         {"selector": ".notice-card .amount", "type": "list"},
    "effective_from": {"selector": ".notice-card .effective-from", "type": "list"},
    "notice_pdf":     {"selector": ".notice-card a.pdf", "type": "href"},
}

r = requests.get(API, headers=KEY, params={
    "url": "https://carrier.sparkproxy.io/tariffs/rate-announcements",
    "render_js": "false",
    "extract_rules": json.dumps(rules),
    "tag": "tariff/announcements/weekly",
}, timeout=120)

print(r.json()["extracted"])

Most quote engines call an internal JSON endpoint that takes the shipment attributes as query parameters. Finding it is worth an afternoon, because it turns a 15 credit rendered form interaction into a 1 credit fetch, and it usually returns the surcharge lines already itemised instead of glued into a rendered total. The technique is covered in how to scrape hidden JSON API endpoints:

r = requests.get(API, headers=KEY, params={
    "url": ("https://carrier.sparkproxy.io/api/quote"
            "?origin=USLAX&dest=USORD&class=70&weight=1200&pallets=2"),
    "render_js": "false",
    "transparent_status_code": "true",
    "forward_headers": json.dumps({
        "Accept": "application/json",
        "Referer": "https://carrier.sparkproxy.io/quote",
    }),
    "json_response": "true",
    "tag": "ltl/USLAX-USORD/c70",
}, timeout=120)

transparent_status_code is doing real work there. It mirrors the target's own HTTP status, and it only applies with render_js=false, so a calculator throttling you with its own 429 shows up as a throttle rather than as a successful response that happens to contain no rate. Those two outcomes must never collapse into the same row.

When a quote genuinely requires a form, drive it with js_scenario. The instructions array takes one action per entry:

scenario = {"instructions": [
    {"click": "#cookie-accept"},
    {"fill": {"selector": "#origin", "value": "CNSHA"}},
    {"fill": {"selector": "#destination", "value": "NLRTM"}},
    {"fill": {"selector": "#equipment", "value": "40HC"}},
    {"fill": {"selector": "#ready-date", "value": "2026-09-08"}},
    {"click": "#get-rate"},
    {"wait_for": ".quote-result"},
    {"wait": 1500},
]}

r = requests.post(API,
    headers={"X-API-Key": "YOUR_API_KEY", "Content-Type": "application/json"},
    json={
        "url": "https://carrier.sparkproxy.io/quote",
        "country_code": "DE",
        "render_js": True,
        "js_scenario": scenario,
        "tag": "fcl/CNSHA-NLRTM/40HC/DE",
    }, timeout=240)

For the wide lane matrix, comma-separate the URLs. The whole batch costs 1 credit with render_js=false, and the response carries a results array with one entry per URL:

lanes = [("USLAX", "USORD"), ("USLAX", "USDFW"), ("USATL", "USMIA")]
urls = ",".join(
    "https://carrier.sparkproxy.io/api/quote?origin={}&dest={}&class=70&weight=1200".format(o, d)
    for o, d in lanes
)

r = requests.get(API, headers=KEY, params={
    "url": urls,
    "render_js": "false",
    "tag": "ltl/matrix/nightly",
}, timeout=180)

for row in r.json()["results"]:
    if row["success"]:
        store_quote(row["url"], row["body"])
    else:
        requeue(row["url"], reason=row["error"])

The country matrix, with the language header aligned to the exit country so the tariff and the copy agree:

MARKETS = {
    "US": "en-US,en;q=0.9",
    "DE": "de-DE,de;q=0.9",
    "SG": "en-SG,en;q=0.9",
    "BR": "pt-BR,pt;q=0.9",
}

def quote(url, country):
    return requests.get(API, headers=KEY, params={
        "url": url,
        "country_code": country,
        "render_js": "true",
        "forward_headers": json.dumps({"Accept-Language": MARKETS[country]}),
        "json_response": "true",
        "tag": "parcel/tariff/{}".format(country),
    }, timeout=180)

For a portfolio sweep, go asynchronous so one slow carrier site does not stall the batch. Pass a callback_url and the API returns 202 Accepted with a job_id, then POSTs the completed result to your endpoint:

requests.get(API, headers=KEY, params={
    "url": quote_url,
    "country_code": "NL",
    "render_js": "true",
    "callback_url": "https://rates.sparkproxy.io/hooks/quote-complete",
    "tag": "fcl/nightly-sweep",
})

Capture the notice itself when a surcharge changes, because carrier announcement pages get replaced rather than versioned:

pdf = requests.get(API, headers=KEY, params={
    "url": "https://carrier.sparkproxy.io/tariffs/notice/2026-09-gri-eur",
    "render_js": "true",
    "format": "pdf",
    "tag": "tariff-evidence/2026-09-gri-eur",
}, timeout=240)

with open("gri_2026_09_eur.pdf", "wb") as f:
    f.write(pdf.content)

Costs, taken from the documented credit table:

Handle failures with the same care as the data. A 530 means the scrape failed and credits are refunded, so the correct response is a requeue. A 429 returns retry_after_seconds, which you should honour rather than hammer. Never write a null rate into the quote table because a page timed out. A missing rate and an unavailable rate are different facts, and conflating them is how a system starts reporting phantom capacity shortages.

That validity row is the part most programmes miss. Before carriers raise rates on a tightening lane, they shorten how long a quote stays honourable, from 30 days to 14, then to 7, then to "on request". The amount has not moved yet, so a monitor watching only the number sees nothing at all. Duration of the validity window is cheap to capture, arrives in the same response you were already collecting, and tends to move earlier than the rate does. Track it as a first class series.

Set the deadband per mode rather than globally. Published tariffs are exact, so any difference is a real change and the deadband is zero. Negotiated and marketplace spot quotes carry rounding and FX noise, so a small percentage threshold on ocean and air prevents an alert storm. Run the comparison on effective_from where it exists rather than on observed_at, otherwise every announcement fires an alert weeks before the money actually changes.

Public pages with no login and no accepted terms sit in ordinary web collection territory: respect robots.txt, keep request rates modest, identify your client, and cache hard so you are not refetching an unchanged notice page every hour. In United States law, hiQ Labs v. LinkedIn (Ninth Circuit, 2022) and Van Buren v. United States (2021) together indicate that accessing genuinely public data is not unauthorised access under the Computer Fraud and Abuse Act. Neither decision says anything about breach of contract, database rights, or the position in the European Union and United Kingdom.

Portals you hold credentials for are a different question, and a simpler one. Your carrier or forwarder agreement governs, and freight contracts commonly prohibit automated extraction outright. Ask for the API instead. Every major container line and integrator publishes rate, booking and tracking APIs, and getting access is usually a commercial conversation rather than an engineering problem. Licensed index data has the same shape: reading a headline number off a public landing page may be fine, while republishing it or rebuilding the series from it is a licence breach regardless of how it was collected.

None of this is legal advice. Freight is a contract heavy industry, and the contract is almost always the binding constraint rather than the statute.