How to Scrape Kayak Flight Prices Without Losing Fares

Kayak holds no inventory. On most routes it sells no tickets at all. What it does is broadcast your query to dozens of suppliers, airlines and online travel agencies alike, then rank whatever comes back. Booking Holdings owns it: the Priceline Group announced the acquisition in November 2012 and closed it in May 2013 for roughly $1.8 billion, and Priceline renamed itself Booking Holdings in February 2018.

The ownership matters less than the architecture. A single Kayak results page is not one document. It is N supplier responses, merged, deduplicated, and re-sorted client side as they land. Compare the three data-arrival models you meet in travel scraping:

If you already built a scraper for Google Flights, the instinct you carry over is the wrong one. There, the hard part is constructing the request. Here the request is trivial, and the hard part is knowing when to read.

Open DevTools, run a search, and watch the network panel instead of the page. The shape is consistent across routes:

1. Session creation. The GET on /flights/JFK-LHR/2026-09-15 returns an application shell. The server mints a search identifier for this query and hands it to the client.
2. Fan-out. Kayak's backend dispatches the query to its supplier connections. Some answer in under two seconds. Some take twenty. A few time out and never appear at all.
3. Polling. The page opens a repeating XHR loop, roughly every one to two seconds, asking for anything new. Each response carries a batch of fresh itineraries plus a progress signal.
4. Re-sort. The DOM is rebuilt as batches land. Under the default "Best" sort, a fare that arrives late can jump to the top and push the card you read three seconds ago down the list.
5. Completion. Polling stops. The result count freezes. The page is now a stable snapshot, and it stays valid only as long as the session behind it does.

Those polling endpoints sit under Kayak's internal search paths, and their payload shape is undocumented and rotates. You can call them directly, using the same technique covered in scraping hidden JSON API endpoints, but then you inherit a contract that can change on any deploy, and you have to reproduce the session tokens and cookies that authorise each poll. Rendering the page is slower per search and much cheaper to maintain. This guide takes the rendered route and treats the poll loop as something to observe, not to reimplement.

Here is the failure in its simplest form:

import requests

# WRONG. Returns an application shell, or at best a very early partial set.
html = requests.get(
    "https://www.kayak.com/flights/JFK-LHR/2026-09-15",
    headers={"User-Agent": "Mozilla/5.0"},
).text

print("data-resultid" in html)   # almost always False

Adding a headless browser and a five-second sleep feels like the fix. It isn't. It replaces "no data" with something worse: plausible data that is quietly wrong.

The bias is not random noise. The same suppliers are slow on every run, because latency is a property of their connection, not of your scraper. Budget OTAs and consolidators, exactly the ones that undercut the airline's own price, are consistently among the last to report. Cut every search off at five seconds and you build a price series that is repeatably too high. Average it over a month and the error does not cancel out, because it was never centred on zero.

Three specific things break in a fixed-timeout scrape:

• Your minimum is wrong. min(prices) at t=5s and at completion are different numbers on any route with more than a handful of suppliers.
• Your count is wrong. Reporting "42 options found" when the finished search holds 180 makes route-liquidity comparisons meaningless.
• Your provider mix is wrong. If you use Kayak to answer "who is cheapest on this route", an early cut answers "who is fastest", which is a different question with a different winner.

Measure the completion curve for your route

Before you write the production scraper, spend a few credits finding out how long your routes actually take. Snapshot the same search at increasing dwell times and count results:

import requests, re, json

API = "https://scrape.sparkproxy.io/api/v1"
KEY = {"X-API-Key": "YOUR_API_KEY"}
URL = "https://www.kayak.com/flights/JFK-LHR/2026-09-15?sort=price_a"

def snapshot(seconds):
    r = requests.get(API, headers=KEY, params={
        "url": URL,
        "render_js": "true",
        "premium_proxy": "true",
        "country_code": "US",
        "stealth": "true",
        "wait": str(seconds),        # 0-30s of extra dwell after page load
        "json_response": "true",
    }, timeout=180)
    body = r.json()["body"]
    cards = len(re.findall(r"data-resultid=", body))
    prices = [int(p.replace(",", "")) for p in re.findall(r"\$([0-9,]{2,7})", body)]
    return {"wait": seconds, "results": cards, "min": min(prices) if prices else None}

curve = [snapshot(s) for s in (3, 6, 10, 15, 20, 25, 30)]
print(json.dumps(curve, indent=2))

One run on a busy transatlantic route produced the shape below. Read it as an illustration of the pattern, not as a benchmark, because the numbers move with route, date, day of week, and point of sale:

Two useful readings come out of that. The curve flattens well before 30 seconds on this route, so padding every request to the maximum burns credits for nothing. And the cheapest fare fell by about 20% between the three-second and twenty-second reads, which is the entire margin most fare-monitoring projects exist to detect. Thin domestic routes settle in under eight seconds. Multi-city and flexible-date searches can still be growing past thirty. Run the curve once per route family and set policy from data instead of from a guess.

Kayak's URL grammar is path-based, which makes it pleasant to generate. The core pattern is /flights/{ORIGIN}-{DEST}/{DEPART} with an optional return date:

A round trip for two in business class, sorted by price:

https://www.kayak.com/flights/JFK-LHR/2026-09-15/2026-09-22/2adults/business?sort=price_a

Set sort=price_a on every request. The default "Best" ranking blends price with duration and stop count, so the top card is not the cheapest fare, and a scraper that reads position one and calls it the minimum is wrong even on a fully completed search. Sorting by price also simplifies your completion check, because the cheapest fare stays where you expect it while the rest of the list churns.

Kayak's fares only exist after JavaScript runs and the poll loop drains, which puts it firmly in the dynamic JavaScript site category. The SparkProxy Scraping API handles the browser, the residential exit, and the fingerprint hardening in one call. The base URL is https://scrape.sparkproxy.io/api/v1 and auth goes in the X-API-Key header.

The minimum viable request:

curl -H "X-API-Key: YOUR_API_KEY" \
  --get "https://scrape.sparkproxy.io/api/v1" \
  --data-urlencode "url=https://www.kayak.com/flights/JFK-LHR/2026-09-15?sort=price_a" \
  --data-urlencode "render_js=true" \
  --data-urlencode "premium_proxy=true" \
  --data-urlencode "country_code=US" \
  --data-urlencode "stealth=true" \
  --data-urlencode "wait=20"

Every parameter is there for a named reason:

A completed search costs roughly 35 to 40 credits: 5 for rendering, 10 to 25 for the premium proxy, and 5 each for stealth and a JS scenario. That is real money at volume, and it is the practical argument for measuring the completion curve once rather than padding every single request to 30 seconds forever.

A fixed wait is fine for a route whose curve you have measured. It gets fragile the moment traffic, travel date, or market changes. The durable approach is to wait until the result count stops growing.

Do not anchor on a "search complete" element. Kayak's class names are hashed and rotate between deploys, and any selector printed in a tutorial has a short shelf life. What survives is the shape of the data: result containers carry a data-resultid attribute, and their count only ever increases within a single search. When it holds steady across consecutive reads, the search is done.

import requests

API = "https://scrape.sparkproxy.io/api/v1"

# Watch the live DOM inside the browser session; return once the count settles.
watcher = """
  (async () => {
    let last = -1, stable = 0;
    for (let i = 0; i < 40; i++) {
      const n = document.querySelectorAll('[data-resultid]').length;
      if (n === last && n > 0) { stable++; } else { stable = 0; }
      if (stable >= 3) return JSON.stringify({results: n, settled: true, ticks: i});
      last = n;
      await new Promise(r => setTimeout(r, 1000));
    }
    return JSON.stringify({results: last, settled: false, ticks: 40});
  })()
"""

scenario = {"instructions": [
    {"wait_for": "[data-resultid]"},   # first batch has landed
    {"scroll": 1200},                  # pull more cards into the DOM
    {"evaluate": watcher},             # block until the count settles
]}

r = requests.post(
    API,
    headers={"X-API-Key": "YOUR_API_KEY", "Content-Type": "application/json"},
    json={
        "url": "https://www.kayak.com/flights/JFK-LHR/2026-09-15?sort=price_a",
        "render_js": True,
        "premium_proxy": True,
        "country_code": "US",
        "stealth": True,
        "js_scenario": scenario,
        "json_response": True,
    },
    timeout=240,
)

data = r.json()
print(data["status_code"], data["credits_used"], data["duration_ms"])
html = data["body"]

Three ticks of no growth is a sensible default. Two is too twitchy, because a slow supplier can leave a two-second gap in the middle of a healthy search. Five wastes credits on routes that finished long ago. The escape hatch matters as much as the condition: when the loop exits with settled: false, tag that record as incomplete rather than filing it beside your clean rows. A dataset that knows which observations are untrustworthy is worth far more than one that silently mixes them.

Once the page has settled, extraction is ordinary work. Offload it to the API with extract_rules when the shape is simple:

rules = {
  "cards": {
    "selector": "[data-resultid]",
    "type": "list",
    "output": {
      "price": "[class*='price-text']",
      "airline": "img[alt]@alt",
      "duration": "[class*='duration']",
      "stops": "[class*='stops-text']",
      "provider": "[class*='provider']"
    }
  }
}

Substring selectors like price-text outlive full hashed class names, but assume you will re-derive them eventually. Build the parser so a selector change fails loudly instead of returning an empty list:

from bs4 import BeautifulSoup
import re, datetime

PRICE = re.compile(r"[0-9][0-9,]{1,6}")

def parse(html, route, point_of_sale):
    soup = BeautifulSoup(html, "html.parser")
    now = datetime.datetime.now(datetime.timezone.utc).isoformat()
    rows = []
    for card in soup.select("[data-resultid]"):
        text = card.get_text(" ", strip=True)
        price_el = card.select_one("[class*='price-text']")
        if not price_el:
            continue
        m = PRICE.search(price_el.get_text())
        if not m:
            continue
        rows.append({
            "result_id": card.get("data-resultid"),
            "price": int(m.group(0).replace(",", "")),
            "nonstop": "nonstop" in text.lower(),
            "carriers": sorted({i.get("alt") for i in card.select("img[alt]") if i.get("alt")}),
            "route": route,
            "point_of_sale": point_of_sale,
            "captured_at": now,
        })
    if not rows:
        raise ValueError("zero cards parsed: selectors changed or the page was challenged")
    return rows

The captured_at field is not decoration. Airfares move within the hour, so a price without a timestamp is a claim you cannot defend later. Store the point of sale next to it for the same reason.

Kayak result URLs are not permalinks. The path you requested stays stable, but the results behind it belong to a server-side search session that expires. Reload that URL twenty minutes later and you do not recover your snapshot, you trigger a brand new search with new suppliers, new latencies, and new prices. Two consequences for pipeline design:

• Never cache the URL as if it were the data. Cache the parsed rows. If you need the same query again, run the search again and store the result as a separate observation with its own timestamp.
• Never resume a session from a different exit IP. The cookies Kayak sets during a search are bound to the identity that started it. Handing them to another IP mid-search is a clean bot signal. Keep one cookie jar per proxy identity for the life of a search, then throw it away. The general pattern is in handling cookies and sessions in web scraping.

Reusing one identity across hundreds of searches is the other common failure. A single residential IP running fifty flight searches an hour looks nothing like a shopper, and metasearch operators watch for exactly that. Rotate per search, or at minimum per small batch, and hold concurrency per identity at one.

Currency is the obvious difference between markets. Inventory is the interesting one. A search run from a US exit sees a different set of OTAs and consolidators than the identical search from Germany, because supplier distribution agreements are regional. So the cheapest fare on a route is not one number. It is one number per point of sale, and for fare-aggregation work the spread between markets is often the whole reason for the project.

MARKETS = [("www.kayak.com", "US"), ("www.kayak.co.uk", "GB"), ("www.kayak.de", "DE")]

def multi_market(route, depart):
    rows = []
    for domain, cc in MARKETS:
        url = f"https://{domain}/flights/{route}/{depart}?sort=price_a"
        html = fetch_settled(url, cc)        # the js_scenario call from above
        rows.extend(parse(html, route, cc))
    return rows

Keep the domain and the exit country aligned. Requesting kayak.de from a US IP produces a page no real German shopper would ever see, and those fares will not reconcile against anything. That market-by-market discipline is the core of travel fare aggregation with proxies, and it is why geo-targeting here is a data-correctness feature rather than an anti-blocking trick.

The characteristic Kayak failure is a 200 response with no fares in it. Status codes will not save you, so classify the body:

def classify(html):
    low = html.lower()
    if "data-resultid" in html:
        return "ok"
    if any(s in low for s in ("verify you are a human", "unusual traffic",
                             "captcha", "are you a robot")):
        return "challenged"
    if any(s in low for s in ("no results", "we couldn't find", "try different dates")):
        return "empty"          # genuinely zero flights, do not retry
    return "incomplete"         # captured too early, or the poll loop stalled

Each class deserves a different response, and conflating them is how scrapers end up hammering a route that simply has no flights:

Only challenged and incomplete earn retries, and they want different backoff shapes: exponential with jitter for the first, a single longer attempt for the second. The general policy lives in retry and backoff strategies for web scraping.

One more habit worth building early: deduplicate on the itinerary, not on the card. The same physical flight often appears several times from different booking providers at different prices, which is the point of metasearch. If the question is "what is the cheapest way to fly this itinerary", group by carrier plus flight numbers plus departure times, take the minimum inside each group, and keep the winning provider name on the row. Skip that step and you inflate option counts, which makes two comparable routes look differently liquid when they are not.

Fares displayed on Kayak are public information, and collecting public data is generally not a criminal matter in the US after the hiQ Labs v. LinkedIn rulings narrowed how the Computer Fraud and Abuse Act applies to publicly accessible pages. That is not the same as permission. Kayak's Terms of Use restrict automated access, and its robots.txt disallows crawling of the search paths, so treat this as a contractual question and read both documents before you scale anything.

Practical guardrails that keep a monitoring project defensible: collect only public search results and no personal data, keep concurrency low enough that your traffic is a rounding error on theirs, cache aggressively so you never re-run a search you already hold, and never resell raw Kayak pages. If you need bookable, licensed inventory rather than observed prices, the honest path is a supplier or GDS API such as Amadeus, Duffel, or Kiwi's Tequila, or Kayak's own affiliate and whitelabel programme for approved partners. Scraping is the right tool for market observation. It is the wrong tool for transacting.