How to Scrape Indeed Jobs Without Getting Blocked

Get the framing right before you write a line of code, because Indeed is more restrictive than a typical e-commerce site.

In the United States, the Ninth Circuit's ruling in hiQ Labs v. LinkedIn (2022) held that scraping data that is publicly accessible, meaning no login and no authentication wall, generally does not violate the Computer Fraud and Abuse Act. That decision is about unauthorized access, not a blanket license. Two separate things still constrain you on Indeed:

• Terms of Service. Indeed's Terms prohibit automated data collection, so scraping can be a breach of contract even where it is not a CFAA violation. "Public" answers the access question, not the contract one.
• robots.txt. Indeed's robots.txt disallows crawler access to paths like /jobs and /viewjob for general bots. robots.txt is not law, but ignoring it is a clear signal you are acting against the site operator's stated wishes, and courts and Indeed both notice.

The responsible path depends on what you are building:

• For production or commercial use, the compliant route is Indeed's official programs (the Employer and Job Sync APIs, or a licensed data partner) rather than scraping. That is the answer if you need a durable feed.
• For small-scale personal research on public postings, the guardrails below keep a project defensible: collect public listing data only (title, company, salary, job ID), never anything behind a login and never personal data about applicants; rate-limit yourself and back off on errors so you do not degrade the service; respect robots.txt; and get legal advice before any commercial launch. This guide is engineering guidance, not legal advice.

Labor-market research, salary benchmarking, and hiring-trend analysis are legitimate reasons to scrape job listings from public pages. For the business context around that kind of collection, see Using Proxies for Market Research and Data Collection.

A public Indeed search result card and the job detail page behind it expose a consistent set of fields. The class names churn, but the fields are stable. Here is the reference set worth pulling, with selectors that work as of mid-2026. Indeed obfuscates and rotates class names, so treat every selector as one candidate in a list, not a guarantee.

Search result card (https://www.indeed.com/jobs?q=...):

Job detail page (https://www.indeed.com/viewjob?jk=):

The jk value is the anchor for everything. It is the hexadecimal job key (for example a1b2c3d4e5f6a7b8) that uniquely names a posting. Store it as your primary key and hang every other field off it. One caveat that matters later: the same real-world role is often posted under several different jk values by different staffing agencies, so jk uniqueness is not the same as job uniqueness.

Indeed sits behind Cloudflare and runs its own bot detection on top. Four things break a naive Indeed job scraper:

The Cloudflare challenge. When Cloudflare suspects automation, it serves an interstitial titled "Just a moment..." (or "Attention Required!") that runs a JavaScript and Turnstile check before it hands over the real page. A plain HTTP client never runs that JavaScript, so it receives the challenge HTML, often with a 403 status, and never sees a single job. Sometimes the challenge returns 200 with challenge markers in the body, so trusting the status code alone will store a block as if it were data.

IP reputation. Plain datacenter IP ranges get flagged fast on Cloudflare-fronted sites. Residential IPs blend in with real job seekers and last far longer. Rotating the exit IP per request is the difference between a scraper that runs for ten minutes and one that runs for a day.

Obfuscated, shifting markup. Indeed A/B tests its result layout and mangles class names, so a selector that works today can return nothing next week. data-testid attributes are more stable than visual classes, which is why the fields table leans on them. Match a list of candidates and expect drift.

Pagination and rate limits. Fire requests too fast and Indeed throttles or hard-blocks the IP. It also caps how deep you can page into any single query, so broad searches leave most postings unreachable.

A managed scraping API absorbs the first two for you and helps with the fourth. The markup drift stays your problem, because it lives in the HTML. For the proxy-side theory behind staying unblocked, How to Avoid Getting Your Proxy Blocked goes deep on it.

The SparkProxy Scraping API takes a target URL and handles the proxy, rotation, browser rendering, and anti-bot layer for you. You send one request; you get the rendered HTML back. For Cloudflare-fronted Indeed, four parameters carry the weight:

• render_js=true: Indeed and Cloudflare both depend on JavaScript, so a raw fetch never clears the challenge. A real Chromium render does.
• premium_proxy=true: routes through residential IPs, which survive Cloudflare where datacenter IPs get flagged.
• stealth=true: adds the extra fingerprint and behavior layers that clear Turnstile-style checks (it requires render_js=true).
• country_code: the ISO alpha-2 code of the market you are targeting (US, GB, DE), which sets the exit country so localized results match the domain.

The base endpoint is https://scrape.sparkproxy.io/api/v1 and auth is a single header, X-API-Key. A minimal request in cURL:

curl -G "https://scrape.sparkproxy.io/api/v1" \
  -H "X-API-Key: YOUR_API_KEY" \
  --data-urlencode "url=https://www.indeed.com/viewjob?jk=a1b2c3d4e5f6a7b8" \
  --data-urlencode "render_js=true" \
  --data-urlencode "premium_proxy=true" \
  --data-urlencode "stealth=true" \
  --data-urlencode "country_code=US"

The full parameter list and response fields live in the Scraping API docs. If you are weighing this against running your own proxy pool and headless browsers, Web Scraping API vs Self-Managed Proxies lays out the trade-off.

Start with one posting. Wrap the request so every call carries the Indeed-specific parameters, add wait_for so the API waits until the job body has actually rendered, and give it a generous timeout since a rendered request drives a real browser.

import requests

API = "https://scrape.sparkproxy.io/api/v1"
API_KEY = "YOUR_API_KEY"

def fetch(url: str, wait_for: str, country: str = "US") -> str:
    resp = requests.get(
        API,
        headers={"X-API-Key": API_KEY},
        params={
            "url": url,
            "render_js": "true",      # clear the Cloudflare JS challenge
            "premium_proxy": "true",  # residential IPs survive Cloudflare
            "stealth": "true",        # extra layers for Turnstile-style checks
            "country_code": country,  # match the market to the domain
            "wait_for": wait_for,     # block until this selector exists
        },
        timeout=90,
    )
    resp.raise_for_status()
    return resp.text

def fetch_job(jk: str, country: str = "US") -> str:
    url = f"https://www.indeed.com/viewjob?jk={jk}"
    return fetch(url, wait_for="#jobDescriptionText", country=country)

Before you trust the HTML, check whether Cloudflare handed you a challenge instead of a job. Because a challenge can arrive with a 200 status, raise_for_status() will not catch it. Scan the body for the telltale markers:

def is_blocked(html: str) -> bool:
    """Cloudflare/Indeed can return a challenge with HTTP 200, so the status lies."""
    markers = (
        "Just a moment...",
        "Attention Required! | Cloudflare",
        "cf-browser-verification",
        "challenge-platform",
        "Enable JavaScript and cookies to continue",
        "Additional Verification Required",
    )
    return any(m in html for m in markers)

Now a single fetch is honest: it either returns a real posting or tells you it was blocked so you can retry.

For parsing at scale, use selectolax (a C-backed HTML parser) rather than the pure-Python default. It parses Indeed's large DOM roughly an order of magnitude faster than html.parser, which matters when you process tens of thousands of pages. Install it with pip install selectolax.

The key move is trying a list of selectors per field and preferring data-testid attributes, which survive Indeed's class-name churn better than visual classes:

from selectolax.parser import HTMLParser

def _first_text(tree, selectors):
    for sel in selectors:
        node = tree.css_first(sel)
        if node and node.text(strip=True):
            return node.text(strip=True)
    return None

TITLE = ["h1.jobsearch-JobInfoHeader-title",
         '[data-testid="jobsearch-JobInfoHeader-title"]']
COMPANY = ['[data-testid="inlineHeader-companyName"] a',
           '[data-testid="inlineHeader-companyName"]']
LOCATION = ['[data-testid="inlineHeader-companyLocation"]',
            '[data-testid="jobsearch-JobInfoHeader-companyLocation"]']

def parse_job(html: str, jk: str) -> dict:
    tree = HTMLParser(html)
    desc = tree.css_first("#jobDescriptionText")
    pay = tree.css_first("#salaryInfoAndJobType")
    return {
        "job_id": jk,
        "title": _first_text(tree, TITLE),
        "company": _first_text(tree, COMPANY),
        "location": _first_text(tree, LOCATION),
        "salary_raw": pay.text(strip=True) if pay else None,
        "description": desc.text(strip=True) if desc else None,
        "url": f"https://www.indeed.com/viewjob?jk={jk}",
    }

The #salaryInfoAndJobType block bundles pay and employment type in one string like "$70,000 - $90,000 a year - Full-time", so keep the raw text now and normalize it in the analytics step. The description comes back as text here; if you need the formatted HTML (bullet lists of responsibilities and requirements), read desc.html instead of desc.text().

Maintaining CSS selectors across Indeed's markup drift is the tax on self-parsing. The SparkProxy Scraping API can do the extraction server-side with the extract_rules parameter: pass a map of field names to selectors, and the API returns JSON keyed by your names. This turns the scraping endpoint into a lightweight Indeed jobs API where the response is already structured.

import json
import requests

rules = {
    "title": '[data-testid="jobsearch-JobInfoHeader-title"]',
    "company": '[data-testid="inlineHeader-companyName"]',
    "location": '[data-testid="inlineHeader-companyLocation"]',
    "salary": "#salaryInfoAndJobType",
    "description": "#jobDescriptionText",
}

resp = requests.get(
    "https://scrape.sparkproxy.io/api/v1",
    headers={"X-API-Key": "YOUR_API_KEY"},
    params={
        "url": "https://www.indeed.com/viewjob?jk=a1b2c3d4e5f6a7b8",
        "render_js": "true",
        "premium_proxy": "true",
        "stealth": "true",
        "country_code": "US",
        "extract_rules": json.dumps(rules),
    },
    timeout=90,
)
data = resp.json()   # {"title": "...", "company": "...", "salary": "$70,000 ...", ...}

Check the docs for the exact extract_rules syntax your plan exposes, including how to pull an attribute versus text. The trade is the same one from the last section, moved server-side: rules mean you do not ship a parser, but you still update the selectors when Indeed changes them. For a fixed field set like salary benchmarking, extract_rules is usually the lower-maintenance path.

Search pages are where Indeed scraping earns its keep, because they let you discover postings by keyword and location. The search URL takes a few parameters worth knowing:

• q: the query ("data engineer", "registered nurse").
• l: the location ("New York, NY", "Remote", a ZIP).
• start: the pagination offset, in increments of 10.
• fromage: max posting age in days (1, 3, 7, 14), which keeps a run to fresh listings.
• sort=date: newest first, which is what you want for tracking new postings.

Geo-targeting has two layers. Set l= for the location within a market, and set country_code (plus the right domain: indeed.com for the US, uk.indeed.com for the UK, de.indeed.com for Germany) so the exit IP country matches. Mismatching them triggers redirects and region interstitials.

from urllib.parse import urlencode

def search_url(query: str, location: str, start: int = 0,
               fromage: int = 7, sort: str = "date") -> str:
    params = {"q": query, "l": location, "start": start,
              "fromage": fromage, "sort": sort}
    return "https://www.indeed.com/jobs?" + urlencode(params)

def parse_search_cards(html: str) -> list[dict]:
    tree = HTMLParser(html)
    rows = []
    for card in tree.css("div.job_seen_beacon, [data-testid='slider_item']"):
        link = card.css_first("a.jcs-JobTitle, h2.jobTitle a")
        if link is None:
            continue
        jk = link.attributes.get("data-jk")
        if not jk:
            continue
        title = card.css_first("h2.jobTitle span[title], a.jcs-JobTitle")
        company = card.css_first('[data-testid="company-name"]')
        location = card.css_first('[data-testid="text-location"]')
        salary = card.css_first('[data-testid="attribute_snippet_testid"], '
                                '.salary-snippet-container')
        rows.append({
            "job_id": jk,
            "title": title.text(strip=True) if title else None,
            "company": company.text(strip=True) if company else None,
            "location": location.text(strip=True) if location else None,
            "salary_raw": salary.text(strip=True) if salary else None,
        })
    return rows

To walk every page, request &start=N in steps of 10 and stop when a page returns no cards. Dedupe on jk as you go, because Indeed repeats cards across page boundaries:

def scrape_search(query: str, location: str, max_pages: int = 10) -> list[dict]:
    seen, results = set(), []
    for page in range(max_pages):
        url = search_url(query, location, start=page * 10)
        html = fetch(url, wait_for="#mosaic-provider-jobcards")
        if is_blocked(html):
            continue
        cards = parse_search_cards(html)
        if not cards:
            break
        new = [c for c in cards if c["job_id"] not in seen]
        if not new:               # nothing new means we've hit the tail
            break
        for c in new:
            seen.add(c["job_id"])
        results.extend(new)
    return results

One hard limit to design around: Indeed caps how deep organic search pagination goes for a single query, so a broad search like "engineer" in "United States" leaves most postings unreachable. The fix is to narrow. Split by specific title, city, and fromage window, run each tight query to its cap, then dedupe across them by job_id. Twenty narrow queries surface far more of the market than one broad one.

Raw cards are not a dataset yet. Two cleanup steps separate a real job market data scraping pipeline from a pile of HTML: salary normalization and posting deduplication. Both are where most tutorials stop, and both quietly corrupt any analysis you run downstream.

Salary normalization. Indeed shows pay in mixed periods ("$28 an hour", "$70,000 - $90,000 a year", "$4,500 a month") and often as an Indeed-estimated range rather than an employer-posted figure. Mixing estimated and posted salaries, or averaging hourly and annual numbers as-is, produces garbage benchmarks. Parse the raw string into an annualized min and max, and flag whether it was estimated:

import re

HOURS_PER_YEAR = 2080          # 40h x 52 weeks
PERIOD_MULTIPLIER = {"hour": HOURS_PER_YEAR, "year": 1,
                     "month": 12, "week": 52, "day": 260}

def normalize_salary(raw: str | None) -> dict:
    if not raw:
        return {"min": None, "max": None, "estimated": None}
    estimated = "estimat" in raw.lower()   # Indeed labels its own guesses
    period = next((p for p in PERIOD_MULTIPLIER if p in raw.lower()), "year")
    nums = [float(n.replace(",", "")) for n in re.findall(r"[\d,]+\.?\d*", raw)]
    nums = [n for n in nums if n > 5]       # drop stray digits like "40h"
    if not nums:
        return {"min": None, "max": None, "estimated": estimated}
    mult = PERIOD_MULTIPLIER[period]
    annual = [round(n * mult) for n in nums]
    return {"min": min(annual), "max": max(annual), "estimated": estimated}

# normalize_salary("$28 an hour")            -> {'min': 58240, 'max': 58240, ...}
# normalize_salary("$70,000 - $90,000 a year") -> {'min': 70000, 'max': 90000, ...}

Posting deduplication. Because staffing agencies re-post the same role under different jk values, counting by jk overstates demand. Dedupe on a content fingerprint of the normalized title, company, and location so each real opening counts once:

from collections import Counter
from statistics import median

def fingerprint(row: dict) -> str:
    parts = [(row.get(k) or "").strip().lower() for k in ("title", "company", "location")]
    return "|".join(parts)

def analyze(rows: list[dict]) -> dict:
    unique = {fingerprint(r): r for r in rows}.values()   # one per real opening
    by_location = Counter(r["location"] for r in unique if r.get("location"))
    posted = [normalize_salary(r.get("salary_raw")) for r in unique]
    mids = [(s["min"] + s["max"]) / 2 for s in posted
            if s["min"] and not s["estimated"]]            # employer-posted only
    return {
        "total_cards": len(rows),
        "unique_openings": len(list(unique)),
        "top_locations": by_location.most_common(5),
        "median_posted_salary": round(median(mids)) if mids else None,
    }

Filtering to employer-posted salaries before you take a median is the detail that makes the number trustworthy. From here, the same records support hiring-velocity tracking (count new job_ids per day with fromage=1), skill-demand analysis (regex the description text for tools and certifications), and remote-versus-onsite splits. The proxy patterns behind sustained collection like this are covered in Using Proxies for Market Research and Data Collection.

At volume, three things keep the pipeline healthy: retries on soft blocks, backoff so you do not spike a single IP, and modest concurrency. With a scraping API the provider rotates the exit IP for you, so your ceiling is your plan's rate limit rather than the number of proxies you own. Keep worker counts sane (5 to 15 is plenty) and let retries absorb the occasional Cloudflare challenge.

import time
import random
from concurrent.futures import ThreadPoolExecutor, as_completed

def fetch_job_with_retry(jk: str, attempts: int = 3) -> str | None:
    for i in range(attempts):
        html = fetch_job(jk)
        if not is_blocked(html) and "jobDescriptionText" in html:
            return html
        time.sleep(2 ** i + random.random())   # exponential backoff + jitter
    return None

def scrape_jobs(job_ids: list[str], workers: int = 8) -> list[dict]:
    out = []
    with ThreadPoolExecutor(max_workers=workers) as pool:
        futures = {pool.submit(fetch_job_with_retry, jk): jk for jk in job_ids}
        for fut in as_completed(futures):
            html = fut.result()
            if html is None:
                continue
            out.append(parse_job(html, futures[fut]))
    return out

The jitter matters more than it looks. random.random() staggers retries so a batch of failures does not retry in lockstep and re-trigger the same block. Persist results as you go instead of holding everything in memory, so a crash at posting 40,000 does not cost you the first 39,999. A flat CSV is enough to start:

import csv

def save_csv(rows: list[dict], path: str = "indeed_jobs.csv") -> None:
    if not rows:
        return
    with open(path, "w", newline="", encoding="utf-8") as f:
        writer = csv.DictWriter(f, fieldnames=list(rows[0].keys()))
        writer.writeheader()
        writer.writerows(rows)

For a running job-market tracker, append a scraped_at timestamp and the query/location you used to each row, and write to a database keyed on (job_id, scraped_at). That gives you a clean time series where posting counts and salary trends are comparable run to run. The general proxy patterns behind high-volume collection are in Using Datacenter Proxies for Web Scraping.