How to Scrape LinkedIn Data (Public Data Only)

LinkedIn is not an e-commerce catalog. It's a database of real people's careers, so the ethics here carry more weight than on any product-scraping job, and they decide the design before the code does. LinkedIn also fights scrapers harder than almost anyone, in court and in production.

Three separate legal questions matter, and "it's public" only answers the first one. The hiQ case is worth knowing in full, because most write-ups stop at the half that helps them:

• Unauthorized access (CFAA). In April 2022, after the Supreme Court sent the case back in light of Van Buren, the Ninth Circuit reaffirmed that scraping data which is publicly accessible, meaning no login and no authentication wall, generally does not violate the Computer Fraud and Abuse Act. That is a ruling about access, not a license to do anything with what you collect.
• Contract. The story did not end there. Later in 2022 the district court found that hiQ had breached LinkedIn's User Agreement, and the parties settled with hiQ agreeing to a permanent injunction against scraping LinkedIn. So the access question and the contract question came out differently. LinkedIn's User Agreement, Section 8.2, flatly prohibits using bots, crawlers, or scripts to copy profiles or other data. Scraping public pages can win on CFAA and still breach that contract.
• Data protection. A person's name, headline, employer, and location are personal data. Public availability is not an exemption under GDPR or similar laws. This gets its own section below because it's where a LinkedIn project most often goes wrong.

Guardrails that keep a LinkedIn project defensible:

• Collect public data only. No logged-in sessions, no authenticated cookies, no accounts, ever.
• Prefer organizational data. Company pages and job posts carry far less personal-data risk than individual member profiles. This guide leans there on purpose.
• Pull the minimum you need. Aggregate hiring trends and firmographics, not dossiers on named individuals.
• Rate-limit yourself and back off on errors so you never degrade the service for real users.
• Honor deletion. If a post, job, or profile disappears, drop it from your store.
• Get a lawyer involved before anything commercial. This is engineering guidance, not legal advice.

The legitimate reasons to want public LinkedIn data are real: competitive hiring intelligence, market mapping, firmographic enrichment, salary and skills trend research. If your goal is recruiting or talent analytics specifically, the business-side patterns live in Proxies for Recruitment Data. The point of this section is that the use case has to survive scrutiny before the pipeline is worth building.

Before you scrape anything, check whether an official API covers your need, because that's the only route LinkedIn actually blesses, and it sidesteps every wall below.

The catch is that LinkedIn's developer platform is deliberately narrow. There is no public API that hands you an arbitrary member's profile or a competitor's company page on demand. What exists:

• Sign In with LinkedIn (OpenID Connect) returns the authenticated member's own basic profile (name, picture, email) after they log in and consent. It's for building "log in with LinkedIn" flows, not for collecting other people's data.
• Marketing, Share, and Community Management APIs cover posting and analytics for pages you administer.
• Talent Solutions and the Job Posting API are gated to approved ATS and recruiting partners, and they're built to publish jobs, not to read arbitrary listings.

A minimal Sign In with LinkedIn call, once you hold a member's consented access token, reads that member's own profile:

# Sanctioned route: the member's OWN profile, after OpenID Connect consent
curl "https://api.linkedin.com/v2/userinfo" \
  -H "Authorization: Bearer <MEMBER_ACCESS_TOKEN>"

Where the official platform falls short is exactly the public-data collection people usually want: strangers' profiles, other companies' pages, the open job market. There's no blessed endpoint for those. That gap is why people reach for HTML scraping, and it's also where the legal weight lands hardest. If your need fits inside your own pages or a partner integration, stop here and use the API. Everything below is for public data the API genuinely does not reach.

A logged-out visit exposes very different amounts of data depending on the target. Company pages and job posts are relatively open. Personal profiles are heavily gated and often blocked outright. Here's the reference set worth pulling as of mid-2026, grouped by target.

The single most useful thing on this list, and the one most tutorials miss, is that LinkedIn ships a full JSON-LD JobPosting block on every public job page so Google for Jobs can index it. You don't fight fragile HTML for job data. You parse one structured JSON object with title, description, dates, location, and hiring company already typed for you. Company pages carry a similar JSON-LD Organization block. Personal profiles are the opposite story: the fields exist, but the page is usually an authwall, and this is the target you should lean away from.

LinkedIn breaks naive scrapers faster than almost any other target. Four defenses do the damage:

The authwall. LinkedIn gates most member content behind a login or "Join LinkedIn" modal, and the gate frequently returns HTTP 200 with a page that's really a sign-in prompt. If your code trusts the status code, response.ok is True, you save the "page", and you've stored an authwall instead of a profile. You have to inspect the body.

HTTP 999. LinkedIn's signature block is a non-standard 999 status with a Request denied body. No other major site uses it. When you see 999, you've been rate-limited or fingerprinted, and hammering the same IP only deepens the block.

Fingerprinting. LinkedIn reads TLS fingerprints (JA3/JA4), header order, and JavaScript signals. A plain Python urllib3 handshake looks nothing like a browser, so you get walled regardless of your User-Agent. Real content needs a genuine browser or a fingerprint-matching client.

IP reputation. Datacenter ranges are flagged almost instantly on LinkedIn. A single static datacenter proxy dies within a handful of requests. Residential and mobile IPs blend in and survive far longer.

A managed scraping API absorbs all four. The parsing is still yours, because the fields live in the HTML and JSON-LD, but the anti-bot arms race moves off your plate. For the deeper theory on ban avoidance, How to Avoid Getting Your Proxy Blocked covers it end to end.

Proxy choice is the difference between a scraper that runs and one that gets an authwall on request two.

Datacenter proxies are the wrong tool here. LinkedIn flags their IP ranges quickly, so they burn out fast even with rotation. They're fine for easy targets, not for this one. Residential proxies route through real consumer ISP connections, so their IPs carry the reputation of ordinary home users, and LinkedIn trusts them far more. They're the practical default for public company and job collection. Mobile proxies route through cellular carrier IPs, and because carriers share a small pool across thousands of subscribers behind CGNAT, a mobile IP is the hardest to ban without hitting real users. They're the strongest option for the toughest jobs, and the priciest.

If you're fuzzy on the type, What Is a Residential Proxy: Types and Use Cases breaks down how they differ from datacenter and mobile. With the SparkProxy Scraping API you don't manage any pool directly. Setting premium_proxy=true routes the request through residential IPs, so the whole proxy decision collapses into one parameter.

The SparkProxy Scraping API takes a target URL and handles the proxy, rotation, browser rendering, and anti-bot layer for you. You send one request and get the rendered HTML back. For LinkedIn, four parameters carry the weight:

• render_js=true: LinkedIn paints member content in JavaScript, so a raw fetch returns a shell.
• premium_proxy=true: routes through residential IPs that survive LinkedIn's defenses.
• stealth=true: adds extra anti-bot layers, and it requires render_js=true.
• country_code: the ISO alpha-2 exit country, useful when a page or job market is geo-fenced.

The base endpoint is https://scrape.sparkproxy.io/api/v1 and auth is a single header, X-API-Key. A minimal request in cURL against a public company page:

curl -G "https://scrape.sparkproxy.io/api/v1" \
  -H "X-API-Key: YOUR_API_KEY" \
  --data-urlencode "url=https://www.linkedin.com/company/microsoft/about/" \
  --data-urlencode "render_js=true" \
  --data-urlencode "premium_proxy=true" \
  --data-urlencode "stealth=true" \
  --data-urlencode "country_code=US"

Wrap that in one helper so every call carries the LinkedIn-specific parameters. Job search fragments (next section) return static HTML, so you can turn rendering off there and save credits, which is why the helper adds stealth only when it's rendering:

import requests

API = "https://scrape.sparkproxy.io/api/v1"
API_KEY = "YOUR_API_KEY"

def fetch(url: str, render_js: bool = True, country: str = "US") -> str:
    params = {
        "url": url,
        "render_js": "true" if render_js else "false",
        "premium_proxy": "true",       # residential IPs; datacenter dies on LinkedIn
        "country_code": country,
    }
    if render_js:
        params["stealth"] = "true"     # stealth requires render_js=true
    resp = requests.get(API, headers={"X-API-Key": API_KEY}, params=params, timeout=90)
    resp.raise_for_status()
    return resp.text

The full parameter list and response fields live in the Scraping API docs. If you're weighing this against building your own rotation, browser farm, and fingerprint stack, Web Scraping API vs Self-Managed Proxies lays out the build-versus-buy math honestly.

Before you trust any response, check whether LinkedIn handed you an authwall. Because that gate returns 200, raise_for_status() won't catch it, and the 999 block ships its own tell-tale body:

def is_blocked(html: str) -> bool:
    """LinkedIn hides public pages behind an authwall (often HTTP 200) and returns a
    non-standard 999 with a 'Request denied' body. The status code alone misses both."""
    markers = (
        "authwall",
        "please sign in",
        "join linkedin",
        "sign in to see",
        "request denied",   # ships with the HTTP 999 block
    )
    low = html.lower()
    return any(m in low for m in markers)

Company pages are the friendliest LinkedIn target and the lowest personal-data risk, so start here. The reliable source is the JSON-LD Organization block, with og meta tags as a fallback. Use selectolax (a C-backed parser) for speed at volume; install it with pip install selectolax.

import json
from selectolax.parser import HTMLParser

def parse_company(html: str) -> dict:
    tree = HTMLParser(html)

    def meta(prop):
        node = tree.css_first(f'meta[property="{prop}"]')
        return node.attributes.get("content") if node else None

    org = {}
    for ld in tree.css('script[type="application/ld+json"]'):
        try:
            data = json.loads(ld.text())
        except (json.JSONDecodeError, ValueError):
            continue
        blocks = data.get("@graph", [data]) if isinstance(data, dict) else data
        for b in (blocks if isinstance(blocks, list) else [blocks]):
            if isinstance(b, dict) and b.get("@type") == "Organization":
                org = b
                break

    addr = org.get("address") or {}
    same_as = org.get("sameAs")
    return {
        "name": org.get("name") or meta("og:title"),
        "description": org.get("description") or meta("og:description"),
        "website": org.get("url") or (same_as if isinstance(same_as, str) else None),
        "employees": org.get("numberOfEmployees"),
        "city": addr.get("addressLocality"),
        "country": addr.get("addressCountry"),
        "logo": org.get("logo") or meta("og:image"),
    }

Request the /about/ path rather than the root company URL. It's the most content-rich public view and the least likely to bounce a logged-out client. The og:image logo is a signed, expiring CDN link, so rehost it immediately rather than storing the raw URL.

Job data is where public LinkedIn scraping actually shines, for two reasons: the data is about roles and companies rather than individuals, and LinkedIn hands it to you as clean structured data.

Enumerating jobs uses the guest search endpoint, which returns a static HTML fragment of job cards without a login. Each card carries the job's URN, so one regex gives you the IDs:

import re
from urllib.parse import quote_plus

JOBS_SEARCH = "https://www.linkedin.com/jobs-guest/jobs/api/seeMoreJobPostings/search"
JOB_URN_RE = re.compile(r"urn:li:jobPosting:(\d+)")

def search_job_ids(keywords: str, location: str, start: int = 0) -> list[str]:
    q = f"keywords={quote_plus(keywords)}&location={quote_plus(location)}&start={start}"
    html = fetch(f"{JOBS_SEARCH}?{q}", render_js=False)   # static fragment, no JS needed
    seen, out = set(), []
    for jid in JOB_URN_RE.findall(html):
        if jid not in seen:
            seen.add(jid)
            out.append(jid)
    return out

Paginate by incrementing start. LinkedIn returns a batch of cards per call, and the page size shifts, so don't hardcode it. Walk start in steps and stop when a call returns no new IDs. That self-terminating loop is more durable than assuming 25 results a page.

Now the payoff. Each public job page at /jobs/view/ embeds a JSON-LD JobPosting object, the same structured data Google for Jobs consumes. Parse that instead of the visible HTML and you get typed fields for free:

def parse_job(job_id: str) -> dict:
    html = fetch(f"https://www.linkedin.com/jobs/view/{job_id}")
    if is_blocked(html):
        return {"id": job_id, "blocked": True}

    tree = HTMLParser(html)
    posting = {}
    for ld in tree.css('script[type="application/ld+json"]'):
        try:
            data = json.loads(ld.text())
        except (json.JSONDecodeError, ValueError):
            continue
        if isinstance(data, dict) and data.get("@type") == "JobPosting":
            posting = data
            break

    org = posting.get("hiringOrganization") or {}
    loc = (posting.get("jobLocation") or {}).get("address", {})
    return {
        "id": job_id,
        "title": posting.get("title"),
        "company": org.get("name"),
        "employment_type": posting.get("employmentType"),
        "date_posted": posting.get("datePosted"),
        "valid_through": posting.get("validThrough"),
        "city": loc.get("addressLocality"),
        "country": loc.get("addressCountry"),
        "description_html": posting.get("description"),
    }

The description field arrives as HTML, so strip tags before analysis if you want plain text. If you'd rather scrape a job board that's far more scraper-friendly than LinkedIn, How to Scrape Indeed Job Listings walks the same pipeline against an easier target.

Personal profiles are the hardest target, the least reliable, and the one that carries the most legal weight. Most logged-out profile requests hit the authwall, and even when a page renders, you're collecting personal data on a named individual. Treat this as the exception, not the goal, and let the parser bail the moment LinkedIn serves a wall.

def parse_profile(html: str) -> dict:
    if is_blocked(html):
        return {"blocked": True}   # LinkedIn served an authwall, not a profile

    tree = HTMLParser(html)
    person = {}
    for ld in tree.css('script[type="application/ld+json"]'):
        try:
            data = json.loads(ld.text())
        except (json.JSONDecodeError, ValueError):
            continue
        graph = data.get("@graph", [data]) if isinstance(data, dict) else data
        for b in (graph if isinstance(graph, list) else [graph]):
            if isinstance(b, dict) and b.get("@type") == "Person":
                person = b
                break

    works_for = person.get("worksFor")
    if isinstance(works_for, list) and works_for:
        company = works_for[0].get("name")
    elif isinstance(works_for, dict):
        company = works_for.get("name")
    else:
        company = None

    return {
        "name": person.get("name"),
        "headline": person.get("description") or person.get("jobTitle"),
        "company": company,
        "location": (person.get("address") or {}).get("addressLocality"),
        "url": person.get("url"),
    }

Notice how little this returns compared to the logged-in profile you see as a user. That gap is by design, and trying to close it means defeating an access control, which puts you back on the wrong side of the CFAA line the hiQ ruling drew. Public means public. If a field only appears after login, it is out of scope for this pipeline.

LinkedIn tracks pressure per IP, per fingerprint, and per session, and it punishes bursts hard. The block shows up as an authwall, a 429, or the 999 code before it hardens into a temporary IP ban. Four habits keep a LinkedIn scraper healthy:

• Rotate the exit IP per request. With the Scraping API this is automatic; premium_proxy=true hands you a fresh residential IP each call.
• Keep concurrency low. Two to five workers is plenty. LinkedIn is stricter than most targets, and job data does not need to arrive fast.
• Back off with jitter. On a block, retry with exponential backoff plus a random delay so a batch of failures doesn't retry in lockstep.
• Cache aggressively. A company's employee range or a job's description does not change minute to minute. Read from your own store before you re-scrape.

import time, random

def fetch_with_retry(url: str, render_js: bool = True, attempts: int = 3) -> str | None:
    for i in range(attempts):
        try:
            html = fetch(url, render_js=render_js)
        except requests.HTTPError:
            html = ""
        if html and not is_blocked(html):
            return html
        time.sleep(2 ** i + random.random())   # exponential backoff + jitter
    return None

Persist as you go rather than holding a run in memory, so a crash at job 8,000 doesn't cost you the first 7,999. Stamp each row with a scraped_at timestamp so you can build clean time series and prove exactly when a data point was collected, which matters for the compliance record too.

This is where a LinkedIn project lives or dies, so read it before you scale anything. It's also why this guide pushed you toward company pages and job posts and away from member profiles.

Under GDPR, a person's name, headline, employer, and location are personal data, and Article 4 defines that broadly. Data being publicly available is not an exemption. If any of your data subjects are in the EU or UK, collecting and storing their public LinkedIn data is processing, and processing needs a legal footing:

• A lawful basis (Article 6). Usually legitimate interest, which requires a documented balancing test weighing your purpose against the individual's rights. Consent is rarely workable at scrape scale.
• No special-category data (Article 9). Do not infer or store health, religion, political views, union membership, or sexual orientation, even where a profile hints at them.
• Transparency (Articles 13 and 14). You generally have to inform people you hold their data. Article 14(5) offers a narrow "disproportionate effort" exemption, and regulators read it strictly.
• Absolutely no biometrics. Do not run face recognition on scraped photos. France's CNIL fined Clearview AI 20 million euros in 2022 for scraping public photos into a facial-recognition database, and other EU regulators followed with similar penalties. That is the cautionary tale for anyone who thinks "public photos are fair game."

In August 2023, twelve data-protection authorities led by the UK's ICO issued a joint statement making clear that publicly accessible personal data is still protected and that scrapers carry obligations. Regulators are watching this space specifically, and LinkedIn is one of the platforms they name.

Practical compliance that keeps you defensible: prefer firmographic and job-market data over individual profiles, minimize to aggregate signals, avoid building searchable profiles of named people, never touch biometric data, run a Data Protection Impact Assessment before a large collection, honor erasure requests, and delete anything a user removes. When in doubt, collect less. The safest LinkedIn dataset studies companies and roles, not people.