How to Scrape Twitter/X Data (Public Data Only)

X is not a product catalog. Almost every field you can pull describes a real person and their speech, so the ethics carry more weight than on a price-scraping job, and they shape the design before any code gets written.

Three separate questions matter, and "it's public" only answers the first:

• Access. In the US, scraping data that's publicly accessible with no login has generally not been treated as a Computer Fraud and Abuse Act violation. That is about access, not a license to do anything you want with what you collect.
• Contract. X's Terms of Service prohibit automated collection outright. Scraping public pages can still breach that contract even where it isn't a computer-crime issue, and X has taken scrapers to court. Two different questions.
• Data protection. A handle, a photo, and a post are personal data under GDPR and similar laws, and public availability is not an exemption. This one gets its own treatment near the end.

Guardrails that keep an X project defensible:

• Collect public data only. No logged-in scraping of other people's timelines, no private or protected accounts, ever.
• Pull the minimum you need. Aggregate counts and public post text, not dossiers on named individuals.
• Rate-limit yourself and back off on errors so you never degrade the service for real users.
• Honor deletion. If a tweet or account disappears, drop it from your store.
• Talk to a lawyer before anything commercial. This is engineering guidance, not legal advice.

The legitimate reasons to want public X data are real: brand monitoring, trend research, crisis tracking, academic study. If that is your use case, the business-side patterns live in Using Proxies for Social Media Monitoring. The point of this section is that the use case has to survive scrutiny before the pipeline is worth building.

If you follow a 2021-era tutorial, it will fail, and understanding why saves you days.

For years, Twitter let anyone read tweets, profiles, and search results without an account. A whole ecosystem grew on top of that open access: snscrape parsed the logged-out search endpoints, Nitter proxied the site through guest tokens, and dozens of libraries scraped profile timelines with no auth at all.

That era ended in stages. In mid-2023 X briefly forced a login to view any tweet, reverted it days later after backlash (see TechCrunch's report), then settled into a durable middle ground: individual tweet pages sometimes render logged-out, but profile timelines and search now demand a login, and the guest-token flow that powered the old tools was throttled into uselessness through 2023 and 2024. Nitter instances went dark one after another for the same reason.

So the modern reality is blunt. There is no reliable, no-login way to scrape an arbitrary profile's timeline or run a keyword search on x.com. Those need the official API or an authenticated session. A single public tweet is the one exception, and it has a clean route covered below.

Before you scrape anything, check whether the official API covers your need, because it is the only route X blesses and the only one that scales without a compliance cloud over it.

X repriced its API more than once since 2023, and the model changed again in February 2026, when new developers were moved to pay-per-use. Confirm the live numbers on developer.x.com before you budget, because X moves them.

On June 1, 2026, X began auto-migrating remaining legacy Basic subscribers to pay-per-use. New signups can no longer buy Basic or Pro, so your practical choices today are pay-per-use or Enterprise.

The API base is https://api.x.com/2 (the old api.twitter.com host still resolves) and auth is a bearer token. Recent search covers roughly the last seven days:

import requests

BEARER = "YOUR_X_API_BEARER_TOKEN"

def search_recent(query, max_results=10):
    r = requests.get(
        "https://api.x.com/2/tweets/search/recent",
        headers={"Authorization": f"Bearer {BEARER}"},
        params={
            "query": query,                     # e.g. 'from:nasa -is:retweet'
            "max_results": max_results,         # 10 to 100 per page
            "tweet.fields": "created_at,public_metrics,lang",
            "expansions": "author_id",
            "user.fields": "username,verified,public_metrics",
        },
        timeout=30,
    )
    r.raise_for_status()
    return r.json()

Full-archive search (anything older than a week) is a Pro or Enterprise feature. Profile lookups are a separate endpoint that returns the public counts most projects actually want:

def user_by_username(username):
    r = requests.get(
        f"https://api.x.com/2/users/by/username/{username}",
        headers={"Authorization": f"Bearer {BEARER}"},
        params={"user.fields": "created_at,description,public_metrics,verified,location"},
        timeout=30,
    )
    r.raise_for_status()
    return r.json()

public_metrics on a user hands you followers_count, following_count, tweet_count, and listed_count in one call, with no HTML parsing and no proxy. If your need fits inside the API budget, stop here and use it. Everything below is for the public tweet data the API bills you for but a public endpoint gives away.

Here is the reference set of public fields and where each one actually comes from, as of mid-2026.

The split is the important part. Everything on a single tweet is reachable without login. Everything about a profile as a whole (followers, bio, timeline) needs the API. Plan your pipeline around that line instead of fighting it.

Here is the piece most current tutorials miss. X's own embed system fetches public tweet data from a CDN endpoint that needs no login and no API key: https://cdn.syndication.twimg.com/tweet-result. It is what renders embedded tweets across the web, and it is what Vercel's open-source react-tweet library calls under the hood.

The endpoint wants a tweet ID and a token derived from that ID. The token is a lightweight plausibility check, not a signature, and react-tweet computes it like this:

// The scheme Vercel's react-tweet uses
const getToken = (id) =>
  ((Number(id) / 1e15) * Math.PI)
    .toString(6 ** 2)          // base 36
    .replace(/(0+|\.)/g, "");

A Python port that mirrors the same base-36 encoding:

import math, re

_DIGITS = "0123456789abcdefghijklmnopqrstuvwxyz"

def _to_base36(value: float, frac_digits: int = 12) -> str:
    n = int(value)
    frac = value - n
    out = "0" if n == 0 else ""
    while n:
        out = _DIGITS[n % 36] + out
        n //= 36
    out += "."
    for _ in range(frac_digits):
        frac *= 36
        d = int(frac)
        out += _DIGITS[d]
        frac -= d
    return out

def tweet_token(tweet_id: str) -> str:
    raw = _to_base36((int(tweet_id) / 1e15) * math.pi)
    return re.sub(r"(0+|\.)", "", raw)

Fetch the tweet and treat an empty body as a miss (deleted, protected, or a token that needs recomputing). Fall back to the official oEmbed endpoint or the API if you get nothing back:

import requests

def fetch_tweet(tweet_id: str) -> dict | None:
    r = requests.get(
        "https://cdn.syndication.twimg.com/tweet-result",
        params={"id": tweet_id, "token": tweet_token(tweet_id), "lang": "en"},
        headers={"User-Agent": "Mozilla/5.0"},
        timeout=20,
    )
    if r.status_code != 200 or not r.text.strip():
        return None
    return r.json()

The JSON is rich. Flatten the parts you care about:

def parse_tweet(data: dict) -> dict:
    user = data.get("user", {})
    return {
        "id": data.get("id_str"),
        "text": data.get("text"),
        "created_at": data.get("created_at"),
        "likes": data.get("favorite_count"),
        "replies": data.get("conversation_count"),
        "lang": data.get("lang"),
        "author_name": user.get("name"),
        "author_handle": user.get("screen_name"),
        "author_verified": user.get("is_blue_verified"),
        "hashtags": [h["text"] for h in data.get("entities", {}).get("hashtags", [])],
        "media": [m.get("media_url_https") for m in data.get("mediaDetails", [])],
    }

One honest caveat: this payload carries likes (favorite_count) and replies (conversation_count) but not retweet or impression counts. Those live behind the authenticated API. Don't build a dashboard that assumes fields the embed route never returns.

The syndication endpoint runs fine from your laptop for a dozen tweets. Push past that and cdn.syndication.twimg.com rate-limits per IP, so you start seeing empty responses and 429s. Routing the same request through the SparkProxy Scraping API puts a fresh IP behind each call and takes the rotation off your plate.

Because the endpoint returns plain JSON with no browser rendering needed, keep render_js=false, which bills at 1 credit per request instead of 5. Add premium_proxy=true to route through residential IPs that shrug off the per-IP limit. If you are unsure why datacenter IPs get flagged first here, What Is a Residential Proxy: Types and Use Cases explains the reputation difference.

import requests

SPARK_API = "https://scrape.sparkproxy.io/api/v1"
SPARK_KEY = "YOUR_API_KEY"

def fetch_tweet_via_spark(tweet_id: str) -> dict | None:
    target = (
        "https://cdn.syndication.twimg.com/tweet-result"
        f"?id={tweet_id}&token={tweet_token(tweet_id)}&lang=en"
    )
    r = requests.get(
        SPARK_API,
        headers={"X-API-Key": SPARK_KEY},
        params={
            "url": target,
            "render_js": "false",       # plain JSON endpoint, 1 credit
            "premium_proxy": "true",    # residential IPs dodge the per-IP limit
        },
        timeout=60,
    )
    if r.status_code != 200 or not r.text.strip():
        return None
    return r.json()

The base endpoint is https://scrape.sparkproxy.io/api/v1 and auth is a single header, X-API-Key. The full parameter list lives in the Scraping API docs. If you are weighing this against building your own rotation and IP pool, Web Scraping API vs Self-Managed Proxies lays out the build-versus-buy math.

With a fetcher that rotates IPs, batching is mostly discipline: retry with backoff, cache what you already have, and persist as you go so a crash at tweet 8,000 doesn't cost you the first 7,999.

import time, random

def collect(ids):
    out = {}
    for tid in ids:
        for attempt in range(3):
            data = fetch_tweet_via_spark(tid)
            if data and data.get("text"):
                out[tid] = parse_tweet(data)
                break
            time.sleep(2 ** attempt + random.random())   # backoff + jitter
    return out

Four habits keep a Twitter scraper healthy at volume:

• Deduplicate by id_str. A tweet ID is stable, so a set of seen IDs stops you paying to fetch the same tweet twice.
• Cache hard. A tweet's text never changes and its like count barely moves after a day. Re-scraping settled tweets is wasted spend.
• Keep concurrency modest. Even through rotating IPs, a slower steady rate draws less attention than a burst.
• Stamp every row with a scraped_at timestamp. It gives you clean time series and a record of exactly when each data point was true, which matters for the compliance trail.

For anti-block theory beyond this, How to Avoid Getting Your Proxy Blocked goes deep on the mechanics.

The syndication route is per-tweet only. For a profile timeline or a keyword search, the honest answer is that the no-login web paths are gone. Here is the full decision matrix.

The official oEmbed endpoint is a clean, no-key way to grab a single tweet's embed markup and author details:

curl -G "https://publish.twitter.com/oembed" \
  --data-urlencode "url=https://x.com/nasa/status/1234567890123456789" \
  --data-urlencode "omit_script=true"

There is a fourth path, and it deserves a warning rather than a snippet. Some scrapers drive a logged-in browser session against x.com and read the internal GraphQL endpoints (UserByScreenName, UserTweets, SearchTimeline) using the account's bearer, guest token, and CSRF cookie. It works. It also means automating your own X account against Terms that prohibit scraping, which can get the account suspended and, at commercial scale, invite legal exposure. You can pass session cookies to the Scraping API through the cookies or forward_headers parameters, but weigh what you are risking first. For anything that has to survive scrutiny, the API or Enterprise access is the route that holds up.

Copy-paste scrapers fail on X for four specific reasons. Knowing the signal tells you which route to take instead.

The takeaway is not "try harder to scrape the HTML." It is "match each need to the route that still works": syndication for single tweets, the API for profiles and search, and rotating residential IPs behind whichever endpoint you hit at volume.

This is where an X project lives or dies, so read it before you scale.

The contract. X's Terms of Service ban scraping and automated data collection. That is true regardless of how "public" the data is, and it is a separate question from whether scraping is a crime. X has litigated it: in X Corp. v. Bright Data (N.D. Cal., 2024), a court dismissed X's claims over scraping publicly available data, siding with the scraper, while the broader fight over Terms enforcement continues in other cases. The law here is unsettled and jurisdiction-specific, which is exactly why the API is the low-drama path.

Data protection. Under GDPR, a person's handle, photo, and posts are personal data, and Article 4 reads that broadly. Public availability is not an exemption. If any of your data subjects sit in the EU or UK, collecting and storing their public tweets is processing, and processing needs a lawful basis (usually a documented legitimate-interest assessment), a transparency plan, and hard limits: no special-category inference, no biometric analysis of profile photos, and prompt deletion when someone removes content. When in doubt, collect less. The safest X dataset studies trends and cohorts, not named individuals.

Practical compliance that keeps you defensible: minimize to public post text and aggregate counts, avoid building searchable profiles of specific people, honor erasure, and get legal sign-off before anything commercial. This is engineering guidance, not legal advice.