How to Scrape Twitch Data: Streams, Channels, Clips

Twitch has two data surfaces, and confusing them is the first mistake people make.

The Helix API lives at https://api.twitch.tv/helix. It is documented at dev.twitch.tv, it needs a registered Client ID and an access token, and it is governed by the Twitch Developer Services Agreement. This is the compliant, stable route. Build on it first.

The private GraphQL API lives at https://gql.twitch.tv/gql. It is the same backend the twitch.tv web app talks to, so it returns things Helix never will: directory ordering as a real viewer sees it, richer clip metadata, panel content, recommendation shelves. It is undocumented, it can change any day, and Twitch does not sanction third-party use of it.

Here is the practical split:

The rate-limit column is the detail that decides your architecture, and section 8 comes back to it. Helix throttles you by client ID, so rotating IPs buys you nothing there. GraphQL throttles you by IP, so proxies are the whole game. Default to Helix. Drop to GraphQL only for public data Helix cannot return, and route those calls through proxies.

Before writing a line of code, know which endpoint holds the field you want. Twitch scatters data across narrow endpoints, and a few fields people expect are simply gone.

Two gaps trip people up. Total channel views were removed from GET /users years ago, so there is no lifetime view count in Helix anymore. Live viewer count only exists while a channel is live, and it comes from GET /streams, not from any channel endpoint. A channel that is offline returns zero rows from /streams, which is exactly how you detect that it went offline.

Every Helix call needs a Client ID plus a token. For scraping public data with no logged-in user, you want an app access token through the client credentials grant.

Register an application at dev.twitch.tv/console/apps, set any OAuth redirect (it is unused for this flow), and copy the Client ID and Client Secret. Then exchange them:

curl -X POST 'https://id.twitch.tv/oauth2/token' \
  -H 'Content-Type: application/x-www-form-urlencoded' \
  -d 'client_id=YOUR_CLIENT_ID&client_secret=YOUR_CLIENT_SECRET&grant_type=client_credentials'

The response carries a token that is valid for roughly 60 days:

{ "access_token": "abcd1234...", "expires_in": 5011271, "token_type": "bearer" }

Cache that token and reuse it. Requesting a new one on every run is a common way to waste calls and confuse yourself when the old one still works. In Python:

import requests

def get_app_token(client_id, client_secret):
    r = requests.post(
        "https://id.twitch.tv/oauth2/token",
        data={
            "client_id": client_id,
            "client_secret": client_secret,
            "grant_type": "client_credentials",
        },
        timeout=15,
    )
    r.raise_for_status()
    return r.json()["access_token"]

HELIX = "https://api.twitch.tv/helix"

def helix_headers(client_id, token):
    return {"Client-Id": client_id, "Authorization": f"Bearer {token}"}

An app token grants access to public data only. It cannot read a user's follows, chat, or private settings; those need a user token with explicit scopes, which is a different flow and outside a pure scraping job.

GET /streams is the workhorse. With no filters it returns the most-watched live channels globally, ordered by viewer count, 100 per page.

curl -H 'Authorization: Bearer ACCESS_TOKEN' \
     -H 'Client-Id: YOUR_CLIENT_ID' \
     'https://api.twitch.tv/helix/streams?first=100'

Each row is one live channel:

{
  "id": "40952121085",
  "user_login": "somestreamer",
  "user_name": "SomeStreamer",
  "game_id": "509658",
  "game_name": "Just Chatting",
  "type": "live",
  "title": "friday night co-op",
  "viewer_count": 18342,
  "started_at": "2026-08-10T17:04:11Z",
  "language": "en",
  "thumbnail_url": "https://.../{width}x{height}.jpg"
}

Filter with game_id, language, or up to 100 user_login values in one call. A helper that takes the common filters:

def top_streams(client_id, token, first=100, game_id=None, language=None):
    params = {"first": first}
    if game_id:
        params["game_id"] = game_id
    if language:
        params["language"] = language
    r = requests.get(f"{HELIX}/streams",
                     headers=helix_headers(client_id, token),
                     params=params, timeout=15)
    r.raise_for_status()
    return r.json()["data"]

# Live English-language "Just Chatting" streams, most-watched first
rows = top_streams(client_id, token, game_id="509658", language="en")
for s in rows:
    print(s["user_login"], s["viewer_count"], s["title"])

Want a specific channel's live state? Pass its login and check whether a row comes back:

def is_live(client_id, token, login):
    rows = requests.get(
        f"{HELIX}/streams",
        headers=helix_headers(client_id, token),
        params={"user_login": login}, timeout=15,
    ).json()["data"]
    return rows[0] if rows else None  # None means offline

That single call gives you the live flag, the current viewer count, the title, and the category in one shot.

Streams give you live data. For the profile behind a channel, use GET /users, which resolves a login into a stable numeric id you will reuse for clips and videos.

def get_user(client_id, token, login):
    data = requests.get(
        f"{HELIX}/users",
        headers=helix_headers(client_id, token),
        params={"login": login}, timeout=15,
    ).json()["data"]
    return data[0] if data else None

user = get_user(client_id, token, "somestreamer")
user_id = user["id"]

For what the channel is set to right now, its title, its language, its configured category, and its stream tags, call GET /channels:

def channel_info(client_id, token, broadcaster_id):
    return requests.get(
        f"{HELIX}/channels",
        headers=helix_headers(client_id, token),
        params={"broadcaster_id": broadcaster_id}, timeout=15,
    ).json()["data"][0]

Viewer count is the field people ask about most, so be precise about where it lives. There is exactly one source: the viewer_count field on a GET /streams row, and it exists only while the channel is live. To track a channel's concurrent viewers over time, poll /streams for that login on an interval and store each (timestamp, viewer_count) pair yourself. Twitch does not hand out a historical viewership time series through Helix; you build it by sampling. Follower totals are a separate endpoint, GET /channels/followers, and that one needs a user token with moderator:read:followers, so a pure app-token scraper cannot read it.

Twitch calls games and categories the same thing in the API. Rank them with GET /games/top:

def top_categories(client_id, token, first=20):
    return requests.get(
        f"{HELIX}/games/top",
        headers=helix_headers(client_id, token),
        params={"first": first}, timeout=15,
    ).json()["data"]

To go from a category name to the game_id you feed into /streams, resolve it once with GET /games:

def game_id_for(client_id, token, name):
    data = requests.get(
        f"{HELIX}/games",
        headers=helix_headers(client_id, token),
        params={"name": name}, timeout=15,
    ).json()["data"]
    return data[0]["id"] if data else None

jc = game_id_for(client_id, token, "Just Chatting")  # -> "509658"

One caveat that matters for anyone measuring category popularity: GET /games/top ranks by current viewership, but it is the global picture. The order a viewer in Berlin or Sao Paulo actually sees on the site is regionalized, and Helix has no parameter for that. Section 10 is how you capture the regional view.

Clips come from GET /clips, filtered by broadcaster_id, game_id, or a specific clip id. Add a started_at and ended_at window (RFC 3339 timestamps) to pull, say, the last 30 days of a channel's clips:

def channel_clips(client_id, token, broadcaster_id, started_at=None, ended_at=None):
    params = {"broadcaster_id": broadcaster_id, "first": 100}
    if started_at:
        params["started_at"] = started_at
    if ended_at:
        params["ended_at"] = ended_at
    r = requests.get(f"{HELIX}/clips",
                     headers=helix_headers(client_id, token),
                     params=params, timeout=15)
    r.raise_for_status()
    return r.json()

Each clip carries view_count, created_at, duration, the creator, and vod_offset, the second-offset into the source VOD where the clip starts. That offset is the field that lets you line clips up against the broadcast they came from.

Past broadcasts use GET /videos. Query by user_id and a type of archive (past streams), highlight, or upload:

def channel_vods(client_id, token, user_id, kind="archive"):
    return requests.get(
        f"{HELIX}/videos",
        headers=helix_headers(client_id, token),
        params={"user_id": user_id, "type": kind, "first": 100}, timeout=15,
    ).json()

Both endpoints return more than one page for active channels, which brings us to pagination.

Helix pages with an opaque cursor. Every response includes a pagination.cursor; pass it back as after to get the next page, and stop when the cursor disappears. first maxes out at 100.

import time

def paginate(endpoint, client_id, token, params):
    params = dict(params)
    while True:
        r = requests.get(f"{HELIX}/{endpoint}",
                         headers=helix_headers(client_id, token),
                         params=params, timeout=15)
        if r.status_code == 429:
            reset = int(r.headers.get("Ratelimit-Reset", "0"))
            time.sleep(max(reset - time.time(), 1))
            continue
        r.raise_for_status()
        payload = r.json()
        for item in payload["data"]:
            yield item
        cursor = payload.get("pagination", {}).get("cursor")
        if not cursor:
            break
        params["after"] = cursor

# Every clip a channel has, walked page by page
all_clips = list(paginate("clips", client_id, token,
                          {"broadcaster_id": user_id, "first": 100}))

Now the rate limits, and the insight that reshapes how you scale. Helix uses a token-bucket limiter. Every response tells you where you stand:

Most GET endpoints cost one point per call, so you get roughly 800 requests per minute before a 429. Here is the part almost every Twitch scraping guide gets wrong: that bucket is keyed to your client ID, not your IP address. Spreading Helix calls across a hundred proxies does not raise the ceiling, because Twitch counts them all against the same client ID. Rotating IPs is wasted money on the Helix path. The right way to go faster is to read Ratelimit-Remaining, pace yourself before you hit zero, and back off to Ratelimit-Reset when you do. If you genuinely need more Helix throughput, that is a conversation with Twitch about a higher limit, not a proxy purchase. For volume tactics that do apply once you leave Helix, see scraping high-volume data without rate limiting.

When Helix does not expose what you need, the twitch.tv frontend talks to https://gql.twitch.tv/gql, and you can talk to it too. It is a POST endpoint that speaks GraphQL, and the site hardcodes a public Client-ID into its own HTML so the browser can reach it: kimne78kx3ncx6brgo4mv6wki5h1ko.

Twitch uses persisted queries. Instead of sending full query text, the client sends an operation name plus a SHA-256 hash of a pre-registered query, batched inside a JSON array:

curl -X POST 'https://gql.twitch.tv/gql' \
  -H 'Client-Id: kimne78kx3ncx6brgo4mv6wki5h1ko' \
  -H 'Content-Type: application/json' \
  --data '[{
    "operationName": "DirectoryPage_Game",
    "variables": { "slug": "just-chatting", "limit": 30 },
    "extensions": {
      "persistedQuery": {
        "version": 1,
        "sha256Hash": "PASTE_CURRENT_HASH_FROM_DEVTOOLS"
      }
    }
  }]'

Two things about those hashes. They are not secret, and they rotate. Open twitch.tv with the browser Network tab filtered to gql, watch the request that loads the data you want, and copy its operationName and sha256Hash. When Twitch ships a frontend build, some hashes change and your stored ones return an error, so treat them as config you refresh, not constants you hardcode forever. The mechanics are identical to any persisted-query backend; the general pattern is covered in how to scrape GraphQL APIs and hidden JSON API endpoints.

Be aware of two moving hurdles. Twitch has added a Client-Integrity token requirement to some operations, especially anything touching video playback, and read-only public queries like directory listings generally still resolve with just the Client-ID. And because this endpoint throttles by IP, high-volume GraphQL scraping does need address rotation. Point your requests at a proxy pool:

proxies = {
    "http":  "http://USER:PASS@proxy.sparkproxy.io:PORT",
    "https": "http://USER:PASS@proxy.sparkproxy.io:PORT",
}
requests.post("https://gql.twitch.tv/gql", headers={
    "Client-Id": "kimne78kx3ncx6brgo4mv6wki5h1ko",
    "Content-Type": "application/json",
}, json=[payload], proxies=proxies, timeout=20)

Use the exact host and port from your SparkProxy dashboard. This is the opposite of the Helix rule: on GraphQL, clean rotating IPs are what keep you unblocked.

Twitch personalizes and regionalizes the directory. The category order, the recommended shelves, and which streams surface first all shift with the viewer's country. Helix can filter streams by language, but language is not location, and there is no Helix parameter for "show me the front page a viewer in Germany sees."

The directory is a client-rendered single-page app, so you need a fetch that runs JavaScript and originates from the target country. The SparkProxy Scraping API does both in one call: render_js executes the page, and country_code picks the exit region.

def regional_directory(api_key, slug, country):
    r = requests.get(
        "https://scrape.sparkproxy.io/api/v1",
        headers={"X-API-Key": api_key},
        params={
            "url": f"https://www.twitch.tv/directory/category/{slug}",
            "render_js": "true",
            "country_code": country,          # DE, BR, JP, ...
            "wait_for": '[data-a-target="card-slot"]',
        },
        timeout=60,
    )
    r.raise_for_status()
    return r.text  # fully rendered directory HTML for that region

de_html = regional_directory(API_KEY, "just-chatting", "DE")

The wait_for selector holds the render until the stream cards mount; confirm the current selector by inspecting the page, since Twitch renames DOM hooks now and then. Swap country_code to BR, JP, or KR and you capture how the same category ranks in each market, which is the regional signal Helix cannot give you. The Scraping API bills five credits for a rendered request and adds five for geo-targeting, and new accounts start with 1,000 free credits. Because it renders JavaScript on a clean IP for you, it also handles channel pages and directory pages without your own headless browser fleet. For the general technique behind rendering SPA content, see scraping dynamic JavaScript websites.

Scraping Twitch responsibly is mostly about picking the sanctioned path and respecting the obvious lines.

• Prefer Helix. It is the route Twitch supports under its Developer Services Agreement. Build there first and only drop to GraphQL for public data Helix genuinely lacks.
• The GraphQL endpoint is unsanctioned. It is undocumented and not offered for third-party consumption, so using it can conflict with Twitch's Terms of Service and it may break or lock down at any time. Keep it to public, non-authenticated data, and never use it to impersonate the app or bypass an integrity check.
• Collect public data only. Live streams, category rankings, clip and VOD metadata are public. Do not harvest private user information, and do not touch anything that requires a real user's consent.
• Respect the limits. Honor the Ratelimit-* headers, cache tokens and responses, and space your requests. Politeness is also what keeps you unblocked.

For a fuller treatment of rate limits, caching, and request etiquette, read the guide on ethical scraping and rate limiting. If you are also pulling other platforms, the approach mirrors how to scrape YouTube data: official API first, private endpoints only for the gaps.