How to Scrape GitHub Data (Repos, Users, Stars)

GitHub runs a first-party REST API at https://api.github.com and a GraphQL API at https://api.github.com/graphql. For repositories, users, stars, followers, topics, and languages, the API is the sanctioned path and it should be your default. It is documented, versioned, and generous: authenticate with a personal access token and you get 5,000 requests per hour instead of the 60 per hour an anonymous client gets.

So when would you scrape the HTML at all? Only when a page shows data the API does not expose. The clearest example is github.com/trending, which has no official endpoint. For everything else, reaching for a headless browser to parse markup that the API returns as structured JSON is slower, more fragile, and easier to block. Treat HTML scraping as the exception, not the plan.

If you are weighing whether to build your own collector or route through a managed service, the trade-offs in web scraping API vs self-managed proxies apply directly. GitHub is unusual in that the free API is so good the build-vs-buy question mostly only shows up for the handful of pages the API skips.

The REST API is a set of predictable endpoints. Point at a resource, get JSON back. First, authenticate. GitHub also requires a User-Agent header on every API request and returns 403 if you omit it, so set one.

curl -s \
  -H "Authorization: Bearer YOUR_GITHUB_TOKEN" \
  -H "Accept: application/vnd.github+json" \
  -H "User-Agent: research-tool/1.0" \
  "https://api.github.com/repos/octocat/Hello-World"

The same call in Python, pulling the fields you usually want to scrape GitHub repositories for:

import requests

HEADERS = {
    "Authorization": "Bearer YOUR_GITHUB_TOKEN",
    "Accept": "application/vnd.github+json",
    "User-Agent": "research-tool/1.0 (contact you@sparkproxy.io)",
}

def get_repo(owner, name):
    r = requests.get(
        f"https://api.github.com/repos/{owner}/{name}",
        headers=HEADERS, timeout=15,
    )
    r.raise_for_status()
    d = r.json()
    return {
        "full_name": d["full_name"],
        "stars": d["stargazers_count"],
        "forks": d["forks_count"],
        "language": d["language"],
        "topics": d["topics"],
        "open_issues": d["open_issues_count"],
        "watchers": d["subscribers_count"],   # the real watcher count, see below
        "pushed_at": d["pushed_at"],
    }

A user profile is just as direct. GET /users/{username} returns the account, and public_repos, followers, and following are counts you can pull without a second call.

def get_user(username):
    r = requests.get(
        f"https://api.github.com/users/{username}",
        headers=HEADERS, timeout=15,
    )
    r.raise_for_status()
    d = r.json()
    return {
        "login": d["login"],
        "name": d.get("name"),
        "followers": d["followers"],
        "following": d["following"],
        "public_repos": d["public_repos"],
        "created_at": d["created_at"],
    }

For stars, GET /repos/{owner}/{repo}/stargazers lists the accounts that starred a repo. By default you get plain user objects with no timestamp. Here is a detail most guides miss: send the application/vnd.github.star+json media type and each entry gains a starred_at field, which is what you need to chart star growth over time.

def get_stargazers(owner, name, per_page=100):
    r = requests.get(
        f"https://api.github.com/repos/{owner}/{name}/stargazers",
        headers={**HEADERS, "Accept": "application/vnd.github.star+json"},
        params={"per_page": per_page},
        timeout=15,
    )
    r.raise_for_status()
    # each item now looks like {"starred_at": "...", "user": {...}}
    return [(s["starred_at"], s["user"]["login"]) for s in r.json()]

Other endpoints follow the same shape: GET /users/{username}/repos for a user's repositories, GET /users/{username}/followers for followers, and GET /repos/{owner}/{repo}/languages for the byte breakdown per language.

The repository object is large. These are the fields a github scraper actually uses, straight from the GET /repos/{owner}/{repo} response:

Two things trip people up. First, watchers_count does not mean watchers. Because of a legacy API decision, watchers_count returns the same number as stargazers_count. The count of accounts genuinely subscribed to a repo is subscribers_count. If you report "watchers" from watchers_count, you are quietly reporting stars twice. Second, open_issues_count includes open pull requests, since GitHub models PRs as issues. Subtract the PR count if you want issues alone.

The user object is smaller. The fields that matter for a profile pull:

List endpoints return 30 items by default. Raise that with per_page (the maximum is 100) and walk the pages with page. You do not guess when to stop. GitHub tells you through the Link response header, which carries the URLs for the next, previous, first, and last pages.

link: <https://api.github.com/repositories/1300192/stargazers?per_page=100&page=2>; rel="next",
      <https://api.github.com/repositories/1300192/stargazers?per_page=100&page=42>; rel="last"

In Python, the requests library parses that header for you into r.links, so following pagination is a loop that ends when there is no next link:

def paginate(url, params=None):
    params = {**(params or {}), "per_page": 100}
    while url:
        r = requests.get(url, headers=HEADERS, params=params, timeout=15)
        r.raise_for_status()
        yield from r.json()
        url = r.links.get("next", {}).get("url")
        params = None   # the next URL already carries per_page and page

# every repo owned by an org, across all pages
repos = list(paginate("https://api.github.com/users/github/repos"))

Not every endpoint uses numbered pages. Some, like listing all users (GET /users?since=) or events, use a cursor such as since or before/after that you carry forward from the previous response. The rule is the same either way: read the pagination signal the API gives back, do not build page numbers by hand.

One cap to plan around. The Search API only ever returns the first 1,000 results for a query, no matter how you paginate. Any collection plan that assumes you can page through every matching repo will hit that wall, so slice broad searches into narrower queries (by language, by star range, by creation date) to stay under 1,000 each.

Rate limits are the core of polite github api scraping. Authenticate and you get a lot of headroom. The ceilings:

Every REST response carries your budget in headers: x-ratelimit-limit, x-ratelimit-remaining, x-ratelimit-used, and x-ratelimit-reset (a UTC epoch second). Read them and slow down before you hit zero. You can also check your standing with GET /rate_limit, a call that does not count against your limit.

def check_budget():
    r = requests.get("https://api.github.com/rate_limit", headers=HEADERS, timeout=10)
    core = r.json()["resources"]["core"]
    print(f"{core['remaining']}/{core['limit']} left, resets at {core['reset']}")

When you do exceed the primary limit, GitHub returns a 403 or 429 with x-ratelimit-remaining: 0. Do not retry until the time in x-ratelimit-reset. If you trip a secondary limit (bursting too fast), the response includes a retry-after header in seconds, and you honor it.

import time

def get_with_backoff(url, params=None, tries=4):
    for attempt in range(tries):
        r = requests.get(url, headers=HEADERS, params=params, timeout=15)
        if r.status_code in (403, 429) and r.headers.get("x-ratelimit-remaining") == "0":
            reset = int(r.headers.get("x-ratelimit-reset", time.time() + 60))
            time.sleep(max(0, reset - time.time()) + 1)
            continue
        if r.status_code == 429 and "retry-after" in r.headers:
            time.sleep(int(r.headers["retry-after"]) + 1)
            continue
        r.raise_for_status()
        return r
    raise RuntimeError("rate limited after retries")

Here is the trick that stretches 5,000 requests much further, and one competitors rarely mention. GitHub supports conditional requests. Most endpoints return an etag header; send it back as If-None-Match on your next call, and if nothing changed you get a 304 Not Modified. Per GitHub's documentation, a 304 returned to a correctly authorized request does not count against your primary rate limit. For any job that polls the same repos or users on a schedule, this is close to free.

etag_cache = {}

def get_conditional(url):
    headers = dict(HEADERS)
    if url in etag_cache:
        headers["If-None-Match"] = etag_cache[url]
    r = requests.get(url, headers=headers, timeout=15)
    if r.status_code == 304:
        return None            # unchanged, and this call was free
    r.raise_for_status()
    etag_cache[url] = r.headers.get("ETag", "")
    return r.json()

REST makes you fetch a repo, then its stargazers, then its languages as separate calls. GraphQL lets you ask for the exact fields you want across related objects in one request, which cuts round-trips. You send a query to https://api.github.com/graphql with the same Authorization: Bearer token.

QUERY = """
query($owner: String!, $name: String!) {
  repository(owner: $owner, name: $name) {
    nameWithOwner
    stargazerCount
    forkCount
    primaryLanguage { name }
    repositoryTopics(first: 10) { nodes { topic { name } } }
    owner { login }
  }
}
"""

def graphql_repo(owner, name):
    r = requests.post(
        "https://api.github.com/graphql",
        headers=HEADERS,
        json={"query": QUERY, "variables": {"owner": owner, "name": name}},
        timeout=15,
    )
    r.raise_for_status()
    return r.json()["data"]["repository"]

GraphQL bills on a different meter than REST. Its budget is 5,000 points per hour, not 5,000 requests. Point cost is roughly the number of requests needed to fulfill each connection in your query divided by 100 and rounded, with a floor of 1 point per call. So a simple query costs 1 point, and a deeply nested one that fans out across thousands of nodes costs more. A single call also cannot request more than 500,000 total nodes. In practice, GraphQL wins when you need several related fields at once (a repo plus its topics plus its language in one shot) and REST wins for simple, single-resource pulls where its per-request accounting is easier to reason about.

The honest reason to render a GitHub page is that no endpoint returns what it shows. The canonical case is the trending page at github.com/trending. GitHub has never shipped a public trending API, so the ranked list of repositories rising today only exists in the HTML. Topic landing pages and a few discovery views are similar.

The good news is that these pages are server-rendered. The trending list is present in the initial HTML, so you do not need to execute JavaScript to read it. That keeps parsing cheap and avoids a headless browser. Selectors do change when GitHub reskins a page, so treat them as the fragile part and keep them in one place.

from bs4 import BeautifulSoup

def parse_trending(html):
    soup = BeautifulSoup(html, "html.parser")
    repos = []
    for row in soup.select("article.Box-row"):
        link = row.select_one("h2 a")
        full_name = link["href"].strip("/") if link else None
        lang = row.select_one('[itemprop="programmingLanguage"]')
        repos.append({
            "full_name": full_name,
            "language": lang.get_text(strip=True) if lang else None,
        })
    return repos

Fetching github.com/trending from a single server IP works until it doesn't. Unauthenticated web traffic shares that 60-per-hour ceiling by IP, and repeated automated hits from one datacenter address invite challenges. Distributing legitimate reads across clean IPs is standard practice for this kind of collection. For the mechanics of staying under detection thresholds, see how to avoid getting your proxy blocked, and for high-volume HTML collection specifically, using datacenter proxies for web scraping covers the setup.

One clarification worth stating plainly: proxies are not what you need for the authenticated API. A token gives you 5,000 requests an hour per token, so if you are hitting api.github.com, add authentication, not a proxy pool. Proxies earn their place on the HTML pages and on unauthenticated reads, not on the sanctioned API.

Managing clean exit IPs, retries, and block handling for the HTML pages is a small project on its own. The SparkProxy Scraping API does that server-side: you send a target URL, it picks an exit IP, handles the request, and returns the response. Because the trending page is server-rendered, keep render_js off, which holds the call at 1 credit instead of 5.

import requests

r = requests.get(
    "https://scrape.sparkproxy.io/api/v1",
    headers={"X-API-Key": "YOUR_API_KEY"},   # key format: sk-...
    params={
        "url": "https://github.com/trending?since=daily",
        "render_js": "false",     # server-rendered HTML, no browser -> 1 credit
        "country_code": "us",     # geo-target the exit IP
    },
    timeout=60,
)
r.raise_for_status()
trending = parse_trending(r.text)   # reuse the BeautifulSoup parser above

The base endpoint is https://scrape.sparkproxy.io/api/v1, authentication is the X-API-Key header carrying a key of the form sk-... from your dashboard, and country_code sets the exit geography. If you want the API to return only the rows you care about instead of the full page, pass extract_rules with CSS selectors and let it hand back structured data. Reserve render_js=true for the rare GitHub view that hydrates client-side, since that path runs a real browser and costs more credits.

For the JSON API itself you generally do not need this at all. Add a token and call api.github.com directly. The scraping API is for the pages GitHub does not expose through an endpoint. If you are collecting broadly across public repositories and profiles as part of a research or market dataset, the patterns in using proxies for market research and data collection cover how to structure that responsibly.

"Public" is not a blank check. GitHub's Acceptable Use Policies set real limits, and a few rules keep a collection project defensible:

• Prefer the API, and authenticate. It is the sanctioned path with a documented 5,000-per-hour budget. Authenticating is also politer, since it ties your traffic to an identity GitHub can rate-limit fairly.
• Do not scrape for spam or to harvest personal data. GitHub's Acceptable Use Policies prohibit using information from the service, whether scraped or pulled through the API, for sending unsolicited communications or for selling personal information. Collecting user emails to cold-message people is exactly what the policy forbids.
• Know the narrow scraping allowances. GitHub's policy states that researchers may scrape public, non-personal information for research, but only if resulting publications are open access, and that archivists may scrape public data for archival purposes. Commercial harvesting of personal data is not in that list.
• Send a descriptive User-Agent. The API requires one and rejects requests without it. It also identifies you honestly, which is the point.
• Cache with ETags and rate-limit politely. Conditional requests cut load on GitHub and cost you nothing under the 5,000-per-hour limit. Sleep between HTML page fetches and honor retry-after.
• Respect robots.txt for the pages you do scrape. Check what GitHub disallows before you point a crawler at it.

Ethical collection and clean engineering point the same way. A job that authenticates, takes only public non-personal data, respects rate limits, and identifies itself honestly is both less likely to get blocked and on far firmer legal ground.