How to Scrape Google Scholar: Papers, Citations, Authors

Scholar is a special case, so start here before you write any code. Google offers no Scholar API, and its Terms of Service prohibit sending automated queries to the service. That is a contractual restriction, not a criminal one, but it sets the tone: you are not a welcome guest.

Two things matter in practice. First, the data itself. Bibliographic facts such as a paper's title, authors, venue, year, and citation count are facts, and in the United States facts are not copyrightable under Feist Publications v. Rural Telephone (1991). Abstract text is expression owned by the publisher, so treat snippets as short quotations, not content to republish. Second, access. The Ninth Circuit's hiQ Labs v. LinkedIn line of rulings held that scraping publicly visible pages generally does not violate the Computer Fraud and Abuse Act, but Scholar's own Terms still say no, and violating them can get your access cut. None of this is legal advice. For commercial use, get counsel.

Here is the honest engineering call almost no tutorial makes: for most citation and metadata needs, you should not scrape Scholar at all. Semantic Scholar, OpenAlex, and Crossref publish free, documented APIs that cover the same papers, authors, and citation edges, with rate limits you can actually plan around. Reach for scraping only when you need a field that lives on the Scholar page and nowhere else, such as Scholar's own cited-by number or a public profile's exact h-index. If you are collecting academic data at scale, our guide to using proxies for academic research data collection walks through those API-first sources, and the guide on ethical scraping and rate limiting covers the pacing rules that keep you inside acceptable use.

Scraping Google Search results is already work. Scholar is harder, and it helps to know why before you fight it.

• No API and no intent to build one. Google has never shipped a Scholar API and has said it will not. There is no rate-limited-but-supported path. Every request is the front end or nothing.
• A hair-trigger CAPTCHA. Scholar counts requests per IP over a short window. Cross the threshold and it stops serving results, redirecting to google.com/sorry/ with a reCAPTCHA. Solve it or that IP is cold for a while.
• Datacenter ranges are pre-flagged. Whole cloud subnets are treated as suspect on arrival. A fresh datacenter IP can get the block page on request one. Residential exits look like real readers and last far longer.
• Thin session tolerance. Scholar is stateless HTML with no login for public search, so you cannot warm a session to earn trust. Each IP is judged almost entirely on its request rate and network reputation.

The upside is that the pages themselves are simple. Scholar renders server-side HTML, no client-side framework, so once a request gets through you can parse it with nothing more than an HTML parser. The whole difficulty is getting the bytes, not reading them.

Two page types carry almost everything. The search results page (/scholar?q=...) lists papers, and the author profile page (/citations?user=...) lists a researcher's metrics and publications. A third view, the cited-by page, is just the search page filtered to one cluster.

Open any Scholar search page and every result follows the same shape. The outer wrapper is div.gs_r.gs_or.gs_scl and it carries a data-cid attribute, which is the cluster id for that paper. Inside sits div.gs_ri, the block you actually parse.

  
# Attention is all you need
A Vaswani, N Shazeer, N Parmar - Advances in neural, 2017 - proceedings.org
    
The dominant sequence transduction models are based on complex recurrent ...
    

      Cited by 98213
      Related articles
      All 47 versions
    
  </div>
</div>

Notice that the data-cid on the wrapper and the number inside the cites= and cluster= links are the same value. That single id is the most important thing on the page. Keep it, and you can walk outward to everyone who cited the paper and inward to every version of it. Most tutorials extract the title and stop, which throws away the entire reason Scholar is interesting.

Here is the selector reference for a parser:

You need residential exits and you need them rotating. The SparkProxy Scraping API gives you both behind one call, so IP rotation and geo-targeting are parameters instead of infrastructure. If you are new to why residential IPs beat datacenter ranges on a target like this, our explainer on what a residential proxy is has the background.

The base is https://scrape.sparkproxy.io/api/v1 and auth is the X-API-Key header. A raw fetch of a Scholar page looks like this:

curl -X POST "https://scrape.sparkproxy.io/api/v1" \
  -H "X-API-Key: sk-your-api-key" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://scholar.google.com/scholar?q=graph+neural+networks&hl=en",
    "render_js": false,
    "premium_proxy": true,
    "country_code": "US"
  }'

Two parameter choices matter for Scholar specifically. Set render_js to false, because Scholar is server-rendered and skipping the headless browser makes each request cheaper and faster. Set premium_proxy to true, because the residential tier is the only thing that survives here. Add stealth: true on the toughest queries when the plain residential fetch still trips the wall.

Wrap it once in Python and every later step reuses it:

import requests

API = "https://scrape.sparkproxy.io/api/v1"
API_KEY = "sk-your-api-key"  # from the SparkProxy dashboard at https://www.sparkproxy.io

def scholar_get(target_url, country="US", stealth=False):
    payload = {
        "url": target_url,
        "render_js": False,       # Scholar is server-rendered
        "premium_proxy": True,    # residential exit is non-negotiable
        "country_code": country,
        "stealth": stealth,
    }
    r = requests.post(API, headers={"X-API-Key": API_KEY}, json=payload, timeout=90)
    r.raise_for_status()
    return r.text

With the HTML in hand, parsing is straightforward. Iterate the wrappers so you can read data-cid, then pull each field from the inner block. This function returns one clean dict per paper, including the cluster id and the forward-citations id.

from bs4 import BeautifulSoup
from urllib.parse import urljoin, urlparse, parse_qs

BASE = "https://scholar.google.com"

def parse_results(html):
    soup = BeautifulSoup(html, "html.parser")
    items = []
    for wrap in soup.select("div.gs_r"):
        block = wrap.select_one("div.gs_ri")
        if not block:
            continue
        title_el = block.select_one("h3.gs_rt a")
        byline = block.select_one("div.gs_a")
        snippet = block.select_one("div.gs_rs")

        cited_by, cites_id, versions_url = None, None, None
        for a in block.select("div.gs_fl a"):
            text = a.get_text(strip=True)
            href = a.get("href", "")
            if text.startswith("Cited by"):
                cited_by = int(text.replace("Cited by", "").strip() or 0)
                cites_id = parse_qs(urlparse(href).query).get("cites", [None])[0]
            elif "version" in text:
                versions_url = urljoin(BASE, href)

        items.append({
            "cluster_id": wrap.get("data-cid"),
            "title": title_el.get_text(" ", strip=True) if title_el else None,
            "url": title_el["href"] if title_el else None,
            "byline": byline.get_text(" ", strip=True) if byline else None,
            "snippet": snippet.get_text(" ", strip=True) if snippet else None,
            "cited_by": cited_by,
            "cites_id": cites_id,
            "versions_url": versions_url,
        })
    return items

The div.gs_a line packs three fields into one string, formatted as authors - venue, year - domain. Split it when you need structured columns:

def split_byline(byline):
    parts = [p.strip() for p in byline.split(" - ")]
    authors = parts[0] if parts else ""
    venue_year = parts[1] if len(parts) > 1 else ""
    year = next((tok for tok in venue_year.replace(",", " ").split()
                 if tok.isdigit() and len(tok) == 4), None)
    return {"authors": authors, "venue": venue_year, "year": year}

If you would rather have the API return structured rows and skip BeautifulSoup, SparkProxy also accepts an extract_rules object that maps selectors to fields on the server. Point it at div.gs_ri as a list and read the JSON back. Check the Scraping API docs for the exact nested shape, since a hand-rolled parser gives you more control over the cluster id and the footer links.

This is where the cluster id pays off. Once you have cites_id (or the cluster_id from data-cid), you rebuild the URLs Scholar uses for its own links and fetch them like any other search page. The response is the same gs_ri layout, so parse_results handles it unchanged.

def cited_by(cites_id, start=0):
    url = f"{BASE}/scholar?cites={cites_id}&hl=en&start={start}"
    return parse_results(scholar_get(url))

def all_versions(cluster_id, start=0):
    url = f"{BASE}/scholar?cluster={cluster_id}&hl=en&start={start}"
    return parse_results(scholar_get(url))

That is the entire citation graph in two functions. Start from a seed query, collect each paper's cites_id, and cited_by gives you every paper that cited it. Feed those results' ids back in and you can traverse forward citations as deep as you are willing to pace. all_versions collapses preprints, publisher copies, and mirrors of the same work into one set, which is how you dedupe a corpus.

Scholar shows 10 results per page and paginates with the start offset, not a page number. Page two is start=10, page three is start=20, and so on. Two ceilings matter: Scholar rarely honors more than 10 results per page even if you ask for num=20, and it stops serving new results at roughly start=990, so about 1000 results per query is the practical wall. Narrow with a year filter (as_ylo and as_yhi) when you need to go deeper than that.

import time, random
from urllib.parse import quote_plus

def search(query, pages=5, per_page=10):
    out = []
    for p in range(pages):
        start = p * per_page
        url = f"{BASE}/scholar?q={quote_plus(query)}&hl=en&start={start}"
        html = scholar_get(url)
        if is_blocked(html):        # defined in the pacing section
            time.sleep(random.uniform(60, 120))
            continue
        page_items = parse_results(html)
        if not page_items:
            break                   # ran out of results
        out.extend(page_items)
        time.sleep(random.uniform(10, 25))   # never hammer
    return out

Notice the sleep is inside the loop and randomized. That single line does more to keep you unblocked than any header trick.

Author pages are the richest single URL on Scholar. Each one carries the researcher's total citations, h-index, and i10-index, plus their full publication list with a per-paper cited-by count. The URL is citations?user=, where the id is the 12-character token from a profile link.

The metrics live in a small table, #gsc_rsb_st, with three rows (Citations, h-index, i10-index) and two columns each (All time, and a recent "Since" window). The publications are tr.gsc_a_tr rows.

def parse_author(html):
    soup = BeautifulSoup(html, "html.parser")
    name = soup.select_one("#gsc_prf_in")
    affil = soup.select_one("#gsc_prf_i .gsc_prf_il")

    stats, labels = {}, ["citations", "h_index", "i10_index"]
    for label, tr in zip(labels, soup.select("#gsc_rsb_st tbody tr")):
        cells = tr.select("td.gsc_rsb_std")
        stats[label] = {
            "all": int(cells[0].get_text()),
            "recent": int(cells[1].get_text()),
        }

    pubs = []
    for tr in soup.select("tr.gsc_a_tr"):
        title = tr.select_one("a.gsc_a_at")
        cites = tr.select_one("a.gsc_a_ac")
        year = tr.select_one("span.gsc_a_h")
        cites_txt = cites.get_text(strip=True) if cites else ""
        pubs.append({
            "title": title.get_text(strip=True) if title else None,
            "cited_by": int(cites_txt) if cites_txt.isdigit() else 0,
            "year": year.get_text(strip=True) if year else None,
        })

    return {
        "name": name.get_text(strip=True) if name else None,
        "affiliation": affil.get_text(strip=True) if affil else None,
        "stats": stats,
        "publications": pubs,
    }

The profile only shows the first 20 publications by default, then loads more behind a "Show more" button. That button is really the cstart and pagesize parameters, and pagesize accepts up to 100. So instead of simulating clicks, request the whole list directly:

def author_all(user_id):
    profile = None
    pubs = []
    cstart = 0
    while True:
        url = f"{BASE}/citations?user={user_id}&hl=en&cstart={cstart}&pagesize=100"
        data = parse_author(scholar_get(url))
        if profile is None:
            profile = {k: data[k] for k in ("name", "affiliation", "stats")}
        if not data["publications"]:
            break
        pubs.extend(data["publications"])
        cstart += 100
        time.sleep(random.uniform(10, 25))
    profile["publications"] = pubs
    return profile

One or two calls pulls a complete publication history, where the point-and-click path would take a dozen. That pagesize=100 trick is the fastest way to snapshot a researcher's entire record.

If you would rather not maintain selectors, the open-source scholarly package (version 1.x) wraps all of this behind Python objects. It parses the same pages, so it hits the same wall, which is why its own docs say you must run it behind a proxy for anything beyond a couple of lookups. Point its ProxyGenerator at a SparkProxy residential endpoint and it routes every request through a rotating exit.

from scholarly import scholarly, ProxyGenerator

pg = ProxyGenerator()
# Your SparkProxy residential gateway (host, port, and credentials from the dashboard)
gateway = "http://USER:PASS@gateway.sparkproxy.io:11000"
pg.SingleProxy(http=gateway, https=gateway)
scholarly.use_proxy(pg)

# A paper search
pub = next(scholarly.search_pubs("graph neural networks"))
scholarly.pprint(pub)          # title, author, venue, num_citations, cites_id

# An author profile by id
author = scholarly.search_author_id("SOME_USER_ID")
scholarly.fill(author, sections=["basics", "indices", "publications"])
print(author["hindex"], author["i10index"], author["citedby"])

Because the SparkProxy residential gateway rotates the exit per request, one SingleProxy URL still gives you IP rotation under the hood. If you switch proxy methods mid-run, build a fresh ProxyGenerator first, which the library notes to avoid stale-state bugs. The tradeoff: scholarly is convenient but a moving target, since it breaks whenever Scholar reshuffles a class name. Rolling your own parser, as above, means you fix one selector instead of waiting for a release.

Every technique above fails without pacing. Scholar's defense is rate-based, so your job is to look slow and human. The rules that actually work:

• One worker. Concurrency is the fastest way to the CAPTCHA. Run a single sequential loop, not a thread pool.
• Slow, randomized delays. Sleep 10 to 30 seconds between requests, jittered. Steady intervals are themselves a bot signal.
• Rotate residential exits. Let SparkProxy hand you a new IP per request or per short session. A single IP, however clean, burns out under repeated Scholar queries.
• Detect the wall and back off. When a response is the block page, stop, wait a minute or more, and rotate. Do not retry into a ban.

Detecting the block page is a two-line check. Scholar redirects to /sorry/ and returns a reCAPTCHA with tell-tale markers:

def is_blocked(html):
    markers = ("gs_captcha", "/sorry/", "unusual traffic",
               "not a robot", 'id="captcha')
    low = html.lower()
    return any(m in low for m in markers)

When is_blocked fires, the right move is patience, not force. Sleep, rotate to a fresh exit, and only then retry, optionally with stealth: true. Solving CAPTCHAs in a loop just trains the filter on you. For the wider playbook on staying under detection thresholds, see how to avoid CAPTCHAs when web scraping and how to scrape high-volume data without rate limiting. On Scholar the honest ceiling is low: a few thousand careful requests a day, not the millions you might run against a friendlier site. Plan the project around that, and lean on the Semantic Scholar and OpenAlex APIs for the bulk so Scholar only serves the fields nothing else has.