Scrape Hacker News: Stories, Comments, Points, Users

Hacker News gives you three ways to read its data, and two of them beat parsing HTML by a wide margin. Reach for the APIs first.

The Firebase API at hacker-news.firebaseio.com is the official one, run by Y Combinator. It is read-only JSON, needs no key, and returns stories, comments, jobs, polls, and user profiles by id. It also exposes the live front page, the newest submissions, and a running maximum item id you can walk for a full backfill. What it does not do is search.

The Algolia search API at hn.algolia.com fills that gap. It indexes every item and lets you search full text, filter by points or date or author, and, best of all, pull an entire nested comment thread in a single request. It is the query layer that sits on top of the same data.

HTML scraping of news.ycombinator.com is the last resort, useful only for the handful of fields the APIs do not model or when you want the page exactly as rendered. Since the content is already served as clean JSON, treating a site's own API as the primary source is the same principle covered in how to scrape hidden JSON API endpoints: find the data feed before you parse the markup.

The base URL is https://hacker-news.firebaseio.com/v0. The single most useful fact about it: everything is an item. A story, a comment, a job listing, a poll, and a poll option all come from the same endpoint, /item/.json, and share one schema. You write one parser and branch on the type field.

# Every Hacker News object is an "item", fetched by id
curl -s "https://hacker-news.firebaseio.com/v0/item/8863.json"
# -> a story item: title, by, score (points), time, kids (comment ids), descendants

Two Python helpers cover almost everything you will do against Firebase. Note that a missing id returns JSON null, and usernames are case-sensitive.

import requests

HN = "https://hacker-news.firebaseio.com/v0"

def hn_item(item_id):
    r = requests.get(f"{HN}/item/{item_id}.json", timeout=15)
    r.raise_for_status()
    return r.json()   # None if the id does not exist

def hn_user(username):
    r = requests.get(f"{HN}/user/{username}.json", timeout=15)
    r.raise_for_status()
    return r.json()   # usernames are case-sensitive: "pg" != "Pg"

Beyond fetching by id, Firebase exposes the live site as flat arrays of ids. These are the lists that power the front page and the section tabs:

The official docs state plainly: "There is currently no rate limit." They also advise clients to "gracefully handle additional fields they don't expect, and simply ignore them," so read the fields you need with .get() and do not assume every optional field is present.

An item carries only the fields that apply to its type. A comment has no title; a story has no parent. Here are the ones you will actually read:

Two field details save hours of confusion. First, score is the story's points, and it exists only on stories and polls. Hacker News does not expose comment scores through any public API, so if your goal is per-comment points, stop now: that data is not available. Second, time is a Unix timestamp, so convert before you compare against anything human-readable:

from datetime import datetime, timezone
posted_at = datetime.fromtimestamp(item["time"], tz=timezone.utc)

The user object is smaller and just as clean. It is genuinely public data with no email or private field anywhere in it:

Getting the current front page with points is a two-step job: read topstories.json for the ordered ids, then fetch each item. The order of that array mirrors the site's front-page ranking, so the index is the rank. Fetch the items concurrently, because you are making a lot of small independent requests.

from concurrent.futures import ThreadPoolExecutor

def top_stories(n=30):
    ids = requests.get(f"{HN}/topstories.json", timeout=15).json()[:n]
    with ThreadPoolExecutor(max_workers=20) as pool:
        items = list(pool.map(hn_item, ids))
    return [
        {
            "rank": i + 1,                    # topstories order = front-page rank
            "id": it["id"],
            "title": it.get("title"),
            "points": it.get("score"),        # 'score' holds the story's points
            "by": it.get("by"),
            "comments": it.get("descendants", 0),
            "url": it.get("url"),             # missing on Ask HN text posts
        }
        for i, it in enumerate(items) if it
    ]

The descendants field already holds the total comment count, so you can size a job without walking the thread. Users are a single call:

u = hn_user("pg")
print(u["id"], u["karma"], u["created"])   # created is a Unix timestamp
# u["submitted"] is a list of item ids; u["about"] is HTML; there is no email

For a full backfill you do not need search at all. maxitem.json gives you the newest id in the whole system, and every id below it is a real item you can walk downward:

# Firehose backfill: walk ids down from the newest item
newest = requests.get(f"{HN}/maxitem.json", timeout=15).json()
for item_id in range(newest, newest - 500, -1):
    item = hn_item(item_id)
    if item and item.get("type") == "story":
        handle(item)

Comments form a tree. Each item's kids array lists its direct replies by id, and each of those replies has its own kids. The array is ordered the way Hacker News displays it, so a depth-first walk over kids reproduces the thread exactly as it reads on the page. No HTML, no fragile CSS selectors.

def comment_tree(item_id, depth=0):
    item = hn_item(item_id)
    if not item or item.get("deleted") or item.get("dead"):
        return []                             # skip removed and flagged nodes
    rows = []
    if item.get("type") == "comment":
        rows.append({
            "id": item["id"],
            "by": item.get("by"),
            "depth": depth,
            "text": item.get("text", ""),     # body is HTML, not plain text
        })
    for kid_id in item.get("kids", []):       # kids: ranked display order
        rows.extend(comment_tree(kid_id, depth + 1))
    return rows

story = hn_item(8863)
print(story.get("descendants"), "comments to fetch")
threads = comment_tree(8863)                  # story is depth 0, replies follow

The catch is request count. A busy thread with a thousand comments is a thousand Firebase calls, one per node. That is fine at low volume, but if you are pulling whole threads in bulk, the Algolia API returns the entire nested tree in a single request. That trade-off is the next section.

One thing to remember about text: it is HTML with entities and

tags, not clean prose. Unescape the entities and strip the tags before you store it, or keep the raw HTML if you plan to render it later.

Firebase can fetch anything by id but cannot answer "every story about Postgres over 100 points." That is what the hacker news algolia api at https://hn.algolia.com/api/v1 is for. Two search endpoints matter: /search ranks results by popularity (a blend of points and comments), and /search_by_date returns them newest first.

# Ranked by popularity
curl -s "https://hn.algolia.com/api/v1/search?query=web+scraping&tags=story"

# Same query, newest first
curl -s "https://hn.algolia.com/api/v1/search_by_date?query=web+scraping&tags=story"

The real power is in tags and numericFilters. Tags restrict by object type or scope, and numeric filters slice by points, comment count, or Unix creation time. Combine them to build precise feeds:

Paging is zero-indexed, and the response tells you nbPages so you know when to stop:

def algolia_search(query, tags="story", min_points=0, pages=5):
    hits = []
    for page in range(pages):
        params = {
            "query": query,
            "tags": tags,
            "numericFilters": f"points>{min_points}",
            "hitsPerPage": 100,
            "page": page,
        }
        data = requests.get(
            "https://hn.algolia.com/api/v1/search_by_date",
            params=params, timeout=15,
        ).json()
        hits.extend(data["hits"])
        if page >= data["nbPages"] - 1:
            break
    return hits

# Every story about "postgres" with more than 100 points
popular = algolia_search("postgres", min_points=100)

Now the feature that makes Algolia worth using even when you are not searching. /items/ returns a story with its entire comment tree already nested under a children key. One request replaces the thousand recursive Firebase calls from the last section.

# Firebase needs one request per comment. Algolia returns the whole
# nested thread in a single call via /items/<id>
def algolia_thread(story_id):
    data = requests.get(
        f"https://hn.algolia.com/api/v1/items/{story_id}", timeout=20,
    ).json()

    def walk(node, depth=0):
        out = [{
            "id": node["id"],
            "by": node.get("author"),
            "depth": depth,
            "text": node.get("text"),
        }]
        for child in node.get("children", []):
            out.extend(walk(child, depth + 1))
        return out

    return walk(data)

So the split is simple. Firebase is for fetch-by-id, live lists, and backfills. Algolia is for search, filters, and pulling a full scrape hacker news comments thread in one shot. Most real pipelines use both. Remember the ceiling: the Algolia API is rate limited to 10,000 requests per hour per IP, which is generous but real.

With two JSON APIs covering stories, points, comments, and users, the honest answer is that you rarely need to touch the HTML at all. There are a few exceptions:

• You want the page precisely as rendered, including the exact ordering and any moderation state the APIs smooth over.
• You need something the item schema does not carry, such as the visual flag/vouch controls or a page the APIs do not model.
• Both APIs are unavailable and you need a stopgap.

If you do hit the HTML, respect the rules. The news.ycombinator.com/robots.txt sets Crawl-delay: 30 for all bots and disallows the action endpoints (/login, /vote?, /reply?, /flag?, /hide?, /x?, and similar). The listing pages themselves are server-rendered plain HTML, so no JavaScript execution is required. The markup is old-school table rows: each story is a tr.athing, and its points sit in the following row.

from bs4 import BeautifulSoup

soup = BeautifulSoup(html, "html.parser")
for row in soup.select("tr.athing"):
    rank = row.select_one(".rank").get_text(strip=True).rstrip(".")
    link = row.select_one(".titleline > a")
    subtext = row.find_next_sibling("tr").select_one(".subtext")
    score = subtext.select_one(".score")          # job posts have no .score
    print(rank, link.get_text(), score.get_text() if score else "0 points")

That Crawl-delay: 30 is the whole reason HTML is the slow path: 30 seconds between page loads makes any volume painful. The APIs have no such delay, which is one more reason to keep HTML as the exception, not the plan.

Here is the honest version most vendor guides skip: for the Firebase and Algolia APIs, you do not need a proxy at all. They are public, keyless, and un-throttled (or generously throttled), and they hand back clean JSON. Route them straight from your own code.

Proxies and a managed scraper earn their keep in two situations. The first is the HTML fallback above, where you are hitting news.ycombinator.com and would rather not manage the Crawl-delay, retries, and the occasional block from a single busy IP. The second is a large Algolia backfill where you want to distribute requests across IPs so no single address trips the 10,000-per-hour ceiling.

For the HTML case, the SparkProxy Scraping API fetches the page through a clean IP and returns the HTML. Hacker News is server-rendered, so keep render_js off and the call stays at 1 credit instead of 5.

import requests
from bs4 import BeautifulSoup

r = requests.get(
    "https://scrape.sparkproxy.io/api/v1",
    headers={"X-API-Key": "YOUR_API_KEY"},    # key format: sk-...
    params={
        "url": "https://news.ycombinator.com/news",
        "render_js": "false",     # HN serves plain HTML -> 1 credit, no browser
        "country_code": "us",     # geo-target the exit IP
    },
    timeout=60,
)

soup = BeautifulSoup(r.text, "html.parser")   # same parsing as before
rows = soup.select("tr.athing")

Authentication is the X-API-Key header carrying a key of the form sk-... from your dashboard. If you would rather have the API extract fields server-side instead of returning raw HTML, the extract_rules parameter takes CSS selectors and returns structured JSON. Deciding whether to build and babysit this yourself or route through a managed endpoint is the same trade covered in web scraping API vs self-managed proxies. For a target this light, the calculus usually favors the plain APIs, with the managed path reserved for the HTML edge cases where blocks are covered in how to avoid getting your proxy blocked.

Hacker News is one of the friendliest scraping targets on the web, which is exactly why you should not abuse it. A few rules keep a collection job defensible and unblocked:

• Cache aggressively. The Firebase docs say there is no rate limit, but that is an invitation to be efficient, not a dare. Store items by id and refetch only what changes. The updates.json endpoint lists recently changed items and profiles so you can poll deltas instead of re-crawling.
• Stay under the Algolia ceiling. 10,000 requests per hour per IP is plenty for most jobs. Back off on a 429, and distribute a big backfill rather than hammering from one address.
• Honor the HTML Crawl-delay: 30. If you scrape the site directly, pace yourself and do not touch the disallowed action endpoints.
• Collect public data only. Everything the APIs return is already public, including usernames and karma. That does not make it ethical to build a dossier on an individual. Aggregating one person's entire comment history to profile or target them crosses from research into surveillance, so do not.
• Honor deletions. Items marked deleted or dead are gone for a reason. Skip them, and do not resurrect removed content from a cache.

A polite client backs off when told to. This tiny wrapper handles the only push-back you are likely to see, an Algolia 429:

import time

def polite_get(url, params=None, tries=4):
    for attempt in range(tries):
        r = requests.get(url, params=params, timeout=20)
        if r.status_code == 429:                  # Algolia: 10,000/hour per IP
            time.sleep(2 ** attempt)
            continue
        r.raise_for_status()
        return r
    raise RuntimeError("rate limited after retries")

Clean engineering and ethical scraping point the same way. A job that caches, paces itself, takes only public data, and respects deletions is both harder to block and easier to defend. The same discipline shows up in using proxies for market research and data collection.