How to Scrape Google News: RSS, Headlines, and Feeds

Google News RSS is a feed Google publishes on purpose, so reading it is not the same as breaking into a private endpoint. The nuance is what sits inside the feed.

In the United States, the Ninth Circuit's ruling in hiQ Labs v. LinkedIn (2022) held that scraping publicly accessible data, meaning no login and no authentication wall, generally does not violate the Computer Fraud and Abuse Act. Headlines and article URLs are also facts, and under Feist Publications v. Rural Telephone (1991) facts themselves are not copyrightable. What is copyrighted is the article body written by each publisher. So the safe pattern is: collect headlines, links, timestamps, and publisher names freely, store short snippets under fair use, and never republish full article text you pulled after decoding a link.

Two housekeeping checks belong in any serious project. Respect each publisher's robots.txt once you follow a link off Google and onto their site, and rate-limit yourself so you are not degrading anyone's service. If the data feeds a commercial product, run it past a lawyer. This is engineering guidance, not legal advice. If you are building an ongoing tracker rather than a one-off pull, the operational side is covered in Proxies for News Monitoring and Media Aggregation.

Google News is an aggregator, not a publisher. A feed hands you metadata that points at other people's articles. Here is the field set that comes out of a single RSS item.

The important detail is that link never points at the publisher directly. It points at a Google redirect that resolves in the browser. Getting from that redirect to https://www.reuters.com/markets/... is the part most tutorials get wrong, and it has its own section below.

Every feed lives under https://news.google.com/rss and takes the same three edition parameters. The path selects the feed type.

The topic path accepts a fixed set of section names: WORLD, NATION, BUSINESS, TECHNOLOGY, ENTERTAINMENT, SPORTS, SCIENCE, and HEALTH. Pass any of those in uppercase. There is also a numeric variant, .../rss/topics/, where the id is the long base64 string you see in a topic page's own URL; that is how you follow a niche topic Google curates but does not expose by name.

The geo path takes a place name (Chicago, London, Mumbai) or a Google location id. Every feed caps at roughly 100 items, and none of them paginate. When 100 headlines is not enough coverage, you narrow the query rather than ask for page two, which is exactly what the search operators in the next section are for.

These three parameters decide which country and language edition Google serves. Get one wrong and you get an empty feed or the wrong country's stories.

• hl is the interface language as a language-region tag: en-US, en-GB, de, fr, pt-BR.
• gl is the country as an uppercase ISO 3166 code: US, GB, DE, FR, IN.
• ceid is the edition id in COUNTRY:language form: US:en, GB:en, DE:de. It must agree with hl and gl.

The ceid is the one people forget, and it is the one that silently breaks feeds. If gl=DE but ceid=US:en, Google resolves the mismatch in ways that are not documented and often returns thin or empty results. Keep all three consistent.

Setting these parameters tells Google which edition you want. It does not fully control which edition you get, because the request IP still weighs in. That gap is why the geo section pairs these params with a proxy exit in the same country.

You can hit an RSS feed with a plain HTTP client, and for a handful of requests that is fine. The moment you poll many feeds every few minutes, or you start decoding article links at volume, Google starts returning 429s and the occasional consent redirect. Routing through the SparkProxy Scraping API turns IP rotation, geo-targeting, and optional browser rendering into request parameters.

The base URL is https://scrape.sparkproxy.io/api/v1, and it authenticates with an X-API-Key header. RSS is static XML, so turn rendering off with render_js=false to keep each call at the 1-credit tier instead of paying for a browser you don't need.

curl -G "https://scrape.sparkproxy.io/api/v1" \
  --data-urlencode "url=https://news.google.com/rss?hl=en-US&gl=US&ceid=US:en" \
  --data-urlencode "render_js=false" \
  -H "X-API-Key: YOUR_API_KEY"

A thin Python wrapper keeps the rest of the guide readable:

import requests

API = "https://scrape.sparkproxy.io/api/v1"
KEY = "YOUR_API_KEY"

def fetch(url, **params):
    """GET a URL through the SparkProxy Scraping API and return the body."""
    params["url"] = url
    r = requests.get(API, params=params,
                     headers={"X-API-Key": KEY}, timeout=120)
    r.raise_for_status()
    return r.text

Every later snippet calls fetch(...). The parameters that matter for Google News are render_js (leave it false for RSS, true only when you resolve an article link in a browser), premium_proxy=true to route through residential IPs, and country_code to pick the exit country. All three are documented in the Scraping API reference.

Grab the top-stories feed and parse it with feedparser, which handles the RSS namespaces and the element for you.

import feedparser

def get_feed(url):
    xml = fetch(url, render_js="false", premium_proxy="true", country_code="US")
    return feedparser.parse(xml)

feed = get_feed("https://news.google.com/rss?hl=en-US&gl=US&ceid=US:en")

for entry in feed.entries[:5]:
    # Search and topic feeds format the title as "Headline - Publisher".
    headline, _, from_title = entry.title.rpartition(" - ")
    publisher = getattr(entry, "source", {}).get("title") or from_title
    print({
        "headline":  headline or entry.title,
        "publisher": publisher,
        "published": entry.get("published"),
        "gnews_link": entry.link,          # still encoded at this stage
        "gnews_id":   entry.id,            # the encoded article id
    })

Two parsing notes save you time. First, read the publisher from entry.source.title when it exists; the - Publisher suffix on the headline is a fallback because some items omit it. Second, do not try to read a clean summary out of entry.summary on search feeds. Google packs an HTML list of related links in there, not a readable snippet, so treat description as structured navigation rather than article text.

The search feed is where Google News gets useful for monitoring, because the q parameter accepts the same operators as the Google News search box. Recency is the big one: news queries without a time window return stale, high-authority pages instead of what broke this hour.

Because the query goes in a URL, encode it. Here is a builder that assembles a search feed and pulls the last hour of headlines for a brand:

from urllib.parse import quote_plus

def search_feed_url(query, hl="en-US", gl="US", ceid="US:en"):
    return (f"https://news.google.com/rss/search?"
            f"q={quote_plus(query)}&hl={hl}&gl={gl}&ceid={ceid}")

url  = search_feed_url('intitle:"rate cut" OR "rate hike" when:1h')
feed = get_feed(url)
print(f"{len(feed.entries)} fresh items")

Two behaviors worth knowing. There is no page two: if when:1h returns 100 items you are already at the cap, so tighten the query or poll more often rather than trying to paginate. And site: filters by the source Google attributes the story to, which is a fast way to watch a single outlet without scraping that outlet directly.

This is the section every other Google News tutorial fumbles. The link and guid in each item look like https://news.google.com/rss/articles/CBMiV..., and they are encrypted redirects, not publisher URLs.

For years the fix was to base64-decode the string after articles/. When it started with CBMi the decoded bytes were a small protobuf that contained the real URL in plain text. That method broke in 2024. Google changed the encoding so the decoded blob no longer carries the destination, and every scraper that relied on the base64 shortcut now returns garbage. If a guide you are reading tells you to base64-decode the id, it is out of date.

There are two working approaches in 2026.

The edition parameters ask for a country's news. The request IP decides whether Google honors that cleanly, and it introduces one gotcha that stops a lot of scrapers cold.

The EU consent wall. A request to news.google.com from a European IP with no consent cookie gets redirected to consent.google.com, and your parser sees a cookie-consent page instead of RSS. You have two clean fixes. Exit from a country that does not show the wall by setting country_code=US, or, when you genuinely need an EU edition, inject the consent cookie so the feed loads from a local IP.

def eu_feed(url, country):
    return fetch(
        url,
        render_js="false",
        premium_proxy="true",
        country_code=country,          # e.g. "DE" for the German edition
        cookies='[{"name":"SOCS","value":"CAISNQ",'
                '"domain":".google.com"}]',
    )

german = eu_feed(
    "https://news.google.com/rss?hl=de&gl=DE&ceid=DE:de", "DE")

Matching the exit to the edition. Even outside the EU, pairing gl/ceid with a same-country country_code gives you the edition a local reader actually sees, and it keeps Google from second-guessing a US IP that claims to want German news. For a German feed, set hl=de&gl=DE&ceid=DE:de and route through a country_code=DE residential exit. The concept behind why the exit location changes results is covered in What Does Geo-Targeting Mean in Proxies.

This is also the difference between Google News scraping and general search scraping. If you also pull ranked results from google.com/search, the geo and blocking mechanics differ enough that they get their own playbook in How to Scrape Google Search Results.

A single feed pull is trivial. A monitoring system polling dozens of queries across several editions every few minutes is a different animal, and it fails in three predictable ways: 429s from over-polling one IP, the consent wall on EU exits, and duplicate stories from wire copy that AP or Reuters syndicate to hundreds of outlets.

Dedupe on the resolved publisher URL, not the Google link, because the same story arrives under different encoded ids across feeds. Normalize the URL (drop tracking query params) before hashing.

import time
from urllib.parse import urlsplit, urlunsplit

seen = set()

def canonical(url):
    parts = urlsplit(url)
    return urlunsplit((parts.scheme, parts.netloc, parts.path, "", ""))

def poll(feed_urls, delay=1.5):
    for url in feed_urls:
        feed = get_feed(url)
        for entry in feed.entries:
            resolved = resolve_via_browser(entry.link)
            if not resolved:
                continue
            key = canonical(resolved)
            if key in seen:
                continue          # wire copy already collected elsewhere
            seen.add(key)
            yield {"url": key, "headline": entry.title,
                   "published": entry.get("published")}
        time.sleep(delay)         # polite pacing between feeds

Three rules keep this alive in production. Poll each query no faster than the news actually moves; when:1h queries every 10 to 15 minutes catch everything without hammering. Let the proxy layer rotate IPs so no single address carries the whole load, and retry a 429 with exponential backoff plus jitter rather than an immediate re-hit. The general pattern for staying under rate limits is worth reading in full: How to Scrape High-Volume Data Without Rate Limiting. If a feed exit gets blocked despite pacing, the checklist in How to Avoid Getting Your Proxy Blocked applies directly.