How to Scrape Medium Articles: RSS, JSON, and Claps
Scrape Medium articles cleanly: pull stories from RSS feeds, read the ?format=json payload past its XSSI prefix, and collect tags, authors, and clap counts.
To scrape Medium articles you almost never need to fight Medium's React front end or its Cloudflare edge. Medium hands you three quieter interfaces: an RSS feed for every profile, publication, and tag; a ?format=json payload sitting behind almost every URL; and a block of JSON-LD baked into each article page. This guide shows how to read all three, how to pull the fields that only exist in one of them (clap counts, the member-only flag, per-tag post counts), how to get past the RSS 10-item cap, and the one parsing gotcha that makes json.loads fail on Medium every single time. Every request routes through SparkProxy's Scraping API, so IP rotation and geo-targeting are one parameter instead of an infrastructure project.
Key Takeaways
- Medium exposes RSS feeds at
medium.com/feed/@user,/feed/tag/, and/feed/, and each returns the full HTML body of public posts incontent:encoded. No browser needed.- The
?format=jsonendpoint carries the fields RSS omits:virtuals.totalClapCountfor claps,virtuals.tagswith per-tag post counts, andisLockedfor the paywall. Its response is prefixed with])}while(1);, which you must strip before parsing.- Member-only stories return
isLocked: trueand only a preview in RSS. Scrape public content, skip the locked bodies, and never republish full article text.
Is scraping Medium legal?
Medium publishes RSS feeds and a JSON view on purpose, so reading public stories is not the same as breaking into a locked endpoint. The line that matters is public versus member-only, and facts versus expression.
In the United States, the Ninth Circuit's ruling in hiQ Labs v. LinkedIn (2022) held that scraping publicly accessible data, meaning content with no login wall, generally does not violate the Computer Fraud and Abuse Act. Separately, under Feist Publications v. Rural Telephone (1991), facts are not copyrightable. A story's title, its author, its tags, its publish date, and its clap count are facts about the post. The article body is creative expression owned by the author, so it is protected. That split gives you a clean rule: collect metadata freely, keep only short snippets of body text under fair use, and never republish someone's full article.
Two checks belong in any serious build. Read Medium's Terms of Service and its robots.txt before you crawl at scale, and rate-limit yourself so you are not degrading the service for anyone else. If the data feeds a commercial product, run it past a lawyer. This is engineering guidance, not legal advice. The general pattern for polite collection is worth reading in full in our guide on ethical scraping and rate limiting.
Scraping at scale? Skip the blocks.
Fast, unblockable datacentre proxies with unlimited bandwidth.
The Medium RSS feeds
Every Medium profile, publication, and tag has an RSS feed. The feeds return clean RSS 2.0 XML, they include the full post body for public stories, and they never require rendering. This is the same feed-first approach that makes an aggregator like Google News scriptable, covered in How to Scrape Google News.
| Feed | URL pattern |
|---|---|
| User profile | `https://medium.com/feed/@username` |
| Publication (on medium.com) | `https://medium.com/feed/publication-slug` |
| Publication (custom domain) | `https://blog.example.com/feed` |
| Tag | `https://medium.com/feed/tag/python` |
| Sitewide latest | `https://medium.com/feed/latest` |
Each title, link, a guid of the form https://medium.com/p/, one pubDate, and @ in a profile feed is required; medium.com/feed/username without it will not resolve.
One hard limit shapes everything downstream: a Medium RSS feed returns only the latest 10 items and does not paginate. For a low-frequency monitor that is fine. For a backfill you need the archive pages in the pagination section below.
The ?format=json endpoint and the )}while(1) prefix
Append ?format=json to almost any Medium URL and you get that page's underlying data as JSON: a profile, a single post, a tag page, a publication. This is where claps, the isLocked paywall flag, reading time, and richly-typed tags live. It is the same "hidden JSON behind the HTML" technique described in How to Scrape Hidden JSON API Endpoints, with one Medium-specific twist that trips up everyone the first time.
Medium does not return clean JSON. It prefixes every response with an anti-hijacking guard:
])}while(1);</x>{"success":true,"payload":{ ... }}
That leading ])}while(1); is deliberate. It is XSSI (cross-site script inclusion) protection: if some other site tries to load the URL as a