How to Scrape Reddit Data with Proxies

Reddit runs an official Data API at oauth.reddit.com. It is the sanctioned way to pull posts and comments, and for anything commercial or high volume it should be your default. You register an app at reddit.com/prefs/apps as a "script" type, get a client ID and secret, authenticate with OAuth, and use a wrapper like PRAW (the Python Reddit API Wrapper).

import praw

reddit = praw.Reddit(
    client_id="YOUR_CLIENT_ID",
    client_secret="YOUR_CLIENT_SECRET",
    user_agent="research-tool/1.0 by u/yourname",  # descriptive, required
)

for post in reddit.subreddit("programming").hot(limit=100):
    print(post.id, post.score, post.num_comments, post.title)

The free tier gives you 100 requests per minute per OAuth client ID, measured as an average over a rolling 10-minute window. Unauthenticated clients get 10 per minute. Since Reddit's June 2023 pricing change, large commercial usage above the free tier is paid, so budget for it if you plan to pull millions of objects a month.

So why would you scrape the public endpoints at all? Because for light, read-only collection of public data, the .json endpoints return the identical objects with no OAuth setup, and they are handy when you want a quick pull or a data shape the API's per-endpoint limits make awkward. The two paths return the same t1 and t3 objects, so your parsing code is the same either way.

If you are weighing whether to build your own scraper or route through a managed service, the trade-offs in web scraping API vs self-managed proxies apply directly here. Reddit is a case where the managed path often wins because the target defends aggressively.

Any listing or content page on Reddit has a JSON twin. Append .json to the path and Reddit returns the raw data. This reddit json endpoint trick is the fastest way to get clean, structured data without a headless browser.

curl -s -A "research-tool/1.0 by u/yourname" \
  "https://www.reddit.com/r/programming/hot.json?limit=25"

The common endpoints:

• Subreddit listings: /r//hot.json, /new.json, /rising.json, /top.json?t=week, /controversial.json
• A single post with its comments: /r//comments/.json
• User activity: /user//submitted.json, /user//comments.json
• Search: /search.json?q=web+scraping&sort=new&restrict_sr=1

Every listing response is wrapped in the same envelope. A Listing holds a children array, and each child has a kind (the type prefix) and a data object with the actual fields:

{
  "kind": "Listing",
  "data": {
    "after": "t3_1a2b3c",
    "before": null,
    "dist": 25,
    "children": [
      { "kind": "t3", "data": { "title": "...", "score": 428, "author": "..." } }
    ]
  }
}

One practical note on the two Reddit frontends. The modern www.reddit.com UI is a client-rendered React app, so scraping its HTML needs JavaScript execution. The .json suffix sidesteps that entirely. If you specifically need HTML rather than JSON, old.reddit.com is server-rendered and parses with plain requests, no browser required. Reach for JSON first, old.reddit.com second, and a rendered browser only when you truly need the new UI.

A post is a t3 object. These are the fields you will use most, drawn straight from the data block of a listing child:

Two details that trip people up. First, score and ups are deliberately fuzzed. Reddit adds noise to vote counts to frustrate vote manipulation, so you cannot recover exact tallies from any public source. Treat them as close approximations, not ground truth. Second, created_utc is a Unix timestamp in UTC, so convert it before you compare against local time:

from datetime import datetime, timezone

posted_at = datetime.fromtimestamp(post["created_utc"], tz=timezone.utc)

Reddit tags every object with a type prefix, called a fullname. You need these because pagination cursors and comment kinds use them:

Fetching one page is easy. Set a real User-Agent, request up to 100 items, and pull the fields you care about.

import requests

HEADERS = {"User-Agent": "research-tool/1.0 (contact you@sparkproxy.io)"}

def parse_post(child):
    d = child["data"]
    return {
        "id": d["id"],
        "title": d["title"],
        "author": d["author"],
        "subreddit": d["subreddit"],
        "score": d["score"],
        "upvote_ratio": d["upvote_ratio"],
        "num_comments": d["num_comments"],
        "created_utc": d["created_utc"],
        "permalink": "https://www.reddit.com" + d["permalink"],
        "url": d["url"],
        "is_self": d["is_self"],
        "selftext": d.get("selftext", ""),
    }

To go past 100 posts you paginate with cursors. Reddit does not use page numbers. It returns an after token (a t3_ fullname) that you feed into the next request. When after comes back null, you have reached the end.

import time

def iter_subreddit(sub, sort="new", pages=10):
    after = None
    for _ in range(pages):
        params = {"limit": 100, "after": after, "count": 0}
        r = requests.get(
            f"https://www.reddit.com/r/{sub}/{sort}.json",
            headers=HEADERS, params=params, timeout=15,
        )
        r.raise_for_status()
        data = r.json()["data"]
        for child in data["children"]:
            yield parse_post(child)
        after = data["after"]
        if after is None:      # no more pages
            break
        time.sleep(2)          # stay polite, stay under the limit

Here is the limit competitors rarely mention. A Reddit listing caps at roughly 1,000 items. After about 1,000 posts, after returns null no matter how many pages you request. You cannot page deeper into a subreddit's history through a single sort. To reach older content, slice by time with top.json?t=month across multiple windows, use search.json with date-scoped queries, or move to the official API. Any "reddit data scraper" promising the full multi-year history of a large subreddit through plain listing pagination is glossing over this cap.

Use before instead of after when you poll for new posts. Store the newest fullname you have seen, then request before= on the next run to collect only what arrived since.

The comments endpoint behaves differently from listings. It returns a two-element array: the first Listing holds the post, the second holds the comment tree.

def fetch_comments(sub, post_id):
    r = requests.get(
        f"https://www.reddit.com/r/{sub}/comments/{post_id}.json",
        headers=HEADERS, params={"limit": 500}, timeout=15,
    )
    r.raise_for_status()
    post_listing, comment_listing = r.json()   # array of two Listings

    def walk(children, depth=0):
        for child in children:
            if child["kind"] != "t1":   # 'more' stubs and non-comments
                continue
            c = child["data"]
            yield {
                "id": c["id"],
                "author": c["author"],
                "score": c["score"],
                "body": c["body"],
                "depth": depth,
                "created_utc": c["created_utc"],
            }
            replies = c.get("replies")
            if isinstance(replies, dict):   # nested replies are another Listing
                yield from walk(replies["data"]["children"], depth + 1)

    return list(walk(comment_listing["data"]["children"]))

Two things to handle. A comment is a t1 object, and its replies field is either an empty string (no replies) or a nested Listing you recurse into. Deep threads also contain kind: "more" stubs, which are collapsed branches Reddit did not expand inline. To fetch those you call the /api/morechildren endpoint with the stub's child IDs, or you request the deep-linked comment permalink directly. For most reddit scraping jobs, walking the t1 tree and skipping more stubs captures the visible conversation.

Most Reddit blocks are not about your IP. They are about your User-Agent. Reddit explicitly asks clients to send a unique, descriptive User-Agent, and it returns 429 Too Many Requests to defaults like python-requests/2.x or a bare curl string almost immediately. Fix that one header and a surprising amount of throttling disappears.

# Bad: near-instant 429
HEADERS = {}                       # sends python-requests/2.x

# Good: unique and descriptive, per Reddit's guidance
HEADERS = {"User-Agent": "research-tool/1.0 by u/yourname (contact you@sparkproxy.io)"}

The rate ceilings that apply:

On the OAuth API, every response carries X-Ratelimit-Used, X-Ratelimit-Remaining, and X-Ratelimit-Reset headers. Read them and sleep before you run out rather than after. When you do hit a 429, honor the Retry-After header if present, and back off:

import time

def get_with_backoff(url, params=None, tries=4):
    for attempt in range(tries):
        r = requests.get(url, headers=HEADERS, params=params, timeout=15)
        if r.status_code == 429:
            wait = int(r.headers.get("Retry-After", 2 ** attempt))
            time.sleep(wait)
            continue
        r.raise_for_status()
        return r
    raise RuntimeError("rate limited after retries")

Once your User-Agent is clean and your pacing is polite, the remaining blocker is IP reputation. Reddit blocks and challenges traffic from known cloud and datacenter subnets. A collection job running from a single AWS or GCP address will see 403s and 429s that the same code never triggers from a residential connection.

Proxies solve two real problems here. They give you clean, residential-grade exit IPs that are not on Reddit's datacenter block lists, and they let you distribute a legitimate volume of public reads across many addresses so no single IP crosses a throttle. Rotating IPs is standard practice for distributed collection. For the mechanics of staying under detection thresholds, see how to avoid getting your proxy blocked.

Be honest about intent. Proxies are for distributing legitimate, public, rate-respecting collection and for getting geographically correct results, not for battering past a limit you have already hit or evading a ban you earned. Teams that track brand mentions or subreddit sentiment lean on the same clean-IP setup covered in using proxies for social media monitoring, and researchers pulling public discussion at scale use the patterns in using proxies for market research and data collection.

Managing residential IPs, retries, and block handling yourself is a project. The SparkProxy Scraping API does it server-side: you send a target URL, it picks a clean exit IP, handles the request, and returns the response. To scrape Reddit posts through it, point it at the .json endpoint and keep render_js off, because a JSON endpoint has nothing to render. That keeps the call at 1 credit instead of 5.

import requests

r = requests.get(
    "https://scrape.sparkproxy.io/api/v1",
    headers={"X-API-Key": "YOUR_API_KEY"},   # key format: sk-...
    params={
        "url": "https://www.reddit.com/r/programming/top.json?t=week&limit=100",
        "render_js": "false",     # JSON endpoint, no browser needed -> 1 credit
        "country_code": "us",     # geo-target the exit IP
        "premium_proxy": "true",  # residential IPs dodge datacenter blocks
    },
    timeout=60,
)

listing = r.json()   # the same Reddit Listing, fetched through a clean IP
posts = [c["data"]["title"] for c in listing["data"]["children"]]

Authentication is the X-API-Key header carrying a key of the form sk-... from your dashboard. premium_proxy routes through the residential pool, which is what beats Reddit's datacenter blocking, and country_code sets the exit geography.

When you genuinely need the modern UI's HTML rather than JSON, such as a rendered search page, switch render_js on and give the client-side app a moment to hydrate:

r = requests.get(
    "https://scrape.sparkproxy.io/api/v1",
    headers={"X-API-Key": "YOUR_API_KEY"},
    params={
        "url": "https://www.reddit.com/search/?q=web+scraping&sort=new",
        "render_js": "true",      # new UI is client-rendered -> 5 credits
        "wait": "3",              # let the React app populate results
        "premium_proxy": "true",
    },
    timeout=90,
)
html = r.text

A sensible split: send .json pulls with render_js=false for the cheap, high-volume path, and reserve rendered requests for the handful of pages that actually require the browser.

Public data is scrapable, but "public" is not a blank check. A few rules keep a collection project defensible:

• Prefer the official API for commercial or high-volume work. The Data API Terms exist for a reason, and the free tier covers a lot of research use cases.
• Collect public data only. Do not touch private, quarantined, or gated content you have not been granted access to, and do not use scraping to route around access controls.
• Honor deletions. When an author or moderator removes content, author becomes [deleted] and body/selftext becomes [removed]. Do not resurrect removed content from caches, and refresh your dataset so deletions propagate.
• Do not build profiles that harm people. Aggregating one user's entire post history to identify or target them crosses from research into surveillance. Reddit's User Agreement and content policy prohibit it, and so should you.
• Rate-limit politely. Sleep between requests, respect Retry-After, and do not run parallel floods against a single subreddit.
• Check the rules of the road. Reddit's robots.txt and User Agreement set expectations, and Pushshift, the old bulk archive, is now restricted to moderators, so it is not a public backfill option.

Ethical scraping and clean engineering point the same direction. A job that respects rate limits, uses honest identification, and takes only public data is both less likely to get blocked and on far firmer legal ground.