How to Scrape YouTube Data with Proxies

There are exactly two reliable sources for YouTube video data, and they answer different questions:

Most projects should start with the API and only scrape when they hit a wall. The two are not mutually exclusive. A common pattern is to use the API for the bulk of structured metadata and scrape only the specific pages where you need data the API will not give you, such as the exact rendered like count on a video that hides it from the API, or the regional recommendation shelf. If you are weighing a managed scraping service against running your own proxy pool for the scraping half, our breakdown of a web scraping API versus self-managed proxies walks through the cost and maintenance trade-offs.

Scraping public web data is not illegal in itself. In hiQ Labs v. LinkedIn, U.S. courts held that accessing publicly available data does not violate the Computer Fraud and Abuse Act. Still, YouTube's Terms of Service prohibit accessing the service through automated means except through the official API, so scraping watch pages is a Terms of Service question, not a criminal one. Treat that distinction seriously and stay on the right side of it.

Practical rules to keep your project ethical and low-risk:

• Public data only. Never scrape private or unlisted videos, age-gated content behind a login, or anything that requires you to defeat an access control.
• Prefer the official API. It exists precisely so you do not have to scrape. Use it as your default.
• Respect rate limits. Do not hammer YouTube. Pace your requests the way a human would and honor the crawl rate.
• Mind personal data. Comment author names and channel handles are personal data under GDPR and similar laws. If you store them, have a lawful basis and a retention policy.
• Do not republish copyrighted content. Metadata for analysis is one thing. Rehosting videos or transcripts wholesale is a copyright problem.

This same discipline applies to any platform monitoring work. Our guide on using proxies for social media monitoring covers the compliance posture in more depth.

Both paths expose roughly the same public fields, but they live in different places and carry different guarantees. This is the reference table to keep open while you build:

One field to set expectations on: dislikeCount has been unavailable since Google removed public dislikes in December 2021. No API or scrape will return a trustworthy dislike total. Third-party "return dislikes" numbers are estimates, not YouTube data.

Start here. Enable the "YouTube Data API v3" in a Google Cloud project, create an API key, and you can pull structured metadata with a single request. Grabbing a video's core fields costs one quota unit:

import requests

API_KEY = "YOUR_GOOGLE_API_KEY"
VIDEO_ID = "dQw4w9WgXcQ"

r = requests.get(
    "https://www.googleapis.com/youtube/v3/videos",
    params={
        "part": "snippet,statistics,contentDetails",
        "id": VIDEO_ID,
        "key": API_KEY,
    },
    timeout=15,
)
item = r.json()["items"][0]

print(item["snippet"]["title"])
print(item["statistics"]["viewCount"], "views")
print(item["statistics"].get("likeCount", "hidden"), "likes")
print(item["contentDetails"]["duration"])   # ISO 8601, e.g. PT3M33S

The catch is the quota. Every project gets 10,000 units per day by default, and the cost per call is wildly uneven:

That table explains the single most important quota rule: avoid search.list when you can. At 100 units a call you get 100 searches a day and nothing else. If you already know the video or channel IDs, videos.list and playlistItems.list are a hundred times cheaper. A better pattern than searching is to resolve a channel's uploads playlist once, then page through playlistItems.list:

def channel_uploads_playlist(channel_id, api_key):
    r = requests.get(
        "https://www.googleapis.com/youtube/v3/channels",
        params={"part": "contentDetails", "id": channel_id, "key": api_key},
        timeout=15,
    )
    return r.json()["items"][0]["contentDetails"]["relatedPlaylists"]["uploads"]

You can request a quota increase through Google's audit form, but it takes weeks and approval is not guaranteed. When 10,000 units a day is not enough and you cannot wait, scraping becomes the practical answer.

Reach for scraping only when the API genuinely cannot do the job. The honest list is short:

• Volume beyond the quota. You need metadata on hundreds of thousands of videos a day and cannot get a quota bump in time.
• Fields the API omits or rounds. The rendered like count on a video that hides likes from the API, the "Streamed live on" label, or the exact recommendation shelf for a given region.
• Per-region snapshots. What the homepage, trending shelf, or search results look like from a specific country, which the API does not model.
• No API key at all. Quick one-off pulls where standing up a Google Cloud project is overkill.

For everything else, the API wins on stability and effort. Watch-page structure changes without notice, so a scraper needs monitoring and maintenance that an API client does not. If your volume is high and steady, datacenter IPs are the cost-effective workhorse for this. Our guide on using datacenter proxies for web scraping covers how to size a pool for throughput.

A YouTube watch page ships its data inside two JavaScript variables in the initial HTML: ytInitialPlayerResponse (video metadata) and ytInitialData (the surrounding UI, including likes and comments). You do not need a headless browser to read them, but you do need to get past consent walls and bot checks, which is where a managed fetch layer earns its keep.

The SparkProxy Scraping API takes a target URL, handles the proxy, rendering, and geo, and returns the page. The base endpoint is https://scrape.sparkproxy.io/api/v1 and auth is the X-API-Key header:

import requests

SPARK_KEY = "YOUR_API_KEY"

def fetch_watch_page(video_id: str, country: str = "US") -> str:
    r = requests.get(
        "https://scrape.sparkproxy.io/api/v1",
        headers={"X-API-Key": SPARK_KEY},
        params={
            "url": f"https://www.youtube.com/watch?v={video_id}",
            "render_js": "true",       # execute the page, clear the consent interstitial
            "country_code": country,   # ISO 3166-1 alpha-2, e.g. US, GB, DE
            "wait_for": "ytd-watch-flexy",  # wait until the player shell mounts
        },
        timeout=60,
    )
    r.raise_for_status()
    return r.text

render_js=true runs the page in a real Chromium instance, which clears the EU consent interstitial that would otherwise return a consent page instead of the video. country_code routes the request through an exit in that country so you see the regional variant. If you only want the raw JSON blobs and not the rendered DOM, you can drop render_js, since ytInitialPlayerResponse is present in the server-rendered HTML, but rendering is the reliable default because it also handles the consent redirect.

Here is the part most tutorials get wrong. They pull the JSON out with a lazy regex like ytInitialPlayerResponse\s=\s(\{.+?\}), which breaks the moment a description or title contains a } character or a nested object. The non-greedy match stops at the first closing brace inside a string and hands you invalid JSON.

The correct approach is to find where the object starts and let a real JSON parser read exactly one value. Python's json.JSONDecoder().raw_decode() does this: it decodes one complete JSON value and ignores whatever trailing JavaScript follows the closing brace. It handles nested braces and braces inside strings correctly, because it is a real parser, not a pattern:

import json

def extract_json_var(html: str, var_name: str) -> dict:
    """Pull a JS-assigned JSON object (var = {...};) out of page HTML safely."""
    idx = html.find(var_name)
    if idx == -1:
        raise ValueError(f"{var_name} not found")
    brace = html.find("{", idx)
    obj, _ = json.JSONDecoder().raw_decode(html[brace:])
    return obj

With a reliable extractor, mapping the fields is straightforward:

def parse_video(html: str) -> dict:
    player = extract_json_var(html, "ytInitialPlayerResponse")
    d  = player["videoDetails"]
    mf = player.get("microformat", {}).get("playerMicroformatRenderer", {})
    return {
        "video_id":     d.get("videoId"),
        "title":        d.get("title"),
        "channel":      d.get("author"),
        "channel_id":   d.get("channelId"),
        "views":        int(d.get("viewCount", 0)),
        "duration_s":   int(d.get("lengthSeconds", 0)),
        "keywords":     d.get("keywords", []),
        "description":  d.get("shortDescription"),
        "publish_date": mf.get("publishDate"),
        "category":     mf.get("category"),
    }

The like count is not in videoDetails. It lives in ytInitialData, buried under the like button view model, and its exact path shifts between layout experiments. A resilient way to find it is to walk the tree for the likeCount or accessibilityText on the like toggle rather than hard-coding a path. If you want the field without fighting YouTube's DOM, let the Scraping API's extract_rules pull the rendered text for you and parse the label:

import json

params = {
    "url": f"https://www.youtube.com/watch?v={video_id}",
    "render_js": "true",
    "json_response": "true",
    "extract_rules": json.dumps({
        "title": "h1.ytd-watch-metadata yt-formatted-string",
        "likes": "like-button-view-model button",
    }),
}

Selectors are less stable than the JSON blob, so treat extract_rules as a convenience for a couple of fields and ytInitialPlayerResponse as your source of truth for the rest.

Pagination is where the API-versus-scraping choice pays off most, because the API's paging is trivial and scraping's is not.

YouTube has three defenses you will actually hit, in rough order of frequency:

The "confirm you're not a bot" wall got noticeably more aggressive across 2024 and 2025, especially against high-volume datacenter traffic. The reliable pattern is to escalate only when you actually get blocked, so you are not paying residential rates on every request:

BLOCK_MARKERS = ("Sign in to confirm", "consent.youtube.com", "captcha")

def is_blocked(html: str) -> bool:
    return any(m in html for m in BLOCK_MARKERS)

def fetch_with_escalation(video_id: str, country: str = "US") -> str:
    base = {
        "url": f"https://www.youtube.com/watch?v={video_id}",
        "render_js": "true",
        "country_code": country,
    }
    # Tier 1: rotating datacenter exit (cheap)
    r = requests.get("https://scrape.sparkproxy.io/api/v1",
                     headers={"X-API-Key": SPARK_KEY}, params=base, timeout=60)
    if not is_blocked(r.text):
        return r.text
    # Tier 2: residential exit + stealth (only when blocked)
    hard = {**base, "premium_proxy": "true", "stealth": "true"}
    r = requests.get("https://scrape.sparkproxy.io/api/v1",
                     headers={"X-API-Key": SPARK_KEY}, params=hard, timeout=90)
    return r.text

premium_proxy=true moves the request to a residential exit, and stealth=true adds anti-detection hardening on the rendered browser. Both cost more credits, which is exactly why you gate them behind is_blocked. The broader anti-detection playbook, from TLS fingerprints to header consistency, is in our guide on how to avoid getting your proxy blocked.

View counts and like totals are global, so you do not need geo-targeting for those. What is regional is availability, trending, search ranking, and recommendations. A video can be blocked in one country and available in another, and the trending shelf is entirely per-country. When those differences matter, set country_code to the market you care about:

for market in ("US", "GB", "DE", "JP", "BR"):
    html = fetch_watch_page(video_id, country=market)
    row = parse_video(html)
    row["market"] = market
    save(row)

For scale, three rules keep a YouTube scraper healthy:

1. Batch through the API first. videos.list accepts up to 50 IDs per one-unit call. Collapsing 50 lookups into one request is the cheapest optimization you have.
2. Cache aggressively. Titles, durations, and publish dates never change. Re-fetch only the volatile fields (views, likes, comment count) on a schedule that matches how fresh your data needs to be.
3. Rotate and pace. One IP pulling thousands of watch pages an hour is the fastest way to trip the bot wall. Spread load across a pool and keep per-IP request rates modest.

Done right, most of your traffic stays on the cheap, sanctioned API, and scraping handles only the slice the API cannot serve.