How to Scrape Stock Market Data

"Stock market data" is four different things, and they carry different rules:

• Quotes: the last price, change, volume, and intraday range you see on Yahoo Finance, Google Finance, or MarketWatch. Mostly delayed on free pages.
• Historical prices: daily or intraday open/high/low/close/volume (OHLCV) bars going back years.
• Fundamentals: revenue, earnings, P/E, balance-sheet items. In the US these ultimately trace back to filings on SEC EDGAR, which is public-domain government data.
• News and sentiment: headlines, filings alerts, analyst notes.

Public here means "reachable without a login." It does not mean "unlicensed." A quote page is public, but the real-time number behind it is licensed by the exchange that produced it. That gap between accessible and usable for your purpose is the single most misunderstood thing in financial data scraping, and section 5 is where it bites people.

If your goal is broader competitive or market intelligence rather than raw ticks, the workflow in using proxies for market research and data collection frames the same collection problem at a higher level.

Before you write a single line of a scraper, check whether an official API already returns what you need. An API is the sanctioned path: it is more stable than HTML that changes weekly, it keeps you inside terms of service by design, and it hands you clean JSON instead of a DOM you have to reverse-engineer. Reach for scraping only when no API exposes the specific field, market, or granularity you want.

Here are the providers worth knowing, with rough free-tier limits. Limits change often, so treat these as a starting point and confirm against each provider's current docs.

A quote from Alpha Vantage is four lines. The GLOBAL_QUOTE function returns the last price and change for a symbol:

import requests

r = requests.get("https://www.alphavantage.co/query", params={
    "function": "GLOBAL_QUOTE",
    "symbol": "AAPL",
    "apikey": "YOUR_KEY",
})
quote = r.json()["Global Quote"]
print(quote["05. price"], quote["10. change percent"])

The yfinance Python library is the popular shortcut for Yahoo data. It is an unofficial community wrapper, not a sanctioned Yahoo API, so it breaks whenever Yahoo changes its endpoints. It is fine for research and prototypes and a poor choice for anything you need to run unattended in production:

import yfinance as yf

aapl = yf.Ticker("AAPL")
print(aapl.fast_info["last_price"])      # current-ish price
hist = aapl.history(period="1mo")        # a month of daily OHLCV

For US fundamentals, EDGAR is the source of record and it is genuinely free to reuse. It does require a descriptive User-Agent that identifies you, a rule the SEC enforces:

import requests

# SEC fair-access policy: declare who you are, stay under 10 requests/sec
headers = {"User-Agent": "SparkProxy Research research@sparkproxy.io"}
cik = "0000320193"  # Apple Inc.
url = (
    f"https://data.sec.gov/api/xbrl/companyconcept/CIK{cik}"
    "/us-gaap/RevenueFromContractWithCustomerExcludingAssessedTax.json"
)
data = requests.get(url, headers=headers).json()

This is the distinction that separates a working data pipeline from a misleading one. The price on a free Yahoo Finance page for a US stock is usually delayed by about 15 minutes. It is not a bug and no proxy fixes it. Real-time consolidated US equity quotes come from the exchange SIP feeds (the CTA and UTP plans), and exchanges charge licensing fees for them. Free public pages show delayed data precisely because real-time redistribution costs money and requires an agreement.

The practical takeaway: if you scrape a public quote and label it "real-time" in a product, you have both a data-accuracy problem and a licensing problem. For backtesting, screening, valuation work, and end-of-day analytics, delayed and historical data is completely fine and freely scrapeable within terms. For live trading signals, pay for a real-time licensed feed. Do not try to scrape your way around an exchange license.

A single quote page carries most of what a stock data scraper needs. These are the standard fields and where they surface. Values are illustrative.

Yahoo makes these unusually easy to target. The numbers on a quote page live inside custom elements with a data-field attribute and a machine-readable data-value, so you read the raw number without parsing formatted text:

from bs4 import BeautifulSoup

soup = BeautifulSoup(html, "html.parser")

def field(name):
    el = soup.select_one(f'fin-streamer[data-field="{name}"]')
    return el["data-value"] if el and el.has_attr("data-value") else None

print(field("regularMarketPrice"))         # e.g. 231.45
print(field("regularMarketChangePercent")) # e.g. 0.0079
print(field("regularMarketVolume"))        # e.g. 41250000
print(field("marketCap"))                  # e.g. 3510000000000

Pulling data-value instead of the visible text saves you from stripping commas, currency symbols, and the "T"/"B" suffixes on market cap.

Scraping public pages is not automatically a crime. In hiQ Labs v. LinkedIn, the Ninth Circuit found that scraping publicly accessible data generally does not violate the Computer Fraud and Abuse Act. That ruling is narrow, though: it is about the anti-hacking statute, not about contracts or copyright. Breaching a site's terms of service and redistributing licensed data are separate exposures the CFAA question never touches. Treat these as the working rules:

• Read robots.txt and the terms of service for each source before scraping it. Many finance sites explicitly restrict automated access or bulk collection.
• Do not redistribute licensed market data. Real-time exchange quotes are owned by the exchanges. Scraping and reselling them, or embedding them in a commercial product, needs a licensing agreement no scraper can substitute for.
• Personal research is a lighter footprint than commercial resale. Screening stocks for yourself sits very differently from selling a data feed built on scraped quotes.
• SEC EDGAR filings are public-domain government records. You can freely reuse the underlying fundamentals. That is often the cleanest legal path to company financials.
• Never present delayed data as real-time. It is inaccurate and, in a paid product, it invites a licensing dispute.

Scraping for private analysis, backtesting, or academic work is standard practice and low-risk when you respect robots.txt, rate limits, and copyright. Building a commercial data product on scraped real-time quotes is where teams get letters from lawyers. When in doubt, license the feed.

When no API fits, Yahoo Finance is the most scrapeable public source, and there is a right and a wrong endpoint to hit. Most guides that teach you to scrape Yahoo Finance point at the old quote endpoint, which no longer works cleanly. Here is what actually holds up.

Use the chart endpoint for prices. query1.finance.yahoo.com/v8/finance/chart/{symbol} returns JSON with price and historical bars and does not demand the authentication dance the quote endpoint now does:

import requests

url = "https://query1.finance.yahoo.com/v8/finance/chart/AAPL"
r = requests.get(
    url,
    params={"interval": "1d", "range": "5d"},
    headers={"User-Agent": "Mozilla/5.0"},
)
result = r.json()["chart"]["result"][0]
meta = result["meta"]
print(meta["regularMarketPrice"], meta["currency"])
# OHLCV bars live in result["indicators"]["quote"][0]

Know the crumb trap on the old quote endpoint. Since late 2023, query1.finance.yahoo.com/v7/finance/quote?symbols=AAPL requires a valid consent cookie plus a crumb token. Call it without them and you get an HTTP 401 with an "Invalid Cookie" or "Invalid Crumb" body, not helpful documentation. You must first hit fc.yahoo.com to receive an A1 cookie, then request a crumb from query1.finance.yahoo.com/v1/test/getcrumb using that cookie, and pass the crumb on every quote call. This crumb requirement is the reason so many old Yahoo scraping snippets suddenly return 401, and it is exactly the plumbing yfinance handles for you behind the scenes. When you can, prefer the v8 chart endpoint and skip the crumb entirely.

Fall back to HTML only when you need a field the JSON omits. Parse it with the fin-streamer selector from the fields section above. Expect Yahoo to rename CSS classes periodically, so anchor on the stable data-field attribute rather than on layout classes.

Hit any public finance source too fast from one IP and it pushes back. Yahoo commonly answers with HTTP 429 (Too Many Requests) and occasionally a curious HTTP 999 that is its own soft rate-limit signal. SEC EDGAR caps you at 10 requests per second and will block an IP that ignores it. The fixes are boring and they work.

Send a real User-Agent, throttle yourself, and back off on failure instead of retrying in a tight loop:

import time, requests

RETRYABLE = {429, 500, 502, 503, 504, 999}
HEADERS = {"User-Agent": "Mozilla/5.0 (compatible; research-bot)"}

def get_with_backoff(url, params=None, max_retries=4):
    for attempt in range(max_retries + 1):
        r = requests.get(url, params=params, headers=HEADERS, timeout=20)
        if r.status_code == 200:
            return r
        if r.status_code not in RETRYABLE or attempt == max_retries:
            r.raise_for_status()
        sleep = min(2 ** attempt, 30)          # 1s, 2s, 4s, 8s, capped
        time.sleep(sleep)
    return None

Three habits keep you unblocked: space requests out (a small delay between symbols beats a burst), rotate the source IP once you scrape more than a few hundred symbols so no single address crosses a threshold, and cache aggressively so you never re-request a quote you already have this minute. For the full playbook on staying under detection thresholds, see how to avoid getting your proxy blocked.

Some finance pages sit behind Cloudflare or similar anti-bot layers that fingerprint your TLS handshake, not just your User-Agent. A plain requests client fails those regardless of headers. That is the point where a headless browser or a managed scraping API earns its keep.

Financial pages localize. The same URL can serve different currencies, exchange listings, or availability depending on where the request appears to originate. A ticker dual-listed in New York and London may resolve differently, some regional exchange data is restricted outside its home country, and localized pages format numbers by locale. If you need the US view of a symbol, request from a US IP; for the London view, request from the UK.

Datacenter proxies handle most finance targets well because the pages are content, not high-security logged-in flows, and datacenter IPs are fast and cheap at volume. The trade-offs and setup are covered in using datacenter proxies for web scraping. Route each request through an exit in the country whose market view you want, and keep the geography consistent within a single symbol's collection so you are not mixing a US price with a UK volume.

Rotating IPs, matching TLS fingerprints, and running a browser for JavaScript-heavy quote pages is a lot of moving parts to maintain. The SparkProxy Scraping API collapses it into one request: you send a target URL, it picks and rotates the proxy, renders JavaScript when needed, and returns the page. A basic quote fetch, geo-targeted to the US:

curl "https://scrape.sparkproxy.io/api/v1?url=https://finance.yahoo.com/quote/AAPL&render_js=true&country_code=us" \
  -H "X-API-Key: YOUR_API_KEY"

The same call in Python, with structured extraction so you get fields back instead of raw HTML:

import json, requests

extract = {
    "price":      'fin-streamer[data-field="regularMarketPrice"]@data-value',
    "change_pct": 'fin-streamer[data-field="regularMarketChangePercent"]@data-value',
    "volume":     'fin-streamer[data-field="regularMarketVolume"]@data-value',
    "market_cap": 'fin-streamer[data-field="marketCap"]@data-value',
}

resp = requests.get(
    "https://scrape.sparkproxy.io/api/v1",
    headers={"X-API-Key": "YOUR_API_KEY"},
    params={
        "url": "https://finance.yahoo.com/quote/AAPL",
        "render_js": "true",       # quote pages hydrate values with JavaScript
        "country_code": "us",      # US market view
        "premium_proxy": "true",   # residential IPs for tougher targets
        "extract_rules": json.dumps(extract),
    },
    timeout=90,
)
resp.raise_for_status()
print(resp.json())

Set render_js=false on static pages to keep the request cheaper, and enable it only where the numbers load through JavaScript (Yahoo's do). country_code sets the exit geography, premium_proxy upgrades to residential IPs when a target is defended, and extract_rules returns parsed fields so you skip the BeautifulSoup step for well-structured pages. Because the API assigns a fresh IP per call and keeps fingerprints current, the rate-limit and anti-bot work from the earlier sections is handled for you.

Whether that is worth it over running your own pool comes down to target difficulty and volume. The web scraping API vs self-managed proxies breakdown gives you a break-even you can run against your own numbers. A common split for financial data: pull structured fundamentals from official APIs and EDGAR, scrape delayed quotes off public pages with your own pool at volume, and send the JavaScript-heavy or Cloudflare-protected pages to the Scraping API.