How to Scrape SEC EDGAR Filings (10-K, 10-Q, 8-K)

Most scraping guides fight a website that does not want to be scraped. EDGAR is the opposite. The SEC runs EDGAR (Electronic Data Gathering, Analysis, and Retrieval) as a public utility, and it actively publishes machine-readable APIs so you do not have to parse HTML at all. Every disclosure a U.S. public company files, the annual 10-K, the quarterly 10-Q, the ad-hoc 8-K, proxy statements, insider trades, is free and in the public domain.

That changes your whole strategy. You are not bypassing anti-bot systems here. You are consuming an official data feed, and the only thing standing between you and a clean pipeline is following the SEC's stated access rules. Get those right and EDGAR is one of the most reliable data sources on the internet. Get them wrong and you get blocked, not because you outsmarted a defense, but because you ignored a posted sign.

A quick note on scope. If you want live share prices or intraday quotes, EDGAR is the wrong tool, and our guide to scraping stock market data covers that. EDGAR is for regulatory filings and the financial statements inside them.

The SEC exposes four REST endpoints on the data.sec.gov host. None of them need an API key. All of them return JSON. Learn these four and you rarely need to parse a filing by hand.

The ########## is a 10-digit zero-padded CIK, which we cover in a moment. The submissions and company facts APIs are your workhorses. Frames is the one people forget exists, and it is the fastest way to pull the same line item from thousands of companies at once.

Before you write a single request, understand the SEC's fair-access policy, because it is the difference between a pipeline that runs for years and one that dies in an afternoon.

You must send a descriptive User-Agent header. The SEC requires a User-Agent that identifies your application and a contact email, formatted like Sample Company AdminContact@sample.com. If you send the default Python or curl User-Agent, or none at all, EDGAR returns 403 Forbidden. This is not a rumor. When we tested plain automated fetches against sec.gov while writing this guide, every request without a proper User-Agent came back 403. The header is the price of entry.

You must stay under 10 requests per second. The SEC caps automated traffic at a maximum of 10 requests per second per source. Cross that line and your IP can be blocked for a period. Treat 10 as a ceiling you never touch, not a target. In practice, cap yourself at 8 or fewer and cache aggressively so you never re-fetch the same document.

Here is a session and a small rate limiter that bakes both rules in. Reuse this session for every call in the rest of the guide.

import time
import threading
import requests

# A descriptive User-Agent is mandatory. Use a real contact address.
HEADERS = {
    "User-Agent": "SparkProxy Research research@sparkproxy.io",
    "Accept-Encoding": "gzip, deflate",
}

session = requests.Session()
session.headers.update(HEADERS)

class RateLimiter:
    """Cap outgoing requests below the SEC's 10 req/sec ceiling."""
    def __init__(self, max_per_sec=8):
        self.min_interval = 1.0 / max_per_sec
        self.lock = threading.Lock()
        self.last = 0.0

    def wait(self):
        with self.lock:
            gap = self.min_interval - (time.monotonic() - self.last)
            if gap > 0:
                time.sleep(gap)
            self.last = time.monotonic()

limiter = RateLimiter(max_per_sec=8)

One more point that matters for the compliance section later. Rotating through proxies to push past 10 requests per second is a violation of the SEC's fair-access policy, not a clever workaround. Proxies have a legitimate role with EDGAR, covered below, but evading the published rate limit is not it.

Every filer on EDGAR has a Central Index Key, or CIK. It is the primary key for all four APIs. You almost never know a CIK by heart, so you map a ticker or company name to it first.

The SEC publishes a single file with every ticker and its CIK: https://www.sec.gov/files/company_tickers.json. Download it once, cache it, and build a lookup.

def load_cik_map():
    url = "https://www.sec.gov/files/company_tickers.json"
    limiter.wait()
    data = session.get(url, timeout=30).json()
    # data is keyed by row index: {"0": {"cik_str": 320193, "ticker": "AAPL", ...}, ...}
    return {row["ticker"].upper(): row["cik_str"] for row in data.values()}

def cik_for(ticker, cik_map):
    cik = cik_map[ticker.upper()]
    return f"{cik:010d}"   # 320193 -> "0000320193"

cik_map = load_cik_map()
print(cik_for("AAPL", cik_map))   # 0000320193

The zero-padding is the part people miss. The data.sec.gov APIs want a 10-digit CIK, so Apple's raw CIK 320193 becomes 0000320193. Skip the padding and you get a 404. Here is the trap that follows: the filing archive path (next section) uses the un-padded integer CIK instead. Same company, two formats, depending on which host you hit.

The submissions API returns a company's recent filings as a set of parallel arrays. You zip them together and filter by form type.

def get_submissions(cik10):
    url = f"https://data.sec.gov/submissions/CIK{cik10}.json"
    limiter.wait()
    return session.get(url, timeout=30).json()

def recent_filings(subs, forms=("10-K", "10-Q", "8-K")):
    recent = subs["filings"]["recent"]
    rows = zip(
        recent["accessionNumber"],
        recent["form"],
        recent["filingDate"],
        recent["primaryDocument"],
    )
    for accession, form, filed, doc in rows:
        if form in forms:
            yield {
                "accession": accession,     # 0000320193-23-000106
                "form": form,               # 10-K
                "filed": filed,             # 2023-11-03
                "primary_document": doc,    # aapl-20230930.htm
            }

subs = get_submissions("0000320193")
for f in recent_filings(subs, forms=("10-K",)):
    print(f["filed"], f["form"], f["accession"])

The filings.recent block holds roughly the last 1,000 filings. For a company with a longer history, the JSON includes a filings.files array pointing to additional pages, each a separate JSON file you fetch the same way. Loop through those only if you actually need filings older than what recent returns.

Here is a cheat sheet of the form types you will filter on most:

The submissions API tells you a filing exists and gives you its accession number and primary document name. To fetch the document itself, you build a URL into the EDGAR archive on www.sec.gov.

def document_url(cik10, accession, primary_document):
    cik = int(cik10)                       # archive path uses the UN-padded CIK
    folder = accession.replace("-", "")    # 0000320193-23-000106 -> 000032019323000106
    return (
        f"https://www.sec.gov/Archives/edgar/data/"
        f"{cik}/{folder}/{primary_document}"
    )

# https://www.sec.gov/Archives/edgar/data/320193/000032019323000106/aapl-20230930.htm
url = document_url("0000320193", "0000320193-23-000106", "aapl-20230930.htm")
limiter.wait()
html = session.get(url, timeout=60).text

Two format quirks live in that one function. The CIK in the archive path drops its zero-padding. The accession number drops its dashes to form the folder name but keeps them everywhere else. Modern filings are inline XBRL (iXBRL), which means the .htm document is human-readable HTML with machine-readable financial tags embedded in it. You can parse the visible tables with a library like BeautifulSoup, but if all you want is the numbers, the XBRL APIs below skip the parsing entirely.

Some older filings and exhibits ship as PDFs rather than HTML. If you need to pull text out of those, our guide to extracting data from PDFs walks through the tooling.

This is the part that makes EDGAR special. Because filings are tagged in XBRL, the SEC can hand you clean numbers without you touching the document. XBRL (eXtensible Business Reporting Language) tags each financial value with a standard concept name, a unit, and a reporting period.

The Company Facts API returns every fact a company has reported, organized by taxonomy (us-gaap for standard accounting concepts, dei for entity metadata) and then by concept.

def company_facts(cik10):
    url = f"https://data.sec.gov/api/xbrl/companyfacts/CIK{cik10}.json"
    limiter.wait()
    return session.get(url, timeout=30).json()

facts = company_facts("0000320193")
# Drill into one concept:
revenue = facts["facts"]["us-gaap"]["RevenueFromContractWithCustomerExcludingAssessedTax"]
for row in revenue["units"]["USD"]:
    print(row["fy"], row["fp"], row["end"], row["val"], row["form"])

If you only want one line item, the Company Concept API is lighter. It returns the full history of a single concept for one company.

def company_concept(cik10, tag, taxonomy="us-gaap"):
    url = f"https://data.sec.gov/api/xbrl/companyconcept/CIK{cik10}/{taxonomy}/{tag}.json"
    limiter.wait()
    return session.get(url, timeout=30).json()

# Apple's annual revenue, straight from the 10-Ks:
data = company_concept("0000320193", "RevenueFromContractWithCustomerExcludingAssessedTax")
annual = [r for r in data["units"]["USD"] if r.get("form") == "10-K" and r.get("fp") == "FY"]
for r in annual:
    print(r["fy"], r["val"])

The Frames API flips the question around. Instead of one concept across time for one company, it gives you one concept for one period across every company that reported it. This is how you build a cross-sectional dataset in a single request.

def frame(tag, period, unit="USD", taxonomy="us-gaap"):
    url = f"https://data.sec.gov/api/xbrl/frames/{taxonomy}/{tag}/{unit}/{period}.json"
    limiter.wait()
    return session.get(url, timeout=30).json()

# Every filer's Q1 2024 revenue in one call:
rows = frame("RevenueFromContractWithCustomerExcludingAssessedTax", "CY2024Q1")["data"]
print(len(rows), "companies reported this concept")

The period string in Frames trips people up, so here is the rule. CY2024 is a full calendar year. CY2024Q1 is a quarter measured over its duration, correct for flow items like revenue. CY2024Q1I with the trailing I is instantaneous, a single point in time, correct for balance-sheet items like cash or total assets. Ask for a duration period on a balance-sheet concept and you get an empty result and a confused afternoon.

A few high-value us-gaap concept tags to get you started:

One honest caveat: companies do not all tag identically. Older filings may use retired revenue tags such as Revenues or SalesRevenueNet, so a complete pipeline checks a short list of candidate tags per concept rather than assuming one. That messiness is the real work in EDGAR analysis, and it is worth knowing before you promise someone a clean number.

The APIs above assume you already know which company you want. Full-text search flips that: you search the actual words inside every filing submitted since 2001 and get back the ones that match. The endpoint is https://efts.sec.gov/LATEST/search-index and it returns JSON.

def full_text_search(query, forms=None, startdt=None, enddt=None, ciks=None, frm=0):
    params = {"q": query, "from": frm}
    if forms:
        params["forms"] = ",".join(forms)
    if ciks:
        params["ciks"] = ",".join(ciks)
    if startdt and enddt:
        params["dateRange"] = "custom"
        params["startdt"] = startdt
        params["enddt"] = enddt
    limiter.wait()
    r = session.get("https://efts.sec.gov/LATEST/search-index", params=params, timeout=30)
    return r.json()

# Every 10-K filed in 2024 that mentions the exact phrase "climate risk":
res = full_text_search('"climate risk"', forms=["10-K"], startdt="2024-01-01", enddt="2024-12-31")
print("total hits:", res["hits"]["total"]["value"])
for hit in res["hits"]["hits"]:
    src = hit["_source"]
    print(src["file_date"], src["display_names"], hit["_id"])

Wrap a phrase in double quotes inside the q value to force an exact-phrase match instead of matching the individual words. The key parameters:

Each hit's _id looks like 0000320193-23-000106:aapl-20230930.htm, which is the accession number and file name joined by a colon. Split on the colon and you have exactly what document_url() needs to fetch the filing. Two limits to plan around: the API returns results in pages, and the total reachable result set is capped in the low thousands. Narrow by form type and date range so your query lands under the cap instead of getting truncated. Full-text coverage starts in 2001, so anything filed before then is not in this index.

Here is the honest positioning, because it matters more than a sales pitch. For the JSON APIs on data.sec.gov, you do not need proxies. Plain requests with a correct User-Agent and the rate limiter above is the right tool, and reaching for anything heavier is over-engineering.

Proxies and a managed scraper earn their place in three specific situations:

1. Parsing filing documents at volume. When you are pulling thousands of 10-K HTML documents from the archive and converting each to clean text, a managed API gives you automatic retries, timeout handling, and one-call HTML-to-markdown conversion instead of a pile of BeautifulSoup.
2. Rendered pages. A handful of EDGAR viewer pages and exhibits render content with JavaScript. A headless-browser API handles those without you running a browser fleet.
3. Resilience. If your single IP gets temporarily throttled, clean managed egress plus retry logic keeps a long harvest moving.

The SparkProxy Scraping API fits all three, and critically, it lets you keep the SEC-required User-Agent while it works. Set it with the custom_ua parameter so every request through the pool still identifies you the way the SEC demands.

import requests

SPARK_KEY = "sk-your-key"   # from your SparkProxy dashboard

def fetch_filing_markdown(filing_url):
    resp = requests.get(
        "https://scrape.sparkproxy.io/api/v1",
        headers={"X-API-Key": SPARK_KEY},
        params={
            "url": filing_url,
            "render_js": "false",       # EDGAR filings are static HTML
            "format": "md",             # convert the 10-K straight to clean markdown
            "custom_ua": "SparkProxy Research research@sparkproxy.io",  # SEC-required UA
        },
        timeout=120,
    )
    return resp.text

markdown = fetch_filing_markdown(
    "https://www.sec.gov/Archives/edgar/data/320193/000032019323000106/aapl-20230930.htm"
)

That format=md conversion is genuinely useful when you feed filings into a language model, because a raw 10-K is hundreds of kilobytes of nested HTML tables and a markdown version is far cheaper to process. If you prefer to pass the header yourself, forward_headers takes a JSON object: forward_headers='{"User-Agent": "SparkProxy Research research@sparkproxy.io"}'.

The rule does not change when you add a proxy: keep your aggregate request rate at or below the SEC's ceiling and keep the descriptive User-Agent attached. The proxy handles rendering, retries, and clean egress. It does not buy you permission to go faster than fair access allows. For the broader pattern of backing off politely under load, see our retry and backoff strategies guide, and for warehousing what you pull, how to store scraped data.

Run through this before you turn a scraper loose on EDGAR:

• Send a descriptive User-Agent with your app name and a real contact email on every request. No exceptions, or it is 403 on the first call.
• Cap your rate below 10 requests per second in aggregate across all threads and all IPs. Slower is safer.
• Cache everything locally. A filing never changes after it is accepted, so re-fetching the same document wastes the SEC's bandwidth and yours.
• Do not use proxy rotation to beat the rate limit. That is the one proxy use case the fair-access policy rules out.
• Zero-pad CIKs to 10 digits for data.sec.gov, but use the un-padded integer in the Archives path.
• Handle amendments. A 10-K/A supersedes or corrects an earlier 10-K; do not treat them as duplicates.
• Match the frame period to the concept. Instantaneous (I suffix) for balance-sheet items, duration for flows.
• Check multiple XBRL tags per concept, because tagging conventions changed over the years and one tag rarely covers a full history.

Get those eight right and EDGAR behaves like the reliable public feed it is meant to be. For a wider view of doing this responsibly across any source, our ethical scraping and rate limiting guide is the companion read, and the financial data collection use case shows where filings fit alongside market data.