How to Extract Data from PDF Files at Scale

A single PDF is easy. A thousand PDFs from forty different sources is a different job, because the failures are not evenly distributed. Roughly speaking you deal with three kinds of file:

Two of those three look identical to a naive script that just calls extract_text(). The scanned file returns an empty string. The broken-encoding file returns text that is technically non-empty and completely wrong. If you route on "did I get any text back", you will silently ship garbage from the third bucket. The fix is a quality gate before parsing, which is the first thing most tutorials skip.

The second scaling issue is the download itself. Public reports, filings, price lists, and datasheets are often linked from pages that rate-limit or block automated clients, or the PDF host geoblocks by region. That part is a scraping problem, not a parsing problem, and it belongs behind proxies or a scraping API. PDFs are a heavily used source in market research and data collection, so this is a common wall to hit.

Before you parse anything you have to fetch it. When the PDF links live on a page that runs JavaScript or throws a bot check, a plain requests.get returns a challenge page instead of HTML. The SparkProxy Scraping API renders the page, rotates the exit IP, and hands back the HTML, so you can pull the PDF hrefs out of it.

import requests
from bs4 import BeautifulSoup
from urllib.parse import urljoin

API = "https://scrape.sparkproxy.io/api/v1"
KEY = "sk-your-key"   # from app.sparkproxy.io

def get_html(url, render=True):
    r = requests.get(API, headers={"X-API-Key": KEY}, params={
        "url": url,
        "render_js": str(render).lower(),  # true renders JS (5 credits)
        "premium_proxy": "true",           # residential IP for anti-bot hosts
    }, timeout=90)
    r.raise_for_status()
    return r.text

listing = "https://www.sparkproxy.io/reports"
soup = BeautifulSoup(get_html(listing), "html.parser")
pdf_links = [urljoin(listing, a["href"]) for a in soup.select("a[href$='.pdf']")]

Now download each PDF. With render_js=false the API does a plain HTTP fetch and returns the raw body, which for a .pdf URL is the file bytes. Write response.content straight to disk.

def download_pdf(url, path):
    r = requests.get(API, headers={"X-API-Key": KEY}, params={
        "url": url,
        "render_js": "false",   # plain fetch, 1 credit, returns the PDF bytes
        "country_code": "us",   # exit region if the host geoblocks
    }, timeout=120)
    r.raise_for_status()
    with open(path, "wb") as f:
        f.write(r.content)

for i, link in enumerate(pdf_links):
    download_pdf(link, f"pdfs/{i}.pdf")

Each API call spends a credit, so for high-volume bulk downloads where the host is not heavily defended, routing requests directly through datacenter proxies is cheaper per file:

proxies = {
    "http":  "http://user:pass@dc.sparkproxy.io:10000",
    "https": "http://user:pass@dc.sparkproxy.io:10000",
}
r = requests.get(link, proxies=proxies, timeout=120)
open(path, "wb").write(r.content)

The trade-off between managing your own proxy pool and letting an API handle rotation, retries, and rendering is a real one, laid out in web scraping API vs self-managed proxies. A practical rule: send the pages that are hard to reach (JS-rendered listings, defended hosts) through the API, and pull the raw PDF bytes through datacenter proxies when the file host is cooperative. If your fetches start returning challenge pages or 403s, how to avoid getting your proxy blocked covers the header and pacing fixes.

One more option worth knowing: the API's format=pdf parameter renders any web page to a PDF (it requires render_js=true). That is handy when the data you want lives in an HTML page and you would rather snapshot it as a PDF and run the same extraction pipeline over everything.

This is the routing decision that everything downstream depends on, and "does extract_text() return anything" is the wrong test. A file can hand you a full string of characters that decode to nonsense.

The tell is the (cid:NN) token. When a PDF embeds a font subset without a ToUnicode map, the text layer stores glyph IDs that no library can turn back into real characters, so pdfminer and PyMuPDF surface them as literal (cid:12), (cid:34) strings. That file has a non-empty text layer and is still unreadable. Treat it exactly like a scanned page and send it to OCR.

import fitz  # PyMuPDF

def classify(path, sample_pages=5):
    doc = fitz.open(path)
    chars = cid_hits = 0
    for page in doc[:sample_pages]:
        text = page.get_text()
        chars += len(text.strip())
        cid_hits += text.count("(cid:")   # missing ToUnicode map = broken glyphs
    doc.close()
    if chars < 50:
        return "scanned"          # no real text layer
    if cid_hits > 3:
        return "broken-encoding"  # text present but garbage
    return "text"

Route on the result: text goes to pdfplumber or PyMuPDF, and both scanned and broken-encoding go to the OCR path. Sampling the first five pages keeps the check cheap; a 300-page file does not need a full scan to know which bucket it is in.

There is no single best library, and picking by popularity instead of by the shape of your file is why extractions come out wrong. Match the tool to the job:

The short version: reach for PyMuPDF when you want speed and clean reading order, pdfplumber when you need word-level positions or the tables have lines, camelot or tabula when the document is table-heavy, and pytesseract only when there is no usable text layer. Most real pipelines use two or three of these together, chosen per file by the classifier above.

For a digital PDF, pdfplumber gives you readable text with almost no ceremony:

import pdfplumber

with pdfplumber.open("report.pdf") as pdf:
    text = "\n".join(page.extract_text() or "" for page in pdf.pages)

The or "" matters. extract_text() returns None for a page with no extractable text, and "\n".join throws on a None, so the guard keeps one blank page from killing a 200-page run.

PyMuPDF is the faster option and gives you more control over reading order, which is the thing that breaks on complex layouts. Its "blocks" mode returns each text block with its bounding box, so you can sort top-to-bottom and left-to-right instead of trusting the order the characters happen to sit in the file:

import fitz

doc = fitz.open("report.pdf")
page = doc[0]
blocks = page.get_text("blocks")  # (x0, y0, x1, y1, text, block_no, block_type)
blocks.sort(key=lambda b: (round(b[1]), b[0]))  # by top edge, then left edge
text = "\n".join(b[4] for b in blocks if b[4].strip())
doc.close()

Rounding the y-coordinate before sorting groups words that sit on the same visual line even when their baselines differ by a fraction of a point. Without that rounding, a single line can shatter into out-of-order fragments.

Tables are where most extractions fall apart, because a PDF table is not a table. It is text positioned to look like a grid, with no cell structure underneath. The right tool depends on whether the table has drawn lines.

camelot is the strongest option for tables with visible ruling lines. Its two flavors are the whole game: lattice reads tables bordered by lines, stream infers columns from whitespace gaps. Pick wrong and you get merged or split columns.

import camelot

# lattice: tables with drawn borders. stream: whitespace-separated columns.
tables = camelot.read_pdf("statement.pdf", pages="all", flavor="lattice")
print(tables[0].parsing_report)  # {'accuracy': 98.7, 'whitespace': 12.3, ...}
tables[0].df.to_csv("table_0.csv", index=False)

Do not guess the flavor by eye across a large batch. Run lattice, read parsing_report["accuracy"], and fall back to stream when it comes back low. That single check automates a decision people otherwise make by hand on every file.

def best_table(path, page):
    lattice = camelot.read_pdf(path, pages=str(page), flavor="lattice")
    if lattice and lattice[0].parsing_report["accuracy"] > 80:
        return lattice[0].df
    stream = camelot.read_pdf(path, pages=str(page), flavor="stream")
    return stream[0].df if stream else None

pdfplumber handles borderless tables reasonably well and needs no system dependencies, which makes it the easy first try:

with pdfplumber.open("statement.pdf") as pdf:
    for page in pdf.pages:
        for table in page.extract_tables():
            for row in table:
                print(row)  # row is a list of cell strings

tabula-py is worth keeping for wide financial tables that camelot mangles, with the caveat that it shells out to a Java runtime, so a JRE has to be installed:

from tabula import read_pdf

dfs = read_pdf("statement.pdf", pages="all", lattice=True)  # returns DataFrames

For anything you would call "parse PDF tables" at volume, the pattern that holds up is camelot first with the accuracy fallback, pdfplumber for borderless grids, and tabula reserved for the specific files that beat both.

When the classifier returns scanned or broken-encoding, there is no text to read, so you rasterize each page and OCR the image. You can skip the usual pdf2image plus poppler dependency by rendering with PyMuPDF directly into a PIL image:

import fitz, pytesseract
from PIL import Image

def ocr_pdf(path, dpi=300, lang="eng"):
    doc = fitz.open(path)
    pages = []
    for page in doc:
        pix = page.get_pixmap(dpi=dpi)  # 300 DPI: the accuracy floor for OCR
        img = Image.frombytes("RGB", (pix.width, pix.height), pix.samples)
        pages.append(pytesseract.image_to_string(img, lang=lang))
    doc.close()
    return "\n".join(pages)

Two numbers drive OCR quality. DPI is the first: 150 DPI produces visibly worse results than 300, and going past 400 mostly buys you slower runs, not better text. The lang parameter is the second. Tesseract ships language packs, and running an English model over a French or German scan drops accuracy hard, so set lang to match the document (fra, deu, or a combination like "eng+fra").

Pytesseract needs the Tesseract binary installed on the machine, not just the pip package. On Debian or Ubuntu that is apt-get install tesseract-ocr, plus tesseract-ocr-fra and similar for extra languages.

A two-column page is the classic failure. Extract it naively and the parser reads straight across the page, interleaving the left and right columns line by line into unreadable text. The fix is to crop each column and extract them in order.

with pdfplumber.open("two-column.pdf") as pdf:
    page = pdf.pages[0]
    w, h = page.width, page.height
    left  = page.crop((0, 0, w / 2, h)).extract_text() or ""
    right = page.crop((w / 2, 0, w, h)).extract_text() or ""
    text = left + "\n\n" + right

Splitting at w / 2 works for a clean symmetric layout. For pages where the column boundary sits elsewhere, or the count changes per page, detect the gap from word positions instead of hardcoding it:

with pdfplumber.open("report.pdf") as pdf:
    page = pdf.pages[0]
    xs = sorted(word["x0"] for word in page.extract_words())
    # a wide horizontal gap between consecutive word starts is the column break
    gaps = [(b - a, a) for a, b in zip(xs, xs[1:]) if b - a > 40]
    boundary = max(gaps)[1] if gaps else page.width / 2

The same block-sorting trick from the PyMuPDF section is the other half of the answer. When headers, footers, and sidebars scatter across the page, sorting blocks by their bounding box rebuilds reading order that the raw character stream lost.

Extraction is only useful if it lands in a shape you can query. Normalize every file into a list of records, then let the shape of the data decide the format. Flat rows go to CSV, nested or ragged data goes to JSON.

import csv, json

records = [
    {"date": "2026-07-01", "invoice": "INV-1042", "amount": 1290.00},
    {"date": "2026-07-03", "invoice": "INV-1043", "amount": 640.50},
]

with open("out.csv", "w", newline="", encoding="utf-8") as f:
    writer = csv.DictWriter(f, fieldnames=records[0].keys())
    writer.writeheader()
    writer.writerows(records)

with open("out.json", "w", encoding="utf-8") as f:
    json.dump(records, f, indent=2, ensure_ascii=False)

Two settings save real pain here. encoding="utf-8" keeps accented names and currency symbols intact instead of crashing on Windows, and ensure_ascii=False writes readable JSON rather than é escapes. When the output is a table you got as a DataFrame from camelot or tabula, skip the manual loop and use df.to_csv("out.csv", index=False) or df.to_json("out.json", orient="records").

The mistake at volume is treating download and parse as one loop. They have opposite bottlenecks. Downloading is IO-bound and often anti-bot-bound, so it wants concurrency and lives behind the API or proxies. Parsing is CPU-bound, so it wants one process per core and gains nothing from threads because of the GIL. Split the pipeline at that seam and each half scales on its own axis.

from concurrent.futures import ThreadPoolExecutor, ProcessPoolExecutor

# Stage 1: download is IO/anti-bot bound -> threads, behind the Scraping API
with ThreadPoolExecutor(max_workers=16) as pool:
    paths = list(pool.map(fetch_one, pdf_links))

# Stage 2: parse is CPU bound -> one process per core
with ProcessPoolExecutor() as pool:
    records = list(pool.map(parse_one, [p for p in paths if p]))

parse_one is where the classifier earns its place: it opens the file, decides text versus OCR versus table, and dispatches to the matching extractor so no thread wastes cycles OCR-ing a file that already had clean text.

def parse_one(path):
    kind = classify(path)
    if kind == "text":
        return extract_text_records(path)   # pdfplumber / PyMuPDF
    if kind in ("scanned", "broken-encoding"):
        return extract_from_ocr(ocr_pdf(path))
    return None

Three things keep a large run from falling over. Wrap each file's work in a try/except and log the failing path rather than letting one corrupt PDF halt the batch. Deduplicate by hashing file bytes, because the same report often links from several pages and re-parsing it wastes the most expensive step. Cache downloads to disk so a re-run does not re-fetch and re-spend credits on files you already have. Those three habits are the difference between a script that works on ten files and one that finishes on ten thousand.