How to Clean Scraped Data: Parse, Normalize, Validate

Scraping gets you the bytes. Turning those bytes into records you can trust is a second job, and it is where most pipelines quietly rot. To clean scraped data well you fix five separate problems in a specific order: broken text encoding, HTML noise, inconsistent whitespace, numbers and dates that are trapped inside strings, and rows that are duplicated or invalid. Get the order wrong and each fix undoes the last one. This guide walks every stage with runnable Python, shows how the SparkProxy Scraping API hands you structured fields at the source so there is less to clean, and finishes with one pipeline that takes raw scraper output to a validated Parquet file.

Cleaning is not one function. It is a pipeline, and the sequence is load-bearing. Run the steps out of order and you create bugs that look like data-quality problems but are really ordering problems.

Here is the order that holds up, with the reason each step has to come before the next:

1. Fix encoding first. If the bytes decode wrong, a euro sign shows up as € and your currency regex never matches. Every later step reads corrupted input.
2. Strip HTML second. Tag removal and entity decoding happen on text you can now read correctly.
3. Normalize whitespace third. Collapse non-breaking spaces, zero-width characters, and runs of blanks before you try to parse anything by pattern.
4. Parse typed fields fourth. Now that the string is clean, pull the price into a Decimal and the date into a UTC datetime.
5. Validate, then deduplicate. Reject rows that fail their schema, and drop duplicates using keys built from the normalized values, not the raw ones.

The two mistakes almost everyone makes: running a price or date regex before fixing encoding, and hashing records for dedupe before Unicode normalization. Both produce silent data loss. We will hit each one in its section.

Here is the library cheat sheet for the whole pipeline:

Install the third-party ones in one go:

pip install ftfy charset-normalizer beautifulsoup4 lxml Babel python-dateutil pydantic pyarrow

The cheapest cleaning is the cleaning you never do. Before you parse a single tag, ask the scraper to return structured fields instead of a page of HTML. The SparkProxy Scraping API supports this with extract_rules, a JSON object of CSS selectors that runs server side. Base URL is https://scrape.sparkproxy.io/api/v1, and you authenticate with the X-API-Key header.

import requests

rules = {
    "title": "h1.product-title",
    "price_text": ".price-now",
    "availability": ".stock-status",
    "images": {"selector": "img.gallery", "type": "src"},
    "specs": {"selector": "ul.spec-list li", "type": "list"},
}

resp = requests.post(
    "https://scrape.sparkproxy.io/api/v1",
    headers={"X-API-Key": "YOUR_API_KEY", "Content-Type": "application/json"},
    json={
        "url": "https://www.sparkproxy.io/sample/product/42",
        "render_js": True,
        "extract_rules": rules,
        "json_response": True,
    },
    timeout=60,
)

record = resp.json()["extracted"]
# {"title": "...", "price_text": "$1,299.00", "availability": "In stock", ...}

That one request replaces the DevTools spelunking, the render_js browser, and the tag-walking code you would otherwise write per field. Set json_response to true and you also get status, credits used, and page metadata in the envelope. If the site serves its data from a private JSON endpoint, skipping HTML entirely is even better; see how to scrape JSON API endpoints behind a page for that route, and web scraping API vs self-managed proxies for when to reach for the managed API at all.

Notice what extract_rules does not do: price_text still comes back as "$1,299.00", a string with a currency symbol and a thousands separator. Structured extraction gets you the right field. Turning that field into a number you can sum is still your job, and that is the rest of this guide.

When you do have raw HTML, get to plain text with a parser, never with a regex. re.sub(r"<[^>]+>", "", html) breaks on comments, """ soup = BeautifulSoup(html, "lxml") for tag in soup(["script", "style", "noscript", "template"]): tag.decompose() text = soup.get_text(separator=" ", strip=True) # "Great laptop. 16GB RAM & 1TB SSD."

Three details decide whether the output is usable. Call decompose() on script and style first, or their inner JavaScript and CSS end up in your text. Pass separator=" " to get_text, because the default glues

• One
• Two

into OneTwo. And strip=True trims each fragment before joining. When you want the visible chunks as a list instead of one blob, iterate soup.stripped_strings.

For extracting one field out of a big page, target it first and read text from that node only:

price_node = soup.select_one("span.price")
price_text = price_node.get_text(strip=True) if price_node else None

Mojibake is the garbled text you get when bytes encoded as UTF-8 are decoded as Latin-1 or Windows-1252. café becomes café, € becomes €, a curly apostrophe becomes ’. It looks like a display bug. It is actually a data bug, because your downstream regex and joins operate on the broken characters.

The ftfy library ("fixes text for you") repairs the common cases in one call:

import ftfy
import unicodedata

raw = "Müller’s Café  €1.299,00 / £25.50"
fixed = ftfy.fix_text(raw)
# "Müller's Café  €1.299,00 / £25.50"

canonical = unicodedata.normalize("NFC", fixed)

After repair, normalize the Unicode form. The same visible character can be encoded two ways: é as one code point (U+00E9) or as e plus a combining accent (U+0301). They look identical and compare as unequal, which wrecks joins, dedupe, and search. Pick one form and apply it everywhere.

• NFC composes characters into their single-code-point form. Use it for display and storage.
• NFKC also folds compatibility characters: full-width digits become ASCII, the fi ligature becomes fi, ① becomes 1. Use it for keys you compare or hash.

Here is the reference for the mojibake you will actually see:

When you are decoding raw bytes yourself rather than getting a string, do not trust the Content-Type header. Detect the encoding:

from charset_normalizer import from_bytes

result = from_bytes(raw_bytes).best()
text = str(result)          # decoded with the detected encoding
print(result.encoding)      # e.g. "utf_8" or "windows-1252"

charset-normalizer is the library requests now depends on, and it beats the older chardet on speed and on modern web pages.

Scraped strings are full of whitespace that is not the space key. Non-breaking spaces ( ), zero-width spaces (​), tabs, newlines from pretty-printed HTML, and the leftover BOM () all count as "not equal" to a normal space, so "in stock" == extracted_value fails for reasons you cannot see.

import re
import unicodedata

def collapse_ws(s: str) -> str:
    s = s.replace(" ", " ").replace("​", "").replace("", "")
    s = unicodedata.normalize("NFKC", s)
    return re.sub(r"\s+", " ", s).strip()

collapse_ws("  In stock \n  ")   # "In stock"

NFKC turns exotic Unicode spaces into ordinary ones, then \s+ collapses any run of whitespace to a single space, and strip() trims the ends. Run this on every text field before you compare it, key on it, or store it.

Prices are the field people get most wrong, because number formatting is locale-specific and the naive fix corrupts half the world's data. In the United States, 1,299.00 is one thousand two hundred ninety-nine. In Germany, 1.299,00 is the same amount with the separators swapped. A quick float(s.replace(",", "")) reads the German string as 1.29900, off by a factor of a thousand, with no error raised.

Use Babel for locale-aware parsing, and Decimal for the result so you never inherit binary float rounding on money:

import re
from decimal import Decimal
from babel.numbers import parse_decimal

def to_amount(raw: str, locale: str) -> Decimal:
    # keep digits and the two separators only, drop symbols and spaces
    digits = re.sub(r"[^\d.,]", "", raw)
    return parse_decimal(digits, locale=locale, strict=False)

to_amount("$1,299.00", "en_US")   # Decimal('1299.00')
to_amount("€1.299,00", "de")      # Decimal('1299.00')
to_amount("£25.50", "en_GB")      # Decimal('25.50')

The currency itself carries meaning, so capture it as a separate field rather than throwing the symbol away:

SYMBOLS = {"$": "USD", "€": "EUR", "£": "GBP", "¥": "JPY", "₹": "INR"}

def detect_currency(raw: str) -> str | None:
    for sym, code in SYMBOLS.items():
        if sym in raw:
            return code
    m = re.search(r"\b(USD|EUR|GBP|JPY|CAD|AUD|INR)\b", raw)
    return m.group(1) if m else None

Store price and currency in two columns. A Decimal of 1299.00 means nothing until you know whether it is dollars or yen. For a target-specific walkthrough, see how to scrape ecommerce prices.

Dates arrive in every format a designer ever invented: March 3, 2026, 03/04/2026, 2026-03-03T09:30:00+02:00, 2 hours ago. Store them one way, as UTC in ISO 8601, so sorting and range queries work.

python-dateutil parses almost anything:

from datetime import timezone
from dateutil import parser as dtparser

def to_utc_iso(raw: str, dayfirst: bool = False) -> str:
    dt = dtparser.parse(raw, dayfirst=dayfirst)
    if dt.tzinfo is None:
        dt = dt.replace(tzinfo=timezone.utc)   # assume UTC if none given
    return dt.astimezone(timezone.utc).isoformat()

to_utc_iso("March 3, 2026")             # "2026-03-03T00:00:00+00:00"
to_utc_iso("2026-03-03T09:30:00+02:00") # "2026-03-03T07:30:00+00:00"
to_utc_iso("03/04/2026", dayfirst=True) # 4 March, European reading

The 03/04/2026 case is the trap. It is March 4th to an American site and April 3rd to a British one, and dateutil cannot know which. Set dayfirst from the site's locale, do not leave it to a guess. Two more habits worth keeping: always attach a timezone before you convert, and convert everything to UTC on the way in so your stored data has no ambiguity.

Relative strings like 2 hours ago or gestern need a different tool. The dateparser library resolves them against a reference time and across languages:

import dateparser

dateparser.parse("2 hours ago")          # datetime relative to now
dateparser.parse("hace 3 días")          # Spanish, "3 days ago"

Scrapes produce duplicates: paginated listings that repeat, retries that write twice, the same product under two URLs. The fix is a stable key per record, but the key has to be built from normalized values or it will not catch anything.

import hashlib
import unicodedata

def record_key(rec: dict, fields: list[str]) -> str:
    parts = []
    for f in fields:
        v = rec.get(f, "")
        if isinstance(v, str):
            v = unicodedata.normalize("NFKC", v).strip().casefold()
        parts.append(str(v))
    blob = "|".join(parts)
    return hashlib.sha256(blob.encode("utf-8")).hexdigest()

seen, unique = set(), []
for rec in records:
    key = record_key(rec, ["brand", "sku", "title"])
    if key not in seen:
        seen.add(key)
        unique.append(rec)

This is the second ordering bug from the intro. Hash the raw strings and Café (composed) and Café (decomposed) produce different digests, so you keep both. iPhone and iphone also survive as two records. Normalize with NFKC and casefold before hashing, and those collapse to one key. casefold is the aggressive sibling of lower and is the correct choice for cross-language text.

Exact hashing catches identical records. For near-duplicates, titles that differ by a stray word or punctuation, compare with a similarity score:

from rapidfuzz.fuzz import token_sort_ratio

token_sort_ratio("Apple MacBook Air 13", "MacBook Air 13 - Apple")  # ~100

Cluster records above a threshold like 90 and keep one per cluster. Save fuzzy matching for the fields that need it, because it is far slower than a hash set.

Cleaning transforms values. Validation decides which records are allowed through at all. A schema turns "I hope the price parsed" into an explicit contract, coerces types for you, and quarantines the rows that fail instead of letting them poison the dataset. pydantic v2 is the standard tool:

from datetime import datetime
from decimal import Decimal
from pydantic import BaseModel, field_validator, ValidationError

class Product(BaseModel):
    title: str
    price: Decimal
    currency: str
    in_stock: bool
    scraped_at: datetime

    @field_validator("title")
    @classmethod
    def title_present(cls, v: str) -> str:
        v = v.strip()
        if not v:
            raise ValueError("empty title")
        return v

    @field_validator("price")
    @classmethod
    def price_positive(cls, v: Decimal) -> Decimal:
        if v <= 0:
            raise ValueError("price must be positive")
        return v

clean, rejected = [], []
for raw in candidates:
    try:
        clean.append(Product.model_validate(raw))
    except ValidationError as e:
        rejected.append({"row": raw, "errors": e.errors()})

pydantic coerces "true" to a real bool and "1299.00" to a Decimal, and it fails loudly when a required field is missing or a price is zero. The pattern that matters is the two-bucket loop: valid records go to clean, failures go to rejected with their error detail. Never silently drop the rejects. Log them and count them, because a rejection rate that jumps from 2% to 40% overnight is usually the first sign a site changed its markup.

The last step is serialization, and the format decides whether your types survive. Match it to who reads the data next.

import csv
import json
import pyarrow as pa
import pyarrow.parquet as pq

rows = [p.model_dump() for p in clean]

# JSON Lines: stream-friendly, one record per line
with open("products.jsonl", "w", encoding="utf-8") as f:
    for r in rows:
        f.write(json.dumps(r, ensure_ascii=False, default=str) + "\n")

# CSV with a BOM so Excel reads UTF-8 correctly
with open("products.csv", "w", newline="", encoding="utf-8-sig") as f:
    w = csv.DictWriter(f, fieldnames=list(rows[0].keys()))
    w.writeheader()
    w.writerows(rows)

# Parquet: typed, compressed, columnar
table = pa.Table.from_pylist(rows)
pq.write_table(table, "products.parquet", compression="zstd")

Three flags earn their place. ensure_ascii=False writes café as café, not café, which keeps the file readable and smaller. utf-8-sig prepends the BOM that makes Excel show accented characters instead of mojibake, the exact bug you just spent a section fixing. And Parquet preserves Decimal and datetime as real types with compression, where CSV stringifies everything and loses the distinction between the number 1299.00 and the text "1299.00".

For CSV or a stream, that is the finish line. Once the data needs indexes, upserts, or a price history you query over time, move it into a database. How to store scraped data covers schema design, content-hash upserts, and time-series storage in depth.

Here is the whole thing wired end to end: one function that cleans a raw record, then the validate, dedupe, and write stages in the order from the first section.

from datetime import datetime, timezone

def clean_record(raw: dict, locale: str = "en_US") -> dict:
    title = collapse_ws(
        BeautifulSoup(ftfy.fix_text(raw["title"]), "lxml").get_text(" ", strip=True)
    )
    price_text = ftfy.fix_text(raw["price_text"])
    return {
        "title": title,
        "price": to_amount(price_text, locale),
        "currency": detect_currency(price_text),
        "in_stock": "in stock" in collapse_ws(raw["availability"]).casefold(),
        "scraped_at": datetime.now(timezone.utc),
    }

candidates = [clean_record(r) for r in extracted_records]

clean, rejected = [], []
for row in candidates:
    try:
        clean.append(Product.model_validate(row))
    except ValidationError as e:
        rejected.append({"row": row, "errors": e.errors()})

seen, unique = set(), []
for p in clean:
    key = record_key(p.model_dump(), ["title", "currency"])
    if key not in seen:
        seen.add(key)
        unique.append(p.model_dump())

pq.write_table(pa.Table.from_pylist(unique), "products.parquet", compression="zstd")
print(f"kept {len(unique)}  rejected {len(rejected)}")

Encoding fix runs first inside clean_record, HTML stripping and whitespace next, typed parsing after that, then validation, then dedupe on normalized keys, then output. Log rejected somewhere durable and alert on its size. When a target changes its layout, that count moves before anything downstream notices, and it tells you exactly which fields broke.