/html/body/div[1]/div/div[3]/section/div[2]/span

# Matches the exact class token "price", never "price-old" or "unit-price"
'//span[contains(concat(" ", normalize-space(@class), " "), " price ")]'

The CSS approach is `table.specs tr:nth-child(2) td`. It breaks the day a "Colour" row is inserted, and it breaks per product if the table is variable-length, which spec tables usually are. XPath anchors on the label instead:

python

from parsel import Selector

sel = Selector(text=html)

sku = sel.xpath(

"//table[contains(@class,'specs')]"

"//th[normalize-space()='SKU']"

"/following-sibling::td[1]/text()"

).get(default="").strip()

That selector does not care about row order, row count, or class hashes. It cares that a header cell says SKU, which is the same thing a human reader uses to find it.

#### Ancestor traversal: find the child, extract from the parent

You often identify a record by something deep inside it and then need fields from the container. CSS cannot climb.

python

Find the card containing an "In stock" badge, then read that card's title

titles = sel.xpath(

"//span[normalize-space()='In stock']"

"/ancestor::*[@data-testid='product-card'][1]"

"//h3/text()"

).getall()

The `[1]` on the ancestor step matters. `ancestor::` returns every matching ancestor, and without the predicate a nested card structure hands you the outermost match instead of the nearest one. Test against nested markup before you trust it.

#### Text predicates: use `.`, not `text()`

This is the subtlest bug in the whole topic. In XPath 1.0, `contains()` takes a string as its first argument. Give it a node-set and the spec converts that node-set to a string by taking the string-value of the **first node only**. So:

html

Ships from Berlin in 2 days

python

sel.xpath("//p[contains(text(), 'days')]") # no match: first text node is "Ships from "

sel.xpath("//p[contains(., 'days')]") # matches: "." is the whole string-value

`text()` returns the element's direct text-node children. `.` returns the concatenated string-value of the element and all of its descendants. For "does this element mention X", `.` is nearly always what you want. Pair it with `normalize-space()` for exact label matching, because real markup is full of newlines and indentation:

python

"//th[normalize-space()='SKU']" # tolerant of "\n SKU\n "

"//th[text()='SKU']" # fails on exactly the same markup

XPath 1.0 has no regex, but lxml exposes the [EXSLT regular expressions](https://exslt.github.io/regexp/) module, which earns its keep on messy labels:

python

from lxml import html as lhtml

doc = lhtml.fromstring(page_html)

ns = {"re": "http://exslt.org/regular-expressions"}

rows = doc.xpath(

"//th[re:test(., '^(SKU|Part ?No\\.?)$', 'i')]/following-sibling::td[1]",

namespaces=ns,

)

## Positional logic, optional nodes, and index errors {#positional}

Two failure modes account for most parser exceptions in production.

#### The `//div[1]` trap

XPath predicates bind to the step, and positions are relative to the parent context, not the document. So `//div[1]` means "every `div` that is the first `div` child of its parent", which on a normal page returns dozens of nodes. To get the first match in document order, parenthesise:

python

sel.xpath("//div[@class='result'][1]") # first result inside EACH container

sel.xpath("(//div[@class='result'])[1]") # the first result on the page

XPath indices are 1-based. Python list slicing on the result is 0-based. Mixing the two inside one comprehension is a reliable way to be off by one.

#### Optional and repeated nodes

XPath and CSS both return an empty node-set for no match, never an error. The exception comes from your own code:

python

Raises IndexError the first time a product has no sale badge

price = sel.css(".sale-badge::text").getall()[0]

parsel gives you the safe forms directly. `.get()` returns `None` or a supplied default for zero matches, and `.getall()` always returns a list:

python

price = sel.css("[data-testid=price]::text").get(default="").strip()

badges = sel.css(".badge::text").getall() # [] when absent, no branch needed

href = sel.css("a.product-link::attr(href)").get()

With raw lxml, `.xpath()` returns a list, so use `next(iter(...), None)` or index defensively:

python

def first(node, expr, default=None):

hits = node.xpath(expr)

return hits[0] if hits else default

For repeated records, iterate the container and select **relative to it**, with a leading dot. A missing dot restarts the search at the document root, so every card gets its field values from the first card on the page:

python

for card in sel.css("[data-testid=product-card]"):

yield {

"title": card.xpath(".//h3/text()").get(default="").strip(), # correct

"title": card.xpath("//h3/text()").get() # every card returns card #1

}

That missing dot is the most common XPath bug in scraping code, and it produces plausible-looking output rather than a crash, which is exactly why it survives code review.

## Namespaces: why //div matches nothing in XHTML and XML {#namespaces}

Parse a real XHTML document, an Atom feed, or a sitemap with an XML parser and your selectors return nothing at all, with no error.

The cause: the document declares a default namespace such as `xmlns="http://www.w3.org/1999/xhtml"`, so every element is `{http://www.w3.org/1999/xhtml}div`, not `div`. XPath 1.0 has no concept of a default namespace inside an expression. An unprefixed name means "no namespace", so `//div` correctly matches zero nodes.

Three ways out, in order of preference.

**Use an HTML parser for HTML.** `lxml.html` and parsel's default HTML mode ignore namespaces entirely, which is what you want for anything served as `text/html`, even when it carries an XHTML doctype. The [WHATWG HTML parsing spec](https://html.spec.whatwg.org/multipage/parsing.html) builds the tree without prefixes, so parsel's HTML selectors just work.

**Bind a prefix** when you genuinely need XML semantics:

python

from lxml import etree

doc = etree.fromstring(xml_bytes)

ns = {"x": "http://www.w3.org/1999/xhtml", "atom": "http://www.w3.org/2005/Atom"}

titles = doc.xpath("//atom:entry/atom:title/text()", namespaces=ns)

**Strip namespaces** when you are only reading, not round-tripping. parsel has this built in:

python

sel = Selector(text=feed_xml, type="xml")

sel.remove_namespaces()

links = sel.xpath("//loc/text()").getall() # now works on a sitemap

One trap worth naming: the `local-name()` workaround, `//*[local-name()='div']`, is portable but defeats libxml2's name-based lookups and gets noticeably slower on large documents. Use it for one-off exploration, not in a hot parsing loop.

SVG inside HTML is the sneaky case. SVG elements sit in the SVG namespace even when embedded in `text/html`, so `//svg//title` behaves differently from what you expect if something upstream switched you to an XML parser.

## Real extraction code with both languages {#real-code}

Here is a parser that uses each language for what it is good at: CSS for flat class and attribute matching, XPath for label anchoring and upward traversal.

python

import parsel

def parse_product(html: str) -> dict:

sel = parsel.Selector(text=html)

CSS: stable data attributes, flat matching, terse

title = sel.css("[data-testid=product-title]::text").get(default="").strip()

price = sel.css("[itemprop=price]::attr(content)").get()

imgs = sel.css("figure.gallery img::attr(src)").getall()

XPath: label-anchored spec table, independent of order and length

specs = {}

for row in sel.xpath("//table[contains(@class,'specs')]//tr"):

key = row.xpath("normalize-space(./th)").get()

val = row.xpath("normalize-space(./td)").get()

if key:

specs[key] = val or ""

XPath: identify by text, then climb to the container

stock = sel.xpath(

"//*[@data-testid='availability']"

"/ancestor::section[1]//span[contains(., 'in stock')]/text()"

).get(default="").strip()

return {

"title": title,

"price": price,

"images": imgs,

"specs": specs,

"in_stock": bool(stock),

}

Note `normalize-space(./th)` used as the whole expression rather than as a predicate. XPath string functions can be the result, not just a filter, which saves a `.strip()` and collapses internal whitespace in the same step. parsel returns the computed string as a single-item result, so `.get()` works as usual.

If you are working in Node instead of Python, Cheerio implements the CSS half of this API and no XPath at all, which is worth knowing before you pick a stack. See our guide to [web scraping with Cheerio and Node.js](https://www.sparkproxy.io/blog/web-scraping-with-cheerio-and-nodejs) for the equivalent patterns. For the wider Python pipeline around this parser, the [Python web scraping tutorial](https://www.sparkproxy.io/blog/python-web-scraping-tutorial-extract-any-website-in-2026) covers fetching, retries, and storage.

## Server-side extraction with the SparkProxy Scraping API {#api-extraction}

Selectors do not help if the fields are painted by JavaScript after load. The [SparkProxy Scraping API](https://www.sparkproxy.io/docs/scraping-api/) renders the page in headless Chromium and can run your selectors server side, returning JSON instead of HTML. Base URL is `https://scrape.sparkproxy.io/api/v1`, and you authenticate with the `X-API-Key` header.

`extract_rules` accepts **CSS selectors only**, which mirrors the split described above: flat, attribute-anchored matching in the API, and anything needing text or ancestor logic back in your own parser.

python

import requests

resp = requests.post(

"https://scrape.sparkproxy.io/api/v1",

headers={"X-API-Key": "sk-your-key", "Content-Type": "application/json"},

json={

"url": "https://www.sparkproxy.io/pricing",

"render_js": True,

"wait_for": "[data-testid=plan-card]",

"json_response": True,

"extract_rules": {

"title": "h1",

"plans": {"selector": "[data-testid=plan-name]", "type": "list"},

"prices": {"selector": "[data-testid=plan-price]", "type": "list"},

"docs_link": {"selector": "a.docs-cta", "type": "href"},

},

},

timeout=90,

)

data = resp.json()

The selector types are a plain string for `querySelector` text content, `"list"` for `querySelectorAll`, and `"href"` or `"src"` for resolved absolute URLs. `wait_for` also takes a CSS selector, so gate on the element you are about to extract rather than on a fixed sleep.

When you need XPath, ask for the rendered HTML and parse it yourself:

python

import parsel, requests

html = requests.get(

"https://scrape.sparkproxy.io/api/v1",

headers={"X-API-Key": "sk-your-key"},

params={"url": "https://www.sparkproxy.io/pricing", "render_js": "true",

"wait_for": "[data-testid=plan-card]"},

timeout=90,

).text

sel = parsel.Selector(text=html)

row = sel.xpath("//li[contains(., 'Concurrent requests')]/following-sibling::li[1]/text()").get()

Rendering in the API and running XPath locally covers the case that pure CSS extraction cannot reach. If the page is heavily client-rendered, read [how to scrape JavaScript-rendered websites](https://www.sparkproxy.io/blog/how-to-scrape-dynamic-javascript-websites) first, and check whether the data is available from a JSON endpoint before writing any selector at all, as covered in [scraping hidden JSON API endpoints](https://www.sparkproxy.io/blog/how-to-scrape-hidden-json-api-endpoints).

## Build a selector-drift detector {#drift-detector}

Here is the monitoring gap almost nobody closes. A scraper that returns HTTP 200, parses without exception, and writes 400 rows of empty strings looks healthy to every dashboard you own. Uptime is green. Error rate is zero. The data is worthless.

The fix is cheap: record how many nodes each selector matched, compare against an expected range, and fail loudly when the count leaves the band.

python

from dataclasses import dataclass, field

@dataclass

class FieldSpec:

name: str

expr: str

kind: str = "css" # "css" or "xpath"

min_matches: int = 1

max_matches: int | None = None

required: bool = True

@dataclass

class DriftReport:

pages: int = 0

matches: dict = field(default_factory=dict) # name -> list[int]

def record(self, name, count):

self.matches.setdefault(name, []).append(count)

def alerts(self, specs, null_rate_threshold=0.05):

out = []

for spec in specs:

counts = self.matches.get(spec.name, [])

if not counts:

continue

zeros = sum(1 for c in counts if c == 0)

rate = zeros / len(counts)

if spec.required and rate > null_rate_threshold:

out.append(f"{spec.name}: {rate:.0%} of {len(counts)} pages matched 0 nodes")

over = [c for c in counts if spec.max_matches and c > spec.max_matches]

if over:

out.append(f"{spec.name}: {len(over)} pages exceeded max_matches "

f"({max(over)} > {spec.max_matches})")

return out

def extract(sel, specs, report):

report.pages += 1

row = {}

for spec in specs:

nodes = sel.css(spec.expr) if spec.kind == "css" else sel.xpath(spec.expr)

report.record(spec.name, len(nodes))

row[spec.name] = nodes[0].get() if nodes else None

return row

Wire it into the end of a run:

python

SPECS = [

FieldSpec("title", "[data-testid=product-title]::text", max_matches=1),

FieldSpec("price", "[itemprop=price]::attr(content)", max_matches=1),

FieldSpec("sku", "//th[normalize-space()='SKU']/following-sibling::td[1]/text()",

kind="xpath", max_matches=1),

FieldSpec("badges", ".badge::text", min_matches=0, required=False),

]

report = DriftReport()

rows = [extract(parsel.Selector(text=h), SPECS, report) for h in pages]

problems = report.alerts(SPECS)

if problems:

raise SystemExit("SELECTOR DRIFT:\n" + "\n".join(problems))

```

Four things make this worth the fifty lines.

Aggregate, never per-page. A single product legitimately lacking a sale badge is normal. Ninety percent of products lacking a price is a redesign. The threshold is what separates the two, so alert on the rate across a batch rather than on any individual miss.

Alert on too many matches as well as too few. When a site swaps a unique id for a repeated class, a max_matches=1 field starts matching 40 nodes and your parser quietly takes the first one, which is often a hidden template node or a skeleton loader. A count-too-high check catches that. A null check never will.

Version the expectations. Store the observed count band from a known-good run and diff against it, so the alert tells you what changed instead of just that something did.

Keep an HTML fixture per target in the repo and run the selectors against it in CI. That separates "our selector is wrong" from "the site changed", which are different fixes and different on-call responses. Pair the drift detector with block detection too, since a challenge page also produces zero matches for every field. Once drift is caught and the parse is right again, the downstream normalisation work is covered in our guide on how to clean scraped data.