How to Scrape Baidu Search Results Accurately

Baidu holds the majority of desktop search share in mainland China, which makes it the only sensible SERP source for Chinese-market rank tracking, brand monitoring, or competitor research. The engine is structurally different from Google in ways that break naive scrapers.

Four differences matter for engineering:

• Wrapped destination URLs. Organic result anchors point at https://www.baidu.com/link?url=. The token is an opaque signed value. You cannot decode it offline, so a scrape that stores the href stores nothing useful.
• Mixed legacy encoding. Baidu predates the UTF-8 consensus and still carries GB2312/GBK behaviour on some surfaces and query forms. A wrong decode gives you 锟斤拷 instead of Chinese text, and it survives into your database silently.
• First-party blocks inside the organic stream. Baijiahao (Baidu's own publishing platform), Zhidao (Q&A), Baike (encyclopedia), Tieba (forums), and Wenku (documents) appear in the same result list as third-party sites. Counting them as organic positions makes your rank data wrong in a direction that flatters Baidu.
• Offset pagination. Baidu uses pn as a result offset, not a page index. Page 3 is pn=20, not pn=3.

If you already run a Google pipeline, the anti-block layer transfers cleanly. Our guide to scraping Google search results covers the pacing and IP-reputation half of the problem, and all of it applies here. The parsing and data-integrity half does not transfer at all, and that is what the rest of this article is about.

The endpoint is https://www.baidu.com/s. These are the parameters worth knowing.

Build it with urlencode, and encode the query yourself so you control the byte encoding.

from urllib.parse import urlencode

def baidu_url(query, page=1, per_page=10, encoding="utf-8"):
    params = {
        "wd": query,
        "pn": (page - 1) * per_page,
        "rn": per_page,
        "ie": encoding,
        "tn": "baidu",
    }
    return "https://www.baidu.com/s?" + urlencode(params, encoding=encoding)

print(baidu_url("跨境电商 代理", page=3))
# https://www.baidu.com/s?wd=%E8%B7%A8%E5%A2%83%E7%94%B5%E5%95%86+%E4%BB%A3%E7%90%86&pn=20&rn=10&ie=utf-8&tn=baidu

The encoding= argument on urlencode is the part people miss. If you ever need the legacy path, switching it to gbk changes the percent-escapes entirely. The word 代理 is %E4%BB%A3%E7%90%86 in UTF-8 and %B4%FA%C0%ED in GBK. Send GBK bytes with ie=utf-8 and Baidu returns results for a nonsense query, usually as a plausible-looking page with zero relevance. That failure raises no error, which is why it deserves a unit test.

Read https://www.baidu.com/robots.txt yourself before you build anything. It is short, it names specific crawler user agents with specific rules, and the wildcard section at the bottom is restrictive. Treat any blog summary of it, including this one, as potentially stale, because it changes.

Practically that means three commitments. Collect only the public SERP, never anything behind a Baidu account login. Keep request rates low enough that your traffic resembles a handful of researchers rather than a crawler. Store no personal data: Zhidao answers and Tieba posts carry usernames, and a rank-tracking dataset has no business keeping them. Our guide on ethical scraping and rate limiting turns those principles into concrete concurrency numbers.

There is no general-purpose public Baidu web-search API for third parties in the way Bing once offered one, so HTML collection is the available route. This is engineering guidance, not legal advice. Chinese data and network regulation is its own specialism, and a commercial project running against a Chinese platform deserves counsel who works in that jurisdiction.

Baidu's web SERP renders server-side. The organic block is fully present in the initial HTML, so you do not need a browser for the core result list. Keep render_js=false, which is both faster and five times cheaper.

The simplest working call:

curl -G "https://scrape.sparkproxy.io/api/v1" \
  -H "X-API-Key: YOUR_API_KEY" \
  --data-urlencode "url=https://www.baidu.com/s?wd=%E4%BB%A3%E7%90%86%E6%9C%8D%E5%8A%A1%E5%99%A8&pn=0&rn=10&ie=utf-8" \
  --data-urlencode "render_js=false" \
  --data-urlencode "premium_proxy=true" \
  --data-urlencode "country_code=cn" \
  --output serp.html

In Python, with the pieces you actually want in production:

import requests

API = "https://scrape.sparkproxy.io/api/v1"
KEY = "YOUR_API_KEY"

def fetch_serp(query, page=1):
    target = baidu_url(query, page=page)
    r = requests.get(
        API,
        headers={"X-API-Key": KEY},
        params={
            "url": target,
            "render_js": "false",
            "premium_proxy": "true",
            "country_code": "cn",
            "block_resources": "true",
            "tag": f"baidu-serp-p{page}",
        },
        timeout=90,
    )
    r.raise_for_status()
    return r.content   # bytes, not text. This matters. See the encoding section.

Three deliberate choices there. render_js=false because the organic list is server-rendered. country_code=cn because Baidu's results are geo-sensitive. And r.content rather than r.text, because handing the raw bytes to your own decoder is the only way to be sure what you got. The timeout=90 is not paranoia either: cross-border routing into mainland China adds real latency, and a 10-second timeout that works fine against a US target will manufacture spurious failures here.

If you only need a few fields and want the API to do the extraction, extract_rules works against Baidu's markup:

rules = {
    "total_text": ".nums_text",
    "titles": {"selector": "div.result.c-container h3.t a", "type": "list"},
    "wrapped_links": {"selector": "div.result.c-container h3.t a", "type": "href"},
}

r = requests.post(
    API,
    headers={"X-API-Key": KEY, "Content-Type": "application/json"},
    json={
        "url": baidu_url("代理服务器"),
        "render_js": False,
        "premium_proxy": True,
        "country_code": "cn",
        "extract_rules": rules,
    },
    timeout=90,
)
print(r.json()["extracted"])

That gets you titles fast, but the hrefs it returns are still the wrapped redirects. Read on.

Baidu tailors results to the requesting IP. A query issued from Frankfurt and the same query issued from Guangzhou can return different result sets, different local blocks, and different first-party promotion. For rank tracking this is not a detail, it is the whole measurement. If you report a client's position from a European exit, you are reporting a position no Chinese user sees.

Set country_code=cn on every request and pair it with premium_proxy=true so the exit is a residential address rather than a datacenter range. If the reasoning is new to you, what geo-targeting means in proxies explains what the exit IP actually changes on the server side.

Two operational realities follow from the network path. Cross-border requests into and out of mainland China carry higher latency and more packet loss than a domestic route, so budget generous timeouts and expect a tail of connection resets that are transport failures rather than blocks. Network filtering along that path can also terminate connections in ways that look identical to a target-side block from your side of the wire.

Distinguish them by symptom. A genuine Baidu block returns a valid HTTP response containing a verification page. A transport failure gives you a reset, a TLS handshake error, or a timeout with no body at all. Retry the second class aggressively and the first class never. The patterns in retry and backoff strategies for web scraping are the right shape for this, provided you branch on that distinction first.

This is where most Baidu scrapers quietly corrupt their data.

https://www.baidu.com/s serves UTF-8 today and declares it in the Content-Type header. Other Baidu surfaces, older query forms, and anything you reach with ie=gbk can come back as GB2312 or GBK. The requests library guesses: if the header names a charset it uses that, otherwise it falls back to ISO-8859-1 for text/* responses, and you get mojibake with no exception raised.

Two rules fix all four rows.

Rule one: decode explicitly from the meta tag or the header, never from r.text.

Rule two: when the answer is GBK or GB2312, decode as gb18030. GB18030 is a strict superset of both, so it decodes everything they can plus the characters they cannot, including many rarer Han characters. Decoding GBK content as gb18030 never loses data. Decoding GB18030 content as gbk throws. There is no case where gbk is the better choice, so make gb18030 your default for the whole legacy family.

import re

META_CHARSET = re.compile(rb'charset\s*=\s*["\']?\s*([\w-]+)', re.I)

def decode_baidu(raw: bytes, header_charset: str | None = None) -> str:
    """Decode Baidu HTML bytes without guessing wrong."""
    declared = header_charset
    if not declared:
        m = META_CHARSET.search(raw[:4096])   # meta charset lives in <head>
        if m:
            declared = m.group(1).decode("ascii", "ignore")

    enc = (declared or "utf-8").lower()
    if enc in ("gbk", "gb2312", "gb-2312", "cp936", "ms936"):
        enc = "gb18030"          # superset: decodes everything gbk/gb2312 can, plus more
    elif enc in ("iso-8859-1", "latin-1", "latin1"):
        enc = "utf-8"            # the requests fallback is almost always wrong here

    try:
        return raw.decode(enc)
    except UnicodeDecodeError:
        return raw.decode("gb18030", errors="replace")


BAD_MARKERS = ("锟斤拷", "�", "代", "æµ")

def assert_clean(html: str, sample: int = 8000) -> None:
    """Fail loudly on corruption instead of letting it reach the warehouse."""
    hits = [m for m in BAD_MARKERS if m in html[:sample]]
    if hits:
        raise ValueError(f"Encoding corruption detected: {hits}")

Run assert_clean on every fetched page. It costs microseconds and catches an entire class of bug that is otherwise invisible until someone reads the dashboard. The same discipline belongs in post-processing, and cleaning scraped data covers the rest of the normalisation layer, including full-width to half-width punctuation, which Chinese text needs and English text does not.

Every organic anchor looks like this:

<h3 class="t"><a href="http://www.baidu.com/link?url=Xy7kQ...&amp;wd=&amp;eqid=..." target="_blank">
  <em>代理</em>服务器是什么
</a></h3>

The url token is not base64, not URL-encoded plaintext, and not decodable offline. People burn hours on this. It is an opaque server-side reference, and the only way to get the destination is to ask Baidu.

There are three routes, in order of cost.

Route one: read the mu attribute. Many result containers carry the destination as a plain attribute, either directly or inside a data-log JSON blob. Zero extra requests when it is present.

import json
from bs4 import BeautifulSoup

def destination_from_attrs(container) -> str | None:
    mu = container.get("mu")
    if mu:
        return mu
    raw = container.get("data-log")
    if raw:
        try:
            return json.loads(raw.replace("'", '"')).get("mu")
        except (ValueError, AttributeError):
            return None
    return None

Coverage is good but not universal. Baidu's own templates (Zhidao, Baike, Baijiahao) usually expose mu; several third-party organic templates omit it.

Route two: resolve the redirect with one HTTP request. Baidu does not always answer with a 302. A large share of link?url= responses are HTTP 200 pages carrying a JavaScript relocation or a meta refresh, which is why allow_redirects=True on its own leaves you holding a Baidu URL and no destination.

import re, requests

JS_REDIRECT = re.compile(r'''window\.location\.replace\(["'](.+?)["']\)''')
META_REFRESH = re.compile(r'''<meta[^>]+http-equiv=["']?refresh["']?[^>]+URL=['"]?([^'">]+)''', re.I)

def resolve_link(wrapped_url: str, session: requests.Session) -> str | None:
    r = session.get(wrapped_url, allow_redirects=False, timeout=30)

    if r.status_code in (301, 302, 303, 307, 308):
        return r.headers.get("Location")

    charset = r.headers.get("Content-Type", "")
    charset = charset.split("charset=")[-1] if "charset=" in charset else None
    body = decode_baidu(r.content, charset)

    m = JS_REDIRECT.search(body) or META_REFRESH.search(body)
    return m.group(1) if m else None

Route three: batch-resolve through the API. Resolving 10 links per SERP page individually is 10 billable calls. SparkProxy's batch mode takes comma-separated URLs with render_js=false and returns an array of results for a single credit, which is the right tool for redirect chasing because the destinations are ordinary sites, not protected endpoints.

def batch_resolve(wrapped_urls, api_key=KEY):
    r = requests.post(
        API,
        headers={"X-API-Key": api_key, "Content-Type": "application/json"},
        json={"url": ",".join(wrapped_urls), "render_js": False},
        timeout=120,
    )
    out = {}
    for src, res in zip(wrapped_urls, r.json()["results"]):
        if not res.get("success"):
            out[src] = None
            continue
        m = JS_REDIRECT.search(res["body"]) or META_REFRESH.search(res["body"])
        out[src] = m.group(1) if m else None
    return out

The practical pipeline is: try mu first, collect the misses, batch-resolve them in one call, and store both the wrapped URL and the resolved destination. Keep the wrapped one. When a resolution fails you want the original token so you can retry later rather than losing the row.

One more detail that saves a rewrite: resolve to the destination but do not follow it further. The first hop is the site the SERP actually pointed at. Following the site's own redirects (http to https, canonical host, tracking parameters) changes the URL you compare against your target list and creates phantom rank changes.

Do not parse Baidu by CSS class alone. Parse by the tpl attribute, which names the template that rendered each block. It is more stable across redesigns than class names, and it tells you exactly what kind of result you are looking at.

<div class="result c-container" tpl="se_com_default" mu="https://example.com/page" id="3">
<div class="result-op c-container" tpl="bk_polysemy" mu="https://baike.baidu.com/item/...">

Note the class difference too. result c-container is a standard organic listing; result-op c-container marks a Baidu "operation" block, meaning a first-party or specially formatted unit.

from bs4 import BeautifulSoup

FIRST_PARTY_HOSTS = (
    "baijiahao.baidu.com", "zhidao.baidu.com", "baike.baidu.com",
    "tieba.baidu.com", "wenku.baidu.com", "jingyan.baidu.com",
    "haokan.baidu.com", "mbd.baidu.com",
)

def parse_serp(html: str, page: int = 1, per_page: int = 10):
    soup = BeautifulSoup(html, "lxml")
    rows = []
    slot = (page - 1) * per_page

    for node in soup.select("div.c-container[tpl]"):
        slot += 1
        title_el = node.select_one("h3 a") or node.select_one("a")
        if not title_el:
            continue

        dest = destination_from_attrs(node)
        host = dest.split("/")[2] if dest and "//" in dest else None

        rows.append({
            "slot": slot,                                   # position in the visual list
            "tpl": node.get("tpl", ""),
            "is_operation": "result-op" in node.get("class", []),
            "is_first_party": bool(host and host.endswith(FIRST_PARTY_HOSTS)),
            "title": title_el.get_text(strip=True),
            "wrapped_url": title_el.get("href"),
            "destination": dest,
            "matched_terms": [em.get_text() for em in title_el.select("em")],
            "abstract": (node.select_one(".c-abstract").get_text(" ", strip=True)
                         if node.select_one(".c-abstract") else None),
        })
    return rows

The field that earns its place is slot paired with is_first_party. Report both a visual position (what a user sees, first-party blocks included) and a third-party organic position (first-party blocks removed). They diverge, sometimes by three or four places on commercial queries where Baijiahao and Zhidao units stack near the top. A rank report quoting only one of those numbers is answering a question nobody asked. Google's SERP has a version of the same problem with its own features, and the counting discipline matches what we describe for SERP scraping proxies, but the first-party density on Baidu makes it far more consequential.

Total result count sits in .nums_text:

import re

def total_results(soup) -> int | None:
    el = soup.select_one(".nums_text, .nums")
    if not el:
        return None
    digits = re.sub(r"[^\d]", "", el.get_text())
    return int(digits) if digits else None

Treat it as an order-of-magnitude estimate. It moves between requests for the same query and should never be reported as a precise figure.

Paid units are labelled with the character pair 推广 ("promotion") or 广告 ("advertisement") in the result footer. Check for those strings before classifying anything as organic. Template names drift over time, so build the classifier as a mapping table you can update in one place, not as conditionals sprinkled through the parser.

The related-searches block at the foot of page one is worth harvesting on its own. It is Baidu telling you which adjacent queries it associates with yours, which is the cheapest keyword-expansion source available for the Chinese market.

Chinese is written without spaces between words. That single fact breaks the matching logic every Western SERP pipeline relies on.

If your target keyword is 跨境电商代理 and a title reads 跨境电商的代理服务, a Python in test returns False even though the result is a strong match. Substring matching also produces false positives across word boundaries, because a character sequence can span two unrelated words. Neither error is rare.

The usual answer is to segment both strings with a tokeniser such as jieba and compare token sets. That works, and it is the right tool when you need to segment arbitrary Chinese text.

For SERP matching specifically there is a better source of truth: Baidu already tells you how it segmented your query. Matched terms come back wrapped in tags inside titles and abstracts. Those tags are the engine's own tokenisation of your query against that document, which is exactly the ground truth your relevance scoring wants, and extracting it costs nothing.

import jieba

def tokenise(query: str) -> set[str]:
    """Segment the keyword once per query, not once per result."""
    return {t for t in jieba.cut_for_search(query) if len(t.strip()) > 1}

def match_strength(row, query_tokens: set[str]) -> float:
    """Score a result using Baidu's own <em> highlighting, not our guess at segmentation."""
    highlighted = {t.strip() for t in row["matched_terms"] if t.strip()}
    if not highlighted:
        return 0.0
    covered = sum(1 for tok in query_tokens if any(tok in h or h in tok for h in highlighted))
    return covered / max(len(query_tokens), 1)