How to Scrape Yandex Search Results in 2026

Five differences change how you write the scraper.

One engine, several indexes. yandex.ru and yandex.com do not return the same ten results for one query: different ranking models, different language priors, different SERP furniture. Turkey has its own front door, yandex.com.tr.

Region is an explicit URL parameter. Google infers locality from your IP plus a gl hint. Yandex takes an integer, lr, from its own geo tree: two requests from one IP with lr=213 and lr=2 return visibly different SERPs.

Pagination is zero-indexed. p=0 is the first page where Google uses start=0. Everyone arriving from Google gets this wrong once, silently scraping page two as page one.

Query rewriting is aggressive. Yandex lemmatises Russian, repairs keyboard-layout mistakes, and transliterates Latin-typed Russian into Cyrillic, so your scraper can record a query you never sent.

The captcha is Yandex's own. SmartCaptcha is a Yandex Cloud product unrelated to reCAPTCHA or hCaptcha, and its challenge page returns HTTP 200, so a naive status check reports success.

If you have read our guide to scraping Google search results, treat this as the companion, not a reskin: the pacing instincts transfer, almost nothing else does.

Decide the index first: it sets the language of the SERP furniture you parse, and how hard the target defends itself.

The path is identical everywhere, so a minimum viable SERP URL is https://yandex.com/search/?text=proxy%20server&lr=225&p=0. Country front doors are not separate indexes; they are the core index with a regional default, which is why setting lr explicitly beats picking a hostname and hoping.

lr is a numeric ID from Yandex's geo tree. It changes organic ranking, which local businesses appear, which ads are eligible, and the currency inside wizard blocks. No other parameter moves the results as much.

Treat that table as a starting point, not gospel: codes for smaller cities move when Yandex reorganises the tree. Resolve them with the Direct API method Dictionaries.get and DictionaryNames: ["GeoRegions"], which returns GeoRegionId, GeoRegionName, GeoRegionType and ParentId for the same tree lr comes from. Cache it, refresh quarterly, key job configs on region names rather than raw integers.

Two behaviours catch people out. Region autodetection still runs: pass lr=213 from a German exit and Yandex sees a Moscow hint from Frankfurt. Ranking usually honours lr, but the mismatch is a bot signal and some local blocks fall back to the IP-derived region, so match exit country to region. And rstr is the negative form: &rstr=-213 returns everything except Moscow.

Geo accuracy is the whole game for rank tracking, and our explainer on geo-targeting in proxies covers the mechanics.

The query goes in text, percent-encoded as UTF-8, so купить прокси becomes %D0%BA%D1%83%D0%BF%D0%B8%D1%82%D1%8C%20%D0%BF%D1%80%D0%BE%D0%BA%D1%81%D0%B8. Let the HTTP library encode it and never concatenate an already-encoded string, the usual source of double-encoded %25D0 garbage.

from urllib.parse import urlencode

def yandex_url(query, lr=213, page=0, host="yandex.ru", noreask=True):
    params = {
        "text": query,
        "lr": lr,
        "p": page,          # zero-indexed: p=0 is the first page
        "lang": "ru",
    }
    if noreask:
        params["noreask"] = 1   # disable "did you mean" rewriting
    return f"https://{host}/search/?{urlencode(params)}"

print(yandex_url("купить прокси", lr=213, page=0))
# https://yandex.ru/search/?text=%D0%BA%D1%83%D0%BF%D0%B8%D1%82%D1%8C+...&lr=213&p=0&lang=ru&noreask=1

noreask=1 is the parameter most tutorials omit and the one that most affects data quality. Without it Yandex substitutes what it thinks you meant, and three rewrites fire regularly:

• Morphological expansion. Russian is heavily inflected and Yandex matches lemmas, so купить прокси and куплю прокси collapse toward one result set.
• Keyboard-layout repair. A query typed on the wrong layout, ghjrcb, becomes прокси. Helpful for humans, poison for a keyword tracker.
• Transliteration. Latin-typed Russian like kupit proksi maps back to Cyrillic, so a transliterated keyword list measures a query you never stored.

With noreask=1 the suggestion banner still appears, but the results belong to the query you sent. Where you cannot use it, read the misspell indicator and flag those rows.

Yandex also ships a richer operator language than Google, working in HTML search and the XML API alike.

The ! operator is the practical one: if a tracked brand name collides with a common Russian noun, !бренд stops Yandex expanding into every declension and gives you a stable time series.

The SparkProxy Scraping API handles the proxy pool, browser, and fingerprint, so your code makes one HTTP call. Base URL is https://scrape.sparkproxy.io/api/v1 with an X-API-Key header, and every parameter is in the Scraping API docs. Start with a residential fetch of a Moscow SERP:

curl -G "https://scrape.sparkproxy.io/api/v1" \
  -H "X-API-Key: sk-your-key-here" \
  --data-urlencode "url=https://yandex.ru/search/?text=купить+прокси&lr=213&p=0&noreask=1" \
  --data-urlencode "render_js=true" \
  --data-urlencode "premium_proxy=true" \
  --data-urlencode "country_code=RU" \
  --data-urlencode "stealth=true"

--data-urlencode matters: it encodes the Cyrillic and the nested ? and & of the target URL so the API receives one intact parameter. Hand-building that string is the most common reason a first Yandex call returns the homepage instead of a SERP.

The production version is a POST with a wait condition, so you never capture the page before the organic list paints:

import requests

API = "https://scrape.sparkproxy.io/api/v1"
KEY = "sk-your-key-here"

def fetch_yandex(query, lr=213, page=0, host="yandex.ru", country="RU"):
    target = yandex_url(query, lr=lr, page=page, host=host)
    r = requests.post(
        API,
        headers={"X-API-Key": KEY, "Content-Type": "application/json"},
        json={
            "url": target,
            "render_js": True,
            "premium_proxy": True,     # residential exit, not a datacenter ASN
            "country_code": country,   # keep the exit country aligned with lr
            "stealth": True,           # extra anti-bot layers, requires render_js
            "wait_for": "#search-result, .serp-list",
            "device": "desktop",
            "format": "html",
            "tag": f"yandex:{lr}:{page}",
        },
        timeout=120,
    )
    r.raise_for_status()
    return r.text

Each parameter answers a specific Yandex defence, and each costs credits.

Credit costs stack: a residential request with JavaScript is 25 credits, with country_code and stealth adding 5 each. Where the SERP parses without a browser, drop render_js to false and pay 10. Test both on your own keywords; the answer differs by region and query type.

The Turkish index, from Node:

const axios = require('axios');

const target = 'https://yandex.com.tr/search/?text=' +
  encodeURIComponent('vekil sunucu') + '&lr=11508&p=0';

axios.get('https://scrape.sparkproxy.io/api/v1', {
  headers: { 'X-API-Key': 'sk-your-key-here' },
  params: {
    url: target,
    render_js: 'true',
    premium_proxy: 'true',
    country_code: 'TR',
    stealth: 'true'
  }
})
.then(res => console.log(res.data.length, 'bytes'))
.catch(err => console.error(err.response?.status, err.response?.data));

Pagination is a loop over p, and this is where the zero-index bites:

import random, time

def collect(query, lr=213, pages=3, host="yandex.ru"):
    rows = []
    for p in range(pages):              # p = 0, 1, 2 -> pages 1, 2, 3
        html = fetch_yandex(query, lr=lr, page=p, host=host)
        batch = parse_organic(html)
        if not batch:
            break                       # empty page means blocked or exhausted
        for r in batch:
            r["position"] += p * 10     # absolute position across pages
        rows.extend(batch)
        time.sleep(random.uniform(4, 12))   # jitter, never a constant sleep
    return rows

Here is a genuine advantage over Google. Yandex invented BEM and its SERP markup still follows it, so class names read like serp-item, organic__url-text, OrganicTitle-LinkText. Semantic, and they rotate far less often than Google's obfuscated hashes. They do rotate, so anchor on structure and attributes first, classes second.

from bs4 import BeautifulSoup

def parse_organic(html):
    soup = BeautifulSoup(html, "lxml")
    rows = []
    for i, item in enumerate(soup.select("li.serp-item"), start=1):
        # Direct ads carry a "Реклама" label; skip them for organic-only tracking
        if "Реклама" in item.get_text():
            continue
        link = item.select_one("a.OrganicTitle-Link, a.organic__url, a.Link")
        title = item.select_one(".OrganicTitle-LinkText, .organic__url-text")
        if not link or not link.get("href"):
            continue
        rows.append({
            "position": i,
            "url": link["href"],
            "title": title.get_text(strip=True) if title else "",
            "wizard": item.get("data-fast-name"),   # present on wizard blocks
            "snippet": (item.select_one(".OrganicTextContentSpan,"
                                        " .organic__content-wrapper")
                        or item).get_text(" ", strip=True)[:300],
        })
    return rows

The data-fast-name attribute is the useful hook: set on a serp-item, it names the wizard type, classifying a block without pattern-matching visible text in three languages.

Yandex calls its feature blocks "колдунщики", wizards. They are not the set Google ships, so a Google parser mislabels most of them.

Turbo pages deserve a warning. A Turbo result points at a Yandex-hosted copy rather than the publisher's URL, so a naive tracker records yandex.ru/turbo?text=... as the ranking domain and the client's real domain looks like it vanished. The destination sits in that URL's query string, so unwrap it before storing the row. Nobody porting a Google parser catches this first time.

To skip parsing, extract_rules returns structured JSON instead of HTML:

payload = {
    "url": yandex_url("прокси для парсинга", lr=213),
    "render_js": True,
    "premium_proxy": True,
    "country_code": "RU",
    "stealth": True,
    "extract_rules": {
        "titles": {"selector": "li.serp-item .OrganicTitle-LinkText", "type": "list"},
        "links": {"selector": "li.serp-item a.OrganicTitle-Link", "type": "list",
                  "output": "@href"},
    },
}

Keep selectors in config, not code. Yandex ships markup changes without notice, and editing a JSON file beats cutting a release.

SmartCaptcha launched as a Yandex Cloud service in 2022 and also guards Yandex's own properties. It is a checkbox-first challenge that escalates to image and slider tasks on a behavioural score. Not reCAPTCHA, not hCaptcha, and the token flow differs, so solvers built for Google's challenge typically fail against it.

The detection rule that matters: the challenge returns HTTP 200, so check the body and the final URL, never the status code.

class BlockedError(Exception):
    pass

CAPTCHA_MARKERS = (
    "/showcaptcha",          # the challenge path Yandex redirects to
    "SmartCaptcha",
    "captcha__image",
    "Подтвердите, что запросы отправляли вы",   # "confirm you sent these requests"
)

def is_blocked(html: str) -> bool:
    return any(m in html for m in CAPTCHA_MARKERS)

def fetch_checked(query, **kw):
    html = fetch_yandex(query, **kw)
    if is_blocked(html):
        raise BlockedError("SmartCaptcha served")
    if "serp-item" not in html:
        raise BlockedError("no results in response")   # soft block or empty shell
    return html

The /showcaptcha path is the strongest single signal. Yandex redirects to https://yandex.ru/showcaptcha?cc=1&retpath=..., and retpath holds your original query, so log which keyword tripped it.

When it fires:

1. Stop that worker. Retrying the same query on the same session returns the challenge again and burns credits.
2. Rotate the exit, then back off with jitter. Take a fresh residential IP and requeue the keyword at roughly 60 seconds, then 300, then 900, each delay multiplied by a random 0.5 to 1.5 so parallel workers do not retry in lockstep. Park it after the third failure instead of grinding. The general shape is in our post on retry and backoff strategies.
3. Treat a captcha rate above roughly 5 percent as a configuration problem, not a traffic problem. Fingerprint, pacing, or exit geography is wrong, and concurrency makes it worse.
4. Fall back to the XML API for keywords that keep failing. It is sanctioned and never serves a captcha.

Solving SmartCaptcha at scale is expensive and brittle. Better pacing plus the official API for the hard tail costs less.

Unlike Google, Yandex publishes a first-party interface returning real SERP documents, not a cut-down custom-search subset. If the quota fits your volume, reach for it before writing a selector.

Two generations exist. The classic XML endpoint is https://yandex.com/search/xml (or the .ru equivalent), authenticated with a Yandex Cloud folderid plus an apikey. Search API v2 lives at https://searchapi.api.cloud.yandex.net/v2/web/search, speaks REST and gRPC, and adds a deferred mode that buys a larger hourly allowance in exchange for waiting. Both share one documentation set.

curl -G "https://yandex.com/search/xml" \
  --data-urlencode "folderid=b1gxxxxxxxxxxxxxxxxx" \
  --data-urlencode "apikey=AQVNxxxxxxxxxxxxxxxxxxxx" \
  --data-urlencode "query=купить прокси" \
  --data-urlencode "lr=213" \
  --data-urlencode "l10n=ru" \
  --data-urlencode "sortby=rlv" \
  --data-urlencode "filter=none" \
  --data-urlencode "groupby=attr=d.mode=deep.groups-on-page=10.docs-in-group=1" \
  --data-urlencode "page=0"

In groupby, groups-on-page sets results per page and docs-in-group pages per domain. Parsing is plain XML, and the error element comes first.

import requests, xml.etree.ElementTree as ET

def yandex_xml(query, folder_id, api_key, lr=213, page=0):
    r = requests.get("https://yandex.com/search/xml", params={
        "folderid": folder_id, "apikey": api_key,
        "query": query, "lr": lr, "l10n": "ru",
        "sortby": "rlv", "filter": "none", "page": page,
        "groupby": "attr=d.mode=deep.groups-on-page=10.docs-in-group=1",
    }, timeout=60)
    root = ET.fromstring(r.content)

    err = root.find(".//error")
    if err is not None:
        raise RuntimeError(f"Yandex XML error {err.get('code')}: {err.text}")

    out = []
    for doc in root.findall(".//doc"):
        title_el = doc.find("title")
        passage = doc.find(".//passage")
        out.append({
            "url": doc.findtext("url"),
            "domain": doc.findtext("domain"),
            "title": "".join(title_el.itertext()) if title_el is not None else "",
            "snippet": "".join(passage.itertext()) if passage is not None else "",
        })
    return out

Note the itertext() calls. Yandex wraps matched terms in tags inside title and passage, so .text truncates at the first highlighted word. That is behind most "why are my Yandex titles cut off" bug reports.

The documented error codes you will meet:

Two quota behaviours decide the fit.

The quota unit is the hour. Published quotas and limits run to 10,000 synchronous requests per hour at 10 per second, or 35,000 per hour deferred, where a request takes at least five minutes and the result is held for 12 hours. The legacy XML console said the same thing differently: a daily allowance spread across 24 hourly slots. Either way a burst at 09:00 exhausts that hour, and you get a hard error rather than gentle throttling. Size workers to the hourly slice, not the daily total.

Depth is capped too. The same table caps results returned at 250 per request, and HTML pagination dries up around 1,000 documents, so deep-tail rank research is unavailable by either route.

Error 33 exists because the interface is IP-bound, the constraint that surprises proxy users most. You register the addresses allowed to call it, so XML traffic through a rotating residential pool is rejected by design. Run XML calls from a few fixed egress addresses, keep the proxy pool for HTML, and never mix both in one worker. That mix is the most common architecture mistake in Yandex pipelines.

The pragmatic architecture uses both: XML for organic ranking series, HTML for the ad blocks, Market carousels, and local packs the API omits. Our SERP scraping proxy explainer covers that split in the general case.

Be honest about this part: Yandex is harder on datacenter ranges than Google. It runs its own cloud business, classifies ASNs in detail, and its Antirobot system treats cloud egress as suspicious. In practical testing, plain datacenter IPs hitting yandex.ru reach /showcaptcha within a few dozen queries, sometimes on the first request from a subnet with history, while residential exits inside the target country sustain roughly 100 to 200 queries per IP per day at human-like pacing. Those are practitioner ranges rather than a published study, and they move with subnet reputation.

The rules that follow:

• Residential exits in-country for yandex.ru. A Russian query answered from a Moscow residential IP is unremarkable. From a Virginia cloud range it is not. The trade-offs sit in residential vs datacenter proxies.
• Jitter, never a constant sleep. Uniform 5-second gaps are a stronger bot tell than a higher raw rate. Randomise between 4 and 12 seconds.
• Keep sessions short. Yandex sets a yandexuid identifier and a yp preferences cookie carrying region state, so long-lived cookie jars accumulate a behavioural profile. Rotate identity with the IP.
• Do not parallelise into a captcha. When block rate climbs, concurrency makes it climb faster. Cut workers first, then investigate.
• Cache hard. Most rank-tracking questions need daily granularity.

The .tr and .com properties are noticeably more tolerant than .ru, so if you only need the international index, target yandex.com and save residential budget for the queries that need a Russian exit. The wider detection surface is covered in how to avoid getting your proxy blocked.

Yandex's terms restrict automated access outside its own interfaces, which puts HTML SERP scraping in the same contractual grey zone as Google. Scraping public results is not a criminal act in most jurisdictions, but it can breach the contract, and a first-party API weakens any argument that no alternative existed. Use the Search API where it fits, keep volumes proportional, and honour back-off signals.

One factual note on the corporate situation, because it affects whether you can open an account at all. Yandex N.V. completed the sale of its Russian businesses on 15 July 2024 and renamed itself Nebius Group, so the Russian operations, Yandex Cloud in Russia included, sit under separate ownership. Several Russian entities and payment channels fall under EU, UK, and US sanctions programmes, so buying Yandex Cloud quota can be restricted depending on where you are incorporated and which entity you contract with. Confirm current designations with your compliance team.

Collect only public SERP data, never logged-in Yandex surfaces, personal accounts, or Yandex Mail. For your own site's rankings, Yandex Webmaster hands you the numbers with zero terms-of-service exposure.