Web Scraping with Rust and Proxies

Rust is not the obvious pick for scraping. Python has Scrapy, a decade of tutorials, and BeautifulSoup. So why reach for Rust? Three reasons show up in real crawls.

• Throughput per core. tokio schedules thousands of in-flight requests on a handful of OS threads. No GIL, no per-request thread overhead. A single small VM can saturate a proxy pool that would need several Python workers.
• Memory that stays flat. No garbage collector means no pause spikes and predictable memory during a multi-hour crawl. Each idle connection costs bytes, not kilobytes.
• Errors you can't ignore. Every network call returns a Result. The compiler forces you to decide what happens on a timeout, a 429, or a dropped proxy before the code runs, which is exactly the failure mode that breaks long-running scrapers.

The tradeoffs are real too. Compile times are slow, the learning curve is steep if you're new to ownership, and the scraping ecosystem is younger than Python's. There is no Scrapy-grade framework yet. For a quick one-off scrape, Python is faster to ship. For a service that scrapes millions of pages a day and has to stay up, Rust earns its keep. If you're new to the practice itself, start with what web scraping is and come back.

Here are the crates that do the work:

Start a binary and add the dependencies. These versions are current as of July 2026:

# Cargo.toml
[dependencies]
reqwest = { version = "0.12", features = ["json"] }
scraper = "0.20"
tokio = { version = "1", features = ["full"] }
futures = "0.3"
serde_json = "1"

The smallest possible fetch confirms the toolchain works. reqwest's free function builds a throwaway client for you:

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let body = reqwest::get("https://www.sparkproxy.io/")
        .await?
        .text()
        .await?;

    println!("{} bytes fetched", body.len());
    Ok(())
}

That's fine for a hello-world. For anything real you want an explicit Client, because a Client owns a connection pool and reusing it is what makes Rust fast. More on that in the concurrency section.

The reqwest proxy lives on the Client, not the request. You attach a Proxy at build time and every request from that client goes through it. Note the proxy URL uses the http:// scheme even when your target is HTTPS, because reqwest opens an HTTP CONNECT tunnel to the proxy and the tunnel carries the encrypted traffic.

use reqwest::{Client, Proxy};

fn build_client() -> reqwest::Result<Client> {
    let proxy = Proxy::all("http://gate.sparkproxy.io:10000")?
        .basic_auth("your-username", "your-password");

    Client::builder()
        .proxy(proxy)
        .timeout(std::time::Duration::from_secs(20))
        .build()
}

Proxy::all routes every scheme through the proxy. If you need finer control, reqwest gives you three constructors plus an exclusion list:

You can pass credentials two ways. The .basic_auth() builder above is the clean one. You can also embed them in the URL, which is handy when you read proxies from a config file or environment variable:

// Credentials inline in the proxy URL
let proxy = Proxy::all("http://your-username:your-password@gate.sparkproxy.io:10000")?;
let client = Client::builder().proxy(proxy).build()?;

Verify the proxy actually took effect before you trust it. Hit an IP echo endpoint and confirm the address is the proxy's, not your server's:

let ip = client
    .get("https://api.ipify.org")
    .send()
    .await?
    .text()
    .await?;

println!("egress IP: {ip}");

If that prints your real server IP, the proxy silently failed. The usual cause is building the request on a different client than the one you attached the proxy to.

The scraper crate gives you CSS selectors over parsed HTML, close to what you'd write in the browser console. Parse the document once, compile a Selector, then iterate matches.

use scraper::{Html, Selector};

fn extract_titles(body: &str) -> Vec<(String, String)> {
    let document = Html::parse_document(body);
    let selector = Selector::parse("article h2.title a").unwrap();

    document
        .select(&selector)
        .filter_map(|el| {
            let text = el.text().collect::<String>().trim().to_string();
            let href = el.value().attr("href")?.to_string();
            Some((text, href))
        })
        .collect()
}

A few details that trip people up:

• Selector::parse returns a Result because an invalid CSS selector is a runtime error. In production, handle it rather than .unwrap().
• el.text() yields every text node under the element, so .collect::() concatenates them. Add .trim() because HTML whitespace is messy.
• el.value().attr("href") returns an Option<&str>. That &str borrows the document, so call .to_string() if you need to keep it after the document is dropped.

That last point about borrowing is not a style note. It's the seam where async scraping in Rust goes wrong.

Here's the gotcha the intro promised, and it's the single biggest reason people bounce off scraping in Rust. The scraper crate's Html and Selector types are !Send. Internally they use non-atomic reference counting, so the compiler refuses to move them across threads. tokio's multi-threaded runtime can move a task between threads at any .await, and tokio::spawn requires the whole future to be Send. Put those together and this innocent-looking code will not compile:

// DOES NOT COMPILE
tokio::spawn(async move {
    let doc = Html::parse_document(&body);
    let sel = Selector::parse("a.item").unwrap();
    for el in doc.select(&sel) {
        save_to_db(el.inner_html()).await; // await while doc is still alive
    }
});

The error is future cannot be sent between threads safely, and it points at an Rc buried inside Html. The problem is that doc and sel are still alive at the .await, so the compiler has to assume they might cross a thread boundary. They can't.

The fix is a scope. Extract everything you need into owned, Send types like String and Vec, and let the !Send values drop before you await anything:

tokio::spawn(async move {
    // Parse in an inner scope that owns no future
    let rows: Vec<String> = {
        let doc = Html::parse_document(&body);
        let sel = Selector::parse("a.item").unwrap();
        doc.select(&sel)
            .map(|el| el.inner_html())
            .collect()
    }; // doc and sel are dropped HERE, before any .await

    for html in rows {
        save_to_db(html).await; // now Send: only String crosses the boundary
    }
});

If parsing is heavy, push it to a blocking thread instead. spawn_blocking runs the closure on tokio's blocking pool and hands back an owned result, which sidesteps the Send requirement entirely and keeps HTML parsing off the async worker threads:

let rows: Vec<String> = tokio::task::spawn_blocking(move || {
    let doc = Html::parse_document(&body);
    let sel = Selector::parse("a.item").unwrap();
    doc.select(&sel).map(|el| el.inner_html()).collect()
})
.await?;

Remember the shape of this pattern: parse synchronously, extract owned data, then go async. Most Rust scraping guides never mention it, and it's the first wall almost everyone hits.

Fetching URLs one at a time wastes the whole point of Rust. The right pattern is bounded concurrency: fire off many requests at once, but cap how many are in flight so you don't hammer the target or exhaust the proxy pool.

First, build the Client once and clone it per task. Cloning a reqwest::Client is cheap because it's an Arc around a shared connection pool. Building a new client per request throws away connection reuse and is a common performance bug.

use futures::{stream, StreamExt};
use reqwest::Client;

async fn scrape_all(client: &Client, urls: Vec<String>) -> Vec<(String, String)> {
    stream::iter(urls)
        .map(|url| {
            let client = client.clone(); // cheap Arc clone, shares the pool
            async move {
                let body = client
                    .get(&url)
                    .send()
                    .await
                    .ok()?
                    .text()
                    .await
                    .ok()?;
                Some((url, body))
            }
        })
        .buffer_unordered(20) // at most 20 requests in flight
        .filter_map(|res| async move { res })
        .collect()
        .await
}

buffer_unordered(20) is the knob that matters. It keeps exactly twenty requests running and starts a new one each time one finishes, so you get steady throughput without a thundering-herd burst. Set it to match your proxy count and the target's tolerance. Twenty concurrent requests across twenty proxies is one request per IP at a time, which most sites accept without complaint.

If you need per-task results with error handling instead of silently dropping failures, tokio::task::JoinSet collects handles you can await individually.

Here's a reqwest-specific fact that shapes your whole rotation design: the proxy is bound to the Client at build time, and there is no per-request proxy override. You cannot swap the proxy on an existing client. So rotation in Rust takes one of two forms.

Option one: a rotating gateway. Point every request at a single endpoint that rotates the upstream IP for you on the provider side. Your Rust code stays simple because there's only one client and one proxy URL. This is the least code and the approach most managed pools support.

Option two: a pool of clients. Build one Client per proxy and round-robin between them. You need this when you want direct control over which IP handles which request, or when you hold sticky sessions per IP.

use std::sync::atomic::{AtomicUsize, Ordering};
use std::time::Duration;
use reqwest::{Client, Proxy};

struct ProxyPool {
    clients: Vec<Client>,
    cursor: AtomicUsize,
}

impl ProxyPool {
    fn new(proxy_urls: &[&str]) -> reqwest::Result<Self> {
        let clients = proxy_urls
            .iter()
            .map(|url| {
                Client::builder()
                    .proxy(Proxy::all(*url)?)
                    .timeout(Duration::from_secs(20))
                    .build()
            })
            .collect::<reqwest::Result<Vec<_>>>()?;

        Ok(Self { clients, cursor: AtomicUsize::new(0) })
    }

    // Round-robin, thread-safe, no lock needed
    fn next(&self) -> &Client {
        let i = self.cursor.fetch_add(1, Ordering::Relaxed) % self.clients.len();
        &self.clients[i]
    }
}

The AtomicUsize cursor means next() is safe to call from every concurrent task without a Mutex. Each client keeps its own connection pool warm to its own proxy, which is a real speed win over rebuilding clients. The one cost is memory: a hundred proxies means a hundred connection pools. For most jobs that's fine. Beyond a few hundred proxies, a single rotating gateway scales better. If you want the theory behind pool sizing and per-IP limits, the guide on using datacenter proxies for web scraping covers the math.

Proxies fail. IPs get rate-limited. A scraper that holds up retries on a fresh proxy with growing delays rather than pounding a dead IP. error_for_status() turns any 4xx or 5xx into an Err so a single match handles both network errors and bad status codes.

use tokio::time::{sleep, Duration};

async fn fetch_with_retry(
    pool: &ProxyPool,
    url: &str,
    max_attempts: u32,
) -> reqwest::Result<String> {
    let mut attempt = 0;

    loop {
        let client = pool.next(); // new proxy each attempt
        let result = client
            .get(url)
            .send()
            .await
            .and_then(|resp| resp.error_for_status());

        match result {
            Ok(resp) => return resp.text().await,
            Err(err) if attempt + 1 < max_attempts => {
                attempt += 1;
                // 400ms, 800ms, 1.6s, ... capped at 10s
                let backoff = Duration::from_millis(400 * 2u64.pow(attempt))
                    .min(Duration::from_secs(10));
                sleep(backoff).await;
            }
            Err(err) => return Err(err),
        }
    }
}

Two things make this hold up under load. Calling pool.next() inside the loop means every retry lands on a different IP, so a rate-limited proxy never gets hit twice in a row. The exponential backoff, capped at ten seconds, keeps you from retrying a struggling site into the ground. If you keep seeing blocks even with rotation, the problem is usually not the retry logic. Read how to avoid getting your proxy blocked for the header and fingerprint fixes.

You've done everything right, rotated clean IPs, added realistic headers, and a site still returns 403 on the first request. The IP isn't the problem. Your TLS handshake is.

Every HTTPS client has a TLS fingerprint, often summarized as a JA3 or JA4 hash, built from the cipher suites and extensions it advertises in the ClientHello. reqwest's fingerprint does not look like Chrome's. Its HTTP/2 settings frame doesn't either. Anti-bot systems like Cloudflare and DataDome hash these and block clients that claim to be a browser in the User-Agent but handshake like a library. This is why the exact same request can succeed with curl and fail with reqwest, or vice versa: they present different fingerprints.

You have three ways out:

1. Impersonate a browser fingerprint. The rquest crate is a reqwest fork that mimics Chrome and Firefox TLS plus HTTP/2 signatures. The API is nearly identical, so migration is mostly a name swap. This works until the target updates its detection.
2. Drive a real browser. thirtyfour controls chromedriver, giving you a genuine browser fingerprint at the cost of far more CPU and RAM per page.
3. Offload it to a scraping API. Hand the URL to a service that already solves TLS fingerprinting, JavaScript rendering, and rotation, then get clean HTML back. Your Rust code stays a plain reqwest call with no fingerprint to manage.

The third option is the least Rust code and the most reliable against hard targets. If you're weighing the build-versus-buy decision, the breakdown in web scraping API vs self-managed proxies lays out the cost tradeoffs.

The SparkProxy Scraping API handles proxy rotation, browser rendering, and the TLS fingerprint problem on the server side. From Rust it's an ordinary reqwest GET. You send the target URL and options as query parameters and authenticate with the X-API-Key header. The base endpoint is https://scrape.sparkproxy.io/api/v1.

use reqwest::Client;

async fn scrape_via_api(
    client: &Client,
    api_key: &str,
    target: &str,
) -> reqwest::Result<String> {
    client
        .get("https://scrape.sparkproxy.io/api/v1")
        .header("X-API-Key", api_key)
        .query(&[
            ("url", target),
            ("render_js", "true"),     // headless Chromium renders the page
            ("premium_proxy", "true"), // route through residential IPs
            ("country_code", "us"),    // geo-target the request
        ])
        .send()
        .await?
        .text()
        .await
}

The parameters map to the controls you'd otherwise implement yourself:

Set json_response=true when you want status, timing, and credit data alongside the HTML. Parse it with serde_json:

use serde_json::Value;

let envelope: Value = client
    .get("https://scrape.sparkproxy.io/api/v1")
    .header("X-API-Key", api_key)
    .query(&[
        ("url", "https://www.sparkproxy.io/"),
        ("render_js", "true"),
        ("json_response", "true"),
    ])
    .send()
    .await?
    .json()
    .await?;

let html = envelope["body"].as_str().unwrap_or_default();
let status = envelope["status_code"].as_u64().unwrap_or(0);
let credits = envelope["credits_used"].as_u64().unwrap_or(0);

println!("status {status}, {credits} credits, {} bytes", html.len());

You feed html straight into the scraper crate, keeping the same Html::parse_document and Selector extraction from earlier. The API replaces the proxy pool, retry loop, and fingerprint workarounds, so the Rust that's left is fetch and parse.