C# Web Scraping Proxy: HttpClient, Rotation, Parsing

.NET is a strong scraping platform. You get real threads, async/await that scales to thousands of concurrent requests without a thread per socket, and two mature HTML parsers in HtmlAgilityPack and AngleSharp. What .NET cannot do is hide that every request leaves from one IP. Send a few hundred requests at one host from a single address and the site rate-limits you, serves a CAPTCHA, or blocks the IP. A proxy pool spreads that traffic across many addresses so no single one crosses the target's detection threshold. If you want the ground-level concept first, see what web scraping is.

You have a few ways to attach proxies to a .NET scraper, and they suit different jobs:

Most real projects mix them. Datacenter proxies handle high-volume fetching of simple pages at the best cost per request, which is why they are the workhorse for web scraping with datacenter proxies, and a managed API takes the handful of targets that are not worth the fight. Deciding where that line sits is the whole scraping API versus self-managed proxies question. The sections below cover each layer.

You route HttpClient through a proxy by setting a WebProxy on the handler that backs the client, not on the client itself. On modern .NET (Core 2.1 through .NET 10), HttpClientHandler is a thin wrapper over SocketsHttpHandler, and either one accepts a proxy.

using System.Net;
using System.Net.Http;

var handler = new HttpClientHandler
{
    Proxy = new WebProxy("http://proxy-1.sparkproxy.io:10000"),
    UseProxy = true
};

using var client = new HttpClient(handler);
var html = await client.GetStringAsync("https://www.sparkproxy.io/ip");
Console.WriteLine(html);

Set UseProxy = true explicitly. If you leave Proxy null and UseProxy at its default, the client silently falls back to the machine's system proxy (or no proxy), and your requests go out on your real IP with no error to tell you. For anything beyond a quick script, prefer SocketsHttpHandler directly. It is the default engine on .NET Core 2.1 and later and exposes the connection-pool knobs you need for scraping at volume.

using System.Net;
using System.Net.Http;

var handler = new SocketsHttpHandler
{
    Proxy = new WebProxy("http://proxy-1.sparkproxy.io:10000"),
    UseProxy = true,
    MaxConnectionsPerServer = 20,
    PooledConnectionLifetime = TimeSpan.FromMinutes(2)
};

using var client = new HttpClient(handler)
{
    Timeout = TimeSpan.FromSeconds(30)
};

MaxConnectionsPerServer caps how many sockets the client opens to any one destination, which keeps a single greedy target from starving the rest of your pool. PooledConnectionLifetime matters more than it looks, and section four explains why. The WebProxy constructor also takes a host and port pair, or a Uri, if you prefer new WebProxy("proxy-1.sparkproxy.io", 10000).

Commercial proxies need a username and password. Unlike some HTTP stacks, .NET makes this clean: attach a NetworkCredential to the WebProxy, and the handler answers the proxy's 407 challenge for you.

using System.Net;

var proxy = new WebProxy("http://proxy-1.sparkproxy.io:10000")
{
    Credentials = new NetworkCredential("your-user", "your-pass")
};

var handler = new SocketsHttpHandler { Proxy = proxy, UseProxy = true };
using var client = new HttpClient(handler);

One NetworkCredential applies to every request through that proxy, which is what you want when all IPs in a pool share one login. That is the common case with gateway-style endpoints. When each proxy endpoint has its own distinct credentials, a single NetworkCredential cannot express that, and this is where people get stuck. Use a CredentialCache and map each proxy URI to its own login:

using System.Net;

var cache = new CredentialCache
{
    { new Uri("http://proxy-1.sparkproxy.io:10000"), "Basic",
      new NetworkCredential("user-1", "pass-1") },
    { new Uri("http://proxy-2.sparkproxy.io:10000"), "Basic",
      new NetworkCredential("user-2", "pass-2") }
};

// Attach the cache to the proxy object as its Credentials.
IWebProxy proxy = new WebProxy { Credentials = cache };

The handler looks up the right entry by matching the proxy's URI and auth scheme, so per-endpoint logins resolve automatically. Keep credentials out of source: read them from Environment.GetEnvironmentVariable or your configuration provider rather than hard-coding the strings above.

This is the most expensive mistake in .NET networking, and scrapers hit it harder than anyone because they fire so many requests. HttpClient is designed to be created once and reused for the life of the application. Wrapping each request in using var client = new HttpClient(...) looks tidy and disposes the client, but disposing does not free the underlying socket immediately. The socket sits in TIME_WAIT for up to four minutes. Loop that a few thousand times and you run out of ephemeral ports, and every new request throws SocketException: Only one usage of each socket address is normally permitted.

Reuse one client. Because the proxy lives on the handler and the handler is created with the client, a naive singleton bakes in one proxy for the whole app, which fights rotation. The fix for rotation is section five. The fix for lifetime is a long-lived client with a bounded connection lifetime:

public static class Http
{
    // One handler, one client, for the whole process.
    private static readonly SocketsHttpHandler Handler = new()
    {
        UseProxy = true,
        PooledConnectionLifetime = TimeSpan.FromMinutes(2),
        MaxConnectionsPerServer = 50
    };

    public static readonly HttpClient Client = new(Handler)
    {
        Timeout = TimeSpan.FromSeconds(30)
    };
}

PooledConnectionLifetime is the second half of the story that most tutorials leave out. A singleton HttpClient keeps connections open and never re-resolves DNS on them, so if a proxy or target changes IP, your long-running scraper keeps hammering the stale address. Setting PooledConnectionLifetime to a couple of minutes forces connections to retire and DNS to refresh, giving you the socket reuse of a singleton without the stale-DNS bug. In ASP.NET apps, IHttpClientFactory (from Microsoft.Extensions.Http) manages this pooling for you and is the recommended path; the manual handler above is the equivalent for a console scraper.

Here is the trap that sends people down the wrong road. The proxy is a property of the handler, HttpClient caches its handler, and you should keep one client alive. Put those three facts together and it looks like rotation forces you to build a new client per proxy, which throws away connection pooling. It does not. The clean pattern is a custom IWebProxy whose GetProxy returns a different endpoint on each call. The handler asks it per request, so one shared client rotates automatically.

using System.Net;
using System.Threading;

public sealed class RotatingProxy : IWebProxy
{
    private readonly Uri[] _proxies;
    private int _counter = -1;

    public RotatingProxy(IEnumerable<string> proxyUrls)
        => _proxies = proxyUrls.Select(u => new Uri(u)).ToArray();

    // Shared credentials for the whole pool (see CredentialCache for per-proxy).
    public ICredentials? Credentials { get; set; }

    public Uri GetProxy(Uri destination)
    {
        // Interlocked.Increment can overflow to int.MinValue on a long run.
        // Cast to uint before the modulo so the index never goes negative.
        uint next = (uint)Interlocked.Increment(ref _counter);
        return _proxies[next % (uint)_proxies.Length];
    }

    public bool IsBypassed(Uri host) => false;
}

Wire it into the one shared handler and rotation happens for free on every request:

var rotating = new RotatingProxy(new[]
{
    "http://proxy-1.sparkproxy.io:10000",
    "http://proxy-2.sparkproxy.io:10000",
    "http://proxy-3.sparkproxy.io:10000"
})
{
    Credentials = new NetworkCredential("your-user", "your-pass")
};

var handler = new SocketsHttpHandler { Proxy = rotating, UseProxy = true };
using var client = new HttpClient(handler);

// Each GetAsync now leaves through the next proxy in the pool.
var page = await client.GetStringAsync("https://www.sparkproxy.io/ip");

Two details make this correct under load. First, Interlocked.Increment is the thread-safe counter, because a plain _counter++ is a data race the moment two async tasks advance it at once. Second, the uint cast is not decoration. Interlocked.Increment returns an int that eventually wraps from int.MaxValue to int.MinValue, and int.MinValue % length is negative, which throws IndexOutOfRangeException deep into an otherwise healthy run. Casting to uint before the modulo keeps the index non-negative forever, and unsigned overflow is well defined. Do not reach for Math.Abs here, because Math.Abs(int.MinValue) itself throws OverflowException. SocketsHttpHandler pools connections keyed by proxy, so returning different endpoints reuses the right pooled sockets rather than opening fresh ones each time.

HttpClient is async first, and that is how you get throughput in .NET without a thread per request. The mistake is unbounded concurrency: fire Task.WhenAll over ten thousand URLs and you hammer the target, overload the proxy pool, and trip rate limits instantly. Cap in-flight requests with a SemaphoreSlim.

using System.Collections.Concurrent;

async Task<IReadOnlyList<string>> ScrapeAll(
    IEnumerable<string> urls, HttpClient client, int maxConcurrency = 10)
{
    using var gate = new SemaphoreSlim(maxConcurrency);
    var results = new ConcurrentBag<string>();

    var tasks = urls.Select(async url =>
    {
        await gate.WaitAsync();
        try
        {
            var html = await client.GetStringAsync(url);
            results.Add(html);
        }
        finally
        {
            gate.Release();   // always release, even on failure
        }
    });

    await Task.WhenAll(tasks);
    return results.ToArray();
}

The SemaphoreSlim holds concurrency at ten regardless of how many URLs you pass, so you get parallelism without a stampede. Release inside a finally so a thrown request never leaks a permit and starves the pool. Two habits keep async code from biting you: never call .Result or .Wait() on a task, since sync-over-async can deadlock and starves the thread pool, and add ConfigureAwait(false) on awaits inside library code to avoid capturing a synchronization context you do not need. Await all the way from the top of your call stack.

Fetching is half the job. HtmlAgilityPack (the current line is 1.11.x, on NuGet as HtmlAgilityPack) is the long-standing .NET HTML parser. It is forgiving of broken markup and queries with XPath. Fetch with your shared HttpClient, load the string, and select.

using HtmlAgilityPack;

var html = await Http.Client.GetStringAsync("https://www.sparkproxy.io/products");

var doc = new HtmlDocument();
doc.LoadHtml(html);

// SelectNodes returns null (not an empty list) when nothing matches.
var cards = doc.DocumentNode.SelectNodes("//div[@class='product-card']");
if (cards != null)
{
    foreach (var card in cards)
    {
        var name  = card.SelectSingleNode(".//*[@class='name']")?.InnerText.Trim();
        var price = card.SelectSingleNode(".//*[@class='price']")?.InnerText.Trim();
        var href  = card.SelectSingleNode(".//a")?.GetAttributeValue("href", "");
        Console.WriteLine($"{name} | {price} | {href}");
    }
}

The null check is not optional, and it is the single most common HtmlAgilityPack bug. SelectNodes returns null when the XPath matches nothing, so foreach (var card in doc.DocumentNode.SelectNodes(...)) throws NullReferenceException the first time a page is missing the element you expected. Always guard the collection, and use the null-conditional ?. on SelectSingleNode, which also returns null on no match. If you prefer CSS selectors to XPath, add the HtmlAgilityPack.CssSelectors.NetCore package (a Fizzler binding) and query with familiar syntax:

using HtmlAgilityPack;
using Fizzler.Systems.HtmlAgilityPack;   // enables QuerySelectorAll on HtmlNode

var doc = new HtmlDocument();
doc.LoadHtml(html);

foreach (var price in doc.DocumentNode.QuerySelectorAll(".product-card .price"))
    Console.WriteLine(price.InnerText.Trim());

QuerySelectorAll from the Fizzler extension returns an empty sequence rather than null, so it is safer to loop directly. HtmlAgilityPack does not run JavaScript, so anything a page injects with client-side scripts is invisible to it. Route those targets to a headless browser or the Scraping API covered below.

AngleSharp (current 1.1.x, NuGet AngleSharp) is the modern alternative. It parses HTML the way a browser does, following the WHATWG standard, and builds a real DOM you query with native CSS selectors. No extra package for selectors, and the API mirrors what you would write in the browser console.

using AngleSharp.Html.Parser;

var html = await Http.Client.GetStringAsync("https://www.sparkproxy.io/products");

var parser = new HtmlParser();
var document = parser.ParseDocument(html);

// QuerySelector returns null on no match; QuerySelectorAll returns an empty list.
var title = document.QuerySelector("h1")?.TextContent.Trim();

foreach (var card in document.QuerySelectorAll("div.product-card"))
{
    var name  = card.QuerySelector(".name")?.TextContent.Trim();
    var price = card.QuerySelector(".price")?.TextContent.Trim();
    var link  = card.QuerySelector("a")?.GetAttribute("href");
    Console.WriteLine($"{name} | {price} | {link}");
}

Which parser should you pick? Use the table:

Reach for HtmlAgilityPack when you want a light, fast parser and you like XPath, especially for high-volume batch jobs. Reach for AngleSharp when you want browser-accurate parsing, native CSS selectors, and a DOM that behaves like the one in DevTools. Both are excellent, and neither executes JavaScript, which is the shared limit that decides when you offload to the API.

Proxies and targets fail. A proxy times out, a site answers 429 Too Many Requests, or an IP draws a 403. Catch it, back off, and retry. Because your rotating IWebProxy hands out a fresh IP on the next call, a banned address is swapped out on the retry automatically.

async Task<string> FetchWithRetry(HttpClient client, string url, int maxRetries = 4)
{
    for (int attempt = 0; ; attempt++)
    {
        try
        {
            using var res = await client.GetAsync(url);
            if (res.IsSuccessStatusCode)
                return await res.Content.ReadAsStringAsync();

            // 429 and 403 can clear on a fresh IP, so they are worth a retry.
            if (attempt >= maxRetries)
                res.EnsureSuccessStatusCode();   // throws with the status
        }
        catch (HttpRequestException) when (attempt < maxRetries) { }
        catch (TaskCanceledException) when (attempt < maxRetries) { }  // timeout

        var backoff = TimeSpan.FromSeconds(Math.Pow(2, attempt));   // 1s,2s,4s,8s
        var jitter  = TimeSpan.FromMilliseconds(Random.Shared.Next(250));
        await Task.Delay(backoff + jitter);
    }
}

Double the delay each attempt so you stop hammering a rate limit, and add jitter so a batch of tasks that all hit 429 at the same instant do not retry in lockstep and re-trigger it. Random.Shared (.NET 6 and later) is thread-safe, so it is fine to call from many concurrent tasks. A TaskCanceledException is how HttpClient surfaces a timeout, so catch it alongside HttpRequestException. For production, the Polly library (v8, via ResiliencePipelineBuilder().AddRetry(...)) gives you the same exponential-backoff-with-jitter policy declaratively, plus circuit breakers and timeouts, and it plugs straight into IHttpClientFactory. For the wider set of signals that get a request blocked in the first place, from TLS fingerprints to header order, see how to avoid getting your proxy blocked.

Running your own pool gives you control and the best cost per request on simple pages. Some targets are not worth it: they render content with JavaScript that HtmlAgilityPack and AngleSharp cannot execute, fingerprint the TLS handshake, or throw hard anti-bot walls. The SparkProxy Scraping API rotates the exit IP server-side on every call and can run a real browser, so there is no pool, no IWebProxy, and no retry loop to maintain. You send a target URL and get the response back.

using System.Net.Http;

var apiKey = Environment.GetEnvironmentVariable("SPARKPROXY_API_KEY");
var target = Uri.EscapeDataString("https://www.sparkproxy.io/products");

var endpoint = $"https://scrape.sparkproxy.io/api/v1?url={target}"
             + "&render_js=true"       // run a headless browser for JS pages
             + "&premium_proxy=true"    // route through residential IPs
             + "&country_code=us";      // geo-target the exit (ISO 3166-1 alpha-2)

using var req = new HttpRequestMessage(HttpMethod.Get, endpoint);
req.Headers.Add("X-API-Key", apiKey);          // auth header from your dashboard

using var res = await Http.Client.SendAsync(req);
var html = await res.Content.ReadAsStringAsync();

// Feed the returned HTML straight into your parser of choice.
var doc = new HtmlAgilityPack.HtmlDocument();
doc.LoadHtml(html);
Console.WriteLine(doc.DocumentNode.SelectSingleNode("//title")?.InnerText);

Authentication is the X-API-Key header carrying the key from your dashboard. country_code sets the exit geography, render_js runs a headless browser when the page needs JavaScript, and premium_proxy upgrades to residential IPs for tougher targets. Because a fresh IP is assigned per call, the rotation, auth, and retry code above is handled for you. Use a longer client timeout, since a rendered page takes longer than a raw fetch.

If you would rather skip parsing entirely, the API can extract server-side. Send a POST with extract_rules (CSS selectors) and it returns structured JSON.

using System.Text;
using System.Text.Json;

var payload = new
{
    url = "https://www.sparkproxy.io/products",
    render_js = true,
    country_code = "us",
    extract_rules = new
    {
        title  = "h1",
        prices = new { selector = ".price", type = "list" }
    }
};

using var req = new HttpRequestMessage(
    HttpMethod.Post, "https://scrape.sparkproxy.io/api/v1");
req.Headers.Add("X-API-Key", Environment.GetEnvironmentVariable("SPARKPROXY_API_KEY"));
req.Content = new StringContent(
    JsonSerializer.Serialize(payload), Encoding.UTF8, "application/json");

using var res = await Http.Client.SendAsync(req);
var json = await res.Content.ReadAsStringAsync();   // parse with System.Text.Json

A pragmatic split: run your own parser plus proxy pool for high-volume simple pages where per-request cost matters, and send the JavaScript-heavy or heavily defended pages to the API.