PowerShell Web Scraping: Invoke-WebRequest and Proxies

PowerShell is already on every Windows box, so a scraper is one Invoke-WebRequest away with nothing to install. It has real objects instead of text streams, a pipeline, native ConvertTo-Json and ConvertFrom-Json, and it drops straight into Task Scheduler for a nightly run. If you live in a Windows or .NET shop, or you mostly pull JSON from internal APIs, it is the path of least resistance. For the ground-level concept first, see what web scraping is.

Be honest about the ceiling, though. Invoke-WebRequest and Invoke-RestMethod are HTTP clients, not browsers, so they never run JavaScript. The built-in HTML parsing is thinner than Python's lxml or BeautifulSoup unless you add a real parser. And every request leaves from one IP, so a few hundred hits at one host earns you a rate limit, a CAPTCHA, or a block. A proxy pool spreads that traffic across many addresses so no single one trips the target's threshold. Datacenter proxies carry the high-volume, simple-page fetching at the best cost per request, which is why they are the workhorse for web scraping with datacenter proxies.

You have four ways to attach the work, and they suit different jobs:

Most real projects mix them: your own proxy pool for the bulk of simple pages, and a managed API for the handful of targets that fight back.

Both cmdlets sit on the same HTTP engine, and the difference is what they hand back. Invoke-WebRequest (alias iwr) returns a response object with .StatusCode, .Headers, .Content, .RawContent, and parsing helpers like .Links, .Images, and .InputFields. Invoke-RestMethod (alias irm) reads the response Content-Type and deserializes for you: JSON becomes a PSCustomObject, XML becomes an XmlDocument, and RSS or Atom becomes feed items.

$resp = Invoke-WebRequest -Uri 'https://www.sparkproxy.io/ip' -UseBasicParsing
$resp.StatusCode                    # 200
$resp.Headers['Content-Type']       # text/html; charset=utf-8
$resp.Content                       # raw body as a single string
$resp.Links | Select-Object -First 5 -ExpandProperty href

Reach for Invoke-RestMethod the moment the endpoint returns structured data. It saves you the ConvertFrom-Json step and gives you objects you can pipe.

# Invoke-RestMethod deserializes JSON to objects automatically.
$data = Invoke-RestMethod -Uri 'https://www.sparkproxy.io/api/status'
$data.region                        # already a property, no parsing needed
$data.nodes.Count

The rule of thumb: use Invoke-WebRequest when you want HTML, a status code, response headers, or cookies, and Invoke-RestMethod when the server speaks JSON or XML.

One migration trap lives in that last row. In Windows PowerShell 5.1, curl and wget are aliases for Invoke-WebRequest, so a copied curl command silently runs the PowerShell cmdlet with entirely different flags. On PowerShell 7 those aliases are gone, and curl calls the real curl binary. Spell the cmdlet out in scripts you plan to share.

This is the difference that breaks copied tutorials. In Windows PowerShell 5.1, Invoke-WebRequest parses the page with the Internet Explorer engine by default and exposes a full DOM on .ParsedHtml, along with .Forms, .Links, .Images, and .InputFields.

# Windows PowerShell 5.1 only: full IE DOM through .ParsedHtml
$resp  = Invoke-WebRequest -Uri 'https://www.sparkproxy.io/products'
$cards = $resp.ParsedHtml.getElementsByClassName('product-card')
foreach ($c in $cards) {
    $c.getElementsByClassName('price')[0].innerText
}

That convenience carries three costs. It needs Internet Explorer installed and through its first-launch setup, it is slow and can hang on large pages, and it was removed from the shell entirely in PowerShell 6. Run the snippet above on PowerShell 7 and .ParsedHtml is $null, so the loop throws.

Passing -UseBasicParsing on 5.1 skips the IE engine. You lose .ParsedHtml and .Forms, but .Links, .Images, .InputFields, and .Content still populate through a lighter regex-based parse, and the call stops depending on IE. On PowerShell 7 that switch is accepted only for backward compatibility and does nothing, because basic parsing is the sole behavior. The portable move is to write code that never touches .ParsedHtml.

# Portable: works on 5.1 (-UseBasicParsing) and PS 7 (default behavior)
$resp = Invoke-WebRequest -Uri 'https://www.sparkproxy.io/products' -UseBasicParsing
$resp.Links.href                              # every <a href>
$resp.InputFields | Where-Object name -eq 'csrf_token'

Here is the full split, because these gaps are where cross-version scripts fail:

For anything past a link list, both runtimes want a real parser. On PowerShell 7 that means HtmlAgilityPack.

Since .ParsedHtml is gone on PowerShell 7, load HtmlAgilityPack (the current line is 1.11.x on NuGet) to get XPath and CSS selection over a proper node tree. Install it through the NuGet provider or drop the DLL next to your script, then Add-Type the assembly.

# One-time: pull the package, then reference its DLL.
Install-Package HtmlAgilityPack -Source nuget.org -Scope CurrentUser -Force
$dll = (Get-Package HtmlAgilityPack | Get-ChildItem).Where{ $_.Name -eq 'HtmlAgilityPack.dll' }
Add-Type -Path 'C:\path\to\HtmlAgilityPack.dll'

With the type loaded, fetch the raw HTML and select against it. Note the null guard: SelectNodes returns $null, not an empty collection, when nothing matches, so a naive foreach throws on the first page that is missing your element.

$html = (Invoke-WebRequest 'https://www.sparkproxy.io/products' -UseBasicParsing).Content

$doc = New-Object HtmlAgilityPack.HtmlDocument
$doc.LoadHtml($html)

$cards = $doc.DocumentNode.SelectNodes('//div[@class="product-card"]')
if ($cards) {                                   # guard: SelectNodes can be $null
    foreach ($card in $cards) {
        $name  = $card.SelectSingleNode('.//*[@class="name"]').InnerText.Trim()
        $price = $card.SelectSingleNode('.//*[@class="price"]').InnerText.Trim()
        [PSCustomObject]@{ Name = $name; Price = $price }
    }
}

Two follow-ups keep the output clean. HtmlAgilityPack leaves HTML entities encoded, so decode them with [System.Net.WebUtility]::HtmlDecode($text) when you see & in your data. And SelectSingleNode also returns $null on a miss, so wrap optional fields in a check before calling .InnerText. HtmlAgilityPack does not run JavaScript either, so anything a page injects client-side stays invisible and belongs on the Scraping API covered below.

Plenty of "scraping" is really a hidden JSON endpoint that the page calls in the background. Invoke-RestMethod is built for it. A GET returns objects directly, and a POST takes a hashtable you pipe through ConvertTo-Json.

$headers = @{ 'Accept' = 'application/json' }
$body = @{ query = 'proxies'; limit = 50 } | ConvertTo-Json

$result = Invoke-RestMethod -Uri 'https://www.sparkproxy.io/api/search' `
    -Method Post -Headers $headers -ContentType 'application/json' -Body $body

$result.items | ForEach-Object { $_.name }

Watch two version-specific gotchas. ConvertTo-Json on Windows PowerShell 5.1 defaults to a depth of 2 and silently truncates anything deeper into System.Object[], so pass -Depth 5 (or higher) when your payload nests. And 5.1 sometimes decodes a UTF-8 body as ISO-8859-1 when the server omits a charset, which turns accented characters into mojibake; PowerShell 7 handles UTF-8 correctly, so if you are stuck on 5.1, re-decode the raw bytes as UTF-8 yourself. On PowerShell 6.2 and later, ConvertFrom-Json -AsHashtable is handy when keys are dynamic and you do not want fixed properties.

Both cmdlets take -Proxy for the endpoint and -ProxyCredential for the login. Commercial proxies need authentication, and you build the PSCredential without an interactive prompt by converting a secret string.

$proxy = 'http://proxy-1.sparkproxy.io:10000'
$user  = $env:SPARKPROXY_USER
$pass  = ConvertTo-SecureString $env:SPARKPROXY_PASS -AsPlainText -Force
$cred  = [System.Management.Automation.PSCredential]::new($user, $pass)

$resp = Invoke-WebRequest -Uri 'https://www.sparkproxy.io/ip' `
    -Proxy $proxy -ProxyCredential $cred -UseBasicParsing

Keep the username and password in environment variables or a secret store, never hard-coded. To spread load, hold a pool of endpoints and pick a fresh one per request. Get-Random gives you simple rotation, and a banned IP is swapped out on the next call for free.

$pool = @(
    'http://proxy-1.sparkproxy.io:10000',
    'http://proxy-2.sparkproxy.io:10000',
    'http://proxy-3.sparkproxy.io:10000'
)

foreach ($url in $targets) {
    $proxy = $pool | Get-Random                 # fresh exit IP per request
    Invoke-WebRequest -Uri $url -Proxy $proxy -ProxyCredential $cred -UseBasicParsing
    Start-Sleep -Milliseconds (Get-Random -Minimum 500 -Maximum 2000)
}

One honest limit: -Proxy speaks to HTTP and HTTPS proxies, not SOCKS. If you need a SOCKS5 exit, route it through the Scraping API's own_proxy parameter or a local HTTP-to-SOCKS forwarder rather than expecting these cmdlets to dial it directly. For the wider set of signals that get an IP blocked in the first place, from header order to TLS fingerprints, see how to avoid getting your proxy blocked. Deciding when to run your own pool at all is the whole Scraping API versus self-managed proxies question.

The default PowerShell user agent announces itself. On 5.1 it contains WindowsPowerShell/5.1, and on 7 it contains PowerShell/7.x, both trivially matched by a block rule. Set a browser string with -UserAgent. PowerShell ships canned ones so you do not have to memorize the format.

$ua = [Microsoft.PowerShell.Commands.PSUserAgent]::Chrome

Invoke-WebRequest -Uri 'https://www.sparkproxy.io/products' `
    -UserAgent $ua -UseBasicParsing

PSUserAgent also exposes ::Firefox, ::Safari, ::InternetExplorer, and ::Chrome. For a longer run, keep an array of current browser strings and rotate them alongside your proxies so the header set changes with the IP. A real Chrome IP paired with a WindowsPowerShell user agent is an obvious mismatch, and defenders key on exactly that kind of contradiction.

Anything behind a login needs cookies to survive between requests. -SessionVariable creates a session container on the first call, and -WebSession replays it on every call after. The container holds cookies, credentials, headers, the proxy, and the user agent, so the whole scrape looks like one browser.

# The first request creates a session variable named 'sess' (note: no $ prefix).
$login = Invoke-WebRequest -Uri 'https://www.sparkproxy.io/login' -Method Post `
    -Body @{ user = 'you@sparkproxy.io'; pass = $env:SITE_PASS } `
    -SessionVariable sess -UseBasicParsing

# Reuse the cookies on every later request.
$dash = Invoke-WebRequest -Uri 'https://www.sparkproxy.io/dashboard' `
    -WebSession $sess -UseBasicParsing

The classic mistake is right there in the parameter. You pass -SessionVariable sess as a bare name with no $, and PowerShell creates the variable $sess for you. Write -SessionVariable $sess and you hand it an empty value, so the session never forms. Once you have the object, set defaults on it once and they apply to every request that uses it.

$sess.UserAgent = [Microsoft.PowerShell.Commands.PSUserAgent]::Chrome
$sess.Proxy     = [System.Net.WebProxy]::new('http://proxy-1.sparkproxy.io:10000')
$sess.Headers.Add('Accept-Language', 'en-US,en;q=0.9')

Politeness keeps you unblocked, and error handling keeps the run alive. Space requests with Start-Sleep plus jitter so a batch does not fire in lockstep. Retrying is where the runtimes diverge most. PowerShell 7 has built-in retry and non-throwing error inspection.

# PowerShell 7: retry transient failures, and inspect status without a throw.
Invoke-WebRequest -Uri 'https://www.sparkproxy.io/products' `
    -MaximumRetryCount 4 -RetryIntervalSec 3 `
    -SkipHttpErrorCheck -StatusCodeVariable code -UseBasicParsing

if ($code -ge 400) { "target answered $code" }

-MaximumRetryCount and -RetryIntervalSec arrived in PowerShell 6 and retry on failures like 429 and 5xx. -SkipHttpErrorCheck stops a non-2xx response from becoming a terminating error, so you read -StatusCodeVariable and branch instead of catching. Windows PowerShell 5.1 has none of these, and it also defaults to a TLS version below 1.2 on many systems, which throws "Could not create SSL/TLS secure channel" against modern sites. You handle both by hand.

# Windows PowerShell 5.1: no built-in retry, and TLS 1.2 is not the default.
[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12

function Get-WithRetry {
    param([string]$Url, [int]$Max = 4)
    for ($i = 0; $i -lt $Max; $i++) {
        try {
            return Invoke-WebRequest -Uri $Url -UseBasicParsing
        } catch [System.Net.WebException] {
            $status = [int]$_.Exception.Response.StatusCode
            if ($status -notin 429,500,502,503,504 -or $i -eq $Max - 1) { throw }
            Start-Sleep -Seconds ([math]::Pow(2, $i))     # 1s, 2s, 4s, 8s
        }
    }
}

Double the delay each attempt so you stop hammering a rate limit, and read the status off $_.Exception.Response since 5.1 surfaces HTTP failures as a WebException. If you want parallelism on PowerShell 7, ForEach-Object -Parallel -ThrottleLimit 5 runs several fetches at once while capping in-flight work, which is the shell's answer to a bounded worker pool.

Running your own pool gives you control and the best cost on simple pages. Some targets are not worth it: they render content with JavaScript that HtmlAgilityPack cannot execute, fingerprint the TLS handshake, or throw a hard anti-bot wall. The SparkProxy Scraping API rotates the exit IP server-side on every call and can drive a real headless browser, so there is no pool, no rotation loop, and no retry code to maintain. Authentication is the X-API-Key header carrying the key from your dashboard, and the base endpoint is https://scrape.sparkproxy.io/api/v1.

$headers = @{ 'X-API-Key' = $env:SPARKPROXY_API_KEY }
$target  = [uri]::EscapeDataString('https://www.sparkproxy.io/products')
$uri = "https://scrape.sparkproxy.io/api/v1?url=$target&render_js=true&country_code=us"

# render_js returns fully rendered HTML; irm hands back the string to parse.
$html = Invoke-RestMethod -Uri $uri -Headers $headers -TimeoutSec 60

render_js=true runs the headless browser for JavaScript pages, country_code geo-targets the exit with an ISO 3166-1 alpha-2 code, and adding premium_proxy=true upgrades to residential IPs for tougher targets. Because a fresh IP is assigned per call, your rotation, proxy auth, and user-agent work is handled server-side. Give it a longer -TimeoutSec, since a rendered page takes longer than a raw fetch.

If you would rather skip HTML parsing entirely, send a POST with extract_rules and the API returns structured JSON that Invoke-RestMethod deserializes straight into objects.

$body = @{
    url           = 'https://www.sparkproxy.io/products'
    render_js     = $true
    country_code  = 'us'
    premium_proxy = $true
    extract_rules = @{
        title  = 'h1'
        prices = @{ selector = '.price'; type = 'list' }
    }
} | ConvertTo-Json -Depth 5                      # -Depth so extract_rules is not truncated

$data = Invoke-RestMethod -Uri 'https://scrape.sparkproxy.io/api/v1' -Method Post `
    -Headers @{ 'X-API-Key' = $env:SPARKPROXY_API_KEY } `
    -ContentType 'application/json' -Body $body

$data.title
$data.prices

The -Depth 5 is not optional here: the nested extract_rules object is exactly the kind of structure that 5.1's default depth of 2 flattens into unusable text. A pragmatic split is to run your own parser plus proxy pool for high-volume simple pages where per-request cost matters, and send the JavaScript-heavy or heavily defended pages to the API.