cURL Proxy Guide: Web Scraping From the Command Line

curl ships on almost every Linux, macOS, and modern Windows box, so a proxied request needs nothing installed. That makes it the right tool for testing a proxy, debugging why a page blocks you, wiring a quick scrape into a cron job, or piping HTML into grep, jq, or htmlq inside a shell pipeline. It's also the reference you reach for when a library's proxy behavior looks wrong, because a raw curl command shows you exactly what goes on the wire.

Where curl stops being the right tool is large, stateful scrapes: managing thousands of concurrent connections, parsing DOM trees, or defeating JavaScript challenges. Pick by the job.

Most people use curl to prove a proxy works, then port the same flags into code. If you're weighing running your own pool against a managed endpoint, the web scraping API vs self-managed proxies breakdown covers the trade-off in detail.

The -x flag (long form --proxy) routes a request through a proxy. This is the core of any curl -x proxy command.

curl -x http://proxy-1.sparkproxy.io:10000 https://www.sparkproxy.io/ip

That prints the exit IP the target sees, which should be the proxy's address, not yours. For an https:// target curl opens a CONNECT tunnel to the proxy and carries your TLS handshake end to end to the real site, so the proxy never sees your decrypted traffic. Add -v to watch the tunnel get built:

curl -v -x http://proxy-1.sparkproxy.io:10000 https://www.sparkproxy.io/ip
# > CONNECT sparkproxy.io:443 HTTP/1.1
# < HTTP/1.1 200 Connection established

Two defaults bite people. If you leave off the scheme, curl assumes http://, so -x proxy-1.sparkproxy.io:10000 is an HTTP proxy. And if you leave off the port, curl does not fall back to 8080 or 3128, it assumes 1080, the SOCKS default. That silently misroutes to a closed port on an HTTP proxy, so always write the port explicitly. The scheme prefix decides the proxy protocol.

If your provider hands you an HTTPS proxy with a self-signed certificate, --proxy-insecure skips verification of the proxy hop only, without touching verification of the target site. Reach for it rarely, and never confuse it with -k.

Most paid proxies want a username and password. For curl proxy authentication, the flag is -U (long form --proxy-user), and the value is user:password.

curl -x http://proxy-1.sparkproxy.io:10000 \
     -U myuser:mypass \
     https://www.sparkproxy.io/ip

Here is the single most common curl auth mistake: -U and -u are different flags. Lowercase -u (--user) sends credentials to the target site for HTTP auth. Uppercase -U (--proxy-user) sends them to the proxy. Swap them and you get a 407 Proxy Authentication Required that makes no sense until you spot the case. You can also embed the credentials in the proxy URL:

curl -x http://myuser:mypass@proxy-1.sparkproxy.io:10000 https://www.sparkproxy.io/ip

If your username or password contains a reserved character such as @, :, /, or #, URL-encode it before splicing it into the proxy URL, otherwise curl parses the string wrong and authentication fails. Use --proxy-user instead, which takes the raw value and encodes it for you. If your plan uses IP whitelisting rather than credentials, drop -U entirely and send the bare proxy URL. A 407 on a whitelisted setup usually means the request left from an egress IP you never added, which is common on cloud hosts with more than one outbound address.

A curl socks5 proxy has a trap that costs both privacy and geo-targeting, and it comes down to who resolves the hostname. --socks5 resolves DNS on your machine before connecting. --socks5-hostname sends the hostname to the proxy and lets it resolve.

# Leaks the DNS lookup to your local resolver:
curl --socks5 proxy-1.sparkproxy.io:20000 https://www.sparkproxy.io/ip

# Correct: the proxy resolves the hostname:
curl --socks5-hostname proxy-1.sparkproxy.io:20000 https://www.sparkproxy.io/ip

The -x scheme prefixes map to the same split, and the h in socks5h stands for hostname resolution:

Plain --socks5 does the DNS lookup from your own network before the connection opens, which leaks the target domain to your local resolver and defeats geo-targeting, because the lookup resolves from your country instead of the exit's. That matters when a site serves different content or prices by region: you route the request through a German exit but resolve the hostname from your own ISP, and any DNS-based geo split lands on the wrong answer. Use socks5h (or --socks5-hostname) for scraping, always. For a deeper look at when SOCKS5 beats HTTP, see understanding proxy protocols: HTTP, HTTPS, and SOCKS5.

curl reads proxy settings from the environment, which is handy for setting a proxy once per shell and forgetting it, and dangerous when you forget it's set. The variables are http_proxy, https_proxy, all_proxy, and no_proxy.

export https_proxy="http://myuser:mypass@proxy-1.sparkproxy.io:10000"
export http_proxy="http://myuser:mypass@proxy-1.sparkproxy.io:10000"
curl https://www.sparkproxy.io/ip        # goes through the proxy, no -x needed

Now the detail that surprises everyone: http_proxy is honored only in lowercase. curl deliberately ignores an uppercase HTTP_PROXY. The reason is a real vulnerability called httpoxy. In a CGI environment the incoming Proxy: request header is mapped into the HTTP_PROXY variable, so an attacker could set an outbound proxy on your server just by sending a header. curl closed that door by reading only the lowercase http_proxy for HTTP. The other variables (HTTPS_PROXY, ALL_PROXY, NO_PROXY) work in either case. When in doubt, use lowercase everywhere.

no_proxy is a comma-separated list of hosts that should bypass the proxy, and it accepts domain suffixes:

export no_proxy="localhost,127.0.0.1,internal.sparkproxy.io,.sparkproxy.io"

Precedence runs command line over environment: a -x on the request always wins, and --noproxy '*' forces a direct connection for that one call even when the environment says otherwise. If a scrape mysteriously ignores your proxy, or routes through one you never set, an exported variable is almost always the cause. Print them with env | grep -i proxy before you debug anything else.

A default curl request announces itself with a User-Agent of curl/8.x and sends almost none of the headers a browser does. Most anti-bot filters block that on sight. For curl web scraping you send a realistic header set with -A (user agent), -H (any header), and --compressed (accept gzip and decode it).

curl -x http://myuser:mypass@proxy-1.sparkproxy.io:10000 \
     -A "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36" \
     -H "Accept-Language: en-US,en;q=0.9" \
     -H "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8" \
     --compressed \
     -e "https://www.google.com/" \
     https://example.com/products

-e sets the Referer, which some sites check. One nuance worth knowing: -H sends the header to the target site, while --proxy-header sends it only to the proxy on the CONNECT hop, which you rarely need but which matters if a provider asks for a routing header.

Sessions need cookies. Save them to a jar on the first request with -c, then send them back on later requests with -b. Point both at the same file to persist a login across calls:

# First request stores the session cookie
curl -c cookies.txt -x http://myuser:mypass@proxy-1.sparkproxy.io:10000 \
     https://example.com/login

# Later requests replay it
curl -b cookies.txt -c cookies.txt \
     -x http://myuser:mypass@proxy-1.sparkproxy.io:10000 \
     https://example.com/dashboard

The jar is a Netscape-format file curl can both read and write. Realistic headers plus a consistent session go a long way, but they don't beat everything. The field guide on how to avoid getting your proxy blocked covers the request-shaping that keeps IPs alive longer.

One IP hitting a site a thousand times is a pattern. Rotation spreads those requests across many exits. curl has no built-in rotation, so you drive it from the shell. Put one proxy per line in proxies.txt:

http://myuser:mypass@proxy-1.sparkproxy.io:10000
http://myuser:mypass@proxy-2.sparkproxy.io:10000
http://myuser:mypass@proxy-3.sparkproxy.io:10000

Then loop over your target URLs, handing each request the next proxy round-robin:

#!/usr/bin/env bash
set -euo pipefail

mapfile -t proxies < proxies.txt
mapfile -t urls < urls.txt
count=${#proxies[@]}
i=0

for url in "${urls[@]}"; do
  proxy="${proxies[$((i % count))]}"
  curl -sS --compressed \
       -x "$proxy" \
       -A "Mozilla/5.0 (Windows NT 10.0; Win64; x64) Chrome/126.0.0.0 Safari/537.36" \
       -o "out_$i.html" \
       -w "%{http_code} %{remote_ip} %{time_total}s -> out_$i.html\n" \
       "$url"
  i=$((i + 1))
  sleep "$(awk 'BEGIN{srand(); print 1 + rand()*2}')"   # 1-3s jitter
done

The -w format string logs the status, the exit IP the request actually used, and the response time per line, so you can see rotation working and spot a proxy that keeps timing out. The randomized sleep adds jitter so you don't fire on a fixed cadence that reads as automation. To randomize the order instead of round-robin, replace the index math with proxy=$(shuf -n1 -e "${proxies[@]}"). When a request needs a specific outbound port, the proxy ports explained: 80, 443, 8080, and more guide covers which port maps to which protocol.

Proxies fail. One times out, another returns 429, a third resets the socket. curl has built-in retry logic that handles the transient cases without any scripting.

curl -sS --compressed \
     -x http://myuser:mypass@proxy-1.sparkproxy.io:10000 \
     --connect-timeout 10 \
     --max-time 60 \
     --retry 4 \
     --retry-delay 2 \
     --retry-all-errors \
     https://example.com/products

The flags do specific things. --connect-timeout 10 caps how long curl waits to establish the connection, and --max-time 60 caps the whole transfer so a slow proxy can't hang your loop. --retry 4 retries up to four times with an exponentially growing wait. By default --retry only fires on transient HTTP errors and a handful of network conditions, not on a connection refused or a curl error 56 reset, which is why --retry-all-errors (added in curl 7.71.0) matters: it makes curl retry on any error. If you're on an older curl, --retry-connrefused (7.52.0) covers the most common gap.

Built-in retries reuse the same proxy, though. To rotate to a fresh IP on each attempt, wrap curl in a small function and branch on the status code:

fetch() {
  local url="$1" tries="${2:-4}" count=${#proxies[@]}
  for ((n = 0; n < tries; n++)); do
    local proxy="${proxies[$((RANDOM % count))]}"   # fresh proxy each try
    local code
    code=$(curl -sS --compressed -x "$proxy" \
                --connect-timeout 10 --max-time 60 \
                -o "body.html" -w "%{http_code}" "$url" || echo 000)
    if [[ "$code" =~ ^2 ]]; then
      echo "OK $code via $proxy"
      return 0
    fi
    echo "retry: got $code via $proxy" >&2
    sleep $(( 2 ** n ))                              # 1s, 2s, 4s, 8s backoff
  done
  echo "FAILED after $tries tries: $url" >&2
  return 1
}

Three decisions make this hold up. Rotate the proxy on every attempt, so a dead IP never gets a second try in the same call. Capture the HTTP code with -w "%{http_code}" and treat any 2xx as success, so a 503 becomes a branch instead of a crash. And double the backoff (1s, 2s, 4s, 8s) so a rate limit gets a chance to cool off instead of a second hammering.

You can set a perfect Chrome User-Agent and still get blocked, because modern anti-bot systems fingerprint the layer below the headers: the TLS handshake. When curl opens a connection it sends a ClientHello whose cipher list, extensions, and ordering form a signature (the JA3 and newer JA4 fingerprints). curl's ClientHello, built on OpenSSL or whatever TLS library it links, does not match Chrome's or Firefox's. Cloudflare, Akamai, and DataDome compare that fingerprint against the User-Agent you claim, and a curl handshake wearing a Chrome header is an obvious mismatch. HTTP/2 settings frames add a second fingerprint that standard curl also can't reshape.

This is a hard limit, not a flag you're missing. Standard curl cannot impersonate a browser's TLS signature, full stop. Two projects work around it by relinking curl against a browser's TLS stack: curl-impersonate (a drop-in curl binary that mimics Chrome and Firefox handshakes) and curl_cffi (its Python binding). They help against fingerprint-based blocks, but you're now maintaining a patched binary and chasing browser versions as fingerprints change.

The alternative is to hand the TLS problem to a service that keeps its fingerprints current for you. That is what the Scraping API in the next section does.

When a target fights back with TLS fingerprinting, JavaScript challenges, or aggressive rate limits, the SparkProxy Scraping API handles rotation, browser rendering, and anti-bot server-side, so your curl command stays simple. The base endpoint is https://scrape.sparkproxy.io/api/v1, and authentication is the X-API-Key header carrying a key of the form sk-... from your dashboard.

The catch with a GET is that your target URL has its own ? and &, which collide with the API's query string. Use -G with --data-urlencode so curl encodes each value correctly, target URL included:

curl -G "https://scrape.sparkproxy.io/api/v1" \
     --data-urlencode "url=https://example.com/products?page=2&sort=price" \
     --data-urlencode "render_js=true" \
     --data-urlencode "premium_proxy=true" \
     --data-urlencode "country_code=us" \
     -H "X-API-Key: $SPARKPROXY_API_KEY"

render_js runs a headless Chromium browser for pages that build their content in JavaScript, premium_proxy upgrades to residential exits, and country_code geo-targets the exit IP. To get structured data back instead of raw HTML, POST a JSON body with extract_rules and let the API return parsed fields:

curl -X POST "https://scrape.sparkproxy.io/api/v1" \
     -H "X-API-Key: $SPARKPROXY_API_KEY" \
     -H "Content-Type: application/json" \
     -d '{
       "url": "https://example.com/products",
       "render_js": true,
       "premium_proxy": true,
       "country_code": "us",
       "extract_rules": {
         "title": "h1",
         "price": ".price",
         "links": { "selector": "a", "type": "list" }
       }
     }'

The API can also return a screenshot or PDF. Set format=screenshot and write the binary straight to a file with -o:

curl -G "https://scrape.sparkproxy.io/api/v1" \
     --data-urlencode "url=https://example.com" \
     --data-urlencode "format=screenshot" \
     --data-urlencode "render_js=true" \
     -H "X-API-Key: $SPARKPROXY_API_KEY" \
     -o shot.png

Because the API assigns a fresh IP per call, renders JavaScript, and keeps its TLS fingerprints current, the rotation loop, retry function, and header juggling above become optional for the targets that need them most. A practical split: use plain curl -x for simple, high-volume pages where per-request cost matters, and send the JavaScript-heavy or heavily defended pages to the Scraping API.

Every flag in this guide, in one place.