Wget vs cURL for Command-Line Scraping

Reach for wget when the unit of work is a set of pages you have not enumerated yet: mirroring a documentation site, pulling a directory of PDFs, resuming a 4 GB download over a bad link. wget --mirror does link discovery, depth limits, timestamping, and link rewriting with one flag.

Reach for curl when the unit of work is one request you need to control exactly: a POST with a JSON body, a custom header set, a SOCKS5 proxy, HTTP/2 or HTTP/3, response timing metrics, output straight into jq. curl speaks about two dozen protocols and ships as libcurl inside almost everything.

Two defaults decide more real cases than any feature: wget consults robots.txt while crawling and curl never does, and curl exits 0 on an HTTP 404 while wget exits 8. Both facts are covered below, because both cause silent failures.

Neither tool defeats modern anti-bot systems. Both hand over a TLS handshake that no browser would produce, which is the last section here for a reason.

GNU Wget grew out of a 1995 program called Geturl and was renamed in 1996. Its design context was unattended downloading over dial-up links that dropped constantly, which is why retrying 20 times, resuming partial files, and recursing through links are built-in behaviours rather than add-ons. The GNU Wget manual still opens by describing it as a free utility for non-interactive download of files from the web.

curl started as httpget in 1996 and took the name curl with version 4.0 in 1998. Its goal was and remains transferring a URL, once, correctly, across as many protocols as possible. The curl manpage lists the surface: arbitrary methods, header control, upload, and a protocol list that runs from DICT and MQTT to SFTP, SMTP, and WebSockets. The reusable core became libcurl, which is why the same transfer engine sits under PHP, Python's pycurl, git, and a large share of embedded devices.

One practical note on Windows: curl.exe has shipped with Windows 10 since build 17063, but typing curl in Windows PowerShell 5.1 hits an alias for Invoke-WebRequest, which takes completely different arguments. Type curl.exe explicitly. wget is not present on Windows or macOS by default and has to be installed.

This is the capability gap. curl fetches URLs you give it. wget can discover URLs by parsing the HTML it already fetched, then fetch those too.

# Mirror a docs section, keep it browsable offline, stay inside the path
wget --mirror \
     --page-requisites \
     --convert-links \
     --adjust-extension \
     --no-parent \
     --wait=1 --random-wait --limit-rate=200k \
     https://www.sparkproxy.io/docs/

--mirror is shorthand for -r -N -l inf --no-remove-listing: recurse, use timestamping so a re-run only pulls changed files, and set infinite depth. --page-requisites grabs the CSS, images, and scripts each page needs. --convert-links rewrites hrefs so the copy works from the filesystem. --adjust-extension renames ?id=3 responses to .html. --no-parent stops wget climbing above /docs/, which is the flag people forget right before they accidentally mirror an entire domain.

Filtering is where wget earns its keep on file harvests:

# Every PDF under a path, two levels deep, nothing else
wget -r -l 2 -np -nd -A pdf,PDF -P ./reports https://www.sparkproxy.io/resources/

-A accepts extensions, -R rejects them, -D limits domains, and -I or -X include or exclude directories. -nd flattens the tree and -P sets the output directory.

curl's nearest equivalent is URL globbing, which expands a pattern you already know:

curl "https://www.sparkproxy.io/page-[1-50].html" -o "page-#1.html"
curl "https://www.sparkproxy.io/{docs,blog,pricing}/" -o "#1.html"

That is enumeration, not crawling. There is no link discovery, no depth limit, no dedupe of already-seen URLs. If you need those, either use wget or write the queue yourself. The distinction between fetching known URLs and discovering them is the same one covered in web scraping vs web crawling, and it is the single best predictor of which tool you want.

Everything about one HTTP transaction is addressable in curl: method, body, headers, auth, TLS version, protocol version, name resolution, and timing.

curl -X POST "https://www.sparkproxy.io/api/jobs" \
     -H "Authorization: Bearer $TOKEN" \
     -H "Content-Type: application/json" \
     --data-binary @job.json \
     --http2 \
     --compressed \
     -w '\nstatus=%{http_code} dns=%{time_namelookup}s tls=%{time_appconnect}s total=%{time_total}s\n'

-w / --write-out is the piece with no wget counterpart at all. It exposes 50-plus variables (%{http_code}, %{num_redirects}, %{size_download}, %{ssl_verify_result}) and turns curl into a probe you can graph. --resolve host:443:203.0.113.10 pins a hostname to an IP without touching /etc/hosts, which is how you test one specific edge node. --json (added in 7.82.0) sets the body, Content-Type, and Accept in a single flag.

wget did gain --method and --body-data in version 1.15, plus --post-data before that, so simple POSTs work. What it lacks is the reporting and the protocol surface. There is no timing output, no HTTP/2 in the 1.x line, no SOCKS, and no way to script against response metadata beyond parsing the -S header dump. GNU Wget2 (currently 2.2.x) adds HTTP/2 and parallel connections, but it is a separate binary that most systems do not ship.

Here is the difference that changes outcomes and gets almost no coverage.

wget implements the Robots Exclusion Protocol. During recursive retrieval it fetches /robots.txt first and honours Disallow and Allow rules for its user agent, and it also respects nofollow in . The behaviour is documented in the manual's Robot Exclusion section, and the protocol itself was finally standardised as RFC 9309 in September 2022. A plain non-recursive wget https://www.sparkproxy.io/page.html does not fetch robots.txt at all; the check is tied to recursion.

curl never fetches robots.txt. That is an explicit non-goal. curl transfers the URL you named and nothing else, so responsibility for the rules is entirely yours.

The practical consequence: when a wget --mirror run stops after four files and you assume you were blocked, you usually were not. Check first.

# See whether robots.txt is what stopped you
wget -d --mirror https://www.sparkproxy.io/docs/ 2>&1 | grep -i robots

# Override it, only where you have the right to
wget -e robots=off --mirror https://www.sparkproxy.io/docs/

Two details worth knowing. First, -e robots=off is a wgetrc directive passed inline, not a normal flag, which is why --robots=off does not exist. Second, wget's parser handles User-agent, Disallow, and Allow, but it does not implement the non-standard Crawl-delay directive that RFC 9309 also left out. So a site asking for a 10-second gap gets ignored unless you set --wait=10 --random-wait yourself. Honouring the file while hammering the server ten times a second is the worst of both worlds. Pacing, backoff, and what counts as acceptable crawl behaviour are covered in the guide on ethical scraping and rate limiting.

curl treats the proxy as a first-class argument. wget treats it as environment configuration. That one design difference produces most of the confusion.

# curl: per-request, on the command line
curl -x "http://USER:PASS@gateway.sparkproxy.io:11000" https://www.sparkproxy.io/

# curl: SOCKS5 with remote DNS resolution
curl --socks5-hostname "USER:PASS@gateway.sparkproxy.io:13000" https://www.sparkproxy.io/

# wget: no --proxy flag exists, so use the environment
http_proxy="http://USER:PASS@gateway.sparkproxy.io:11000" \
https_proxy="http://USER:PASS@gateway.sparkproxy.io:11000" \
wget https://www.sparkproxy.io/

# wget: same result without exporting anything, credentials kept separate
wget -e use_proxy=yes \
     -e https_proxy=http://gateway.sparkproxy.io:11000 \
     --proxy-user=USER --proxy-password=PASS \
     https://www.sparkproxy.io/

Two gotchas. curl reads HTTPS_PROXY and ALL_PROXY in either case, but http_proxy only in lower case, deliberately, because CGI environments turn a client's Proxy: header into HTTP_PROXY. And anything you pass on the command line, including -x with inline credentials, is visible in ps to every user on the box. Put credentials in ~/.curlrc or ~/.wgetrc with mode 600, or use curl -K secrets.conf and wget --config.

The rotation model also differs. Every curl invocation is a fresh process, so a rotating gateway hands each one a new exit IP. A single wget --mirror run is one long session across hundreds of URLs, so a rotating endpoint may swap your IP mid-crawl and invalidate the session cookies you picked up on page one. Use a sticky session for mirroring and a rotating pool for per-URL loops. The full set of curl proxy flags, including authentication edge cases and DNS leaks, is in web scraping with cURL and proxies.

Both tools read and write the same Netscape cookies.txt format, so a jar written by one is readable by the other. That is genuinely useful when you log in with curl and mirror with wget.

# curl: log in, keep the jar, replay it
curl -c jar.txt -b jar.txt -d "user=you@sparkproxy.io&pass=$PW" https://www.sparkproxy.io/login
curl -b jar.txt https://www.sparkproxy.io/account

# wget: same jar, note the extra flag
wget --save-cookies jar.txt --keep-session-cookies \
     --post-data "user=you@sparkproxy.io&pass=$PW" https://www.sparkproxy.io/login
wget --load-cookies jar.txt https://www.sparkproxy.io/account

--keep-session-cookies is not optional. wget discards cookies without an expiry date when it writes the jar, and login cookies are very often session cookies, so leaving the flag off produces a jar that looks fine and authenticates nothing. curl behaves the opposite way: it saves session cookies by default and you discard them explicitly with -j / --junk-session-cookies.

Header syntax is close enough to be dangerous. Both accept --header "Name: value", and curl also has the short -H. curl's -A and wget's -U both set the user agent. But -e means --referer in curl and "execute a wgetrc command" in wget, so -e https://www.sparkproxy.io/ does two entirely different things depending on which binary you typed.

wget assumes the network is unreliable. curl assumes you will handle it.

--random-wait multiplies your --wait value by a random factor between 0.5 and 1.5, a cheap way to avoid the perfectly periodic request pattern that rate limiters look for. curl has nothing equivalent, so you add sleep $((RANDOM % 3 + 1)) inside the loop.

For curl, the retry set matters. Plain --retry 5 covers transient failures such as timeouts and 408, 429, 500, 502, 503, and 504. Anything else, including a connection reset, is not retried until you add --retry-all-errors (7.71.0 and later).

curl --retry 5 --retry-all-errors --retry-max-time 120 \
     --connect-timeout 10 --max-time 60 \
     -sS -o page.html https://www.sparkproxy.io/

Neither tool implements a retry budget across a whole job, honours Retry-After intelligently, or backs off per host. For those you want a real scheduler; the patterns are in retry and backoff strategies for web scraping.

curl writes the body to stdout and progress to stderr, so curl -sS URL | jq . works with no extra flags. wget writes to a file named after the URL and needs -qO- to behave the same way:

curl -sS https://www.sparkproxy.io/api/status | jq -r '.region'
wget -qO- https://www.sparkproxy.io/api/status | jq -r '.region'

Now the failure mode that costs people hours:

curl -s -o /dev/null -w '%{http_code}\n' https://www.sparkproxy.io/does-not-exist; echo "curl exit: $?"
# 404
# curl exit: 0

wget -q -O /dev/null https://www.sparkproxy.io/does-not-exist; echo "wget exit: $?"
# wget exit: 8

curl considers an HTTP error a successful transfer, because it did transfer the response. In a set -e script or a CI step, that 404 passes as green. Add -f / --fail to make curl return 22 on a 4xx or 5xx, or --fail-with-body (added in 7.76.0, March 2021) if you still want the error page. wget's exit status is granular by default:

If you already ship curl in a monitoring script, curl -fsS is the three-character fix that makes it fail correctly.

Everything above is about ergonomics. This section is about whether the request gets through at all.

Before any HTTP is sent, your client presents a TLS ClientHello: cipher suites in a specific order, extension list, elliptic curves, ALPN values, and for TLS 1.3 the key share groups. Hash that and you get a JA3 or JA4 fingerprint. Cloudflare, DataDome, Akamai, and PerimeterX all check it against the browser your User-Agent claims to be. curl linked against OpenSSL and wget linked against GnuTLS or OpenSSL produce fingerprints that match no browser on earth.

So setting -A "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 ..." makes things worse, not better. You have moved from an honest bot to a client whose TLS handshake contradicts its own user agent, and that mismatch is a stronger block signal than either half alone. The mechanism is broken down in what is TLS fingerprinting.

The asymmetry favours curl. Because libcurl is a library, the community built patched builds that replicate a browser handshake: curl-impersonate swaps the TLS backend for BoringSSL and reorders extensions, and the Python binding curl_cffi exposes that from code with an impersonate argument. The practical setup is in web scraping with curl-cffi.

wget has no equivalent project. Nothing patches its handshake, and wget 1.x negotiates only HTTP/1.1, so on sites where every real browser arrives over h2, the absence of an HTTP/2 fingerprint is itself the tell. If a target does active TLS fingerprinting, wget is not a tool you can fix. That is the honest limit.

Answer these in order and stop at the first yes.

Already have the URL list and nothing exotic to configure? Either one works. wget -i urls.txt and a curl loop both do the job, and the argument about which is faster is noise next to network latency.

When the last row of that table applies, swapping tools does not help. The fix is to move rendering, IP rotation, and anti-bot handling behind an endpoint and keep whichever CLI you like. The SparkProxy Scraping API runs the headless browser, picks and rotates the proxy, and retries blocks. Base URL https://scrape.sparkproxy.io/api/v1, authenticated with an X-API-Key header.

# curl builds the query string for you
curl -fsS -G "https://scrape.sparkproxy.io/api/v1" \
     -H "X-API-Key: YOUR_API_KEY" \
     --data-urlencode "url=https://www.sparkproxy.io/pricing" \
     --data-urlencode "render_js=true" \
     --data-urlencode "premium_proxy=true" \
     --data-urlencode "country_code=US" \
     -o pricing.html

wget has no --data-urlencode, so you encode the target URL by hand. Miss that and an & inside the target URL truncates your parameters:

wget -qO pricing.html \
     --header="X-API-Key: YOUR_API_KEY" \
     "https://scrape.sparkproxy.io/api/v1?url=https%3A%2F%2Fsparkproxy.io%2Fpricing&render_js=true&premium_proxy=true&country_code=US"

Skip parsing entirely by passing CSS selectors and reading back JSON:

curl -fsS -G "https://scrape.sparkproxy.io/api/v1" \
     -H "X-API-Key: YOUR_API_KEY" \
     --data-urlencode "url=https://www.sparkproxy.io/pricing" \
     --data-urlencode "render_js=true" \
     --data-urlencode 'extract_rules={"plan": ".plan-name", "price": ".price"}' | jq '.extracted'

One honest caveat: you cannot point wget --mirror at the API, because every target URL has to be wrapped as a parameter and wget would recurse the API's own response links instead. If you need both recursion and rendering, generate the URL list first (a shallow wget --spider -r pass works), then feed each URL through the API individually.