Web Scraping with PHP and Proxies (Guzzle & cURL)

Send a few hundred requests from one IP and the target starts rate-limiting you, serving CAPTCHAs, or blocking the address outright. A proxy puts a different IP between your PHP process and the site, and a pool of them spreads traffic so no single address crosses the site's detection threshold. If you're new to the mechanics, start with what web scraping is and the field guide on how to avoid getting your proxy blocked.

PHP gives you three sensible ways to make the request. Pick by how much control and defense you need.

Most real scrapers use two of these at once: Guzzle for the bulk of simple pages, and a scraping API for the handful of targets that fight back. The web scraping API vs self-managed proxies breakdown covers when each pays off.

The curl extension ships with almost every PHP install, so a php curl proxy request needs no Composer packages at all. Set the proxy host and the credentials, then run the transfer.

<?php
$ch = curl_init('https://www.sparkproxy.io/ip');

curl_setopt_array($ch, [
    CURLOPT_RETURNTRANSFER => true,
    CURLOPT_PROXY          => 'proxy-1.sparkproxy.io:10000',
    CURLOPT_PROXYUSERPWD   => 'user:pass',
    CURLOPT_TIMEOUT        => 30,
    CURLOPT_FOLLOWLOCATION => true,
]);

$body = curl_exec($ch);
if ($body === false) {
    fwrite(STDERR, 'cURL error: ' . curl_error($ch) . "\n");
}
curl_close($ch);
echo $body;

For an https:// target, you don't do anything special: cURL opens a CONNECT tunnel to the proxy and carries your TLS handshake end to end to the real site. That's why you keep certificate verification on, which the authentication and errors sections come back to.

SOCKS5 is where a subtle bug lives. CURLOPT_PROXYTYPE controls the proxy protocol, and the value you choose decides who resolves the hostname.

<?php
$ch = curl_init('https://www.sparkproxy.io/ip');

curl_setopt_array($ch, [
    CURLOPT_RETURNTRANSFER => true,
    CURLOPT_PROXY          => 'proxy-1.sparkproxy.io:20000',
    CURLOPT_PROXYUSERPWD   => 'user:pass',
    // Resolve DNS on the proxy, not on your machine.
    CURLOPT_PROXYTYPE      => CURLPROXY_SOCKS5_HOSTNAME,
]);

echo curl_exec($ch);
curl_close($ch);

Use CURLPROXY_SOCKS5_HOSTNAME. Plain CURLPROXY_SOCKS5 does the DNS lookup on your own machine before connecting, which leaks the target domain to your local resolver and, worse, defeats geo-targeting because the lookup happens from your country instead of the exit's. The curl CURLOPT_PROXYTYPE manual documents the split. It's a one-constant fix that most PHP tutorials never mention.

Guzzle 7 is the standard PHP HTTP client, and its guzzle proxy support is a single request option. Install it alongside the parsers you'll need later.

composer require guzzlehttp/guzzle symfony/dom-crawler symfony/css-selector

The simplest form passes one proxy URL to the client, and every request inherits it.

<?php
require 'vendor/autoload.php';

use GuzzleHttp\Client;

$client = new Client([
    'timeout' => 30,
    'proxy'   => 'http://user:pass@proxy-1.sparkproxy.io:10000',
]);

$res = $client->get('https://www.sparkproxy.io/ip');
echo $res->getBody();

The proxy option also takes an array so you can route HTTP and HTTPS through different proxies and bypass internal hosts entirely.

<?php
$client = new Client([
    'proxy' => [
        'http'  => 'http://user:pass@proxy-1.sparkproxy.io:10000',
        'https' => 'http://user:pass@proxy-1.sparkproxy.io:10000',
        'no'    => ['localhost', '127.0.0.1', 'internal.sparkproxy.io'],
    ],
]);

Now the trap that eats the most debugging time. Guzzle's default handler is libcurl, and when you don't set a proxy explicitly, libcurl reads the http_proxy, https_proxy, and all_proxy environment variables on its own. So a HTTPS_PROXY your CI runner exported, or a leftover in your shell, silently routes every request through a proxy you never configured, or sends them direct when you thought you'd set one. The curl CURLOPT_PROXY manual confirms the env-var fallback. If you want to guarantee a direct connection, set the option to an empty string:

$res = $client->get('https://www.sparkproxy.io/ip', ['proxy' => '']);

Passing proxy on the request always wins over both the constructor and the environment, which is exactly what you want once you start rotating.

Most paid proxies use one of two auth methods: username and password, or IP whitelisting. For a php proxy with credentials, cURL and Guzzle put them in different places but mean the same thing.

In raw cURL, credentials go in CURLOPT_PROXYUSERPWD as user:pass:

curl_setopt($ch, CURLOPT_PROXYUSERPWD, 'user:pass');

In Guzzle, they go inline in the proxy URL, user:pass@host:port:

$client->get('https://www.sparkproxy.io/ip', [
    'proxy' => 'http://user:pass@proxy-1.sparkproxy.io:10000',
]);

Two things bite people here. If your username or password contains a special character such as @, :, or #, URL-encode it with rawurlencode() before splicing it into a Guzzle proxy URL, otherwise the string parses wrong and you get a 407. And if you whitelist your server's IP in the dashboard instead of using credentials, drop the user:pass entirely and send http://proxy-1.sparkproxy.io:10000. A 407 Proxy Authentication Required on a whitelisted setup usually means the request came from an IP you never added, which is common on multi-homed or cloud hosts with more than one egress address.

One IP hitting a site a thousand times is a pattern. A guzzle rotating proxy pool turns that into a thousand requests from many IPs. The core is a rotator that hands out the next proxy and can bench one that fails. This class covers round-robin selection plus a cooldown map so a dead IP stops poisoning the next call.

<?php
class ProxyPool
{
    private array $proxies;
    private int $index = 0;
    private array $benched = [];  // proxy url => unix time it may be used again

    public function __construct(array $proxies)
    {
        $this->proxies = $proxies;
    }

    private function live(): array
    {
        $now = time();
        return array_values(array_filter(
            $this->proxies,
            fn (string $p): bool => ($this->benched[$p] ?? 0) < $now
        ));
    }

    public function next(): string
    {
        $live = $this->live();
        if (!$live) {
            throw new RuntimeException('No live proxies available');
        }
        $proxy = $live[$this->index % count($live)];
        $this->index++;
        return $proxy;
    }

    public function bench(string $proxy, int $seconds = 60): void
    {
        $this->benched[$proxy] = time() + $seconds;
    }
}

Wire the pool into any request by passing proxy per call. A fresh IP goes out with each one.

$pool = new ProxyPool([
    'http://user:pass@proxy-1.sparkproxy.io:10000',
    'http://user:pass@proxy-2.sparkproxy.io:10000',
    'http://user:pass@proxy-3.sparkproxy.io:10000',
]);

$res = $client->get('https://www.sparkproxy.io/ip', ['proxy' => $pool->next()]);

The bench() method is the part most homemade pools skip. When a proxy times out or returns a 407, benching it for 60 seconds keeps it out of the rotation until it's had a rest, and live() shrinks the pool automatically. When every proxy is benched, next() throws, which is your signal to widen the pool or slow down. If you also write Python, the same round-robin and cooldown idea maps directly onto the approach in using proxies with requests and aiohttp; the difference is PHP runs this synchronously by default, which the concurrency section fixes.

Proxies fail. One times out, another returns 429, a third drops the socket mid-transfer. The pattern that survives production is: catch the failure, rotate to a new proxy, and back off before retrying so you don't hammer a rate limit into a longer ban.

<?php
use GuzzleHttp\Client;
use GuzzleHttp\Exception\ConnectException;
use GuzzleHttp\Exception\RequestException;

function fetch_with_rotation(Client $client, ProxyPool $pool, string $url, int $maxRetries = 4): string
{
    $retryable = [408, 429, 500, 502, 503, 504];
    $lastError = 'unknown';

    for ($attempt = 0; $attempt <= $maxRetries; $attempt++) {
        $proxy = $pool->next();               // fresh proxy on every attempt
        try {
            $res = $client->get($url, [
                'proxy'       => $proxy,
                'http_errors' => false,        // inspect the status ourselves
                'timeout'     => 30,
            ]);
            $status = $res->getStatusCode();
            if ($status < 400 || !in_array($status, $retryable, true)) {
                return (string) $res->getBody();
            }
            $lastError = "HTTP $status";
        } catch (ConnectException | RequestException $e) {
            $lastError = $e->getMessage();     // timeout, reset, 407
            $pool->bench($proxy);              // rest the bad proxy
        }

        if ($attempt < $maxRetries) {
            $backoffMs = min(1000 * (2 ** $attempt), 15000) + random_int(0, 250);
            usleep($backoffMs * 1000);         // usleep takes MICROseconds
        }
    }

    throw new RuntimeException("All retries failed for $url: $lastError");
}

Three decisions make this reliable. Rotate the proxy on every attempt, not just the first, so a dead IP never gets a second try in the same call. Set http_errors to false so a 503 returns a response you can branch on instead of throwing a GuzzleHttp\Exception\ServerException you have to catch separately. And add jitter with random_int(0, 250) so that when fifty concurrent requests all hit a 429 at once, they don't retry in lockstep and re-trigger the limit. One easy-to-miss detail: usleep() counts microseconds, so a 2000 ms backoff is usleep(2000 * 1000). Doubling the delay (1s, 2s, 4s, 8s) capped at 15 seconds keeps retries polite.

If you prefer Guzzle's own middleware, GuzzleHttp\Middleware::retry() on a HandlerStack handles the backoff, though rotating the proxy per attempt still means swapping the request option inside the retry decider. The manual loop above is easier to reason about and rotates cleanly.

A raw HTML string isn't data yet. Symfony DomCrawler turns it into a queryable document, and with symfony/css-selector installed you get CSS selectors instead of raw XPath. This is the cleanest way to extract structured records once you scrape with PHP.

<?php
use Symfony\Component\DomCrawler\Crawler;

$html = fetch_with_rotation($client, $pool, 'https://www.sparkproxy.io/blog');

$crawler = new Crawler($html);

$posts = $crawler->filter('article.post')->each(function (Crawler $node): array {
    return [
        'title' => trim($node->filter('h2')->text('')),
        'url'   => $node->filter('a')->attr('href'),
        'date'  => $node->filter('time')->attr('datetime'),
    ];
});

print_r($posts);

filter() needs symfony/css-selector; without it you'll see an error telling you to install it, or you fall back to filterXPath(). The small trick worth knowing is passing a default to text(''). In Symfony 5.3 and later, text() throws an InvalidArgumentException when the node is empty or missing, and the empty-string default turns that into a safe fallback so one missing element doesn't kill the whole scrape. Wrap per-record extraction in the closure, and a malformed row returns partial data instead of aborting the batch.

PHP runs synchronously by default, so a thousand pages fetched one after another is slow. Guzzle's promise API sends many requests at once over the same rotating pool. The simplest form fires a batch and collects every result, success or failure, without throwing.

<?php
use GuzzleHttp\Client;
use GuzzleHttp\Promise\Utils;

$client = new Client(['timeout' => 30, 'http_errors' => false]);

$urls = [
    'https://www.sparkproxy.io/blog/what-is-web-scraping',
    'https://www.sparkproxy.io/blog/how-to-avoid-getting-your-proxy-blocked',
    // ...hundreds more
];

$promises = [];
foreach ($urls as $url) {
    $promises[$url] = $client->getAsync($url, ['proxy' => $pool->next()]);
}

$results = Utils::settle($promises)->wait();  // settle() never rejects

foreach ($results as $url => $result) {
    if ($result['state'] === 'fulfilled') {
        echo "$url -> " . $result['value']->getStatusCode() . "\n";
    } else {
        echo "$url FAILED: " . $result['reason']->getMessage() . "\n";
    }
}

Utils::settle() waits for all promises and reports each as fulfilled or rejected, so one dead proxy can't sink the batch. The catch is that it launches every request at once, which will exhaust sockets and trip rate limits on a large list. For bounded concurrency, use GuzzleHttp\Pool with a request generator so you assign a rotating proxy per request while capping how many run in parallel.

<?php
use GuzzleHttp\Pool;

$requests = function () use ($urls, $client, $pool) {
    foreach ($urls as $url) {
        yield fn () => $client->getAsync($url, ['proxy' => $pool->next()]);
    }
};

$runner = new Pool($client, $requests(), [
    'concurrency' => 10,  // at most 10 requests in flight at once
    'fulfilled'   => function ($response, $i) { /* store the body */ },
    'rejected'    => function ($reason, $i)   { /* log and move on */ },
]);

$runner->promise()->wait();

The generator matters: it yields one promise at a time so Pool starts a new request only as an old one finishes, holding steady at ten in flight. Ten concurrent workers over a healthy pool clears thousands of pages in a fraction of the sequential time, and each still leaves on its own proxy.

Running your own pool gives you full control. Sometimes you'd rather not. The SparkProxy Scraping API handles rotation, retries, headless rendering, and anti-bot server-side, so from PHP you send one request and get the response. Authentication is the X-API-Key header carrying a key of the form sk-... from your dashboard.

<?php
use GuzzleHttp\Client;

$client = new Client(['base_uri' => 'https://scrape.sparkproxy.io']);

$res = $client->get('/api/v1', [
    'headers' => [
        'X-API-Key' => getenv('SPARKPROXY_API_KEY'),  // key format: sk-...
    ],
    'query' => [
        'url'           => 'https://example.com/products',
        'render_js'     => 'true',  // run a real browser for JS-heavy pages
        'premium_proxy' => 'true',  // route through residential IPs
        'country_code'  => 'us',    // geo-target the exit IP
    ],
    'timeout' => 90,
]);

echo $res->getBody();

render_js runs a headless browser for pages that build their content with JavaScript, premium_proxy upgrades to residential exits for tougher targets, and country_code picks the exit geography. For structured output without touching DomCrawler, POST an extract_rules object and the API returns parsed JSON. Here it is in raw cURL so you can see the wire format.

<?php
$payload = json_encode([
    'url'           => 'https://example.com/products',
    'render_js'     => true,
    'premium_proxy' => true,
    'country_code'  => 'us',
    'extract_rules' => [
        'title' => 'h1',
        'price' => '.price',
        'links' => ['selector' => 'a', 'type' => 'list'],
    ],
]);

$ch = curl_init('https://scrape.sparkproxy.io/api/v1');
curl_setopt_array($ch, [
    CURLOPT_RETURNTRANSFER => true,
    CURLOPT_POST           => true,
    CURLOPT_POSTFIELDS     => $payload,
    CURLOPT_HTTPHEADER     => [
        'Content-Type: application/json',
        'X-API-Key: ' . getenv('SPARKPROXY_API_KEY'),
    ],
]);

$json = curl_exec($ch);
curl_close($ch);
echo $json;

Because the API assigns a fresh IP per call and renders JavaScript for you, the pool, retry loop, and DomCrawler parsing above become optional for those targets. A practical split: run your own ProxyPool and Guzzle for high-volume, simple pages where per-request cost matters, and send the JavaScript-heavy or heavily defended pages to the Scraping API.