Perl Web Scraping With Proxies: LWP and Mojo Guide

Every request carries your IP address. Send too many from one address and the server rate-limits you, then blocks you outright. A proxy sits between your Perl process and the target, swapping your real IP for one from a pool. Rotate that pool and the request pattern that used to look like a single script now looks like ordinary traffic from many machines.

Three problems push most Perl scrapers toward proxies:

• Rate limits and bans. Sites cap requests per IP per minute. Cross the line and you get 429 or 403 no matter how careful your code is.
• Geo-restricted content. Prices, search results, and stock change by country. To read what a visitor in Germany sees, you need a German exit IP.
• IP reputation. Anti-bot systems score the address before they even parse your headers. A datacenter range with a clean history passes where a flagged one does not.

If the mechanics are new to you, what is web scraping covers the fundamentals, and using datacenter proxies for web scraping explains why proxy type matters for the sites you target. For the failure modes specifically, read how to avoid getting your proxy blocked before you scale up.

You want Perl 5.40 or newer (5.42 is the current stable release) and a handful of CPAN modules. LWP::UserAgent and Mojo::UserAgent are the two HTTP clients this guide compares. For HTML you need HTML::TreeBuilder; Mojo::DOM ships inside Mojolicious, so installing Mojolicious covers both the Mojo client and its parser.

List the dependencies in a cpanfile:

# cpanfile
requires 'LWP::UserAgent', '6.77';
requires 'LWP::Protocol::https', '6.14';   # HTTPS support for LWP
requires 'LWP::Protocol::connect';         # tunnel HTTPS through an HTTP proxy
requires 'Mojolicious', '9.39';            # Mojo::UserAgent + Mojo::DOM
requires 'HTML::TreeBuilder', '5.07';      # tree parser with look_down
requires 'Text::CSV_XS';                   # fast CSV output

Install with cpanm and check the versions you actually got:

cpanm --installdeps .
perl -v                                    # v5.40.x or v5.42.x
perl -MMojolicious -e 'print "$Mojolicious::VERSION\n"'
perl -MLWP -e 'print "$LWP::VERSION\n"'

Two of those modules are the ones people forget. Without LWP::Protocol::https, LWP cannot fetch a single HTTPS page and dies with a protocol error. Without LWP::Protocol::connect, LWP cannot tunnel HTTPS through an HTTP proxy, which is the exact setup most scraping uses. Both are covered in the next two sections.

One rule before any of this ships: never hardcode a proxy password or API key. Read them from the environment ($ENV{SPARKPROXY_API_KEY}) so a leaked repo does not leak your account.

LWP::UserAgent is the workhorse of Perl HTTP. For a plain HTTP target, the proxy is one method call:

use strict;
use warnings;
use LWP::UserAgent;

my $ua = LWP::UserAgent->new(
    timeout => 20,
    agent   => 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 '
             . '(KHTML, like Gecko) Chrome/128.0 Safari/537.36',
);

# HTTP targets go straight through the proxy
$ua->proxy('http', 'http://gate.sparkproxy.io:8000');

my $res = $ua->get('http://www.sparkproxy.io/pricing');
if ($res->is_success) {
    print $res->code, "\n";              # 200
    print substr($res->decoded_content, 0, 300), "\n";
} else {
    warn $res->status_line, "\n";
}

Notice the agent argument. LWP's default User-Agent is the string libwww-perl/6.77, and plenty of sites block that token on sight. Setting a current browser agent is the single cheapest anti-block change you can make in Perl, and forgetting it is the most common reason a script that works in your browser returns a 403.

The line above sets the proxy for the http scheme only. That is deliberate, because HTTPS through a proxy in LWP is a different mechanism, and mixing them up is where most people lose an afternoon.

Here is the trap that catches nearly everyone. This looks correct and quietly does the wrong thing:

# WRONG for HTTPS: this does not tunnel, and can leak your real IP
$ua->proxy('https', 'http://gate.sparkproxy.io:8000');

Almost every scraping target is https://, and reaching an HTTPS site through an HTTP proxy needs an HTTP CONNECT tunnel. Plain proxy('https', 'http://...') does not establish that tunnel reliably in LWP. The dependable fix is the connect:// proxy scheme from LWP::Protocol::connect, which tells LWP to open a CONNECT tunnel and run TLS end to end inside it:

use LWP::UserAgent;
use LWP::Protocol::connect;   # registers the connect:// scheme

my $ua = LWP::UserAgent->new(timeout => 20);
$ua->agent('Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 '
         . '(KHTML, like Gecko) Chrome/128.0 Safari/537.36');

# HTTP targets: ordinary proxy. HTTPS targets: CONNECT tunnel.
$ua->proxy('http',  'http://gate.sparkproxy.io:8000');
$ua->proxy('https', 'connect://gate.sparkproxy.io:8000');

my $res = $ua->get('https://www.sparkproxy.io/pricing');
print $res->code, "\n";

The http:// proxy value describes how you reach the proxy. The connect:// value on the https scheme tells LWP to ask that proxy to open a raw tunnel, then carry your TLS handshake through it to the destination. Skip LWP::Protocol::connect and the connect:// line either dies with "Can't locate object method" or silently falls back to a direct connection on your real IP.

There is an environment-variable route too: set $ENV{https_proxy} and call $ua->env_proxy. It is convenient, but the exact tunneling behavior varies by LWP and IO::Socket::SSL version, so for a scraper you want to control the exit IP by hand, prefer the explicit connect:// scheme above.

Mojo::UserAgent, from the Mojolicious distribution, is the modern alternative. It tunnels HTTPS through a proxy natively (no extra CONNECT module), it parses HTML with a built-in CSS engine, and it does non-blocking requests without threads. Setting the proxy takes one chained call:

use Mojo::UserAgent;

my $ua = Mojo::UserAgent->new;

# One call sets both schemes; Mojo handles the HTTPS CONNECT tunnel itself
$ua->proxy->http('http://gate.sparkproxy.io:8000')
          ->https('http://gate.sparkproxy.io:8000');

my $res = $ua->get('https://www.sparkproxy.io/pricing')->result;
print $res->code, "\n";            # 200
print $res->dom->at('title')->text, "\n";

Two Mojo defaults surprise people coming from LWP. First, max_redirects defaults to 0, so Mojo::UserAgent does not follow redirects at all unless you ask. A page that redirects http to https returns a 301 with an empty body until you raise it:

my $ua = Mojo::UserAgent->new(max_redirects => 5, request_timeout => 20);

Second, Mojo's default User-Agent is the string Mojolicious (Perl), which is as obvious a bot signal as LWP's default. Change it once on the transactor and every request carries the new value:

$ua->transactor->name('Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) '
                    . 'AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.0 Safari/605.1.15');

If you would rather read the proxy from the environment, $ua->proxy->detect picks up HTTP_PROXY, HTTPS_PROXY, and NO_PROXY. Set the vars, call detect, and you are done.

Fetching gives you a string of HTML. You need a parser to pull structured data out of it. Perl gives you two good options, and which one you reach for usually follows which client you already picked.

Mojo::DOM comes free with Mojo::UserAgent. A response exposes it through ->dom, and you query with CSS selectors:

my $res = $ua->get('https://www.sparkproxy.io/pricing')->result;

$res->dom->find('div.product-card')->each(sub {
    my $card  = shift;
    my $name  = $card->at('h2.title')  ? $card->at('h2.title')->text  : '';
    my $price = $card->at('span.price') ? $card->at('span.price')->text : '';
    my $link  = $card->at('a')          ? $card->at('a')->attr('href')  : '';
    print "$name | $price | $link\n";
});

find returns a collection you iterate with each; at returns the first match or undef, so guard it before calling text. This is the shortest path from response to data in Perl, because the client and the parser are the same distribution.

HTML::TreeBuilder is the classic tree parser, useful when you are on LWP or need XPath through HTML::TreeBuilder::XPath. You query with look_down, which walks the tree matching tag and attributes:

use HTML::TreeBuilder 5;

my $tree = HTML::TreeBuilder->new_from_content($res->decoded_content);

for my $card ($tree->look_down(_tag => 'div', class => 'product-card')) {
    my $name  = $card->look_down(_tag => 'h2', class => 'title');
    my $price = $card->look_down(_tag => 'span', class => 'price');
    printf "%s | %s\n",
        ($name  ? $name->as_trimmed_text  : ''),
        ($price ? $price->as_trimmed_text : '');
}

$tree->delete;   # free the tree, see the note below

That last line is not optional. HTML::TreeBuilder builds a tree where every node holds a reference to its parent and its children, a circular reference the Perl garbage collector will not reclaim on its own. In a one-shot script it does not matter. In a scraper that loops over ten thousand pages, skipping $tree->delete leaks the memory of every page until the process exits and eventually gets killed by the OOM killer. Call delete when you finish with each tree, or scope it so it is the last thing you use.

Providers authenticate you one of two ways, and your Perl changes with each.

Username and password. You embed credentials in the proxy URL as user:pass@host:port. This works from any machine and any outbound IP, which makes it right for scrapers on ephemeral cloud workers whose IPs change:

# LWP, HTTPS through an authenticated proxy
$ua->proxy('https', 'connect://you%40sparkproxy.io:SECRET@gate.sparkproxy.io:8000');

# Mojo, same idea
$ua->proxy->https('http://you%40sparkproxy.io:SECRET@gate.sparkproxy.io:8000');

Note the %40. When the username is an email address, the @ inside it has to be percent-encoded so the URL parser does not read it as the separator before the host. Encode any reserved character in the user or pass with URI::Escape:

use URI::Escape qw(uri_escape);
my $user = uri_escape('you@sparkproxy.io');       # you%40sparkproxy.io
my $pass = uri_escape($ENV{PROXY_PASS});
my $proxy = "http://$user:$pass\@gate.sparkproxy.io:8000";

IP whitelisting. You register your server's outbound IP in the provider dashboard, then send requests with no credentials in the URL at all. It is simpler and avoids putting a password in your process, but it breaks the moment your server IP changes, so it suits fixed infrastructure. For the deeper mechanics of both schemes, using datacenter proxies for web scraping walks through where each one fits.

Rotation is what actually keeps you unblocked. There are two ways to do it in Perl.

Option A: a rotating gateway. Point every request at one endpoint and let the provider assign a fresh exit IP per request. Your code stays identical to the single-proxy examples above, and the pool is managed for you. Least code, most reliable.

Option B: rotate a list yourself. Hold a list of endpoints and cycle it. Perl has no built-in cycle, so a closure that returns the next entry keeps the state tidy:

sub make_rotator {
    my @list = @_;
    my $n    = 0;
    return sub {
        my $proxy = $list[$n];
        $n = ($n + 1) % @list;   # @list in scalar context is its length
        return $proxy;
    };
}

my $next_proxy = make_rotator(
    'http://gate1.sparkproxy.io:8000',
    'http://gate2.sparkproxy.io:8000',
    'http://gate3.sparkproxy.io:8000',
);

for my $url (@urls) {
    $ua->proxy->https($next_proxy->());   # Mojo: new proxy per request
    my $res = $ua->get($url)->result;
    # parse $res->dom here
    sleep 1 + int(rand 2);                # jittered delay, not a fixed one
}

The sleep 1 + int(rand 2) matters more than it looks. Perfectly even timing is itself a bot signal, so a little jitter between requests makes the traffic read as human.

Now the Perl-specific detail almost every tutorial misses. If you want real concurrency with Mojo, the proxy is set per user-agent, not per request. Set $ua->proxy->https(...) and then fire several non-blocking requests, and they all use whichever proxy was set last, because the calls resolve after your loop finishes. To rotate proxies across concurrent requests, build one Mojo::UserAgent per proxy and spread the URLs across the pool:

use Mojo::UserAgent;
use Mojo::Promise;

my @proxies = (
    'http://gate1.sparkproxy.io:8000',
    'http://gate2.sparkproxy.io:8000',
    'http://gate3.sparkproxy.io:8000',
);

# One UA per proxy: each holds its own exit IP
my @pool = map {
    my $ua = Mojo::UserAgent->new(max_redirects => 5, request_timeout => 30);
    $ua->proxy->http($_)->https($_);
    $ua;
} @proxies;

my @promises;
for my $i (0 .. $#urls) {
    my $ua = $pool[$i % @pool];           # round-robin across the UA pool
    push @promises, $ua->get_p($urls[$i]);
}

Mojo::Promise->all(@promises)->then(sub {
    for my $result (@_) {
        my $tx = $result->[0];
        say $tx->result->code;
    }
})->wait;

This runs all requests on one event loop, no threads or forks, each on its own proxy. Perl threads are heavy and forking a process per request wastes memory; Mojo's promise-based concurrency is the idiomatic way to parallelize a Perl scraper. For managing how hard you hit each IP, pair this with the anti-block tactics in how to avoid getting your proxy blocked.

Even with rotation, requests fail. Connections reset, proxies time out, and a 429 on one IP clears on a retry through another. Wrap the fetch in retry logic with exponential backoff, and read status codes as signals rather than plain success or failure:

sub fetch_with_retry {
    my ($ua, $url, $max) = @_;
    $max //= 3;

    for my $attempt (1 .. $max) {
        my $res = $ua->get($url);

        return $res if $res->is_success;          # 2xx, done

        my $code = $res->code;
        return $res if $code == 404;              # hard miss, do not retry

        if ($code == 403 || $code == 429 || $code >= 500) {
            sleep 2 ** $attempt;                  # 2s, 4s, 8s
            next;                                 # retry on the next proxy
        }
        return $res;                              # anything else, give up
    }
    return;                                        # out of attempts
}

Two more fixes stop most blocks before retries even matter. Send a real User-Agent, as shown in every example above, because both LWP and Mojo announce themselves by default. And match the IP to the content: if a page throws a CAPTCHA on a datacenter IP, another datacenter IP rarely helps, which is the point to switch to residential IPs.

When JavaScript renders the data you need, none of this reaches it. LWP and Mojo fetch raw HTML and do not run scripts. For single-page apps you either drive a headless browser or use an API that renders for you, which is the next section.

Managing pools, headless browsers, and CAPTCHA logic is a project of its own. The SparkProxy Scraping API collapses that into one HTTP call: send a URL, and it handles proxy rotation, JavaScript rendering, and anti-bot handling, then returns the HTML. There is no proxy to set on your client, because the rotation happens server-side.

The endpoint is https://scrape.sparkproxy.io/api/v1, and you authenticate with an X-API-Key header carrying a key of the form sk-... from your dashboard. The parameters you reach for most:

From LWP, build the query with URI and set the header on the request:

use LWP::UserAgent;
use URI;

my $ua  = LWP::UserAgent->new(timeout => 60);
my $api = URI->new('https://scrape.sparkproxy.io/api/v1');
$api->query_form(
    url           => 'https://www.sparkproxy.io/pricing',
    render_js     => 'true',
    premium_proxy => 'true',
    country_code  => 'US',
);

my $res = $ua->get($api, 'X-API-Key' => $ENV{SPARKPROXY_API_KEY});
die $res->status_line unless $res->is_success;
print substr($res->decoded_content, 0, 300), "\n";

From Mojo the same call reads cleaner, and you get Mojo::DOM on the response for free:

use Mojo::UserAgent;
use Mojo::URL;

my $ua  = Mojo::UserAgent->new(request_timeout => 60);
my $url = Mojo::URL->new('https://scrape.sparkproxy.io/api/v1')->query(
    url           => 'https://www.sparkproxy.io/pricing',
    render_js     => 1,
    premium_proxy => 1,
    country_code  => 'US',
);

my $res = $ua->get($url => { 'X-API-Key' => $ENV{SPARKPROXY_API_KEY} })->result;
say $res->dom->at('h1')->text if $res->is_success;

Set json_response => 1 when you want metadata alongside the page. The envelope carries status_code, duration_ms, and credits_used, with the page itself base64-encoded in body:

use Mojo::JSON qw(decode_json);
use MIME::Base64 qw(decode_base64);

my $api = Mojo::URL->new('https://scrape.sparkproxy.io/api/v1')
    ->query(url => 'https://www.sparkproxy.io/pricing', json_response => 1);

my $res     = $ua->get($api => { 'X-API-Key' => $ENV{SPARKPROXY_API_KEY} })->result;
my $payload = decode_json($res->body);

say $payload->{status_code};                  # 200
say $payload->{credits_used};
my $html = decode_base64($payload->{body});   # the rendered page

When does this beat raw proxies? When the target renders with JavaScript, throws CAPTCHAs, or fingerprints aggressively, the API is usually cheaper than the engineering time you would spend fighting it. When you scrape simple, static HTML at high volume, raw datacenter proxies are more economical. Web scraping API vs self-managed proxies breaks that trade-off down with numbers.

Here is the whole thing wired together: fetch through the API with rendering and geo-targeting, parse with Mojo::DOM, and write rows to CSV. Swap the selectors for your target's markup.

#!/usr/bin/env perl
use v5.36;                       # strict, warnings, say, signatures
use Mojo::UserAgent;
use Mojo::URL;
use Text::CSV_XS;

my $KEY = $ENV{SPARKPROXY_API_KEY} or die "set SPARKPROXY_API_KEY\n";
my $ua  = Mojo::UserAgent->new(request_timeout => 60);

sub scrape ($target, $country = 'US') {
    my $url = Mojo::URL->new('https://scrape.sparkproxy.io/api/v1')->query(
        url           => $target,
        render_js     => 1,
        premium_proxy => 1,
        country_code  => $country,
    );
    my $res = $ua->get($url => { 'X-API-Key' => $KEY })->result;
    die 'API error ' . $res->code unless $res->is_success;
    return $res->dom;
}

my @pages = (
    'https://www.sparkproxy.io/products?page=1',
    'https://www.sparkproxy.io/products?page=2',
);

my $csv = Text::CSV_XS->new({ binary => 1, eol => "\n" });
open my $fh, '>:encoding(UTF-8)', 'results.csv' or die "results.csv: $!\n";
$csv->say($fh, [qw(name price url)]);

for my $page (@pages) {
    my $dom   = scrape($page);
    my $count = 0;
    $dom->find('div.product-card')->each(sub ($card, @) {
        my $name  = $card->at('h2.title')   ? $card->at('h2.title')->text   : '';
        my $price = $card->at('span.price') ? $card->at('span.price')->text : '';
        my $link  = $card->at('a')          ? $card->at('a')->attr('href')  : '';
        $csv->say($fh, [$name, $price, $link]);
        $count++;
    });
    say "$page: $count rows";
    sleep 1 + int(rand 2);
}
close $fh;

That is a working scraper in under 40 lines. The API absorbs the proxy rotation and rendering, Mojo::DOM does the extraction, and Text::CSV_XS handles output with proper quoting and UTF-8. Start from this, adjust the selectors, and you have a Perl scraper you can maintain.