Java Web Scraping Proxy: Jsoup and HttpClient Guide

Java is a solid scraping platform: real threads, mature HTTP stacks in both HttpURLConnection and java.net.http.HttpClient, and Jsoup, which is the best HTML parser on the JVM. What Java cannot do is hide that all your requests come from one IP. Fire a few hundred requests at one host from a single address and the site rate-limits you, serves a CAPTCHA, or bans the IP outright. A proxy pool spreads that traffic across many IPs so no single address crosses the target's detection threshold. If you want the ground-level concept first, see what web scraping is.

You have three practical ways to attach proxies to a Java scraper, and they suit different jobs:

Most real projects mix them. Datacenter proxies handle high-volume fetching of simple pages cheaply, which is why they are the workhorse for web scraping with datacenter proxies, and a managed API takes the handful of targets that are not worth the fight. Deciding where that line sits is the whole scraping API versus self-managed proxies question. The sections below cover every layer.

The fastest way to route a Java process through a proxy is the JVM's built-in system properties. Jsoup uses HttpURLConnection under the hood, and HttpURLConnection reads these properties automatically.

public class SystemProxyExample {
    public static void main(String[] args) throws Exception {
        // For https:// targets:
        System.setProperty("https.proxyHost", "proxy-1.sparkproxy.io");
        System.setProperty("https.proxyPort", "10000");

        // For http:// targets (separate settings, easy to forget):
        System.setProperty("http.proxyHost", "proxy-1.sparkproxy.io");
        System.setProperty("http.proxyPort", "10000");

        org.jsoup.nodes.Document doc =
            org.jsoup.Jsoup.connect("https://www.sparkproxy.io").get();
        System.out.println(doc.title());
    }
}

Two things bite people here. First, http.proxyHost and https.proxyHost are separate settings. Set only the HTTP pair, then scrape an HTTPS site, and your request quietly goes out on your real IP with no error. Set both. Second, these properties are global for the entire JVM. Every library that uses HttpURLConnection now routes through that one proxy, and there is no per-request rotation. System properties are fine for a single-proxy batch job. For anything that rotates IPs or needs credentials per pool, use the per-connection approach instead. You can pass the same values on the command line with -Dhttps.proxyHost=... -Dhttps.proxyPort=... if you would rather keep them out of code.

Jsoup added a per-connection proxy in version 1.14.1, and it is the right tool when each request may need a different IP. The current release is 1.18.1. There are two overloads: host plus port, or a java.net.Proxy object.

import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;

Document doc = Jsoup.connect("https://www.sparkproxy.io/ip")
        .proxy("proxy-1.sparkproxy.io", 10000)   // host, port
        .userAgent("Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
                 + "AppleWebKit/537.36 (KHTML, like Gecko) "
                 + "Chrome/125.0.0.0 Safari/537.36")
        .timeout(30_000)
        .get();

System.out.println(doc.body().text());

The java.net.Proxy overload is what you want for rotation, because you can build a list of Proxy objects up front and hand a different one to each request. It is also the only way to use a SOCKS proxy, since the host/port overload assumes HTTP.

import java.net.InetSocketAddress;
import java.net.Proxy;

// HTTP proxy
Proxy http = new Proxy(Proxy.Type.HTTP,
        new InetSocketAddress("proxy-1.sparkproxy.io", 10000));

// SOCKS5 proxy (use Type.SOCKS, not Type.HTTP)
Proxy socks = new Proxy(Proxy.Type.SOCKS,
        new InetSocketAddress("proxy-1.sparkproxy.io", 1080));

Document doc = Jsoup.connect("https://www.sparkproxy.io").proxy(http).get();

Note what is missing: neither overload takes a username or password. Jsoup gives you no way to pass proxy credentials through .proxy(). That limitation trips up almost everyone the first time they point Jsoup at a commercial proxy, and the fix is the next section.

Commercial proxies need credentials. Because .proxy() has no auth argument, Java authenticates the proxy through a global Authenticator. Register one and it supplies the username and password when the proxy asks for them.

import java.net.Authenticator;
import java.net.PasswordAuthentication;

Authenticator.setDefault(new Authenticator() {
    @Override
    protected PasswordAuthentication getPasswordAuthentication() {
        // Guard on PROXY so you never leak proxy creds to the target site.
        if (getRequestorType() == RequestorType.PROXY) {
            return new PasswordAuthentication("your-user",
                    "your-pass".toCharArray());
        }
        return null;
    }
});

That alone works for plain http:// targets and fails silently on https:// ones, which is the trap. Since JDK 8u111, Java ships with jdk.http.auth.tunneling.disabledSchemes set to Basic, which disables Basic authentication on the CONNECT tunnel used to reach HTTPS sites through a proxy. So your HTTP scraping authenticates fine, your HTTPS scraping returns 407 Proxy Authentication Required, and nothing in the error hints at why. Clear the property before the first request.

public class AuthenticatedProxy {
    public static void main(String[] args) throws Exception {
        // Re-enable Basic proxy auth on HTTPS CONNECT tunnels.
        System.setProperty("jdk.http.auth.tunneling.disabledSchemes", "");

        Authenticator.setDefault(new Authenticator() {
            @Override
            protected PasswordAuthentication getPasswordAuthentication() {
                if (getRequestorType() == RequestorType.PROXY) {
                    return new PasswordAuthentication("your-user",
                            "your-pass".toCharArray());
                }
                return null;
            }
        });

        org.jsoup.nodes.Document doc = org.jsoup.Jsoup
                .connect("https://www.sparkproxy.io/ip")
                .proxy("proxy-1.sparkproxy.io", 10000)
                .get();
        System.out.println(doc.body().text());
    }
}

Set the property before any HTTP request runs, because the tunneling scheme list is read once and cached. The RequestorType.PROXY guard is not optional. Without it, your Authenticator hands the proxy username and password to every server you scrape, including the target. That is a real credential leak. Two lines, and authenticated HTTPS proxies work in Jsoup.

Jsoup is a parser first and a fetcher second. Its connection layer is convenient but basic: no HTTP/2, no async, and the awkward global Authenticator dance above. For authenticated rotating pools at scale, a cleaner pattern is to fetch with Java 11's java.net.http.HttpClient and hand the raw HTML to Jsoup.parse. You get connection reuse, HTTP/2, per-client proxy config, and an authenticator scoped to the client instead of the whole JVM.

import java.net.*;
import java.net.http.*;
import java.time.Duration;
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;

public class HttpClientJsoup {
    public static void main(String[] args) throws Exception {
        // Still required for Basic proxy auth over HTTPS.
        System.setProperty("jdk.http.auth.tunneling.disabledSchemes", "");

        HttpClient client = HttpClient.newBuilder()
                .proxy(ProxySelector.of(
                        new InetSocketAddress("proxy-1.sparkproxy.io", 10000)))
                .authenticator(new Authenticator() {
                    @Override
                    protected PasswordAuthentication getPasswordAuthentication() {
                        return new PasswordAuthentication("your-user",
                                "your-pass".toCharArray());
                    }
                })
                .connectTimeout(Duration.ofSeconds(20))
                .build();

        HttpRequest request = HttpRequest.newBuilder(
                        URI.create("https://www.sparkproxy.io"))
                .header("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
                        + "AppleWebKit/537.36 (KHTML, like Gecko) "
                        + "Chrome/125.0.0.0 Safari/537.36")
                .timeout(Duration.ofSeconds(30))
                .GET()
                .build();

        HttpResponse<String> response =
                client.send(request, HttpResponse.BodyHandlers.ofString());

        // Pass a base URI so abs:href and absUrl() resolve relative links.
        Document doc = Jsoup.parse(response.body(), "https://www.sparkproxy.io");
        System.out.println("status=" + response.statusCode()
                + " title=" + doc.title());
    }
}

The base URI argument to Jsoup.parse matters. When you fetch with Jsoup's own connect().get(), Jsoup records the URL and resolves relative links for you. When you parse a string you fetched elsewhere, Jsoup has no idea where the HTML came from, so abs:href and element.absUrl("href") return empty until you pass the base URI as the second argument. Reuse one HttpClient for the whole job; it pools connections internally, and building a new one per request throws that away. The same jdk.http.auth.tunneling.disabledSchemes fix applies here, because HttpClient uses the same tunneling machinery for HTTPS through a proxy.

Rotation is the point of a pool: send each request through a different IP so no single address gets flagged. Java scrapers are usually multithreaded, so the rotation counter is shared state and a plain index++ is a data race. Use AtomicInteger.

import java.net.InetSocketAddress;
import java.net.Proxy;
import java.util.List;
import java.util.concurrent.atomic.AtomicInteger;

public final class ProxyRotator {
    private final List<Proxy> proxies;
    private final AtomicInteger index = new AtomicInteger();

    public ProxyRotator(List<String> hostPorts) {
        this.proxies = hostPorts.stream()
                .map(hp -> {
                    String[] parts = hp.split(":");
                    return new Proxy(Proxy.Type.HTTP, new InetSocketAddress(
                            parts[0], Integer.parseInt(parts[1])));
                })
                .toList();
    }

    public Proxy next() {
        // floorMod, not %, so the index stays valid after the int overflows
        // to Integer.MIN_VALUE (a plain % would go negative and throw).
        int i = Math.floorMod(index.getAndIncrement(), proxies.size());
        return proxies.get(i);
    }
}

Then hand the next proxy to each request:

ProxyRotator rotator = new ProxyRotator(List.of(
        "proxy-1.sparkproxy.io:10000",
        "proxy-2.sparkproxy.io:10000",
        "proxy-3.sparkproxy.io:10000"));

Document doc = Jsoup.connect("https://www.sparkproxy.io")
        .proxy(rotator.next())
        .userAgent(USER_AGENT)
        .timeout(30_000)
        .get();

The Math.floorMod detail is the one most guides skip. AtomicInteger.getAndIncrement() eventually wraps from Integer.MAX_VALUE to Integer.MIN_VALUE, and Integer.MIN_VALUE % size is negative, which throws IndexOutOfBoundsException. Math.floorMod always returns a non-negative result, so the rotation survives billions of requests. Parse the proxy strings into Proxy objects once in the constructor, not on every call, and keep credentials in the JVM-wide Authenticator from the previous section since all IPs in one pool usually share a login. Round-robin is the default; for random or weighted selection, swap the body of next() and keep the AtomicInteger for thread safety.

Jsoup's defaults are tuned for grabbing a small document in a browser-like way, not for scraping. Two defaults will silently break a scraper, and a couple more save you from wasted retries. Set them on every connection.

The maxBodySize trap is the expensive one. A product listing or search results page can easily exceed 2 MB of HTML, and Jsoup cuts it off at the default without any warning. Your selectors return partial data, you assume the site changed its markup, and you burn an hour before checking the body length. Set .maxBodySize(0) for full pages.

Use .execute() instead of .get() when you need the status code, because it returns a Connection.Response you can inspect before parsing.

import org.jsoup.Connection;
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;

Connection.Response res = Jsoup.connect(url)
        .proxy(rotator.next())
        .userAgent(USER_AGENT)
        .timeout(30_000)
        .maxBodySize(0)             // no truncation
        .ignoreHttpErrors(true)     // do not throw on 4xx/5xx
        .followRedirects(true)
        .execute();

if (res.statusCode() == 200) {
    Document doc = res.parse();     // parse only on success
    // ... extract data
}

res.parse() gives you the Document without a second network call, and res.statusCode() lets you route 429 and 403 into your retry logic instead of crashing the run.

Fetching is half the job. Jsoup earns its keep with CSS selectors that read like jQuery. select returns an Elements collection, selectFirst returns a single Element or null, and .text(), .attr(), and .absUrl() pull the data out.

import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;
import org.jsoup.select.Elements;

Document doc = Jsoup.connect("https://www.sparkproxy.io/products")
        .proxy(rotator.next())
        .userAgent(USER_AGENT)
        .maxBodySize(0)
        .get();

// Single value. selectFirst can return null, so guard it.
Element heading = doc.selectFirst("h1");
String title = heading != null ? heading.text() : "";

// A repeating block: loop the Elements collection.
Elements cards = doc.select("div.product-card");
for (Element card : cards) {
    String name  = card.select(".name").text();
    String price = card.select(".price").text();
    // abs:href resolves the relative link against the document base URI.
    String link  = card.select("a").attr("abs:href");
    System.out.printf("%s | %s | %s%n", name, price, link);
}

Two selector facts save time. First, selectFirst returns null when nothing matches, so doc.selectFirst("h1").text() throws a NullPointerException the day a page is missing that element. Guard it, or use doc.select("h1").text(), which returns an empty string for an empty match. Second, use the abs: prefix (or element.absUrl("href")) to turn relative links like /item/42 into full URLs, which only works when the document has a base URI. Jsoup's connect().get() sets that automatically; Jsoup.parse(html) does not unless you pass the base as the second argument. Jsoup's selector syntax also supports :contains(text), attribute matchers like a[href*=product], and positional selectors like li:eq(0), which cover most extraction needs without regex.

Proxies and targets fail. A proxy times out, a site answers 429 Too Many Requests, or an IP gets a 403. Catch it, rotate to a fresh proxy, back off, and retry. Because you pull a new proxy from the rotator each attempt, a banned IP is swapped out automatically.

import org.jsoup.Connection;
import org.jsoup.HttpStatusException;
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import java.io.IOException;
import java.net.SocketTimeoutException;
import java.util.concurrent.ThreadLocalRandom;

public static Document fetch(String url, ProxyRotator rotator, int maxRetries)
        throws IOException {
    IOException last = null;
    for (int attempt = 0; attempt <= maxRetries; attempt++) {
        try {
            Connection.Response res = Jsoup.connect(url)
                    .proxy(rotator.next())      // fresh IP each attempt
                    .userAgent(USER_AGENT)
                    .timeout(30_000)
                    .maxBodySize(0)
                    .ignoreHttpErrors(true)
                    .execute();

            int code = res.statusCode();
            if (code == 200) {
                return res.parse();
            }
            // 403/404 are usually not fixed by retrying the same request,
            // but with rotation a 403 can be an IP block worth one retry.
            last = new HttpStatusException("bad status", code, url);
        } catch (SocketTimeoutException e) {
            last = e;                            // dead or slow proxy
        }
        if (attempt == maxRetries) break;

        long backoff = (1L << attempt) * 1000L;  // 1s, 2s, 4s, 8s
        long jitter  = ThreadLocalRandom.current().nextInt(250);
        try {
            Thread.sleep(backoff + jitter);
        } catch (InterruptedException ie) {
            Thread.currentThread().interrupt();
            throw new IOException("interrupted during backoff", ie);
        }
    }
    throw last != null ? last : new IOException("failed after retries: " + url);
}

Two decisions make this reliable. Double the delay each attempt (1s, 2s, 4s, 8s) so you stop hammering a rate limit, and add a little jitter so a batch of threads that all hit 429 at the same instant do not retry in lockstep and re-trigger it. HttpStatusException extends IOException and carries getStatusCode(), so you can branch on the exact code. Restore the interrupt flag in the InterruptedException branch rather than swallowing it, or you break cancellation for the whole thread pool. For the wider set of signals that get a request blocked in the first place, from TLS fingerprints to header order, see how to avoid getting your proxy blocked.

Running your own pool gives you control and the best cost per request on simple pages. Some targets are not worth it: they render content with JavaScript that Jsoup cannot execute, fingerprint the TLS handshake, or throw hard anti-bot walls. The SparkProxy Scraping API rotates the exit IP server-side on every call and can run a real browser, so there is no pool, no Authenticator, and no retry loop to maintain. You send a target URL and get the response back.

import java.net.URI;
import java.net.URLEncoder;
import java.net.http.*;
import java.nio.charset.StandardCharsets;
import java.time.Duration;
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;

public class SparkProxyApi {
    public static void main(String[] args) throws Exception {
        String apiKey = System.getenv("SPARKPROXY_API_KEY");
        String target = URLEncoder.encode(
                "https://www.sparkproxy.io/products", StandardCharsets.UTF_8);

        String endpoint = "https://scrape.sparkproxy.io/api/v1"
                + "?url=" + target
                + "&render_js=true"       // run a headless browser for JS pages
                + "&premium_proxy=true"   // route through residential IPs
                + "&country_code=us";     // geo-target the exit (ISO 3166-1 alpha-2)

        HttpClient client = HttpClient.newBuilder()
                .connectTimeout(Duration.ofSeconds(90))  // rendered pages are slower
                .build();

        HttpRequest request = HttpRequest.newBuilder(URI.create(endpoint))
                .header("X-API-Key", apiKey)             // auth header from your dashboard
                .timeout(Duration.ofSeconds(120))
                .GET()
                .build();

        HttpResponse<String> res =
                client.send(request, HttpResponse.BodyHandlers.ofString());

        // Feed the returned HTML straight into Jsoup and select as usual.
        Document doc = Jsoup.parse(res.body(), "https://www.sparkproxy.io/products");
        System.out.println(doc.title());
    }
}

Authentication is the X-API-Key header carrying the key from your dashboard. country_code sets the exit geography, render_js runs a headless browser when the page needs JavaScript, and premium_proxy upgrades to residential IPs for tougher targets. Because a fresh IP is assigned per call, the rotation, auth, and retry code above is handled for you. Use a longer timeout, since a rendered page takes longer than a raw fetch.

If you would rather skip Jsoup entirely, the API can parse server-side. Send a POST with extract_rules (CSS selectors) and it returns structured JSON in the extracted field. Text blocks need Java 15 or newer.

String body = """
    {
      "url": "https://www.sparkproxy.io/products",
      "render_js": true,
      "country_code": "us",
      "extract_rules": {
        "title": "h1",
        "prices": { "selector": ".price", "type": "list" }
      }
    }
    """;

HttpRequest request = HttpRequest.newBuilder(
                URI.create("https://scrape.sparkproxy.io/api/v1"))
        .header("X-API-Key", System.getenv("SPARKPROXY_API_KEY"))
        .header("Content-Type", "application/json")
        .POST(HttpRequest.BodyPublishers.ofString(body))
        .build();

A pragmatic split: run your own Jsoup plus proxy pool for high-volume simple pages where per-request cost matters, and send the JavaScript-heavy or heavily defended pages to the API.