Java Proxy Settings: Fix JVM 407 and Ignored Proxies

Match your symptom to a row before changing anything. The exact text tells you which layer failed, and the strings below are copied from the OpenJDK source (read on 2026-10-30), not paraphrased.

The last row is the nastiest because nothing fails. The HTTPS 407 rows are the most common, and they are the reason this post exists. If you only need a refresher on what a 407 challenge is, the proxy authentication explainer covers the header exchange; here we stay inside the JVM.

Work down this list. Each step takes under a minute and rules out a whole layer.

1. Prove the credentials outside Java. If curl fails too, no JVM setting will help.
2. Ask the JVM which proxy it would pick for your exact URL.
3. Confirm the client you use actually reads that choice. HttpClient, HttpURLConnection and OkHttp disagree on this.
4. Confirm your Authenticator is attached to that client and returns credentials for the proxy.
5. Check jdk.http.auth.tunneling.disabledSchemes if only HTTPS URLs fail.
6. Turn on client logging and read what was actually sent.

Step 1, against the SparkProxy rotating gateway:

curl -sS -o /dev/null -w "%{http_code}\n" \
  -x http://gateway.sparkproxy.io:11000 \
  -U "$SPARK_USER:$SPARK_PASS" \
  https://www.sparkproxy.io/

A 200 here and a 407 from Java means the credentials are fine and the problem is JVM configuration. A 407 from curl too means the credentials, the username format, or the IP whitelist on your account is wrong, and you should fix that first. curl sends Basic on CONNECT without complaint, which is exactly why "it works in curl" is so misleading for Java developers.

Step 2 is a three-line program you can drop into any project:

import java.net.ProxySelector;
import java.net.URI;

public class WhichProxy {
    public static void main(String[] args) {
        URI target = URI.create(args.length > 0 ? args[0] : "https://www.sparkproxy.io/");
        System.out.println(ProxySelector.getDefault().select(target));
    }
}

Run it with the same flags as your app. [DIRECT] means the JVM will not use a proxy for that URL, whatever you think you configured. [HTTP @ gateway.sparkproxy.io/:11000] means the default selector found it (that is the literal output on JDK 21; the address stays unresolved until connect time). [SOCKS @ ...] needs special care, covered in the SOCKS5 section.

Most "Java ignores my proxy" reports come down to this table. Each row is taken from the JDK 21 API documentation or the client's own source, read 2026-10-30.

Three facts in that table cause most of the trouble.

HttpClient does read the proxy system properties. The HttpClient.Builder.proxy Javadoc states that a client built without proxy(...) uses the default proxy selector, which "supports a set of system properties related to proxy settings." Passing HttpClient.Builder.NO_PROXY or your own ProxySelector.of(...) replaces that behaviour.

HttpClient does not fall back to the JVM-wide Authenticator. HttpClient.authenticator() returns an empty Optional when none was set on the builder, and the authentication filter in the OpenJDK source has the comment "if no authenticator, let the user deal with 407/401." So code that calls Authenticator.setDefault(...) and then HttpClient.newHttpClient() will get a 407 back even with Basic fully enabled.

OkHttp is a different stack. It picks up -Dhttps.proxyHost through ProxySelector.getDefault(), but proxy credentials go through OkHttpClient.Builder.proxyAuthenticator, and the JDK's tunneling property does nothing to it. The Kotlin scraping guide shows the OkHttp proxyAuthenticator pattern, so it is not repeated here.

The JDK 21 "Networking Properties" page and the shipped conf/net.properties file define these. Here is what each one does and the trap attached to it.

Port defaults. The default selector's source (sun.net.spi.DefaultProxySelector) looks for https.proxyPort, then the legacy proxyPort, then falls back to 443. We ran WhichProxy on JDK 21.0.13 with only -Dhttps.proxyHost=gateway.sparkproxy.io and it printed [HTTP @ gateway.sparkproxy.io/:443]. SparkProxy's HTTP gateway listens on 11000, so a missing port produces a connect timeout that looks like a network problem.

Fallback order. For an https:// URL the selector tries https.proxyHost, then the legacy proxyHost, then socksProxyHost. With only -DsocksProxyHost=gateway.sparkproxy.io -DsocksProxyPort=13000, the same program printed [SOCKS @ gateway.sparkproxy.io/:13000] for an HTTPS URL. Keep that in mind for the SOCKS section.

nonProxyHosts semantics. Entries are separated by | and is a wildcard. When you set a non-empty value, the JDK appends the loopback defaults for you. When you set it to an empty string, the defaults are dropped and even localhost goes through the proxy. A stray in a value copied from a corporate template matches every host and turns the proxy off without a word.

Set the properties on the command line, not in code, unless you have a reason:

java \
  -Dhttps.proxyHost=gateway.sparkproxy.io -Dhttps.proxyPort=11000 \
  -Dhttp.proxyHost=gateway.sparkproxy.io  -Dhttp.proxyPort=11000 \
  "-Dhttp.nonProxyHosts=*.internal.sparkproxy.io|metadata.google.internal" \
  -jar app.jar

Quote the nonProxyHosts argument. In bash and in Windows cmd.exe an unquoted | is a pipe, and the shell will try to run whatever follows it as a command.

When you cannot edit the launch command (a container base image, an IDE run config, a vendor launcher), use JAVA_TOOL_OPTIONS. HotSpot prints Picked up JAVA_TOOL_OPTIONS: ... on stderr at startup, which doubles as your check that the flags arrived:

export JAVA_TOOL_OPTIONS="-Dhttps.proxyHost=gateway.sparkproxy.io -Dhttps.proxyPort=11000 -Djdk.http.auth.tunneling.disabledSchemes="
java -jar app.jar 2>&1 | head -1

Gradle users hit one more trap. org.gradle.jvmargs in gradle.properties configures the Gradle daemon, not your application. For ./gradlew run, put the flags on the application itself:

// build.gradle.kts
application {
    mainClass.set("io.sparkproxy.demo.MainKt")
    applicationDefaultJvmArgs = listOf(
        "-Dhttps.proxyHost=gateway.sparkproxy.io",
        "-Dhttps.proxyPort=11000",
        "-Djdk.http.auth.tunneling.disabledSchemes="
    )
}

Check: rerun WhichProxy with the same flags. With the command above it printed [HTTP @ gateway.sparkproxy.io/:11000] for https://www.sparkproxy.io/ and [DIRECT] for https://db.internal.sparkproxy.io/ in our run.

The proxy host properties carry no credentials. There is no https.proxyUser property in the JDK, despite how often it appears in answers online. Credentials come from a java.net.Authenticator, and the shape of that class matters more than people expect.

import java.net.Authenticator;
import java.net.PasswordAuthentication;

public final class GatewayAuthenticator extends Authenticator {
    private final String proxyHost;
    private final String user;
    private final char[] pass;

    public GatewayAuthenticator(String proxyHost, String user, String pass) {
        this.proxyHost = proxyHost;
        this.user = user;
        this.pass = pass.toCharArray();
    }

    @Override
    protected PasswordAuthentication getPasswordAuthentication() {
        boolean isOurProxy = proxyHost.equalsIgnoreCase(getRequestingHost());
        boolean httpProxyAsk = getRequestorType() == RequestorType.PROXY;
        boolean socksAsk = "SOCKS5".equals(getRequestingProtocol());
        if (isOurProxy && (httpProxyAsk || socksAsk)) {
            return new PasswordAuthentication(user, pass);
        }
        return null; // never hand proxy credentials to a target site
    }
}

Two guards, each for a reason you can verify in the JDK source.

The host check stops your gateway password from being sent to any site that answers with 401 and a Basic challenge. Without it, a scraped page can harvest your proxy credentials. For HttpClient, the requesting host on a proxy challenge is the proxy's host string, taken from the address you passed to ProxySelector.of(...).

The SOCKS clause exists because java.net.SocksSocketImpl asks the Authenticator with protocol "SOCKS5" and never sets a requestor type, so the type stays at its reset value, RequestorType.SERVER. The common snippet that returns credentials only when getRequestorType() == RequestorType.PROXY therefore refuses every SOCKS5 request. The JDK then falls back to java.net.socks.username and, failing that, to your OS login name with no password, and the server answers with an authentication failure.

Check: add a temporary System.err.println(getRequestorType() + " " + getRequestingProtocol() + " " + getRequestingHost()) inside the method. If it never prints, the Authenticator is not attached to the client you are using. If it prints and the request still fails, read the next section.

This is the gotcha in the title. Oracle's JDK 8u111 release notes say the Basic scheme "has been deactivated, by default, in the Oracle Java Runtime, by adding Basic to the jdk.http.auth.tunneling.disabledSchemes networking property." Current OpenJDK still ships it. The conf/net.properties file in the OpenJDK master branch contains:

#jdk.http.auth.proxying.disabledSchemes=
jdk.http.auth.tunneling.disabledSchemes=Basic

The effect is narrow and confusing. Plain http:// requests through the proxy authenticate fine, because they use the proxying list, which is empty. Every https:// request needs a CONNECT tunnel first, uses the tunneling list, and finds Basic disabled. User:pass auth on an HTTP proxy gateway, SparkProxy's included, arrives as a Basic challenge. So HTTP works, HTTPS gets 407, and the credentials were never the problem. The Jsoup scraping guide hits the same wall from the scraping side; the detail that matters here is when the JDK reads the value.

Both JDK clients read it once and cache it:

• sun.net.www.protocol.http.HttpURLConnection loads it into a static final Set disabledTunnelingSchemes in a static initializer.
• HttpClient's jdk.internal.net.http.common.Utils loads it into PROXY_AUTH_TUNNEL_DISABLED_SCHEMES, also a static final field.

The java.net.http module documentation is blunt about it: values can be set as system properties on the command line, and "No guarantee is provided that property values can be set programatically with System.setProperty()". A System.setProperty("jdk.http.auth.tunneling.disabledSchemes", "") call works only if it runs before anything in the JVM touches those classes. In a framework app, a health check, a metrics exporter, or a library's static block can load them first, and your call becomes a silent no-op. Put it on the command line:

java -Djdk.http.auth.tunneling.disabledSchemes= -jar app.jar

The value after = is an empty string. That removes Basic from the disabled list. Leave the proxying list alone unless your proxy demands a scheme the JDK blocks there.

The reason Oracle turned this off is real: Basic sends the password base64-encoded, which is readable by anything between you and the proxy. The CONNECT request to an HTTP proxy is not encrypted, because TLS starts only after the tunnel is up. Re-enabling Basic is reasonable when the path to the gateway is one you trust. If it is not, IP whitelisting avoids sending credentials at all: add your server's egress IP in the SparkProxy dashboard and drop the Authenticator.

Check: run with -Djdk.httpclient.HttpClient.log=errors. If Basic is still disabled for tunnels, HttpClient logs Proxy-Authorization disabled just before you get the 407. If that line disappears and you still get 407, the credentials themselves are wrong. Go back to the curl check.

One more HttpClient limitation from the same source file: its authentication filter only implements Basic. A proxy that offers only Digest, NTLM or Negotiate gets nothing from HttpClient, regardless of these properties. Use HttpURLConnection for those.

Here is the full pattern with an explicit selector, a scoped authenticator and a 407 check that fails loudly. It compiles on JDK 21 (we compiled every Java block in this post with Temurin 21.0.13).

import java.net.InetSocketAddress;
import java.net.ProxySelector;
import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.time.Duration;

public class GatewayFetch {
    static final String PROXY_HOST = "gateway.sparkproxy.io";
    static final int ROTATING_PORT = 11000;

    public static void main(String[] args) throws Exception {
        HttpClient client = HttpClient.newBuilder()
                .proxy(ProxySelector.of(new InetSocketAddress(PROXY_HOST, ROTATING_PORT)))
                .authenticator(new GatewayAuthenticator(PROXY_HOST,
                        System.getenv("SPARK_USER"), System.getenv("SPARK_PASS")))
                .connectTimeout(Duration.ofSeconds(15))
                .build();

        HttpRequest request = HttpRequest.newBuilder(URI.create("https://www.sparkproxy.io/"))
                .timeout(Duration.ofSeconds(30))
                .GET()
                .build();

        HttpResponse<String> response = client.send(request, HttpResponse.BodyHandlers.ofString());
        if (response.statusCode() == 407) {
            throw new IllegalStateException(
                "Proxy refused auth. Started with -Djdk.http.auth.tunneling.disabledSchemes= ?");
        }
        System.out.println(response.statusCode() + " " + response.body().length() + " bytes");
    }
}

Launch it with the flag, not with a setProperty call:

java -Djdk.http.auth.tunneling.disabledSchemes= GatewayFetch.java

The single-file source launcher (java File.java) needs both classes in one file, so paste GatewayAuthenticator in as a nested static class when running it that way.

The same client in Kotlin uses the identical JDK API, so the same flag applies:

import java.net.InetSocketAddress
import java.net.ProxySelector
import java.net.URI
import java.net.http.HttpClient
import java.net.http.HttpRequest
import java.net.http.HttpResponse
import java.time.Duration

fun main() {
    val host = "gateway.sparkproxy.io"
    val client = HttpClient.newBuilder()
        .proxy(ProxySelector.of(InetSocketAddress(host, 11000)))
        .authenticator(GatewayAuthenticator(host, System.getenv("SPARK_USER"), System.getenv("SPARK_PASS")))
        .connectTimeout(Duration.ofSeconds(15))
        .build()

    val response = client.send(
        HttpRequest.newBuilder(URI.create("https://www.sparkproxy.io/")).timeout(Duration.ofSeconds(30)).build(),
        HttpResponse.BodyHandlers.ofString()
    )
    check(response.statusCode() != 407) { "Proxy refused auth: check the tunneling flag" }
    println("${response.statusCode()} ${response.body().length} bytes")
}

Notice what is not here: no Authenticator.setDefault, no reliance on https.proxyHost. Configuring the client explicitly means a library that loads earlier, or a teammate's global setting, cannot change where this client sends traffic. If you prefer JVM-wide properties for an existing codebase, that works too, but pair it with .authenticator(...) on every HttpClient you build.

Check: to prove the exit IP changed, request a page that echoes the caller's address and compare it with a direct request from the same machine. If both show your server's IP, the proxy was bypassed; go back to WhichProxy.

java.net.http.HttpClient does not speak SOCKS. In OpenJDK's HttpRequestImpl.retrieveProxy, the client takes the first proxy from the selector and keeps it only if (p.type() == Proxy.Type.HTTP). Anything else returns null, which means a direct connection. There is no exception and no log line. Combine that with the fallback order from earlier, and -DsocksProxyHost=gateway.sparkproxy.io on an HttpClient app sends every request straight out of your server with its real IP.

For SOCKS5 on the JVM, use a socket-based client. HttpURLConnection with an explicit Proxy works, and so does OkHttp:

import java.net.Authenticator;
import java.net.HttpURLConnection;
import java.net.InetSocketAddress;
import java.net.Proxy;
import java.net.URI;

public class SocksFetch {
    public static void main(String[] args) throws Exception {
        String host = "gateway.sparkproxy.io";
        Authenticator.setDefault(new GatewayAuthenticator(host,
                System.getenv("SPARK_USER"), System.getenv("SPARK_PASS")));

        Proxy socks = new Proxy(Proxy.Type.SOCKS, new InetSocketAddress(host, 13000));
        HttpURLConnection con = (HttpURLConnection) URI.create("https://www.sparkproxy.io/")
                .toURL().openConnection(socks);
        con.setConnectTimeout(15_000);
        con.setReadTimeout(30_000);
        System.out.println(con.getResponseCode());
    }
}

SOCKS authentication happens at the socket layer, before HTTP, so the tunneling property is irrelevant here and Basic never enters the picture. That makes SOCKS5 a legitimate workaround when you cannot change JVM flags but can change code. The GatewayAuthenticator above already accepts the "SOCKS5" protocol ask. For a side-by-side of when SOCKS5 beats an HTTP proxy, the SOCKS5 vs HTTP proxies post covers the protocol trade-offs.

Check: if you get SOCKS : authentication failed, print the requestor fields as shown in the Authenticator section. A SERVER SOCKS5 gateway.sparkproxy.io line with no credentials returned means your guard is the culprit.

SparkProxy exposes two behaviours on the HTTP gateway: Random Rotation on port 11000, where the gateway picks an exit IP for each new upstream connection, and Sticky Sessions on port 11002, which hold one exit IP for about five minutes. The JVM adds a wrinkle neither port can see.

An HTTPS request through an HTTP proxy rides a CONNECT tunnel, and a tunnel is one TCP connection from your JVM, through the gateway, to the target. Every request sent down that tunnel exits from the same IP, because the gateway is just relaying bytes. HttpClient pools connections and keeps idle ones for jdk.httpclient.keepalive.timeout seconds (default 30, per the java.net.http module docs). So twenty quick requests to the same host on port 11000 can all share one tunnel and one exit IP. That's the opposite of what "rotating" suggests.

How to get the behaviour you actually want:

Turning off keep-alive costs a TCP and TLS handshake per request, so measure throughput before and after. For background on when each mode wins, see rotating vs static residential proxies.

Cost shapes this decision less than you might expect. SparkProxy plans start from $75/month (price read from sparkproxy.io, USD) and include unlimited bandwidth, so the billing unit is the plan and its concurrency, not gigabytes. Reopening connections costs latency, not money. A 24-hour free trial with no credit card is enough to run every check in this guide against the real gateway.

Once the tunnel authenticates, some failures remain, and no setting on the proxy side changes them. Say so early in an incident and you save hours.

For the target-side blocks, detecting when your scraper is blocked gives the response markers to test for, and the proxy error codes reference maps the 4xx and 5xx codes you will see from gateways.

When the target is the problem rather than the JVM, you can skip proxy plumbing entirely and call the SparkProxy Scraping API, which needs only an X-API-Key header and no tunnel auth:

import java.net.URI;
import java.net.URLEncoder;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.nio.charset.StandardCharsets;
import java.util.Base64;
import java.util.regex.Matcher;
import java.util.regex.Pattern;

public class ScrapeApiFetch {
    public static void main(String[] args) throws Exception {
        String target = URLEncoder.encode("https://www.sparkproxy.io/", StandardCharsets.UTF_8);
        URI uri = URI.create("https://scrape.sparkproxy.io/api/v1?url=" + target
                + "&render_js=false&json_response=true");

        HttpRequest req = HttpRequest.newBuilder(uri)
                .header("X-API-Key", System.getenv("SPARKPROXY_API_KEY"))
                .build();
        String envelope = HttpClient.newHttpClient()
                .send(req, HttpResponse.BodyHandlers.ofString()).body();

        // The envelope's "body" field is base64. Decode before matching anything in it.
        Matcher m = Pattern.compile("\"body\"\\s*:\\s*\"([^\"]*)\"").matcher(envelope);
        if (m.find()) {
            String html = new String(Base64.getDecoder().decode(m.group(1)), StandardCharsets.UTF_8);
            System.out.println(html.contains("<title>") ? "decoded page OK" : "unexpected body");
        }
    }
}

render_js=false is a plain fetch at 1 credit; leaving render_js at its default of true runs a browser for 5 credits. json_response=true is what produces the envelope with job_id, status_code, duration_ms, credits_used and the base64 body. The format parameter only converts content (html, md, mdx, json, screenshot, pdf) and does not create the envelope. New accounts get 1,000 free credits. Use a real JSON parser such as Jackson in production; the regex keeps the example dependency-free.