Scrapy: Could Not Open CONNECT Tunnel With Proxy (Fixed)

Scrapy builds the message in scrapy/core/downloader/handlers/http11.py. When the proxy's reply to CONNECT parses as HTTP, Scrapy prints a dict with status and reason. When it does not parse, Scrapy prints up to the first 1,000 bytes of whatever came back instead (that limit was raised from 32 characters in Scrapy 2.6.0, per the release notes). So the bracket tells you whether you reached a working HTTP proxy at all.

Here is what each failure looked like in our runs, verbatim, with the exception class Scrapy reports in its stats:

Two of those rows deserve a note. The 403/502 row is the only one we did not reproduce against our gateway, so treat its wording as Scrapy's format rather than a capture. And the exception class names in the second column only exist from Scrapy 2.15.0 onward, which replaced Twisted's exceptions with Scrapy-specific ones such as DownloadTimeoutError. On older versions you will see twisted.internet.error.TimeoutError and friends with the same message text.

Work from the outside in. Each step takes under a minute and rules out a whole row of the table.

1. Read the bracket. A status key means the proxy is reachable and talking HTTP. Everything after that is configuration.
2. Test the same proxy with curl or plain Python (next section). If that also gets a 407, Scrapy is not the problem.
3. Print what Scrapy actually sends. Log request.meta["proxy"] and whether Proxy-Authorization is present right before the download. Most "it works in curl" cases end here.
4. Check the scheme and port against your provider's docs. For SparkProxy that is http://gateway.sparkproxy.io:11000 for Random Rotation over HTTP/HTTPS and port 11002 for Sticky Sessions, which keep one IP for 5 minutes. Port 13000 is SOCKS5, which Scrapy's built-in handler does not speak.
5. Only then look at timeouts and concurrency, because those failures are intermittent and the first four are not.

If curl is installed, one command checks credentials and the tunnel together:

curl -sS -o /dev/null -w "%{http_code}\n" \
  -x http://gateway.sparkproxy.io:11000 \
  -U "your-username:your-password" \
  https://www.sparkproxy.io/

A 200 means the proxy is fine and the bug is in your spider. A 407 here means the credentials or the whitelist are wrong, and no Scrapy setting will change that.

On a server without curl, the standard library can send the same CONNECT. This is the exact script we ran:

import base64
import http.client
import sys

PROXY_HOST, PROXY_PORT = "gateway.sparkproxy.io", 11000
USER, PASSWORD = "your-username", "your-password"
TARGET = "www.sparkproxy.io"

token = base64.b64encode(f"{USER}:{PASSWORD}".encode()).decode()
conn = http.client.HTTPSConnection(PROXY_HOST, PROXY_PORT, timeout=15)
conn.set_tunnel(TARGET, 443, headers={"Proxy-Authorization": f"Basic {token}"})
try:
    conn.request("HEAD", "/")
    print("tunnel OK, target answered", conn.getresponse().status)
except OSError as exc:
    print("tunnel failed:", exc)
    sys.exit(1)
finally:
    conn.close()

From our non-whitelisted box it printed tunnel failed: Tunnel connection failed: 407 Proxy Authentication Required, which is the same verdict Scrapy gave. If you want the protocol background on what CONNECT does before TLS starts, see the HTTP CONNECT method explained.

A 407 has three causes, and only one of them lives in your Scrapy code:

• The credentials are wrong. Typos, an expired trial, or a sub-user password you rotated in the dashboard.
• You rely on IP whitelisting from a machine that isn't on the list. This is the classic "works on my laptop, 407 on the server" case. Either add the server's public IP in the dashboard or switch to user:pass auth. The trade-offs are covered in what IP whitelisting is for proxies.
• Scrapy sent no Proxy-Authorization header, or sent a stale one. That is the next section.

Put credentials in the proxy URL and let HttpProxyMiddleware build the header. Percent-encode them, because a /, # or ? in a password will break URL parsing:

from urllib.parse import quote

import scrapy

USER = "your-username"
PASSWORD = "p@ss/word#1"  # characters that break a raw URL
PROXY = f"http://{quote(USER, safe='')}:{quote(PASSWORD, safe='')}@gateway.sparkproxy.io:11000"


class PriceSpider(scrapy.Spider):
    name = "prices"

    async def start(self):
        yield scrapy.Request("https://www.sparkproxy.io/pricing", meta={"proxy": PROXY})

    def parse(self, response):
        yield {"status": response.status, "title": response.css("title::text").get()}

HttpProxyMiddleware runs unquote() on the username and password before encoding them, so the encoded form is what you want in the URL. It also encodes with HTTPPROXY_AUTH_ENCODING, which defaults to latin-1. A password with characters outside latin-1 needs that setting changed to utf-8, and your provider has to agree. The async def start() form shown here is the current API. Scrapy 2.16.0 removed start_requests() entirely (it had been deprecated since 2.13.0), so a spider copied from an old tutorial may never send the request you are debugging.

The 407 bracket looks identical for all three causes. The curl test separates the first two from the third.

This is the cause behind most "first request works, retries all fail with 407" reports. Since a security fix in Scrapy 2.6.2 (relaxed slightly in 2.7.1), HttpProxyMiddleware follows three rules on every pass, and they still hold in 2.19.0. You can read them in _set_proxy_and_creds() in httpproxy.py:

1. If meta["proxy"] contains credentials, the Proxy-Authorization header is always rewritten to match them.
2. If meta["proxy"] has no credentials, a Proxy-Authorization header survives only while the proxy URL stays the same as when the header was first seen. The middleware remembers that URL in meta["_auth_proxy"].
3. If meta["proxy"] is missing or None, the header is removed.

Rule 2 is the trap. A common rotation pattern sets a credential-free proxy URL and writes the header by hand. On the first pass, that works. When a retry or ban handler picks a different host, the header gets deleted even though your middleware just set it again, because _auth_proxy still points at the old host. We confirmed it by calling the middleware directly:

import base64

from scrapy import Request
from scrapy.downloadermiddlewares.httpproxy import HttpProxyMiddleware

mw = HttpProxyMiddleware()
auth = b"Basic " + base64.b64encode(b"user:pass")

req = Request(
    "https://www.sparkproxy.io/",
    meta={"proxy": "http://gw-a.sparkproxy.io:11000"},
    headers={"Proxy-Authorization": auth},
)
mw.process_request(req)
print(req.headers.get("Proxy-Authorization"))  # b'Basic dXNlcjpwYXNz'

req.meta["proxy"] = "http://gw-b.sparkproxy.io:11000"  # rotate
req.headers["Proxy-Authorization"] = auth  # set again, as many tutorials do
mw.process_request(req)
print(req.headers.get("Proxy-Authorization"))  # None -> the proxy answers 407

The hostnames there are placeholders for two different proxy endpoints. The second request reaches the proxy with no credentials at all. The fix is to stop managing the header yourself and build a credentialed URL on every pass, so rule 1 applies:

import random
from urllib.parse import quote


class CredentialedRotationMiddleware:
    """Sets a proxy URL with credentials on every pass; never writes the header."""

    def __init__(self, hosts, user, password):
        self.hosts = hosts
        self.userinfo = f"{quote(user, safe='')}:{quote(password, safe='')}"

    @classmethod
    def from_crawler(cls, crawler):
        s = crawler.settings
        return cls(s.getlist("PROXY_HOSTS"), s.get("PROXY_USER"), s.get("PROXY_PASS"))

    def process_request(self, request, spider=None):
        host = random.choice(self.hosts)
        request.meta["proxy"] = f"http://{self.userinfo}@{host}"
        return None

Register it below 750 so it runs before HttpProxyMiddleware. If you only use one gateway endpoint, you don't need rotation code at all, because the gateway rotates exits for you. The general middleware setup is in our Scrapy proxy setup guide, so we won't repeat it here.

Two more places credentials disappear. Setting meta["proxy"] = None deliberately disables the proxy and strips the header. And Scrapy removes Proxy-Authorization from the request sent through an open tunnel, so a target site never sees it; if you were checking the header on the target side, its absence there is expected.

Not every proxy error reaches the tunnel stage. These came up in our test runs:

• https:// as the proxy scheme. Scrapy raises NotImplementedError: HTTPS proxies for HTTPS destinations are not supported before it opens a socket. The proxy URL should be http:// even for HTTPS targets, because TLS to the target runs inside the tunnel.
• No scheme at all. gateway.sparkproxy.io:11000 worked in our test; Scrapy prepends http://. It isn't the bug, though writing the scheme keeps intent clear.
• The SOCKS5 port with an http:// scheme. Pointing Scrapy at port 13000 gave no error from the proxy, only took longer than 10.0 seconds, because a SOCKS5 listener waits for a SOCKS handshake that never comes. Scrapy's HTTP/1.1 handler has no SOCKS support, so use the HTTP port.
• A system proxy you didn't set. HttpProxyMiddleware reads http_proxy/https_proxy from the environment for any request without meta["proxy"], and on Windows Python's getproxies() also reads the Internet Options proxy from the registry. If requests you never proxied fail with tunnel errors, check both, or pass meta={"proxy": None}.

Both messages surface as DownloadTimeoutError in 2.15+, but they fail at different points and have different fixes.

User timeout caused connection failure. is Twisted's connect timeout. Scrapy passes DOWNLOAD_TIMEOUT (or the download_timeout meta key) as the TCP connect timeout to the proxy, so this line means the SYN never got an answer within that window. We reproduced it by pointing at a non-routable address. Causes: a firewall dropping outbound traffic to the proxy port, a dead proxy host, or a typo'd IP. Rotation helps only in the dead-host case.

Getting took longer than N seconds. means the TCP connection opened and then the whole download did not finish in time. Protocol mismatches (the SOCKS case above), a slow exit IP, or a slow target all land here.

The default DOWNLOAD_TIMEOUT is 180 seconds and the default RETRY_TIMES is 2, so an unreachable proxy can hold one request for up to nine minutes before Scrapy gives up. Set the timeout from the latency you actually observe:

# settings.py
DOWNLOAD_TIMEOUT = 30  # proxy hop + TLS + target response, per attempt
RETRY_TIMES = 2


# Per request, for a page you know is slow:
# yield scrapy.Request(url, meta={"proxy": PROXY, "download_timeout": 60})

Scrapy records each response's time in response.meta["download_latency"]. Log it for a few hundred requests and set the timeout a comfortable margin above the slow tail, not the average. For a provider-agnostic treatment of picking timeout values, see proxy timeouts and retry logic.

Scrapy's concurrency limits are keyed by the target hostname, not by the proxy. When every request goes through one gateway, the gateway sees up to CONCURRENT_REQUESTS simultaneous tunnels (default 16), whatever CONCURRENT_REQUESTS_PER_DOMAIN (default 8) says. A spider that crawls ten domains at 8 each would want 80, and the global cap of 16 is the only thing holding it back. Raise CONCURRENT_REQUESTS to 64 and the gateway sees 64.

Projects created with scrapy startproject on 2.13.3 or later start with CONCURRENT_REQUESTS_PER_DOMAIN = 1 and DOWNLOAD_DELAY = 1 in settings.py, so a spider that seems slow behind a proxy may simply be throttled by its own template.

SparkProxy plans include unlimited bandwidth and are sold as packs sized by concurrency (entry pricing reads "Starting from $75/month", USD, on the live site as of 2026-10-09), so traffic volume isn't the limit you hit; simultaneous connections are. We have not documented which status the gateway returns once you exceed a plan's concurrency, so the practical sign is a pattern: tunnels that succeed at low concurrency and fail only above a certain level. Cap Scrapy below the plan:

# settings.py
PLAN_CONCURRENCY = 50  # the concurrency figure on your plan, from the dashboard

CONCURRENT_REQUESTS = PLAN_CONCURRENCY - 5  # headroom for retries and other tools
CONCURRENT_REQUESTS_PER_DOMAIN = 8
AUTOTHROTTLE_ENABLED = True
AUTOTHROTTLE_TARGET_CONCURRENCY = 4.0

The 50 is an example value, not a plan size. Remember that every Scrapy process, cron job and test script shares the same plan. Two spiders at 45 each on a 50-connection plan is the same failure as one spider at 90. How providers count connections is explained in understanding concurrent connections in proxies.

TunnelError is in Scrapy's default RETRY_EXCEPTIONS, so a 407 is retried like a network blip. In our runs with RETRY_TIMES = 1, the log read Gave up retrying (failed 2 times) with the same 407 both times. At the default of 2, that is three identical attempts per request. On a big crawl that wastes your connection budget and buries the real error under thousands of identical lines.

This middleware turns a tunnel 407 into an immediate, distinct failure, and the errback stops the crawl:

from scrapy.core.downloader.handlers.http11 import TunnelError
from scrapy.exceptions import CloseSpider, IgnoreRequest


class ProxyAuthFailed(IgnoreRequest):
    pass


class FailFastOn407Middleware:
    def process_exception(self, request, exception, spider=None):
        if isinstance(exception, TunnelError) and "'status': 407" in str(exception):
            raise ProxyAuthFailed(str(exception))
        return None


# In the spider: pass errback=self.on_error on each Request
def on_error(self, failure):
    if failure.check(ProxyAuthFailed):
        raise CloseSpider("proxy_auth_failed")
    self.logger.error("download failed: %r", failure.value)

# settings.py: process_exception runs from high numbers to low,
# so 560 sees the error before RetryMiddleware at 550.
DOWNLOADER_MIDDLEWARES = {
    "myproject.middlewares.CredentialedRotationMiddleware": 350,
    "myproject.middlewares.FailFastOn407Middleware": 560,
}

Against our gateway with three start URLs, the stats showed exactly three TunnelErrors and finish_reason: 'proxy_auth_failed': no retries, and one clear reason in the log. Keep normal retries for timeouts and connection drops, ideally with backoff; the patterns for that are in retry and backoff strategies for web scraping.

Swapping proxies fixes: a dead or overloaded proxy host, an exit IP the target has banned (which shows up as a 403 or CAPTCHA after the tunnel opens, not as a TunnelError), and geo restrictions.

Swapping proxies does not fix: wrong credentials, a missing whitelist entry, a stripped Proxy-Authorization header, an https:// proxy scheme, the SOCKS port, a firewall blocking outbound traffic, or a concurrency cap you are exceeding. Every one of those follows you to the next proxy. Authentication itself is covered in how proxy authentication works.

If your network blocks outbound connections to proxy ports entirely (some corporate and cloud egress rules do), the SparkProxy Scraping API avoids the tunnel: Scrapy makes an ordinary HTTPS request to scrape.sparkproxy.io and the API fetches the target. A plain fetch with render_js=false costs 1 credit, and new accounts get 1,000 free credits:

from urllib.parse import urlencode

import scrapy

API = "https://scrape.sparkproxy.io/api/v1"


class ApiSpider(scrapy.Spider):
    name = "via_api"

    async def start(self):
        params = urlencode({"url": "https://www.sparkproxy.io/pricing", "render_js": "false"})
        yield scrapy.Request(
            f"{API}?{params}",
            headers={"X-API-Key": "YOUR_API_KEY"},
            meta={"proxy": None},  # no proxy hop, so no tunnel to fail
        )

    def parse(self, response):
        yield {"bytes": len(response.body)}

Note that render_js defaults to true (5 credits), so set it to false explicitly for static pages. meta={"proxy": None} keeps any environment proxy from being applied to the API call.