How to Use a Docker Proxy: Daemon, Build, Runtime

Almost every "Docker proxy not working" thread comes down to setting the proxy in one scope and expecting it to cover another. These scopes never share state:

Read the table top to bottom and the rule falls out: the daemon proxy moves image layers, the build proxy feeds package managers, and the runtime proxy carries your app's requests. A build ARG does not survive into the running container, and a docker run -e value was never visible while the image was being built. If your scraper needs a proxy, that is the runtime layer, and it is the one people most often forget.

The proxy URL is the same shape at every layer: http://USER:PASS@HOST:PORT. Credentials live in the URL, and the scheme is http:// even when the target site is HTTPS, because that describes how you talk to the proxy, not the site. If that format is new to you, read how proxy authentication works before wiring it into build files where a typo is hard to see.

The daemon (dockerd) is the process that talks to registries. If your host sits behind a corporate or filtered network and docker pull python:3.12-slim hangs, this is the layer to fix. On Linux with systemd, add a drop-in file:

sudo mkdir -p /etc/systemd/system/docker.service.d
sudo tee /etc/systemd/system/docker.service.d/http-proxy.conf >/dev/null <<'EOF'
[Service]
Environment="HTTP_PROXY=http://USER:PASS@gate.sparkproxy.io:10000"
Environment="HTTPS_PROXY=http://USER:PASS@gate.sparkproxy.io:10000"
Environment="NO_PROXY=localhost,127.0.0.1,::1"
EOF
sudo systemctl daemon-reload
sudo systemctl restart docker

Confirm the daemon picked it up:

sudo systemctl show docker --property=Environment
docker info | grep -i proxy

Since Docker Engine 23.0 (February 2023) you can skip systemd and set the same thing in /etc/docker/daemon.json:

{
  "proxies": {
    "http-proxy": "http://USER:PASS@gate.sparkproxy.io:10000",
    "https-proxy": "http://USER:PASS@gate.sparkproxy.io:10000",
    "no-proxy": "localhost,127.0.0.1,*.internal"
  }
}

Restart the daemon after editing either file. On Docker Desktop (macOS or Windows) there is no systemd, so use Settings > Resources > Proxies instead, which writes the same configuration for you.

One honest caveat: routing image pulls through a metered scraping proxy burns bandwidth on multi-hundred-megabyte layers. The daemon proxy is for reaching registries from a restricted network, not for anonymizing pulls. If your only goal is anonymous scraping, leave the daemon alone and set the proxy at the runtime layer instead.

Here is the trap that eats an afternoon. You set http_proxy on your host, run docker build, and every RUN apt-get update still times out. Build steps run inside isolated containers that do not inherit your shell environment. A proxy in the Dockerfile only works if you get the value into the build container.

Docker gives you a shortcut. A fixed set of proxy variables are predefined build args, so you can reference them without an ARG line, and Docker injects them as environment variables into every RUN step:

HTTP_PROXY   http_proxy
HTTPS_PROXY  https_proxy
FTP_PROXY    ftp_proxy
NO_PROXY     no_proxy
ALL_PROXY    all_proxy

Pass them on the command line and your package managers pick them up automatically:

docker build \
  --build-arg HTTP_PROXY=http://USER:PASS@gate.sparkproxy.io:10000 \
  --build-arg HTTPS_PROXY=http://USER:PASS@gate.sparkproxy.io:10000 \
  --build-arg NO_PROXY=localhost,127.0.0.1 \
  -t scraper:latest .

The Dockerfile itself needs nothing proxy-specific:

# syntax=docker/dockerfile:1
FROM python:3.12-slim

WORKDIR /app
COPY requirements.txt .
# apt and pip below inherit the predefined proxy build args automatically
RUN apt-get update && apt-get install -y --no-install-recommends ca-certificates \
    && rm -rf /var/lib/apt/lists/*
RUN pip install --no-cache-dir -r requirements.txt

COPY scraper.py .
CMD ["python", "scraper.py"]

Two properties of these predefined proxy args are worth knowing, because they are the reason to prefer them over a custom ARG. First, they are excluded from the build cache, so changing the proxy does not force a full rebuild. Second, they are kept out of docker history and the final image metadata, so your credentials do not ship inside the image. Define your own ARG MY_PROXY=... and the value is visible in docker history, which is exactly how proxy passwords leak into shared registries.

BuildKit is the default builder since Docker 23.0 and honors all of this the same way. If you have pinned an older Docker or disabled BuildKit, the legacy builder behaves identically for the predefined proxy args, so no change is needed there.

Typing three --build-arg flags on every build gets old, and it is easy to forget one. The Docker CLI can inject them for you from ~/.docker/config.json. Add a proxies block:

{
  "proxies": {
    "default": {
      "httpProxy": "http://USER:PASS@gate.sparkproxy.io:10000",
      "httpsProxy": "http://USER:PASS@gate.sparkproxy.io:10000",
      "noProxy": "localhost,127.0.0.1,*.internal"
    }
  }
}

Now a bare docker build . sets the proxy build args for you, and docker run sets the matching environment variables in the container. This single file is the cleanest way to run a docker http proxy across every build and container on your machine without editing each Dockerfile or command. Note the camelCase keys here (httpProxy), which differ from the daemon's kebab-case keys (http-proxy) in daemon.json. Mixing them up silently does nothing, so copy the right casing for the right file.

You can also scope proxies per registry host instead of default if only some targets need routing. Keep this file out of version control, since it now holds credentials.

This is the layer that matters for a scraper. The build is done, the image is built, and now your program makes outbound requests. Pass the proxy at docker run time:

docker run --rm \
  -e HTTP_PROXY=http://USER:PASS@gate.sparkproxy.io:10000 \
  -e HTTPS_PROXY=http://USER:PASS@gate.sparkproxy.io:10000 \
  -e NO_PROXY=localhost,127.0.0.1 \
  scraper:latest

The catch: setting the env var only helps if the program inside actually reads it. Env-var proxy support is per-tool, not a Docker feature, and this is where "the env var is set but traffic ignores it" comes from:

Because case handling is inconsistent (curl reads lowercase, others read uppercase), the safe move is to set both:

docker run --rm \
  -e HTTP_PROXY=http://USER:PASS@gate.sparkproxy.io:10000 \
  -e http_proxy=http://USER:PASS@gate.sparkproxy.io:10000 \
  -e HTTPS_PROXY=http://USER:PASS@gate.sparkproxy.io:10000 \
  -e https_proxy=http://USER:PASS@gate.sparkproxy.io:10000 \
  scraper:latest

If your target site speaks SOCKS instead of HTTP, the same env vars accept a socks5:// URL where the tool supports it. The difference between the two, and when SOCKS matters, is covered in understanding proxy protocols: HTTP, HTTPS, and SOCKS5.

Compose separates build-time and runtime cleanly, which is exactly the distinction people miss on the command line. Build args live under build.args, runtime values under environment:, and they are independent:

services:
  scraper:
    build:
      context: .
      args:
        HTTP_PROXY: ${HTTP_PROXY}
        HTTPS_PROXY: ${HTTPS_PROXY}
        NO_PROXY: ${NO_PROXY}
    environment:
      HTTP_PROXY: ${HTTP_PROXY}
      HTTPS_PROXY: ${HTTPS_PROXY}
      NO_PROXY: localhost,127.0.0.1
    restart: unless-stopped

Keep the actual values in a .env file next to compose.yaml so Compose interpolates ${HTTP_PROXY} at load time:

# .env  (git-ignore this)
HTTP_PROXY=http://USER:PASS@gate.sparkproxy.io:10000
HTTPS_PROXY=http://USER:PASS@gate.sparkproxy.io:10000
NO_PROXY=localhost,127.0.0.1

Run docker compose up --build and both stages get the proxy. If you only put the value under environment: and skip build.args, your image builds without a proxy and fails on the first pip install. If you only put it under build.args, the build works and the scraper runs on your real IP. You almost always want both blocks.

Environment variables are convenient, but for a scraper the more reliable pattern is to pass the proxy in code, so it works regardless of how the client library treats env vars. Here is the full container. The requirements.txt holds one line, requests, and scraper.py reads the proxy from an env var and applies it explicitly:

# scraper.py
import os
import requests

PROXY = os.environ["SPARKPROXY_URL"]  # http://USER:PASS@gate.sparkproxy.io:10000
PROXIES = {"http": PROXY, "https": PROXY}

def fetch(url: str) -> str:
    resp = requests.get(url, proxies=PROXIES, timeout=30)
    resp.raise_for_status()
    return resp.text

if __name__ == "__main__":
    # Confirm the container is exiting through the proxy, not your host IP
    who = requests.get("https://www.sparkproxy.io/ip", proxies=PROXIES, timeout=30)
    print("Exit IP:", who.text.strip())
    print(fetch("https://www.example.com")[:200])

Build and run it, injecting the proxy only at runtime:

docker build -t scraper:latest .
docker run --rm \
  -e SPARKPROXY_URL=http://USER:PASS@gate.sparkproxy.io:10000 \
  scraper:latest

Passing the proxy explicitly in requests sidesteps every env-var casing quirk, and reading it from SPARKPROXY_URL keeps the credential out of the image and out of your source. To spread requests across many exit IPs and stay under a site's rate limits, rotate the gateway per session and add retry logic. The blocking side of that, headers, fingerprints, and pacing, is in how to avoid getting your proxy blocked.

You have two ways to reach SparkProxy from inside a container, and they solve different problems.

Option A, the proxy gateway. Point your client at gate.sparkproxy.io and SparkProxy rotates the exit IP for you. This is the runtime pattern from the section above: one SPARKPROXY_URL env var, credentials in the URL, and your code keeps full control of the request.

Option B, the Scraping API. When targets fight back with JavaScript rendering, CAPTCHAs, or aggressive fingerprinting, offload the whole retrieval to SparkProxy's managed Scraping API. The container makes a normal HTTPS call to https://scrape.sparkproxy.io/api/v1, and SparkProxy handles the proxy pool, rotation, and headless rendering on its side. No http_proxy env var, no proxy URL in your code:

# scraper_api.py
import os
import requests

def scrape(url: str) -> str:
    resp = requests.get(
        "https://scrape.sparkproxy.io/api/v1",
        params={
            "url": url,
            "country_code": "US",   # geo-target the exit
            "render_js": "true",    # run headless Chromium
            "premium_proxy": "true" # route via residential IPs
        },
        headers={"X-API-Key": os.environ["SPARKPROXY_API_KEY"]},
        timeout=90,
    )
    resp.raise_for_status()
    return resp.text

if __name__ == "__main__":
    print(scrape("https://www.example.com")[:500])

Run it with the key passed as an env var, never baked into the image:

docker run --rm \
  -e SPARKPROXY_API_KEY=sk-YOUR_API_KEY \
  scraper:latest python scraper_api.py

The API call is a plain HTTPS request, so it works even when the container's daemon and build proxies are unset. Deciding between running your own rotating gateway and letting the API do it is a real trade-off in cost, control, and maintenance, laid out in web scraping API vs self-managed proxies.

NO_PROXY is the setting people skip until it breaks something. It is a comma-separated list of hosts that should bypass the proxy. Inside Docker you almost always want localhost,127.0.0.1 on it so a container talking to a sibling service (a database, another API container) does not route internal traffic out through your paid proxy and back. Add *.internal or your Compose service names when containers call each other by name.

Common failures and their real cause:

• docker pull hangs but curl works on the host. You set the shell proxy, not the daemon proxy. Fix Layer 1 (systemd drop-in or daemon.json), then systemctl restart docker.
• RUN pip install times out during build. Build containers do not inherit host env. Pass the predefined proxy build args or add the proxies block to ~/.docker/config.json.
• Scraper runs on your real IP. The runtime env var is missing, or the library ignores it. Pass the proxy in code as shown in the scraper section.
• 407 Proxy Authentication Required. The credentials in the proxy URL are wrong or URL-unsafe. If your password contains @, :, or /, percent-encode it, since those characters break URL parsing.
• Compose builds but the app has no proxy (or vice versa). You filled build.args or environment: but not both.

To prove which IP a container actually exits from, run a one-off check against the same endpoint the daemon and app would use:

docker run --rm \
  -e HTTPS_PROXY=http://USER:PASS@gate.sparkproxy.io:10000 \
  curlimages/curl:latest -s https://www.sparkproxy.io/ip

If that prints a proxy IP and your app still shows your own, the problem is in your application code or its env casing, not Docker.