http_proxy Not Working With sudo, cron and systemd

Here is the whole problem in four lines:

$ export https_proxy=http://USER:PASS@gateway.sparkproxy.io:11000
$ curl -s https://api.ipify.org
203.0.113.77          # a proxy exit IP
$ sudo curl -s https://api.ipify.org
198.51.100.10         # your server's own IP

The same thing happens when the line runs from cron or from ExecStart= in a unit. In our lab, every case where the variable was lost (sudo curl, sudo -i curl, sudo sh -c "curl ...", a crontab with no proxy lines, a bare systemd service) finished with HTTP 200 and zero connections at the proxy. Nothing failed. The request just went direct.

The cause is how Unix environments work. A variable you export lives in your shell process and is copied only to that shell's children. The cron daemon and systemd (PID 1) were started at boot and are not children of your shell at all. sudo is a child, but it deliberately throws most of the inherited environment away before running the command.

Typical ways this shows up:

• A scraper that never got blocked in testing starts collecting 403s and CAPTCHAs the night it moves to a timer.
• sudo pip install or sudo ./setup.sh hangs on a network that only allows outbound traffic through a proxy.
• Your provider dashboard shows no usage from the scheduled job, while the target's logs show your origin address.

Each launcher assembles a fresh environment from its own sources. This table is what we measured on Ubuntu 24.04.5 with all four proxy variables exported in the calling shell:

The sudoers manual describes env_reset, which is on by default, plainly: sudo "will run the command in a minimal environment containing the TERM, PATH, HOME, MAIL, SHELL, LOGNAME, USER and SUDO_* variables", plus anything listed in env_keep. On this Ubuntu build, sudo -V (run as root) listed the default keep set as XDG_CURRENT_DESKTOP, XAUTHORIZATION, XAUTHORITY, PS2, PS1, PATH, LS_COLORS, KRB5CCNAME, HOSTNAME, DPKG_COLORS, DISPLAY and COLORS. No proxy variable is on it.

For cron, crontab(5) says "SHELL is set to /bin/sh, and LOGNAME and HOME are set from the /etc/passwd line of the crontab's owner". Your .bashrc and .profile are never read.

For systemd, systemd.exec(5) is explicit: "Processes started by the system service manager generally do not inherit environment variables set for the service manager". A bare service in our lab received only INVOCATION_ID, JOURNAL_STREAM, LANG, MEMORY_PRESSURE_WATCH, MEMORY_PRESSURE_WRITE, PATH, PWD, SYSTEMD_EXEC_PID, USER and the one variable we set ourselves.

The /etc/environment column surprises people. On Ubuntu 24.04, both /etc/pam.d/sudo and /etc/pam.d/cron call pam_env.so, so a line in /etc/environment reaches sudo and cron jobs (we confirmed both), while a systemd service ignores it unless the unit sets PAMName=. Check your own distro with grep pam_env /etc/pam.d/sudo /etc/pam.d/cron before relying on it, and remember that the file is mode 0644: every local user can read any password you put there.

Don't guess which variables arrived. Ask each context directly; this takes under a minute per context and settles the question.

# sudo: what does the command actually see?
sudo env | grep -i _proxy || echo "no proxy vars under sudo"
sudo -V | sed -n '/Environment variables to preserve/,/^$/p'   # run as root

# cron: add a throwaway line, wait one minute, read the file, delete the line
* * * * * env | sort > /tmp/cron-env.txt

# systemd: the configured list, then the live process
systemctl show -p Environment -p EnvironmentFiles scraper.service
tr '\0' '\n' < /proc/$(systemctl show -p MainPID --value scraper.service)/environ | grep -i _proxy

/proc//environ is the ground truth for a running process. It's what the kernel handed to execve, after every launcher, PAM module and unit file has had its say.

Variables can be present and still unused, because clients disagree on names and case. Run this small script in the same context as the scraper to see what Python itself will resolve:

#!/usr/bin/env python3
"""dump_env.py: show which proxy settings this process will really use."""
import os
import urllib.request

import requests.utils

TARGET = "https://api.ipify.org"

for name in sorted(os.environ):
    if name.lower().endswith("_proxy"):
        value = os.environ[name]
        print(f"{name:12} = {'(empty)' if not value else value.split('@')[-1]}")

print("urllib resolves:", {k: v.split("@")[-1] for k, v in urllib.request.getproxies().items()})
print("requests would use for", TARGET, "->",
      {k: v.split("@")[-1] for k, v in requests.utils.get_environ_proxies(TARGET).items()})

It prints only the host part after @, so it won't paste your proxy password into a cron mail or the journal.

The best fix is often not to use sudo for the scraper at all. Nothing about fetching pages needs root. If something in the job does need root, pick the narrowest option below. Each one passed in our lab (one connection reached the proxy, HTTP 200).

Pass named variables for one command. The list form arrived in sudo 1.8.21, whose release notes say "--preserve-env command line option has been extended to accept" a list of variables:

sudo --preserve-env=http_proxy,https_proxy,no_proxy python3 /opt/scraper/run.py

Pass everything with -E. sudo -E keeps the whole caller environment. The sudoers policy only permits it when the rule carries the SETENV tag or the command is ALL (which implies SETENV), so the stock %sudo ALL=(ALL:ALL) ALL rule allows it. It also carries your PATH, PYTHONPATH and anything else across, which is more than you asked for.

Make it permanent with env_keep. Create the file with visudo so a syntax error can't lock you out:

sudo visudo -f /etc/sudoers.d/90-proxy
# contents:
Defaults env_keep += "http_proxy https_proxy no_proxy HTTP_PROXY HTTPS_PROXY NO_PROXY"

With that drop-in in place, all three variants we tried kept the proxy: sudo curl, sudo -i curl and sudo -u builder curl. sudo -V then listed the six names under "Environment variables to preserve".

System-wide via /etc/environment. On Ubuntu this works for sudo through pam_env, as the table above shows. It's the bluntest option and exposes credentials to every user, so prefer IP whitelisting if you go this way; SparkProxy accepts both user:pass and IP whitelist authentication, and a whitelisted proxy URL has no secret in it.

Check: sudo env | grep -i _proxy must list the variables, and sudo curl -s https://api.ipify.org must print something other than your origin IP.

A crontab can set variables itself. Lines of the form name = value apply to every job below them:

# crontab -e  (as the user who runs the scraper)
http_proxy=http://USER:PASS@gateway.sparkproxy.io:11000
https_proxy=http://USER:PASS@gateway.sparkproxy.io:11000
no_proxy=localhost,127.0.0.1
17 */6 * * * /usr/bin/python3 /opt/scraper/run.py >> /var/log/scraper/run.log 2>&1

Two details we measured. Values are taken literally: LABHOME=$HOME/x reached the job as the string $HOME/x, unexpanded. And a % inside a variable line survived intact (p%40ss stayed p%40ss).

The command field is different. crontab(5): "A "%" character in the command, unless escaped with a backslash (\), will be changed into newline characters", and everything after it goes to the command's stdin. That bites proxy users twice, because URL-encoded passwords are full of % and so is curl's -w '%{http_code}'. Cron's own log shows the damage:

(builder) CMD (curl -s -m 10 -o /dev/null -x http://u:p)
(builder) CMD (curl -s -m 10 -o /dev/null -w ')

The first line was meant to be -x http://u:p%40ss@127.0.0.1:18080 .... Neither job reached the proxy or wrote its output file. Escaping as p\%40ss fixed it, but the cleaner fix is to keep secrets out of the command field entirely with a wrapper:

#!/bin/sh
# /usr/local/bin/run-scraper  (chmod 750)
set -a                       # export every variable the next file defines
. /etc/scraper/proxy.env     # dash has no "source": it fails with "source: not found"
set +a
exec /usr/bin/python3 /opt/scraper/run.py "$@"

Cron runs commands with /bin/sh, which is dash on Debian and Ubuntu. Write . not source; we got sh: 1: source: not found when we tried. Keep /etc/scraper/proxy.env at mode 0600 and owned by the job's user. Scheduling details (/etc/cron.d user fields, jitter, retries) are covered in how to schedule and automate web scrapers; this post sticks to getting the proxy into the job.

The same proxy.env file (plain name=value lines, no export) also works as a systemd EnvironmentFile=; we used one file for both and both runs reached the proxy with credentials.

Check: add a throwaway line * sh -c 'set -a; . /etc/scraper/proxy.env; curl -s https://api.ipify.org' > /tmp/cron-ip.txt 2>&1, wait a minute, and compare the file with your origin IP.

Put the proxy on the .service unit, never the .timer; the timer only starts the service. Use a drop-in so a package upgrade can't overwrite it. systemctl edit scraper.service opens an override file, or write one by hand:

# /etc/systemd/system/scraper.service.d/proxy.conf
[Service]
EnvironmentFile=/etc/scraper/proxy.env

sudo systemctl daemon-reload
sudo systemctl restart scraper.service
systemctl show -p DropInPaths scraper.service

EnvironmentFile= takes "newline-separated variable assignments" (systemd.exec), and the file is "read shortly before the process is executed", so a changed password needs a restart, not a daemon-reload. Prefix the path with - only if a missing file should be ignored; for a proxy, a missing file ought to stop the job.

You can also write Environment= lines directly, and here's the trap that cost us the most time. systemd performs specifier expansion on Environment= values, and %4 is not a valid specifier. With Environment="http_proxy=http://u:p%40ss@127.0.0.1:18080" the journal printed:

proxy.conf:3: Failed to resolve specifiers in http_proxy=http://u:p%40ss@127.0.0.1:18080, ignoring: Invalid slot

"Ignoring" is literal. The unit started, finished successfully and fetched directly; systemctl show -p Environment no longer listed http_proxy. Writing %%40 (systemd's escape for a single percent) fixed it, and so did moving the same line into an EnvironmentFile=, whose contents were passed through unchanged in our test. systemd-analyze verify prints the same warning, so run it before you restart.

Avoid systemctl set-environment and DefaultEnvironment= for credentials. They worked (a unit started after set-environment used the proxy), but they apply to every unit the manager starts, and systemd.exec ranks them lowest in precedence, below Environment= and EnvironmentFile=. If the scraper runs in a container rather than directly under systemd, the daemon and container layers are a separate problem covered in how to use proxies with Docker.

Check without touching the real unit:

sudo systemd-run --quiet --wait --pipe -p EnvironmentFile=/etc/scraper/proxy.env \
  curl -s https://api.ipify.org

That runs curl as a transient service with the same environment rules, and prints its exit IP to your terminal.

Getting a variable into the process is half the job. The client then has to read the name you set.

curl's documentation says it directly: "The lower case version has precedence. http_proxy is an exception as it is only available in lower case." CPython's getproxies_environment() runs two passes, and in the second, for lowercase names only, an empty value calls proxies.pop(proxy_name, None). So a unit or wrapper that blanks https_proxy while setting HTTPS_PROXY works for curl and silently bypasses the proxy for Python. The same function also drops an uppercase HTTP_PROXY when REQUEST_METHOD is present (the CGI case, CVE-2016-1000110).

The safe habit: set both cases to identical, non-empty values, plus no_proxy for your own hosts. Clients that ignore these variables entirely (Node's built-in fetch is the common one) need configuring in code instead.

"The variable is set" is not proof. Compare three addresses:

The job's IP must differ from the origin. It does not need to match the shell's IP. On SparkProxy's rotating port 11000 (Random Rotation) every new connection can get a different exit, so two runs disagreeing is expected. If you want a stable number for side-by-side checks, use port 11002, Sticky Sessions, which holds an exit for 5 minutes. For a broader test routine (headers, DNS, geo), see how to test if your proxy is working.

If the job's IP is a proxy IP but requests fail with 407 Proxy Authentication Required, the variable arrived and the credentials didn't: look again for a % mangled by cron or systemd, or an @ in the password that was never encoded. How proxy authentication works explains what the gateway expects in the Proxy-Authorization header.

Every failure above degrades silently to a direct connection. Make the job refuse to run instead. This guard uses the same lookup requests does, so it catches the empty-lowercase case too:

#!/usr/bin/env python3
"""proxy_guard.py: refuse to scrape unless this process will really use the proxy."""
import os
import sys
import urllib.request

import requests

IP_ECHO = "https://api.ipify.org"


def main() -> None:
    # The same lookup requests performs when trust_env is on (the default).
    env_proxies = urllib.request.getproxies()
    if "https" not in env_proxies:
        sys.exit("proxy_guard: no usable https proxy in this environment, refusing to run")
    exit_ip = requests.get(IP_ECHO, timeout=15).text.strip()
    origin_ip = os.environ.get("ORIGIN_IP", "")
    if origin_ip and exit_ip == origin_ip:
        sys.exit(f"proxy_guard: exit IP {exit_ip} is this server's own IP, proxy not in path")
    print(f"proxy_guard: ok, exit IP {exit_ip}")


if __name__ == "__main__":
    main()

In our runs it exited with status 1 and "no usable https proxy" both with no variable and with https_proxy= empty beside a set HTTPS_PROXY. When we pointed it at our loopback proxy, which egresses from the same machine, it correctly reported that the exit IP equalled ORIGIN_IP. Call it first in the wrapper or as ExecStartPre=/usr/local/bin/proxy_guard.py in the unit. systemd.service(5) says that if any of those commands (not prefixed with -) fail, "the rest are not executed and the unit is considered failed", so the scraper never starts.

Stronger still, stop depending on the environment. requests can ignore every proxy variable and use only what your code passes:

import os

import requests

PROXY_URL = os.environ["SPARKPROXY_URL"]  # or read from a 0600 config file

session = requests.Session()
session.trust_env = False  # ignore http(s)_proxy, no_proxy and .netrc entirely
session.proxies = {"http": PROXY_URL, "https": PROXY_URL}
print(session.get("https://api.ipify.org", timeout=15).text)

Now a missing SPARKPROXY_URL raises KeyError and the job dies loudly, instead of quietly fetching from your origin.

Most of this article is plain Linux configuration. No proxy plan fixes env_reset, a crontab %, a dropped Environment= line or a client that never reads the variable. Those are on your side of the connection.

Once traffic really flows through the gateway, the proxy does its usual job: a different exit IP from your server, per-IP rate limits spread across a pool of 1M+ premium IPs in 80+ countries, and country targeting. It does not change your TLS fingerprint, render JavaScript, or fix a parser that broke when the site changed.

On cost, SparkProxy plans start from $75/month (USD, from the pricing page as of 9 October 2026), sold as packs with unlimited bandwidth, so a cron job that retries after a failure doesn't add a traffic bill. There's a 24-hour free trial with no credit card.

If you'd rather not manage proxy variables at all, the Scraping API needs only an API key in one header:

import base64
import os

import requests

resp = requests.get(
    "https://scrape.sparkproxy.io/api/v1",
    headers={"X-API-Key": os.environ["SPARKPROXY_API_KEY"]},
    params={
        "url": "https://www.sparkproxy.io/",
        "render_js": "false",      # plain fetch, 1 credit (render_js defaults to true, 5 credits)
        "proxy_type": "premium",
        "json_response": "true",   # envelope: job_id, status_code, credits_used, body
    },
    timeout=60,
)
envelope = resp.json()
html = base64.b64decode(envelope["body"]).decode("utf-8", errors="replace")
print(envelope["status_code"], envelope["credits_used"], len(html))

The envelope's body is base64, so decode it before matching anything in it. format=json is a content conversion of the page, not this envelope. New accounts get 1,000 free credits.