Website Blocks AWS IP but Works Locally: Fix the 403

Before you touch any code, save one failing response with its headers. The body and two or three headers usually name the system that blocked you, and that decides everything after.

Cloudflare documents both codes on its own support pages, Error 1005 and Error 1020. The cf-ray header is the Ray ID the site owner would search for in their logs. AWS keeps its list of CloudFront 403 causes in the CloudFront developer guide.

A 1005 is the cleanest signal you will ever get. It names the ASN as the reason, so you can skip straight to the fix. Everything else needs the test below.

Your laptop exits through a consumer ISP, so IP lookup databases label it as a home connection. Your server exits through AS16509 (Amazon), AS15169 (Google), AS14061 (DigitalOcean) or a similar hosting network, and those labels are what the block rule reads. AWS, Google Cloud and most others publish their full address lists as JSON, so a site can deny every EC2 address with a cron job and no detection logic at all. We cover where those files live and how sites consume them in what cloud provider IP ranges are and why they get blocked, and how ASN classification works in what a datacenter ASN is.

The practical consequence: the request can be byte-for-byte identical to the one that worked at home and still fail on its first attempt, because the rule fires on the source address before your headers are even read.

But the IP is not the only thing that changed when you deployed. The server also has a different region, a different Python or Node build, possibly a headless browser where you ran a headed one, and a fresh environment with none of your local cookies. Any of those can produce a 403 too. The diagnosis order exists to separate them.

Run these in order on the deployed machine itself, not from your laptop. Stop at the first step that gives a clear answer.

Steps 1 and 2 are one paste:

# Step 1: keep the evidence
curl -s -D headers.txt -o body.html -w "%{http_code}\n" \
  -A "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0 Safari/537.36" \
  "https://shop.sparkproxy.io/products?page=1"
grep -iE "^(server|cf-ray|x-amz-cf-id):" headers.txt
grep -oE "error code: [0-9]+|Access denied" body.html | head -3

# Step 2: what the target actually sees
curl -s https://ipinfo.io/json      # check "org" and "country"

Watch out for one trap in step 2. On Lambda inside a VPC, ECS tasks in private subnets, or anything behind a NAT gateway, the public egress address is the NAT gateway's Elastic IP, not the instance or task IP you see in the console. Ask an external echo service, as above, rather than reading ifconfig.

To confirm the ASN independently of IPinfo, Team Cymru answers over plain DNS. Reverse the four octets of the IP and query the origin.asn.cymru.com zone. When we ran it on 2026-10-30 against an AWS address:

nslookup -type=TXT 248.236.94.52.origin.asn.cymru.com 8.8.8.8
# "16509 | 52.94.224.0/20 | US | arin | 1991-12-19"

curl -s https://ipinfo.io/52.94.236.248/org
# AS16509 Amazon.com, Inc.

If the org line names Amazon, Google, Microsoft, DigitalOcean, OVH, Hetzner, Linode/Akamai or Oracle, you have a strong suspect. Step 3 turns the suspicion into proof.

The decisive test is the same request, from the same box, at the same moment, with only the exit IP changed. If the proxied request gets a 200 and the direct one a 403, nothing else is in play: same code, same TLS stack, same headers, same region.

Start with curl, since it removes your scraper's own code from the picture:

TARGET="https://shop.sparkproxy.io/products?page=1"
UA="Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0 Safari/537.36"
PROXY="http://USER:PASS@gateway.sparkproxy.io:11000"

curl -s -o /dev/null -w "direct: %{http_code}\n" -A "$UA" "$TARGET"
curl -s -o /dev/null -w "proxy:  %{http_code}\n" -A "$UA" -x "$PROXY" "$TARGET"
curl -s -x "$PROXY" https://ipinfo.io/org   # confirm the exit is no longer AS16509

Then repeat it from inside your scraper's runtime, because a Python or Node process can behave differently from curl. This script sends a handful of paired requests and labels the vendor on every block it sees:

import os
import requests

TARGET = "https://shop.sparkproxy.io/products?page=1"
PROXY = os.environ["SPARKPROXY_URL"]  # http://USER:PASS@gateway.sparkproxy.io:11000
HEADERS = {"User-Agent": "Mozilla/5.0 (X11; Linux x86_64) Chrome/130.0 Safari/537.36"}


def vendor(resp):
    h = {k.lower(): v for k, v in resp.headers.items()}
    if "cf-ray" in h:
        return "cloudflare"
    if "x-amz-cf-id" in h:
        return "cloudfront"
    return h.get("server", "unknown")


def probe(use_proxy):
    s = requests.Session()
    s.trust_env = False  # ignore any HTTP(S)_PROXY already set on the box
    if use_proxy:
        s.proxies = {"http": PROXY, "https": PROXY}
    r = s.get(TARGET, headers=HEADERS, timeout=30)
    code = None
    if "error code:" in r.text[:200]:
        code = r.text.strip()[:20]
    return r.status_code, vendor(r), code


for i in range(5):
    print("direct", probe(False), "| proxy", probe(True))

Five pairs are enough. A categorical IP rule fails on every direct attempt, including the first one, which is how you tell it apart from rate limiting.

Put the four observations side by side. Step 4 matters more than people expect: run your built container or deployment bundle on the laptop, not your editable source tree.

How to check whether a request was blocked rather than served a thin page is its own topic, covered in how to detect when your scraper is blocked.

A proxy changes one thing, the source address. Several deploy-time failures look like IP blocks and have nothing to do with the address.

The last two rows deserve emphasis because they waste hours. A blocked IP gets an HTTP response. A network you can't leave gets no response at all. AWS documents that a VPC-attached Lambda loses internet access unless the VPC provides it through a NAT gateway and route table entries. If your error is ReadTimeout or ETIMEDOUT, stop reading about IP reputation.

When the target sits behind AWS WAF and you need to understand which managed rules apply, we went through that in bypassing AWS WAF when web scraping.

The fastest-looking fix is setting HTTPS_PROXY on the whole instance or function. In Python it works, and that is the problem.

Requests reads http_proxy, https_proxy, no_proxy and all_proxy from the environment, in either case, per its advanced usage docs. Boto3 does the same: its configuration guide says you "can use the HTTP_PROXY and HTTPS_PROXY environment variables to specify proxy servers." So a global variable sends your S3 uploads, SQS polls, DynamoDB writes and Secrets Manager reads out through the proxy as well. That adds latency to every AWS call, ships traffic to a third party that never needed to see it, and burns proxy concurrency on requests that were never blocked.

Node behaves in the opposite way, which confuses people moving between stacks. Node's built-in proxy support is opt-in: the global agent only honours HTTP_PROXY, HTTPS_PROXY and NO_PROXY when NODE_USE_ENV_PROXY=1 or --use-env-proxy is set, a feature the Node.js HTTP docs mark as added in v24.5.0 and v22.21.0. Without that flag, setting HTTPS_PROXY on a Node Lambda changes nothing and the 403 carries on, which looks like "the proxy didn't help" when the proxy was never used.

If you must set the variable globally for some reason, put AWS endpoints and the instance metadata addresses on the bypass list as a precaution:

NO_PROXY=localhost,127.0.0.1,169.254.169.254,169.254.170.2,.amazonaws.com

The better design is to scope the proxy to the one client that talks to the target.

Give the scraper its own session with trust_env = False, so it ignores the environment entirely, and leave every SDK client on defaults. Use a distinct variable name such as SPARKPROXY_URL that no library reads on its own.

import os

import boto3
import requests

PROXY = os.environ["SPARKPROXY_URL"]

# Scraping traffic: explicit proxy, environment ignored
scrape = requests.Session()
scrape.trust_env = False
scrape.proxies = {"http": PROXY, "https": PROXY}
scrape.headers["User-Agent"] = "Mozilla/5.0 (X11; Linux x86_64) Chrome/130.0 Safari/537.36"

# AWS traffic: untouched, goes straight to AWS endpoints
s3 = boto3.client("s3")


def handler(event, context):
    url = event["url"]
    r = scrape.get(url, timeout=30)
    r.raise_for_status()
    key = "raw/" + url.rsplit("/", 1)[-1] + ".html"
    s3.put_object(Bucket="scrape-output", Key=key, Body=r.content)
    return {"status": r.status_code, "key": key}

With httpx, the argument is proxy= (singular). The changelog shows proxy was added in 0.26.0 and the old proxies argument was removed in 0.28.0; the current release on PyPI is 0.28.1, checked 2026-10-30.

import os

import httpx

client = httpx.Client(
    proxy=os.environ["SPARKPROXY_URL"],
    trust_env=False,
    timeout=30.0,
)
resp = client.get("https://shop.sparkproxy.io/products?page=1")
print(resp.status_code)

Playwright lets you attach the proxy to one browser context rather than the whole browser, per its network docs. That keeps any other contexts, such as a health check against your own app, on the direct path.

import os

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch()
    scrape_ctx = browser.new_context(proxy={
        "server": "http://gateway.sparkproxy.io:11000",
        "username": os.environ["SPARKPROXY_USER"],
        "password": os.environ["SPARKPROXY_PASS"],
    })
    page = scrape_ctx.new_page()
    resp = page.goto("https://shop.sparkproxy.io/products?page=1")
    print(resp.status if resp else "no response")
    browser.close()

In Node, create an undici ProxyAgent and pass it as the dispatcher on the requests that go to the target. The undici ProxyAgent docs describe it as routing every request through an HTTP, HTTPS or SOCKS5 proxy wherever a dispatcher is accepted. Importing fetch from the same package avoids version mismatches with Node's bundled copy.

import { fetch, ProxyAgent } from 'undici';
import { S3Client, PutObjectCommand } from '@aws-sdk/client-s3';

const scrapeAgent = new ProxyAgent(process.env.SPARKPROXY_URL);
const s3 = new S3Client({}); // no proxy: direct to AWS

export async function handler(event) {
  const res = await fetch(event.url, {
    dispatcher: scrapeAgent,
    headers: { 'user-agent': 'Mozilla/5.0 (X11; Linux x86_64) Chrome/130.0 Safari/537.36' },
  });
  const html = await res.text();
  await s3.send(new PutObjectCommand({
    Bucket: 'scrape-output',
    Key: `raw/${Date.now()}.html`,
    Body: html,
  }));
  return { status: res.status };
}

Do not call setGlobalDispatcher(scrapeAgent) here. It turns a scoped fix back into a global one.

The code above only needs one secret. Where it lives depends on the platform.

Lambda. Store the proxy URL as an environment variable named something no library recognises. For production, keep it in Secrets Manager and read it at cold start instead.

aws lambda update-function-configuration \
  --function-name price-scraper \
  --environment "Variables={SPARKPROXY_URL=http://USER:PASS@gateway.sparkproxy.io:11000}"

Authentication choice. SparkProxy supports both IP whitelist and user:pass auth. A Lambda function that is not attached to a VPC has no fixed egress address, so IP whitelisting can't work there; use user:pass. If your functions or tasks run in private subnets behind a NAT gateway, AWS requires an Elastic IP on a public NAT gateway, per the VPC NAT gateway docs, and that one static address is what you whitelist.

Containers. Inject SPARKPROXY_URL at runtime, never at build time, and let the code apply it. The Docker-specific layers (daemon, build args, Compose) are already in how to use proxies with Docker.

A plain VPS. A systemd unit with an environment file keeps the credential out of your repo and out of the shell history:

# /etc/systemd/system/scraper.service
[Service]
EnvironmentFile=/etc/scraper/proxy.env
ExecStart=/usr/bin/python3 /opt/scraper/run.py
Restart=on-failure

proxy.env holds a single line, SPARKPROXY_URL=http://USER:PASS@gateway.sparkproxy.io:11000, with chmod 600.

For targets that care about location, add the country to the username, for example USER-country-de, so a server sitting in us-east-1 exits from Germany. For multi-step flows such as a login followed by paginated requests, use Sticky Sessions on port 11002 so the same exit IP holds for five minutes; for independent page fetches, Random Rotation on port 11000 spreads the load.

Be clear about what the gateway is. SparkProxy's proxy pool is datacenter address space. It is not hosted on AWS, Google Cloud, Azure or DigitalOcean, so it is absent from their published range files, and that is why it gets past the cron-job tier of blocking. It still sits on hosting ASNs. A site that bans hosting networks wholesale, which a Cloudflare 1005 often means, will refuse it just as it refused your EC2 instance, and no datacenter provider can honestly promise otherwise.

For those targets, keep the gateway for everything else and send only the strict targets through the Scraping API with proxy_type=premium, which picks a residential exit. Use json_response=true to get the envelope with the target's real status_code, and decode the body, which arrives base64-encoded:

import base64
import os
from urllib.parse import urlparse

import requests

API = "https://scrape.sparkproxy.io/api/v1"
PROXY = os.environ["SPARKPROXY_URL"]
RESIDENTIAL_HOSTS = {"tickets.sparkproxy.io"}  # targets that returned 1005

gateway = requests.Session()
gateway.trust_env = False
gateway.proxies = {"http": PROXY, "https": PROXY}


def fetch(url):
    if urlparse(url).hostname not in RESIDENTIAL_HOSTS:
        r = gateway.get(url, timeout=30)
        return r.status_code, r.text

    r = requests.get(API, headers={"X-API-Key": os.environ["SPARKPROXY_API_KEY"]}, params={
        "url": url,
        "render_js": "false",
        "proxy_type": "premium",
        "json_response": "true",
    }, timeout=120)
    r.raise_for_status()
    env = r.json()
    html = base64.b64decode(env["body"]).decode("utf-8", "replace")
    return env.get("status_code"), html

Two contract details matter here. format=json is content conversion (headings, links, metadata), not the envelope, so it will not give you status_code. And the docs note that status_code can be null when the page did not respond normally, so treat None as a failure rather than a success.

The two routes are billed differently, which affects how you split targets.

Because the gateway is billed by threads and not by traffic, a scraper pulling large pages costs the same as one pulling small ones. The number to plan around is concurrency: set your worker pool size to your thread count, not to whatever your Lambda reserved concurrency happens to be. The API's credit model works the other way, so reserve it for the hosts that actually refuse datacenter exits. There is a 24-hour free trial with no credit card, and the API account starts with 1,000 free credits, which is plenty to run the A/B test above against every target on your list before you commit.