AWS Lambda Proxy Setup for Scraping: Code and Fixes

A function outside a VPC gets its outbound address from a Lambda-managed network, and that address sits inside the ranges AWS publishes for its own services. Bot managers and WAFs consume those published lists and score or block the whole lot, so the block is about where the packet comes from, not about your headers. Our explainer on cloud provider IP ranges covers how sites ingest those files; for this job, the takeaway is short. No header change, user-agent rotation or retry loop will move your request out of an AWS range. Only a different egress address does that.

Two things a proxy will not fix, so you can stop testing them now: a target that rejects your TLS or browser fingerprint regardless of IP, and a target behind AWS WAF's own bot rules that challenges every non-browser client. The second one has its own playbook in our guide to scraping sites behind AWS WAF.

With a proxy configured, each fetch from the handler opens a TCP connection to the gateway, the gateway picks an exit IP from its pool, and the target sees that exit instead of the Lambda address. For HTTPS, the client sends CONNECT target:443 to the gateway and runs TLS inside the tunnel, so the gateway never sees your request content.

EventBridge Scheduler / SQS
        |
        v
  Lambda handler  --(user:pass or whitelisted IP)-->  gateway.sparkproxy.io:11000 / 11002
        |                                                     |
        |                                               exit IP from pool
        v                                                     v
   S3 / DynamoDB  <------------ parsed result ---------  target site

Three settings in that path are Lambda-specific and catch people out. Credentials have to reach the function without landing in the code bundle. Connections opened in one invocation survive into the next while the environment stays warm. And if you attach the function to a VPC, it loses internet access entirely until a NAT gateway exists, proxy or no proxy.

The SparkProxy gateway listens on port 11000 for Random Rotation (a new exit per connection), 11002 for Sticky Sessions (the same exit for up to 5 minutes) and 13000 for SOCKS5. The examples below use the HTTP ports, because every client here speaks HTTP proxy natively.

You have two sane choices. Encrypted environment variables are the simplest: Lambda encrypts them at rest with KMS, and the handler reads them with os.environ. The aggregate limit is 4 KB for all variables on a function, which is plenty for a username and password. The weakness is rotation: changing the password means updating every function's configuration.

Secrets Manager suits a fleet. Store one JSON secret, grant secretsmanager:GetSecretValue on it, and read it through the AWS Parameters and Secrets Lambda Extension, a layer that serves secrets on localhost:2773 and caches them for 300 seconds by default. Rotating the proxy password then reaches every function within five minutes without a deploy.

# One secret for the whole fleet
aws secretsmanager create-secret \
  --name sparkproxy/scraper \
  --secret-string '{"username":"YOUR_USER","password":"YOUR_PASS"}'

# Or plain env vars for a single function
aws lambda update-function-configuration \
  --function-name price-scraper \
  --environment "Variables={PROXY_USER=YOUR_USER,PROXY_PASS=YOUR_PASS,NO_PROXY=localhost,127.0.0.1}"

One trap deserves a sentence of its own. Do not set HTTPS_PROXY as a function-wide environment variable to "turn the proxy on". boto3 and requests both read proxy variables from the environment, so in a Python function your S3 writes and the secrets lookup would go through the scraping proxy, and the extension call to localhost:2773 fails unless NO_PROXY covers it. Pass the proxy explicitly to the scraping client and leave the environment clean.

If you would rather hold no credentials at all, IP whitelisting is the third option, and it is the reason the NAT section below exists.

This handler reads the secret once per warm environment, builds a requests.Session pointed at the gateway, and stops starting new fetches when the invocation is close to its deadline. trust_env = False keeps the session from picking up stray proxy variables.

import json
import time
import urllib.parse

import boto3
import requests

GATEWAY = "gateway.sparkproxy.io"
PORTS = {"rotate": 11000, "sticky": 11002}
SECRET_ID = "sparkproxy/scraper"
_cache = {"creds": None, "at": 0.0}


def proxy_creds():
    """Fetch proxy credentials via the Parameters and Secrets extension."""
    if _cache["creds"] and time.time() - _cache["at"] < 300:
        return _cache["creds"]
    token = boto3.Session().get_credentials().get_frozen_credentials().token
    resp = requests.get(
        "http://localhost:2773/secretsmanager/get?secretId=" + SECRET_ID,
        headers={"X-Aws-Parameters-Secrets-Token": token},
        timeout=3,
    )
    resp.raise_for_status()
    creds = json.loads(resp.json()["SecretString"])
    _cache.update(creds=creds, at=time.time())
    return creds


def build_session(mode):
    c = proxy_creds()
    user = urllib.parse.quote(c["username"], safe="")
    pwd = urllib.parse.quote(c["password"], safe="")
    proxy = f"http://{user}:{pwd}@{GATEWAY}:{PORTS[mode]}"
    s = requests.Session()
    s.trust_env = False
    s.proxies = {"http": proxy, "https": proxy}
    return s


def lambda_handler(event, context):
    session = build_session(event.get("mode", "rotate"))
    results = []
    for url in event["urls"]:
        left = context.get_remaining_time_in_millis() / 1000
        if left < 15:
            break  # return the rest instead of being killed mid-request
        try:
            r = session.get(url, timeout=(5, min(25, left - 10)))
            results.append({"url": url, "status": r.status_code, "bytes": len(r.content)})
        except requests.exceptions.ProxyError as exc:
            results.append({"url": url, "error": "proxy", "detail": str(exc)[:300]})
        except requests.exceptions.Timeout:
            results.append({"url": url, "error": "timeout"})
    done = {r["url"] for r in results}
    return {"results": results, "unfinished": [u for u in event["urls"] if u not in done]}

Package requests with the function (pip install requests -t .); boto3 ships in the Python runtime, though AWS recommends bundling the SDK version you test against. Percent-encoding the password matters: a @ or : in it otherwise breaks the proxy URL, and the error looks like a DNS failure on a hostname made of half your password.

Notice the session is created inside the handler, not at module scope. A module-level session keeps its kept-alive tunnel across warm invocations, and because a rotating gateway can only pick a new exit when a new tunnel opens, a warm function can keep hitting the target from one IP for many invocations in a row. Build it per invocation when you want spread; build it once when you want the opposite.

Node's global fetch is built on undici, and undici's ProxyAgent is the cleanest way to route it. Add undici to your bundle, since the runtime does not expose it as an importable package. The token option sends a pre-formatted proxy-authorization header, which avoids URL-encoding problems with special characters.

// index.mjs  (nodejs22.x or nodejs24.x)
import { ProxyAgent, fetch } from "undici";

const user = process.env.PROXY_USER;
const pass = process.env.PROXY_PASS;

function agentFor(port) {
  return new ProxyAgent({
    uri: `http://gateway.sparkproxy.io:${port}`,
    token: "Basic " + Buffer.from(`${user}:${pass}`).toString("base64"),
  });
}

export const handler = async (event, context) => {
  const agent = agentFor(event.sticky ? 11002 : 11000);
  const out = [];
  try {
    for (const url of event.urls) {
      if (context.getRemainingTimeInMillis() < 15000) break;
      try {
        const res = await fetch(url, {
          dispatcher: agent,
          signal: AbortSignal.timeout(25000),
        });
        out.push({ url, status: res.status, len: (await res.text()).length });
      } catch (err) {
        out.push({ url, error: String(err.cause ?? err).slice(0, 300) });
      }
    }
  } finally {
    await agent.close();
  }
  return out;
};

Logging err.cause is not optional. undici's top-level message is a bare TypeError: fetch failed, and the useful part (a refused connection, a 407 from the proxy, a timeout) only appears in the cause. Recent Node releases are adding built-in proxy support driven by environment variables (the http.Agent proxyEnv option arrived in v24.5.0 and v22.21.0, per the Node docs), but an explicit dispatcher scopes the proxy to scraping traffic only, which is what you want inside Lambda.

Full Chromium does not fit a zip function comfortably: Lambda caps a function plus its layers at 250 MB unzipped and five layers per function. The usual answer is @sparticuz/chromium, a stripped headless build packaged for Lambda. When we read its repository on 2026-10-30 it had about 1.7k stars and 532 commits, and the releases page listed v153.0.0 (Chromium 153.0.8010.36, published 11 Sep) and v152.0.0 (published 8 Sep), so it is actively released. Its README asks for at least 512 MB of memory and recommends 1600 MB or more; at 1,769 MB a function gets the equivalent of one vCPU, per the Lambda quotas page.

Publish the browser as a layer so code deploys stay small:

# Layer zip from the project's GitHub release, uploaded to your bucket
aws s3 cp chromium-v153.0.0-layer.x64.zip s3://your-artifacts-bucket/layers/
aws lambda publish-layer-version \
  --layer-name chromium-153 \
  --content S3Bucket=your-artifacts-bucket,S3Key=layers/chromium-v153.0.0-layer.x64.zip \
  --compatible-runtimes nodejs22.x nodejs24.x \
  --compatible-architectures x86_64

Chromium takes the proxy as a launch flag, --proxy-server=http://host:port, and that flag has no field for a username or password. You either answer the proxy's 407 challenge from the automation library or remove the challenge by whitelisting the function's egress IP. With Puppeteer:

import chromium from "@sparticuz/chromium";
import puppeteer from "puppeteer-core";

const exePath = chromium.executablePath(); // started once per environment

export const handler = async (event) => {
  const browser = await puppeteer.launch({
    args: await puppeteer.defaultArgs({
      args: [...chromium.args, "--proxy-server=http://gateway.sparkproxy.io:11002"],
      headless: "shell",
    }),
    executablePath: await exePath,
    headless: "shell",
  });
  try {
    const page = await browser.newPage();
    if (process.env.PROXY_USER) {
      await page.authenticate({
        username: process.env.PROXY_USER,
        password: process.env.PROXY_PASS,
      });
    }
    const res = await page.goto(event.url, { waitUntil: "domcontentloaded", timeout: 45000 });
    return { status: res?.status(), title: await page.title() };
  } finally {
    await browser.close();
  }
};

Port 11002 is deliberate. A rendered page pulls scripts, XHR calls and images over many connections, and if each lands on a different exit IP the site sees one "visitor" spread across a dozen addresses. Sticky Sessions keep the whole page load on one exit. Puppeteer's docs also note that page.authenticate turns on request interception behind the scenes, which can cost performance, one more reason the whitelisted-IP route below is attractive for browser jobs. Our Puppeteer proxy guide covers the library side in more depth.

Playwright takes credentials directly in its proxy option. The @sparticuz/chromium README warns that Playwright does not clean its profile directory on a warm Lambda, so /tmp fills until launches fail with ERR_INSUFFICIENT_RESOURCES. Give each invocation its own profile and delete it:

import { chromium as pw } from "playwright-core";
import chromium from "@sparticuz/chromium";
import { mkdtemp, rm } from "node:fs/promises";

export const handler = async (event) => {
  const dir = await mkdtemp("/tmp/pw-");
  const ctx = await pw.launchPersistentContext(dir, {
    args: chromium.args,
    executablePath: await chromium.executablePath(),
    proxy: {
      server: "http://gateway.sparkproxy.io:11002",
      username: process.env.PROXY_USER,
      password: process.env.PROXY_PASS,
    },
  });
  try {
    const page = await ctx.newPage();
    await page.goto(event.url, { timeout: 45000 });
    return await page.title();
  } finally {
    await ctx.close();
    await rm(dir, { recursive: true, force: true });
  }
};

Pin puppeteer-core or playwright-core to a release that supports the Chromium major in your layer. The @sparticuz/chromium version numbers track the Chromium major, and the project warns that breaking changes can land in a patch release, so pin exact versions in package.json.

Lambda adds a wrinkle to the Sticky Sessions choice. Five minutes is shorter than the 15-minute maximum invocation, so a long-running function doing a multi-step flow can change exit mid-flow. Keep each sticky unit of work comfortably inside the window, or split the flow across invocations that each start fresh.

Some setups need every request from the function to leave from one address you can name in advance: a proxy account authenticated by IP whitelist instead of a password, or a partner API that only accepts allowlisted callers. Lambda outside a VPC cannot give you that, because its egress address is not yours and changes. The fix is to attach the function to private subnets and route them through a NAT gateway that owns an Elastic IP. AWS's own guide is explicit that putting the function in a public subnet does not work: "Connecting a function to a public subnet doesn't give it internet access."

EIP=$(aws ec2 allocate-address --domain vpc --query AllocationId --output text)
NAT=$(aws ec2 create-nat-gateway --subnet-id subnet-PUBLIC-A \
  --allocation-id "$EIP" --query NatGateway.NatGatewayId --output text)
aws ec2 wait nat-gateway-available --nat-gateway-ids "$NAT"
aws ec2 create-route --route-table-id rtb-PRIVATE-A \
  --destination-cidr-block 0.0.0.0/0 --nat-gateway-id "$NAT"
aws lambda update-function-configuration --function-name price-scraper \
  --vpc-config SubnetIds=subnet-PRIVATE-A,subnet-PRIVATE-B,SecurityGroupIds=sg-0123456789abcdef0

Confirm the egress before you whitelist anything. Invoke a test function that fetches https://checkip.amazonaws.com with no proxy; the body should be the Elastic IP. Then add that address in the SparkProxy dashboard, and Chromium can run with a bare --proxy-server=http://gateway.sparkproxy.io:11002, no page.authenticate, no credentials in the function at all. Our explainer on IP whitelisting for proxies covers how the gateway matches the source address.

Three consequences people miss:

• The Elastic IP does not unblock anything. It is still an AWS address. It exists so the proxy or partner can recognise you; the proxy exit is what the target sees.
• One NAT per Availability Zone means one IP per zone. AWS recommends a NAT gateway per AZ for resilience, and each has its own Elastic IP. Two zones means two whitelist entries; SparkProxy plans include 5 to 25 whitelist slots depending on tier.
• Everything else in the VPC loses its route too. The secrets extension reaches Secrets Manager through the NAT now, or through a Secrets Manager VPC endpoint if you add one. Without either, the extension call hangs and the function dies with a timeout that looks like a proxy problem.

The default function timeout is 3 seconds. A first proxied HTTPS request, with DNS, the proxy handshake, the CONNECT tunnel and TLS to the target, can eat most of that alone, so a freshly created function often fails with the exact line AWS documents:

2024-04-11T17:22:20.857Z abe12jlc-640a-8157-0249-9be825c2y110 Task timed out after 3.01 seconds

Raise it to fit the job (the maximum is 900 seconds), and then budget inside it: the handlers above check remaining time before each fetch and return unfinished URLs instead of being killed. Our post on proxy timeouts and retry logic explains how to size connect and read timeouts from measured latency; on Lambda the extra rule is that the sum of your retries must fit under the function timeout, or the last retry never reports.

The Init phase, where your module-level code runs, is limited to 10 seconds. If it overruns, Lambda logs INIT_REPORT Init Duration: ... Phase: init Status: timeout and retries init during the first invocation under the function timeout. Launching Chromium or fetching secrets with long timeouts at module scope is the usual cause. Start the browser inside the handler; at module scope, only kick off cheap work such as resolving executablePath().

Other Lambda-only failures worth recognising:

• /tmp keeps its contents across warm invocations and even after an invoke failure resets the environment. Browser profiles, screenshots and downloads accumulate until the 512 MB default fills. Raise ephemeral storage (up to 10,240 MB) or delete per invocation.
• Background work resumes later. An unawaited promise or open socket from one invocation continues when the environment thaws for the next. Close the browser and the proxy agent in finally.
• A timed-out invocation resets the environment, so the next one pays a cold start and a fresh Chromium launch. A function that times out repeatedly gets slower, which makes it time out more.

Work down the list. Each step takes seconds and rules out a whole layer.

1. Does the function reach the internet at all? If it is in a VPC, test https://checkip.amazonaws.com with no proxy. A timeout here is routing (no NAT, wrong route table, function in a public subnet), not a proxy issue.
2. Does the gateway accept you? Run the same request through port 11000 and read the error text.
3. Is the block coming from the target? A 200 from the proxy tunnel but a 403 or CAPTCHA page from the site moves you to fingerprint or rate questions.
4. Is it Lambda's clock? Compare the failure time with the function timeout and the REPORT line's Duration.

A scraper on Lambda is usually a scheduler, a queue and a worker. EventBridge Scheduler fires a small "planner" function that writes URL batches to SQS; an SQS event source mapping drives the worker function; the worker writes results to S3.

aws scheduler create-schedule --name hourly-prices \
  --schedule-expression "rate(1 hour)" \
  --flexible-time-window Mode=OFF \
  --target '{"Arn":"arn:aws:lambda:us-east-1:111122223333:function:planner","RoleArn":"arn:aws:iam::111122223333:role/scheduler-invoke"}'

# Cap worker concurrency so it never exceeds your proxy thread allowance
aws lambda put-function-concurrency --function-name price-scraper \
  --reserved-concurrent-executions 50

Reserved concurrency is the lever that matters for the proxy. Every concurrent worker holds at least one gateway connection, and a Chromium worker holds several. SparkProxy plans are sized by threads (Starter 100, Core 250, Boost 500, Plus 1,000), and Lambda's default account limit of 1,000 concurrent executions can blow straight through a smaller allowance during a backlog. Set reserved concurrency to roughly your threads divided by connections per worker.

Write raw pages to S3 keyed by date and URL hash, so a parser bug never forces a re-scrape:

import datetime
import hashlib

import boto3

s3 = boto3.client("s3")  # module scope: reused while warm


def store(bucket, url, html):
    day = datetime.date.today().isoformat()
    key = f"raw/{day}/{hashlib.sha256(url.encode()).hexdigest()}.html"
    s3.put_object(Bucket=bucket, Key=key, Body=html.encode("utf-8"),
                  ContentType="text/html; charset=utf-8")
    return key

Return the unfinished list from the worker and push it back onto the queue. Failed messages go to a dead-letter queue after a few receives, which gives you a list of URLs that fail consistently instead of silent gaps.

If the Chromium layer, memory sizing and /tmp hygiene are more than the job is worth, the function can call the SparkProxy Scraping API and receive finished HTML. The browser, proxy and retries run on our side, so the function needs only the standard library and an API key.

import base64
import json
import os
import urllib.parse
import urllib.request

API = "https://scrape.sparkproxy.io/api/v1"


def scrape(url, render_js=True, country=None):
    params = {"url": url, "render_js": str(render_js).lower(), "json_response": "true"}
    if country:
        params["country_code"] = country
    req = urllib.request.Request(
        API + "?" + urllib.parse.urlencode(params),
        headers={"X-API-Key": os.environ["SPARK_API_KEY"]},
    )
    with urllib.request.urlopen(req, timeout=240) as resp:
        envelope = json.load(resp)
    # json_response wraps the page; the body is base64 and must be decoded first
    html = base64.b64decode(envelope["body"]).decode("utf-8", "replace")
    return envelope["status_code"], envelope["credits_used"], html

render_js defaults to true and costs 5 credits; render_js=false is a plain fetch at 1 credit. country_code adds 5 credits, and proxy_type=premium moves to the premium pool (10 credits without JS, 25 with, per the docs read 2026-10-30). Marker matching must run on the decoded html, never on the raw envelope. The docs say navigation retries server-side at 90, 120 and 180 seconds and failed jobs return 530 with credits refunded, so set the function timeout to several minutes and invoke asynchronously rather than through an API Gateway request that would give up first. New accounts get 1,000 free credits.

SparkProxy plans start from $75/month (USD, excluding VAT/GST, read 2026-10-30) and are packs sized by threads with unlimited bandwidth, so a function that downloads heavy rendered pages costs the same proxy fee as one that fetches small JSON. Term discounts are 5% for 3 months, 10% for 6 and 15% for 12, and the 24-hour free trial needs no credit card.

The AWS side is where bandwidth starts costing again, but only if you add a VPC. AWS's VPC pricing page, read 2026-10-30, gives a US East (Ohio) example of $0.045 per NAT gateway-hour and $0.045 per GB processed (USD), plus $0.005 per hour for each public IPv4 address. At 730 hours that is about $32.85 a month per NAT gateway before data, and every byte your scraper pulls through the proxy also passes through the NAT and is billed per GB. That flips the "unlimited bandwidth" advantage for rendered scraping, so attach the function to a VPC only when you need the fixed address. Credentials over env vars or Secrets Manager need no NAT.