Crawlee Proxy Configuration: Rotation That Keeps Sessions

Symptom one, the retire loop. Requests to one site keep coming back 401, 403 or 429. Every response retires the session that carried it, the request is retried on a fresh session, the fresh session gets the same status, and the request burns its retries. If the target is genuinely blocking an IP, rotating is correct. If the 403 means "your login cookie expired" or "you sent this from the wrong country", rotation just throws away the one thing that was working, the session's cookies.

Symptom two, the silent logout. Nothing errors. Page seven of a paginated listing simply returns the login page or an empty result set, because the request ran on a session that never logged in, or on one whose exit IP changed under it.

A proxy can fix exactly one class of cause behind these: the target rejecting the address. It cannot fix expired cookies, a missing CSRF token, a TLS or browser fingerprint check, or a site-wide rate limit on your account. The rest of this post is about telling those apart inside Crawlee's own machinery. For a general method of reading block signals, our guide on detecting when your scraper is blocked covers response markers outside Crawlee.

Crawlee changes between releases, and the retry behaviour below is version-specific. As of 30 October 2026, the latest stable JavaScript release on GitHub is v3.18.2 (published 29 September 2026), with v4.0.0-rc.1 published 6 October 2026 as a release candidate. Crawlee for Python's latest release is v1.11.0 (15 October 2026). Both repositories had commits pushed on 30 October 2026, so both projects are actively maintained.

These defaults are read from the v3.18.2 and v1.11.0 source and API reference:

Two numbers matter most here. A session lives 50 minutes by default, and retire() adds the full maxErrorScore to the session in one call, so a single blocked status ends that session immediately.

ProxyConfiguration accepts exactly one of three options, in both languages. Passing two throws.

The simplest working setup is a single URL. With SparkProxy, the gateway host is gateway.sparkproxy.io. Port 11000 gives Random Rotation, a fresh exit per connection; port 11002 gives Sticky Sessions, which hold the same exit for up to 5 minutes; port 13000 is SOCKS5.

// crawl.mjs : Crawlee for JavaScript v3.18.2
import { CheerioCrawler, ProxyConfiguration } from 'crawlee';

const user = encodeURIComponent(process.env.SPARK_USER);
const pass = encodeURIComponent(process.env.SPARK_PASS);

const proxyConfiguration = new ProxyConfiguration({
    proxyUrls: [`http://${user}:${pass}@gateway.sparkproxy.io:11000`],
});

const crawler = new CheerioCrawler({
    proxyConfiguration,
    async requestHandler({ request, $, proxyInfo, session, log }) {
        log.info(`${request.url} via port ${proxyInfo?.port} on ${session?.id}`);
    },
});

await crawler.run(['https://www.sparkproxy.io/blog']);

Encode the username and password. A @, : or # in a raw password produces a URL that parses into the wrong host, and the failure shows up as a connection error, not an auth error.

The Python equivalent:

# crawl.py : Crawlee for Python v1.11.0
import asyncio
import os
from urllib.parse import quote

from crawlee.crawlers import BeautifulSoupCrawler, BeautifulSoupCrawlingContext
from crawlee.proxy_configuration import ProxyConfiguration

USER = quote(os.environ["SPARK_USER"], safe="")
PASS = quote(os.environ["SPARK_PASS"], safe="")


async def main() -> None:
    proxy_configuration = ProxyConfiguration(
        proxy_urls=[f"http://{USER}:{PASS}@gateway.sparkproxy.io:11000"],
    )
    crawler = BeautifulSoupCrawler(proxy_configuration=proxy_configuration)

    @crawler.router.default_handler
    async def handler(context: BeautifulSoupCrawlingContext) -> None:
        session_id = context.session.id if context.session else None
        context.log.info(f"{context.request.url} via {context.proxy_info} on {session_id}")

    await crawler.run(["https://www.sparkproxy.io/blog"])


if __name__ == "__main__":
    asyncio.run(main())

If you whitelist your server's IP instead of using credentials, drop the user:pass@ part. The trade-offs of that choice are in our explainer on IP whitelisting for proxies.

The Crawlee docs say it plainly: once a session ID is passed to newUrl(), "subsequent newUrl() calls with the same sessionId will always return the same proxyUrl." In the v3.18.2 source this is a Map from session ID to URL string, filled the first time a session asks. Python keeps the same thing as a dictionary.

Read that carefully. The binding is between a session and a URL string. Crawlee never sees an IP address. With a list of ten single-IP proxies, URL and IP are the same thing and the binding does what people expect: session A always leaves from proxy 3. With one gateway URL, every session in the pool is bound to the same string, and which exit IP a request gets is decided entirely by the gateway on the far side of that URL.

Retirement follows from this. When a session is retired, the pool creates a new one with a new ID, and the new ID is handed the next URL in the round-robin. With ten URLs, that moves traffic to a different proxy. With one gateway URL, the "new proxy" is the same URL, so the only thing retirement changed was the session's cookies, which are now gone. On Random Rotation the next request would have got a new exit anyway; on Sticky Sessions it may get the same exit back. Either way, the IP change you wanted from retiring came from the gateway, and the cookie loss came from Crawlee.

The general case for keying rotation on a session rather than a request is covered in our session-aware proxy rotation guide; the point here is narrower. In Crawlee, a session is a cookie jar plus a cached URL, and the IP behind that URL is not part of the session at all.

Pick the port by what the target remembers between requests.

The trap on port 11002 is lifetime mismatch. A Crawlee session lives 3000 seconds by default. A Sticky Session holds its exit for up to 5 minutes. Left at defaults, one Crawlee session outlives roughly ten exit IPs, carrying the same cookies across all of them, which is exactly the "same user, new address every five minutes" pattern that trips session-binding checks. Cap the Crawlee session at the sticky window so cookies and exit retire together:

const crawler = new CheerioCrawler({
    proxyConfiguration, // gateway.sparkproxy.io:11002
    persistCookiesPerSession: true,
    sessionPoolOptions: {
        maxPoolSize: 40,
        sessionOptions: {
            maxAgeSecs: 270,     // just under the 5-minute sticky window
            maxUsageCount: 150,  // usage is not the limit here, age is
        },
    },
    async requestHandler({ request, $ }) { /* ... */ },
});

270 seconds leaves margin for a request already in flight when the window closes. Raise maxUsageCount so the session is not ended early by the default 50 uses on a fast target.

One thing the SparkProxy pages we read on 30 October 2026 do not document is how the sticky port maps several concurrent connections on the same credentials onto exits. Don't assume either way. Measure it: run your crawler against an IP-echo endpoint you control, log session.id beside the returned address, and check whether 40 sessions show 40 addresses or fewer. If they share, size maxPoolSize and concurrency with that in mind. The definitional side of sticky behaviour is in our sticky session proxy explainer.

newUrlFunction is the escape hatch when one URL does not fit every request. Crawlee calls it with (sessionId, { request }) for each request, and the docs note that request can be absent, so the function needs a default. Returning null sends that request without a proxy.

const USER = process.env.SPARK_USER;
const PASS = encodeURIComponent(process.env.SPARK_PASS);
const gw = (port, user = USER) =>
    `http://${encodeURIComponent(user)}:${PASS}@gateway.sparkproxy.io:${port}`;

const proxyConfiguration = new ProxyConfiguration({
    newUrlFunction: (sessionId, { request } = {}) => {
        if (!request) return gw(11000);
        if (request.label === 'LOGIN' || request.label === 'LIST') return gw(11002);
        if (request.userData.country === 'us') return gw(11000, `${USER}-country-us`);
        return gw(11000);
    },
});

The user-country-us username pattern is the one shown on the SparkProxy homepage for US targeting. Other country codes follow the same shape, but check your dashboard for the exact form on your plan.

Two consequences of how Crawlee calls this function are easy to miss. First, there is no caching: in both languages newUrlFunction runs on every request, and the session-to-URL map that proxyUrls uses is bypassed. If two requests on the same session carry different labels, that session's cookies travel through two different ports. Route by session, not by request, if continuity matters. Second, an exception or a non-URL return value fails the request; Python wraps it as The provided "new_url_function" did not return a valid URL.

The Python version takes the session ID and request as arguments:

import os
from urllib.parse import quote

from crawlee import Request
from crawlee.proxy_configuration import ProxyConfiguration

USER = os.environ["SPARK_USER"]
PASS = quote(os.environ["SPARK_PASS"], safe="")


def gw(port: int, user: str = USER) -> str:
    return f"http://{quote(user, safe='')}:{PASS}@gateway.sparkproxy.io:{port}"


def pick_proxy(session_id: str | None = None, request: Request | None = None) -> str | None:
    if request is None:
        return gw(11000)
    if request.label in ("LOGIN", "LIST"):
        return gw(11002)
    if request.user_data.get("country") == "us":
        return gw(11000, f"{USER}-country-us")
    return gw(11000)


proxy_configuration = ProxyConfiguration(new_url_function=pick_proxy)

tieredProxyUrls takes a list of lists. Each domain starts on tier 0; when requests to that domain fail, Crawlee moves it up a tier, and it periodically probes the lower tier again to step back down. The docs warn that calling newUrl() directly with this option "will not yield the expected results", so it only works inside a crawler.

const proxyConfiguration = new ProxyConfiguration({
    tieredProxyUrls: [
        [null],          // tier 0: direct, for targets that do not block
        [gw(11000)],     // tier 1: Random Rotation
        [gw(11002)],     // tier 2: Sticky Sessions, for targets that want continuity
    ],
});

proxy_configuration = ProxyConfiguration(
    tiered_proxy_urls=[
        [None],
        [gw(11000)],
        [gw(11002)],
    ],
)

What triggers a climb is worth knowing exactly, because it is broader than "the proxy got blocked". In both codebases, when a request comes back for a retry still carrying the tier it last used, that tier gets 10 points added to its error score for that domain, and the scores of the tiers not in use drop by one on each later prediction, which is how the domain drifts back down. Any failure that leads to a retry counts: a 403, a timeout, a 500 from the site, or an exception thrown by your own parsing code. A selector bug that throws on every page will push a domain to the top tier while the proxy was never the problem.

The session interaction also differs between the two ports of Crawlee. In JS v3.18.2, the tiered branch ignores the session ID and round-robins inside the chosen tier on every request, so a tier holding several URLs gives no session-to-URL stickiness at all. In Python v1.11.0, the session-to-URL cache still applies: a session first bound on tier 0 keeps its tier-0 URL even after the domain escalates, until that session is retired. Keep one URL per tier, as above, and both behave the same.

Here is the chain in JS v3.18.2. CheerioCrawler and the other HTTP crawlers call session.retireOnBlockedStatusCodes(statusCode). If the code is in the pool's blocked list, the session is retired and the crawler throws a plain Error with the message Request blocked - received 403 status code. Because it is a plain error and not a SessionError, it counts against maxRequestRetries (3 by default), so the request gets four attempts in total and then logs Request failed and reached maximum retries.

Network-level proxy failures go down a different path. If the error message contains ECONNRESET, ECONNREFUSED, ERR_PROXY_CONNECTION_FAILED, ERR_TUNNEL_CONNECTION_FAILED or Proxy responded with, the crawler wraps it in a SessionError, which rotates the session and counts against maxSessionRotations (10) instead.

Python v1.11.0 treats a blocked status as a SessionError directly. You will see Encountered "crawlee.errors.SessionError: Assuming the session is blocked based on HTTP status code 403", rotating session and retrying... and the request can rotate up to the session-rotation limit. Other 4xx codes raise HttpClientStatusCodeError, which Python does not retry at all.

So the loop forms like this on a logged-in crawl:

1. The site's login cookie expires, or a CSRF token goes stale, and it answers 403.
2. Crawlee reads 403 as an IP block and retires the session, deleting its cookies.
3. The retry runs on a fresh session that never logged in. The site answers 401 or 403 again.
4. Repeat until the retry or rotation budget is spent, across every request in flight.

The same pattern happens with 429. A rate limit applied to your account or API key is not tied to an IP, yet the default list retires a session for it, and each new session starts the cooldown over.

The fix is to stop letting status codes alone decide, and make your handler classify the response. Work through the causes in order of how cheap they are to check:

In JS, empty the pool's blocked list so 401, 403 and 429 reach your handler, then decide what each one means:

const crawler = new CheerioCrawler({
    proxyConfiguration,
    sessionPoolOptions: {
        blockedStatusCodes: [],           // classify in the handler instead
        sessionOptions: { maxAgeSecs: 270, maxUsageCount: 150 },
    },
    maxRequestRetries: 5,
    async requestHandler({ request, response, $, session, log }) {
        const status = response.statusCode;
        if (status === 401 || $('form#login').length) {
            session.userData.loggedIn = false;   // keep the session, log in again
            throw new Error(`LOGGED_OUT on ${request.url}`);
        }
        if (status === 429) {
            session.markBad();                    // +1 error score, not a retirement
            throw new Error(`RATE_LIMITED, retry-after ${response.headers['retry-after']}`);
        }
        if ($('#challenge-form, iframe[src*="challenges"]').length) {
            session.retire();                     // this one really is about the exit
            throw new Error(`CHALLENGE on ${request.url}`);
        }
        // normal extraction below
    },
});

In Python, ignore_http_error_status_codes both stops the session-block check and stops the 4xx error from being raised, so the handler sees the response. Raise SessionError yourself for real blocks; Crawlee then retires and rotates for you. Don't call mark_bad() for ordinary exceptions: v1.11.0 already marks the session bad when a request errors.

from datetime import timedelta

from crawlee.crawlers import BeautifulSoupCrawler, BeautifulSoupCrawlingContext
from crawlee.errors import SessionError
from crawlee.sessions import SessionPool

crawler = BeautifulSoupCrawler(
    proxy_configuration=proxy_configuration,
    session_pool=SessionPool(
        max_pool_size=40,
        create_session_settings={
            "max_age": timedelta(seconds=270),
            "max_usage_count": 150,
        },
    ),
    ignore_http_error_status_codes=[401, 403, 429],
    max_request_retries=5,
)


@crawler.router.default_handler
async def handler(context: BeautifulSoupCrawlingContext) -> None:
    status = context.http_response.status_code
    if status == 401 or context.soup.select_one("form#login"):
        if context.session:
            context.session.user_data["logged_in"] = False
        raise RuntimeError(f"LOGGED_OUT on {context.request.url}")
    if status == 429:
        raise RuntimeError("RATE_LIMITED")
    if context.soup.select_one("#challenge-form"):
        raise SessionError("challenge page served")

create_session_settings is passed straight to the Session constructor, where max_age is a timedelta, so don't hand it a bare number of seconds. Pair the rate-limit branch with real backoff; our article on retry and backoff strategies covers jitter and Retry-After handling.

Python v1.11.0 has the cleanest answer: a request can be pinned to a session. Enqueue the next page with the current session's ID, and Crawlee will run it on that session or not at all.

from urllib.parse import urljoin

from crawlee import Request


@crawler.router.handler("LIST")
async def list_page(context: BeautifulSoupCrawlingContext) -> None:
    for row in context.soup.select("article h2 a"):
        await context.push_data({"title": row.get_text(strip=True), "url": row["href"]})

    nxt = context.soup.select_one("a[rel=next]")
    if nxt and context.session:
        await context.add_requests([
            Request.from_url(
                urljoin(context.request.url, nxt["href"]),
                label="LIST",
                session_id=context.session.id,
            )
        ])

If that session has been retired by the time the next page runs, the request fails without retrying, and the log reads The Session (id: ...) bound to the Request is no longer available in SessionPool. That is deliberate: it is better than silently serving page eight to an anonymous session. Restart the listing from page one, or from a stored cursor, in your failed_request_handler.

JS v3.18.2 has no equivalent field on Request, so the usual pattern is to make every session able to log itself in. Keep a flag in session.userData and log in from a pre-navigation hook when it is missing:

const crawler = new CheerioCrawler({
    proxyConfiguration,
    persistCookiesPerSession: true,
    sessionPoolOptions: {
        maxPoolSize: 5,                               // one per account you hold
        blockedStatusCodes: [],
        sessionOptions: { maxAgeSecs: 270, maxUsageCount: 500 },
    },
    preNavigationHooks: [
        async ({ session, sendRequest }) => {
            if (session.userData.loggedIn) return;
            const res = await sendRequest({
                url: 'https://www.sparkproxy.io/login',
                method: 'POST',
                form: { email: 'crawler@sparkproxy.io', password: process.env.SITE_PASS },
            });
            session.userData.loggedIn = res.statusCode < 400;
        },
    ],
    async requestHandler({ request, $, session }) { /* classify as above */ },
});

In v3.18.2, sendRequest uses the session's cookie jar and the proxy URL already chosen for the request, so the login cookies land in the session and the login leaves from the same port as the pages after it. Two limits stay true regardless of code. A Sticky Session holds its exit for up to 5 minutes, so a single login that must keep one IP for an hour is not something this setup guarantees; design for a fresh login per session instead. And a pool larger than the number of accounts you own just logs the same account in from several places at once, which is its own detection signal. Cookie handling outside Crawlee is covered in our guide to cookies and sessions in web scraping.

SparkProxy plans are priced by concurrent threads with unlimited bandwidth, so the variable you are paying for is parallelism, not gigabytes. On the SparkProxy homepage, read 30 October 2026, prices in USD excluding VAT/GST, per 30 days:

Longer terms take 5% off for 3 months, 10% for 6 and 15% for 12. A 24-hour free trial needs no credit card. Set Crawlee's maxConcurrency below your thread count and leave headroom: an HTTP crawler uses roughly one connection per request, but a PlaywrightCrawler page pulls scripts, images and XHRs in parallel through the same proxy. Because traffic is unmetered, the per-domain tier climb described above costs nothing extra in bandwidth on these ports; the cost of escalating is only the thread it occupies.

For the pages a proxy cannot fix, such as fingerprint-gated targets, route the failures to the Scraping API from failedRequestHandler rather than raising retries. A rendered fetch (render_js defaults to true) costs 5 credits, a plain fetch with render_js=false costs 1, and country_code adds 5. With json_response=true you get an envelope whose body is base64, so decode it before matching anything:

async failedRequestHandler({ request, log }, error) {
    const qs = new URLSearchParams({ url: request.url, render_js: 'true', json_response: 'true' });
    const res = await fetch(`https://scrape.sparkproxy.io/api/v1?${qs}`, {
        headers: { 'X-API-Key': process.env.SPARK_API_KEY },
    });
    const env = await res.json();                     // job_id, status_code, credits_used, body
    const html = Buffer.from(env.body, 'base64').toString('utf8');
    log.info(`API fallback ${request.url}: ${env.status_code}, ${env.credits_used} credits`);
    await Dataset.pushData({ url: request.url, html, via: 'scraping-api' });
},

Import Dataset from crawlee alongside the crawler. One caution from the Scraping API docs: its session_id parameter is a log label only, it does not persist cookies, and each request gets a fresh browser profile. It is a fallback for anonymous pages, not a replacement for a logged-in Crawlee session. Credit mechanics are broken down in our explainer on Scraping API credit pricing, and the plain Node rotation patterns that sit underneath Crawlee are in how to rotate proxies in Node.js.