How to Handle Cookies and Sessions When Web Scraping

A "session" is just a set of cookies the server uses to remember you between otherwise independent HTTP requests. HTTP is stateless. The only reason a site knows request two came from the same visitor as request one is a cookie it set on the first response and expects back on the second.

Login is the famous case, but it is not the only one. You need session handling any time state carries between requests:

• A cookie-consent or GDPR banner sets consent=1 and refuses to render content until it sees it.
• A store keeps your cart, currency, or shipping country in a cart_id or geo cookie.
• An A/B testing framework pins you to a variant with a bucketing cookie, and pages break if it flips mid-crawl.
• A rate limiter or bot-management layer drops an anonymous session token on the landing page and rejects any deep link that arrives without it.
• Anti-CSRF systems set a token cookie that a later form submission has to echo back.

Every one of these fails the same way: request one works, request three returns a redirect, a 403, or an empty page, and the scraper "randomly" breaks. The fix is always the same discipline. Capture what the server sets, store it correctly, and send it back with matching attributes from a stable IP.

A cookie arrives in a Set-Cookie response header. Everything after the name=value pair is an attribute that controls when the cookie is sent back. Miss one and your client either drops the cookie or refuses to attach it later.

You rarely parse this by hand, but knowing the shape helps you debug. Python's standard library will do it for you:

from http.cookies import SimpleCookie

raw = "session=8f3a91cd; Domain=.sparkproxy.io; Path=/; Secure; HttpOnly; SameSite=Lax; Max-Age=3600"

jar = SimpleCookie()
jar.load(raw)

c = jar["session"]
print(c.value)          # 8f3a91cd
print(c["domain"])      # .sparkproxy.io
print(c["path"])        # /
print(c["max-age"])     # 3600
print(c["samesite"])    # Lax

The takeaway: a cookie is not a string, it is a value plus a set of matching rules. When you copy cookies between tools, you have to carry the Domain and Path too or the destination client will not know when to send them.

There are two lifetimes, and the difference bites people who try to save a jar to disk.

A persistent cookie has a Max-Age or Expires. The browser (and your client) stores it until it expires. A session cookie has neither. It lives only as long as the "browser session," which for your scraper means only as long as the in-memory jar. The moment your process exits, session cookies vanish.

This is the number one reason a saved cookie file comes back empty. Most auth cookies are session cookies by design, so a naive jar.save() writes nothing useful. Every serious cookie-jar API therefore has a flag to override this, which you will see in the next sections as ignore_discard=True. Without it, the jar refuses to write session cookies to disk, and your "saved" login is gone on the next run.

The correct primitive is a session object, not a bare get(). A session keeps a cookie jar, applies whatever the server sets, and replays it on every subsequent request automatically.

With requests:

import requests

session = requests.Session()

# First request receives Set-Cookie headers and stores them in session.cookies
session.get("https://www.sparkproxy.io/account/login")

# Every later request through this session sends those cookies back
resp = session.get("https://www.sparkproxy.io/account/dashboard")

# Inspect the jar: each cookie carries its own domain and path
for c in session.cookies:
    print(c.name, c.value[:8], c.domain, c.path, c.secure, c.expires)

The httpx equivalent is nearly identical, and it is worth knowing because httpx gives you HTTP/2 and an async client that shares the same jar across concurrent requests:

import httpx

with httpx.Client(http2=True, timeout=30) as client:
    client.get("https://www.sparkproxy.io/account/login")
    resp = client.get("https://www.sparkproxy.io/account/dashboard")
    print(dict(client.cookies))

Both clients honor Domain, Path, and Secure automatically. That is the whole point of using a session: you stop hand-managing the Cookie header and let the jar decide what to attach to each URL. If you are moving a scraper from requests to async, the companion post on using proxies with requests and aiohttp covers keeping one jar across an async pool.

This is the failure that eats hours. The server clearly sent Set-Cookie, you can see it in the jar, yet the next request goes out without it. There are only a few causes, and they are all attribute mismatches.

# See exactly what the jar holds, scoped by domain
print(session.cookies.get_dict(domain="sparkproxy.io"))

# Compare against what actually went out on the wire
resp = session.get("https://www.sparkproxy.io/account/dashboard")
print(resp.request.headers.get("Cookie"))

Run those two lines side by side and the gap explains itself. The usual culprits:

• Scheme mismatch. The cookie is Secure and you requested http://. It is dropped. Always request https://.
• Domain mismatch. The cookie is host-only for www.sparkproxy.io but you requested sparkproxy.io, or vice versa. Host-only cookies do not cross the www boundary.
• Path mismatch. A Path=/account cookie will not attach to a request for /.
• Expiry. The cookie already expired, or it was a session cookie you reloaded from disk without ignore_discard, so it never made it into the jar at all.
• A redirect stripped it. If the login POST 302-redirects to another host, the cookie set on the redirect response belongs to that host. Follow the chain and inspect resp.history.

When in doubt, print the jar and print the outgoing Cookie header. The truth is always in the diff between those two.

Logging in on every run is slow and, worse, it looks like a brand new visitor every time, which is exactly the pattern anti-bot systems score against you. Save the jar once, reload it, and only re-authenticate when it actually expires.

The cleanest built-in option is http.cookiejar.LWPCookieJar, which serializes to a readable text format. Assign it to the session and remember ignore_discard so session cookies survive:

import requests
import http.cookiejar

session = requests.Session()
session.cookies = http.cookiejar.LWPCookieJar("spark_cookies.txt")

# ... perform your login here ...

# Save, including session cookies that have no explicit expiry
session.cookies.save(ignore_discard=True, ignore_expires=True)

Reloading on the next run:

session = requests.Session()
session.cookies = http.cookiejar.LWPCookieJar("spark_cookies.txt")
try:
    session.cookies.load(ignore_discard=True, ignore_expires=True)
except FileNotFoundError:
    pass  # first run, no jar yet

Which serialization format you pick depends on where the cookies are going next:

That last row matters more than it looks. The moment you scrape through a headless browser or an API, you need cookies as a plain list of dictionaries, so build the export deliberately:

import json

cookie_list = [
    {"name": c.name, "value": c.value, "domain": c.domain}
    for c in session.cookies
]
json.dump(cookie_list, open("cookies.json", "w"))

Keep that shape in mind. It is exactly what the SparkProxy Scraping API expects, which is covered near the end.

A CSRF token is anti-forgery protection, and it is a session concern because the token is tied to your current session. Sites deliver it in one of two ways, and you handle both by keeping a live session.

The common pattern is a hidden form field. Fetch the page with your session, read the token out of the HTML, and submit it back on the same session so the token cookie and the form value agree:

from bs4 import BeautifulSoup

page = session.get("https://www.sparkproxy.io/account/login")
soup = BeautifulSoup(page.text, "html.parser")
token = soup.select_one('input[name="csrf_token"]')["value"]

# The token only validates alongside the session cookie set on the GET above
resp = session.post(
    "https://www.sparkproxy.io/account/login",
    data={"email": "you@sparkproxy.io", "password": "REDACTED", "csrf_token": token},
)

The second pattern is a double-submit cookie. The server sets a token as a cookie and also expects it in a request header (often X-CSRF-Token or X-XSRF-TOKEN). Read it out of the jar and mirror it into the header:

xsrf = session.cookies.get("XSRF-TOKEN", domain="sparkproxy.io")
resp = session.post(
    "https://www.sparkproxy.io/account/settings",
    headers={"X-XSRF-TOKEN": xsrf},
    json={"timezone": "UTC"},
)

Two rules keep this reliable. Always fetch the token with the same session you will submit with, because a token minted against one session is invalid on another. And never cache a token across runs. It is short-lived and bound to the session cookie you saved, so refetch it after every reload.

Once you have a valid jar, the goal is to log in as rarely as possible. Reload the saved cookies, probe a page that requires auth, and only re-authenticate when the probe fails. A cheap, reliable probe is a request to a protected URL with redirects disabled:

def is_logged_in(session):
    r = session.get(
        "https://www.sparkproxy.io/account/dashboard",
        allow_redirects=False,
    )
    # A bounce to /login means the session is dead
    return not (r.is_redirect and "login" in r.headers.get("Location", ""))

if not is_logged_in(session):
    do_login(session)  # fetch token, POST credentials, then save the jar again
    session.cookies.save(ignore_discard=True, ignore_expires=True)

Watch the signals a site uses to say "your session expired." A 302 to /login, a 401, a JSON body like {"error": "unauthenticated"}, or a login form showing up where content should be are all the same event. Detect it explicitly instead of scraping a login page by accident and writing garbage to your dataset. The full login mechanics, including a headless-browser login for JavaScript-heavy sites, live in the scrape behind a login guide.

Here is the part most cookie tutorials skip. A session is not just cookies. Many sites bind the session to the IP that created it. Log in on IP A, then send request two through IP B from a rotating pool, and the server may invalidate the session as suspicious. Your cookies are perfect and the site still logs you out.

The fix is a sticky session: hold one exit IP for the life of the authenticated session. With requests, that means pointing the whole session object at one sticky proxy endpoint and keeping it there:

session = requests.Session()
session.proxies = {
    "http":  "http://USER:PASS@proxy.sparkproxy.io:PORT",
    "https": "http://USER:PASS@proxy.sparkproxy.io:PORT",
}
# Every request in this session now exits from the same sticky IP,
# matching the IP that was used at login time.

The cookie jar and the exit IP are one unit. Save them together, rotate them together, retire them together. When the sticky window ends and the IP changes, treat the session as potentially dead and be ready to re-authenticate. For how sticky windows work and how long they hold, see what is a sticky session proxy. If you also rotate for unauthenticated crawling, the rotate proxies in Python guide covers keeping rotation and sticky sessions separate so you never rotate an IP out from under a live login.

Managed scraping APIs change the model, and it trips people up. The SparkProxy Scraping API runs each request in a fresh headless browser profile. That has one consequence you have to design around: cookies do not persist across separate API requests. The session_id parameter labels a request in your logs for grouping, but it does not carry a jar from one call to the next. Every call starts clean.

So you reconcile a stateless API with a stateful target the same way a browser would, by supplying the state yourself. Inject your saved cookies on each request with the cookies parameter, which takes exactly the name / value / domain shape you exported earlier:

import os, json, requests

saved = json.load(open("cookies.json"))  # [{"name","value","domain"}, ...]

resp = requests.post(
    "https://scrape.sparkproxy.io/api/v1",
    headers={"X-API-Key": os.environ["SPARK_API_KEY"], "Content-Type": "application/json"},
    json={
        "url": "https://www.sparkproxy.io/account/dashboard",
        "render_js": True,          # full headless render preserves cookies within this request
        "cookies": saved,           # replay your authenticated session
        "country_code": "US",       # consistent geo so the session's IP context matches
    },
)
print(resp.json())

With render_js on, the request runs a real Chromium profile, so any cookies the page sets during that single render (redirects, JavaScript writes, consent handling) are honored inside that request. They just do not survive to the next one, which is why you keep the durable jar on your side.

If you want the login to happen inside one API call instead of managing credentials in your own client, use js_scenario to drive the browser in a single request. The instruction list follows the documented action format, for example waiting for a selector and clicking through a consent wall before the content loads:

json={
    "url": "https://www.sparkproxy.io/account/login",
    "render_js": True,
    "js_scenario": {
        "instructions": [
            {"wait_for": "#login-form"},
            {"click": "#accept-cookies"},
            {"wait_for": ".dashboard"}
        ]
    },
    "json_response": True,   # return metadata alongside the HTML
}

To keep the session's IP stable across a run, pin the exit IP too. The API accepts your own sticky proxy through the own_proxy parameter, so the same IP that established the login can serve the follow-up requests. That is the API-side version of the sticky rule from the previous section. When you decide whether to run your own jar or let the API carry the work, the web scraping API vs self-managed proxies comparison lays out the tradeoff.

One more habit worth keeping: sending clean, consistent cookies is also part of not getting flagged. A jar full of stale or mismatched cookies is a signal. The avoid getting your proxy blocked guide covers the rest of the fingerprint.