How to Scrape a Website Behind a Login

Everything below assumes you have the right to access the data. Read that sentence twice, because logging in changes the legal and ethical picture completely.

Scraping a public page and scraping a page behind a login are not the same activity. Behind a login you have almost always agreed to Terms of Service, and that contract usually says what automated access is and isn't allowed. Ignoring it can breach the ToS, and in some jurisdictions unauthorized access to a computer system carries real legal risk (the US Computer Fraud and Abuse Act and similar laws elsewhere). This is not legal advice, but the practical rule is simple.

Before you write a single line of scraping code, check these:

• You own the account or have explicit permission. Scraping your own dashboard, your own analytics, or a client's account you've been authorized to manage is normal. Harvesting another person's private data is not.
• An official API doesn't already exist. If the site publishes an API, use it. It's faster, it's supported, and it won't break every time the HTML changes.
• The Terms of Service permit automated access. Some sites allow it with rate limits, some forbid it outright. Read the ToS for the section on automated access, bots, and scraping.
• You are not collecting other people's personal data. Under GDPR and similar regimes, scraping personal data behind a login can trigger obligations you don't want to discover after the fact.
• You rate-limit and identify yourself. Hammering a login-protected app looks like credential stuffing or a brute-force attack. Go slow, respect the server, and keep a contact string in your User-Agent when appropriate.

Never share, sell, or store credentials that aren't yours, and never automate account creation to get around access limits. When in doubt, ask the site owner for a data export or API access. Scraping responsibly also keeps your infrastructure clean, which is a large part of how to avoid getting your proxy blocked in the first place.

With that settled, here's how authenticated sessions actually work.

HTTP is stateless. The server forgets you the instant a request finishes. A login flow exists to hand your client a credential it can replay on every future request so the server treats a stream of independent requests as one authenticated session.

Three mechanisms do most of the work, and knowing which one a site uses tells you exactly what your scraper has to store and resend.

A session cookie is the most common. You POST your credentials, the server validates them, and it replies with Set-Cookie: session=.... From then on your browser (or your scraper) attaches that cookie to every request to the same domain. Lose the cookie and you're anonymous again.

The catch is that many sites bind that cookie to signals beyond the cookie value itself: your IP address, your User-Agent, sometimes a TLS fingerprint. Change any of those mid-session and the server may invalidate the cookie and force a re-login. That single fact is why proxy choice matters so much for authenticated scraping, and we come back to it in the sticky proxies section.

For a plain HTML login form with no JavaScript, requests.Session is the fastest tool you have. A Session object keeps a cookie jar and reuses it across every call, so once you log in, you stay logged in.

import requests

session = requests.Session()
session.headers.update({
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
                  "AppleWebKit/537.36 (KHTML, like Gecko) "
                  "Chrome/126.0.0.0 Safari/537.36",
})

# 1. POST your credentials to the login endpoint
login = session.post(
    "https://app.sparkproxy.io/login",
    data={"email": "you@sparkproxy.io", "password": "your-password"},
    timeout=15,
)
login.raise_for_status()

# 2. The Set-Cookie response is now stored in the session's cookie jar
print(session.cookies.get_dict())

# 3. Reuse the SAME session object for every authenticated page
dashboard = session.get("https://app.sparkproxy.io/dashboard", timeout=15)
print("Logged in" if "Sign out" in dashboard.text else "Not authenticated")

The key detail: do not create a new Session per request. That throws the cookie jar away and logs you out. Build one Session, authenticate once, and pass it around.

One trap that costs people hours: the login form might not POST to the same URL you loaded. Open your browser's network tab, submit the form by hand, and copy the exact request. Note the real POST URL, the field names (email vs username vs user[email]), and any hidden fields. Match those exactly.

If you're running this at scale with many accounts or many targets, keep each account on its own Session and its own proxy, and drive them concurrently. The patterns in using proxies with Python requests and aiohttp for async scraping carry over directly to authenticated flows.

Send credentials to a modern login endpoint with nothing else and you'll often get a 403 Forbidden or a "token mismatch" error. That's a CSRF (Cross-Site Request Forgery) token doing its job. The server planted a one-time value in the login page and expects it back with the form.

The fix is a two-step flow: GET the login page first to collect the token and the initial cookies, then POST the token alongside your credentials.

import requests
from bs4 import BeautifulSoup

session = requests.Session()

# 1. GET the login page to collect the CSRF token and the initial cookies
page = session.get("https://app.sparkproxy.io/login", timeout=15)
soup = BeautifulSoup(page.text, "html.parser")

# The field name varies: csrf_token, _token, authenticity_token, __RequestVerificationToken
token = soup.select_one('input[name="csrf_token"]')["value"]

# 2. Send the token back with the credentials, on the SAME session
resp = session.post(
    "https://app.sparkproxy.io/login",
    data={
        "email": "you@sparkproxy.io",
        "password": "your-password",
        "csrf_token": token,
    },
    headers={"Referer": "https://app.sparkproxy.io/login"},
    timeout=15,
)
resp.raise_for_status()

Two things trip people up here. First, the token is tied to the session cookie set on that initial GET, so you must use the same Session for both requests. Fetch the token with one client and post with another and the server sees a mismatch. Second, some frameworks put the token in a tag or return it in a cookie named XSRF-TOKEN that you echo back as an X-XSRF-TOKEN header (Angular and Laravel do this). Check where the value actually lives before assuming it's a hidden input.

If the login form field names look randomized or the page won't render the form without JavaScript, requests has hit its limit. Switch to a real browser.

Single-page apps build the login form in JavaScript, sign requests with values computed at runtime, and sometimes fingerprint the browser before they'll accept a submit. requests can't run any of that. A headless browser can, because it is a real browser.

Playwright drives Chromium, Firefox, or WebKit, fills the form the way a user would, and captures the resulting session. Install it first:

pip install playwright
playwright install chromium

Then log in and save the authenticated state to a file:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page()

    page.goto("https://app.sparkproxy.io/login")
    page.fill('input[name="email"]', "you@sparkproxy.io")
    page.fill('input[name="password"]', "your-password")
    page.click('button[type="submit"]')

    # Wait for the post-login page so cookies are fully set before you save
    page.wait_for_url("**/dashboard")

    # storage_state captures cookies AND localStorage in one file
    page.context.storage_state(path="auth.json")
    browser.close()

storage_state is the piece most tutorials skip. It writes both cookies and localStorage to auth.json. That matters because token-based apps keep the auth token in localStorage, not a cookie, and a cookies-only export would leave you logged out. For login flows that fingerprint the browser or throw a challenge, run headed (headless=False) while you debug, and route Playwright through a proxy so the login IP matches the IP you'll scrape from later. If running browsers and proxies yourself starts to hurt, weigh the trade-off in a managed scraping API versus self-managed proxies.

Logging in on every run is slow and, worse, it looks like a bot. Repeated logins from the same account within minutes are a classic abuse signal. Authenticate once, save the session, and reload it.

Playwright makes reuse trivial. Point a new context at the saved auth.json and skip the login entirely:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)

    # Load the saved session: no second login required
    context = browser.new_context(storage_state="auth.json")
    page = context.new_page()

    page.goto("https://app.sparkproxy.io/billing")
    print(page.title())
    browser.close()

The best pattern for most projects is a hybrid: log in once with a browser to clear the JavaScript and fingerprint checks, then hand the cookies to requests for the actual scraping, which is far faster than driving a browser for every page.

import json
import requests

# Read the cookies Playwright saved in storage_state
with open("auth.json") as f:
    state = json.load(f)

session = requests.Session()
for c in state["cookies"]:
    session.cookies.set(c["name"], c["value"], domain=c["domain"], path=c["path"])

# Now scrape at HTTP speed with the browser's authenticated cookies
resp = session.get("https://app.sparkproxy.io/api/usage", timeout=15)
print(resp.json())

This bridge gives you the best of both: a browser handles the hard login, and lightweight HTTP handles the volume. It only breaks if the site rotates the session on IP change, which is the next problem to solve.

Here's the failure that ends most authenticated-scraping projects: you log in successfully, scrape ten pages, and then every request starts redirecting to the login screen. The cookie is fine. The IP changed.

A rotating proxy assigns a new IP to each request by default. That's exactly what you want for stateless scraping and exactly what breaks a login session, because the server sees your valid cookie arriving from a brand-new IP on every hit and treats it as a hijacked session. The fix is a sticky session: a proxy that pins one exit IP for the full lifetime of your login.

With sticky residential or ISP proxies, you request a session that holds the same IP for a set window (commonly up to 10 to 30 minutes, sometimes longer), which is enough to log in and scrape under one consistent identity. The session is usually selected through the proxy username:

import requests

# A sticky session pins ONE exit IP for the whole login flow.
# The session token in the username tells the gateway to reuse that IP.
proxy = "http://user-session-a1b2c3:your-pass@residential.sparkproxy.io:8000"

session = requests.Session()
session.proxies = {"http": proxy, "https": proxy}

# Log in and scrape on the SAME IP so the cookie stays valid
session.post("https://app.sparkproxy.io/login",
             data={"email": "you@sparkproxy.io", "password": "your-password"},
             timeout=15)
session.get("https://app.sparkproxy.io/dashboard", timeout=15)

The exact username format for sticky sessions lives in your dashboard, and the mechanics of encoding a session token into proxy credentials are covered in how proxy authentication works. Two rules make sticky sessions reliable: keep the User-Agent identical between login and scraping (a User-Agent flip invalidates many sessions just like an IP flip), and give each account its own sticky session so their cookies never cross IPs.

Managing cookies, CSRF tokens, headless browsers, and sticky IPs by hand is a lot of moving parts. The SparkProxy Scraping API collapses them into request parameters. It runs a real browser, holds the proxy, and persists the session for you. See the full parameter list at the Scraping API docs.

Two parameters do the heavy lifting for authenticated work:

• session_id: labels a browser profile (max 128 characters). Reuse the same value and you get the same cookies and the same proxy IP across calls, which is a sticky session with no username juggling.
• cookies: a JSON array of cookie objects injected before the page loads, so you can hand it a session you obtained elsewhere.

If you already have cookies from a browser login, inject them and start scraping straight away:

curl -X POST "https://scrape.sparkproxy.io/api/v1" \
  -H "X-API-Key: sk-your-api-key" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://app.sparkproxy.io/dashboard",
    "render_js": true,
    "session_id": "acct-42-run-1",
    "cookies": [
      {"name": "session", "value": "eyJhbGciOi...", "domain": "app.sparkproxy.io", "path": "/"}
    ]
  }'

Or perform the login inside the API with a js_scenario, then reuse the same session_id to scrape while staying authenticated on one IP:

import requests

API = "https://scrape.sparkproxy.io/api/v1"
KEY = {"X-API-Key": "sk-your-api-key"}

# 1. Log in once. session_id pins the browser profile, its cookies, and the proxy IP.
requests.post(API, headers=KEY, json={
    "url": "https://app.sparkproxy.io/login",
    "render_js": True,
    "session_id": "acct-42",
    "premium_proxy": True,          # route through a residential IP
    "country_code": "US",
    "js_scenario": {"steps": [      # actions follow the docs: fill, click, wait
        {"fill": {"selector": "input[name=email]", "value": "you@sparkproxy.io"}},
        {"fill": {"selector": "input[name=password]", "value": "your-password"}},
        {"click": {"selector": "button[type=submit]"}},
        {"wait_for": {"selector": "text=Dashboard"}}
    ]},
})

# 2. Reuse the SAME session_id: still logged in, same IP, cookies intact.
html = requests.post(API, headers=KEY, json={
    "url": "https://app.sparkproxy.io/billing",
    "render_js": True,
    "session_id": "acct-42",
    "premium_proxy": True,
    "country_code": "US",
}).text

print(len(html), "bytes scraped while authenticated")

The trade-off is the usual one: you hand off cookie, browser, and IP management in exchange for a per-request cost and less low-level control. Our breakdown of a web scraping API versus self-managed proxies walks through when each wins. For login-heavy scraping where sessions keep dropping, the API usually pays for itself in debugging time saved.

Sessions expire. The server times them out, rotates the token, or invalidates the cookie after a security event. A resilient scraper assumes this will happen and checks every response instead of trusting that it's still logged in.

The tell is consistent: a 401 or 403 status, or a silent 302 redirect to /login that returns 200 with the login HTML. Detect it, re-authenticate, and retry once.

def is_authenticated(resp) -> bool:
    """Return False when the session has expired."""
    if resp.status_code in (401, 403):
        return False
    if resp.url.rstrip("/").endswith("/login"):   # redirected to login
        return False
    if "name=\"password\"" in resp.text:          # login form served instead of content
        return False
    return True


def fetch(session, url, relogin):
    resp = session.get(url, timeout=15)
    if not is_authenticated(resp):
        relogin(session)                # refresh cookies, then retry once
        resp = session.get(url, timeout=15)
    return resp

Don't retry forever. If a re-login also fails, stop and alert, because repeated failed logins are the fastest way to get an account locked or flagged. Back off, log the failure, and move on.