How to Rotate User Agents for Web Scraping

A user agent is a request header your HTTP client sends with every request. It identifies the software making the call: the browser, its version, the rendering engine, and the operating system. A real Chrome request on Windows sends something like this:

User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/138.0.0.0 Safari/537.36

Servers read that string to decide what to send back: a mobile layout, a desktop layout, a specific script bundle, or a block page. The string is trivial to set and trivial to fake, which is exactly why anti-bot systems treat it as a first-pass filter rather than a source of truth. Faking it well is table stakes; faking it badly is worse than not faking it at all.

Your default client gives you away immediately. requests sends python-requests/2.32.5. httpx sends python-httpx/0.28.1. aiohttp, curl, Scrapy, and Go's net/http all announce themselves by name. Any site that cares about automated traffic drops those on sight.

Anti-bot systems flag user agents in three ways, and each one catches a different mistake.

Named automation clients. Anything containing python-requests, curl, Go-http-client, Scrapy, Java, or okhttp is a library default. These are the cheapest blocks a site can make.

Impossible or dead versions. Many scraping tutorials copy a user-agent list that was current three years ago. A request claiming Chrome/85.0 in 2026 is a giveaway, because that build was retired long ago and no real user is running it at volume. Some detection engines keep a rolling window of known-good version ranges and reject anything outside it. Scraping a fixed list from a blog post guarantees you drift out of that window within months.

Header sets that contradict the string. This is the one most guides miss. A user agent claiming Chrome 138 that arrives without any Sec-CH-UA client-hint headers is contradictory, because real Chrome 138 always sends them. A Safari user agent that arrives with Sec-CH-UA present is equally wrong, because Safari does not implement client hints at all. The mismatch is a stronger bot signal than a plain library default, since no honest browser produces it.

Rotating user agents is one layer in a larger defense. For the full picture on the other layers, see our guide on how to avoid getting your proxy blocked.

A good pool is small, current, and internally consistent. You do not need thousands of strings. You need a handful of real, recent browser signatures that you keep updated. The table below lists current desktop and mobile user agents as of mid-2026. Treat the version numbers as perishable and refresh them every few weeks.

Two rules make this pool work. First, weight it toward what your target's real audience uses. If you scrape a US retail site, most visitors run Chrome on Windows and Chrome or Safari on mobile, so those should dominate the pool. A pool that is 40% Firefox looks nothing like real traffic. Second, keep the pool honest about client hints: mark which strings must carry Sec-CH-UA and which must not, because you will use that flag in the next sections.

About the fake-useragent library: it is convenient, but it pulls from a data set that can include rare, old, or oddly formatted strings, and a random draw can hand you a Chrome/91 on a bad day. A short curated list you control and update by hand is more predictable for production scraping. Use fake-useragent for quick tests, not for a job you need to stay unblocked for weeks.

Before rotating anything, confirm what you send by default and how to override it. Set the header explicitly on the request or, better, on a session:

import requests

# What requests sends by default
print(requests.utils.default_headers()["User-Agent"])
# python-requests/2.32.5

CHROME_WIN = (
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
    "AppleWebKit/537.36 (KHTML, like Gecko) "
    "Chrome/138.0.0.0 Safari/537.36"
)

# Per-request override
resp = requests.get("https://www.sparkproxy.io", headers={"User-Agent": CHROME_WIN}, timeout=10)

# Per-session override (applies to every request on the session)
session = requests.Session()
session.headers.update({"User-Agent": CHROME_WIN})
resp = session.get("https://www.sparkproxy.io", timeout=10)

Setting User-Agent on a Session is the pattern you want for a scraper, because it keeps the header consistent across the connection reuse that a session provides. In Scrapy, set USER_AGENT in settings.py or override it per request in Request(headers=...). In httpx, pass headers= to the Client. The mechanism is the same everywhere: one header, set once per identity.

The simplest rotation picks a random user agent from the pool for each request. This is the pattern behind most searches for a random user agent python snippet:

import random
import requests

USER_AGENTS = [
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/138.0.0.0 Safari/537.36",
    "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/138.0.0.0 Safari/537.36",
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:141.0) Gecko/20100101 Firefox/141.0",
    "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/18.5 Safari/605.1.15",
]

def get(url: str) -> requests.Response:
    ua = random.choice(USER_AGENTS)
    return requests.get(url, headers={"User-Agent": ua}, timeout=10)

for _ in range(5):
    r = get("https://httpbin.org/user-agent")
    print(r.json()["user-agent"])

random.choice() is thread-safe in CPython, so this is safe to call from a ThreadPoolExecutor without a lock. That covers the mechanics. The problem is that this snippet sends only the User-Agent and nothing else, which is exactly the contradiction described earlier. A Chrome string with no client hints is a red flag. We fix that in the matching-headers section.

The bigger mistake is not which random function you use. It is rotating the user agent at the wrong boundary.

A real browser does not change its user agent between requests. It picks one identity when it launches and keeps it for the entire session: same UA, same TLS handshake, same cookie jar, for hours. If your scraper holds a login cookie or a session token and the user agent flips on every request, you have described a user whose browser mutates mid-session. Many anti-bot systems bind the user agent to the session cookie and flag exactly that.

The rule is simple:

For per-session rotation, assign the identity once when you build the session and never touch it again:

import random
import requests

def new_identity_session() -> requests.Session:
    ua = random.choice(USER_AGENTS)
    session = requests.Session()
    session.headers.update({"User-Agent": ua})
    return session

# One session = one consistent identity for the whole flow
session = new_identity_session()
session.get("https://www.sparkproxy.io/login", timeout=10)
session.get("https://www.sparkproxy.io/dashboard", timeout=10)  # same UA, same cookies

This is the section most user-agent rotation guides skip, and it is the one that decides whether your rotation helps or hurts. A user agent is not a standalone value. Modern Chromium browsers advertise their identity across a set of headers that must agree with the User-Agent string.

The critical set is User-Agent Client Hints, sent as Sec-CH-UA headers. Chrome, Edge, and other Chromium browsers send them on every request. Firefox and Safari do not send them at all. So your rules are:

• Chromium user agent, you must add matching Sec-CH-UA, Sec-CH-UA-Mobile, and Sec-CH-UA-Platform.
• Firefox or Safari user agent, you must not send any Sec-CH-UA headers.
• Sec-CH-UA-Platform has to match the OS in the string. A Windows UA with "macOS" platform is a contradiction.
• Sec-CH-UA-Mobile is ?1 for mobile strings and ?0 for desktop.

The clean way to enforce this is to store each user agent with its full header set, then rotate the whole bundle instead of a lone string:

import random
import requests

# Each profile is a complete, self-consistent identity.
PROFILES = [
    {
        "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
                      "(KHTML, like Gecko) Chrome/138.0.0.0 Safari/537.36",
        "sec-ch-ua": '"Google Chrome";v="138", "Chromium";v="138", "Not.A/Brand";v="24"',
        "sec-ch-ua-mobile": "?0",
        "sec-ch-ua-platform": '"Windows"',
        "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,"
                  "image/webp,*/*;q=0.8",
        "Accept-Language": "en-US,en;q=0.9",
    },
    {
        # Firefox: NO sec-ch-ua headers, different Accept string
        "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:141.0) "
                      "Gecko/20100101 Firefox/141.0",
        "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
        "Accept-Language": "en-US,en;q=0.5",
    },
    {
        # Safari: NO sec-ch-ua headers
        "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) "
                      "AppleWebKit/605.1.15 (KHTML, like Gecko) Version/18.5 Safari/605.1.15",
        "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
        "Accept-Language": "en-US,en;q=0.9",
    },
]

def get(url: str) -> requests.Response:
    headers = random.choice(PROFILES).copy()
    return requests.get(url, headers=headers, timeout=10)

Notice the Accept-Language values differ between profiles, and the Accept string differs between Chrome and Firefox. Those are real, browser-specific defaults, and copying them exactly is what turns a suspicious request into an ordinary one. Accept-Language also has a geo dimension: if you route through a German exit IP, an en-US language header on a request that resolves a German page is a small inconsistency worth avoiding. Set the language to match the region you are presenting as.

Header order matters too, though requests limits how much control you have there. Real browsers send headers in a fixed, browser-specific order, and requests and urllib3 use their own order that differs from Chrome's. For casual targets this is invisible. For hard targets that fingerprint header order over HTTP/2, it is another mismatch. If you need to read more about which headers reveal a client and how proxies add their own, see proxy headers explained: X-Forwarded-For and more.

User-agent rotation and proxy rotation solve different halves of the same problem. The proxy changes the IP the site sees; the user agent changes the client the site sees. Rotate them together so each new IP arrives with a fresh, consistent identity, and hold that pairing for the life of the session.

The anti-pattern is a single IP that cycles through ten user agents, or a single user agent spread across ten IPs. Both look mechanical. Bind one identity to one proxy:

import random
import requests

PROXIES = [
    "http://user-1:pass@gate.sparkproxy.io:10000",
    "http://user-2:pass@gate.sparkproxy.io:10001",
    "http://user-3:pass@gate.sparkproxy.io:10002",
]

def build_worker():
    """Pin one proxy to one browser identity for the whole session."""
    profile = random.choice(PROFILES)   # from the matching-headers section
    proxy = random.choice(PROXIES)
    session = requests.Session()
    session.headers.update(profile)
    session.proxies = {"http": proxy, "https": proxy}
    return session

worker = build_worker()
r = worker.get("https://httpbin.org/anything", timeout=15)
print(r.json()["headers"]["User-Agent"], "via", r.json()["origin"])

For the full set of proxy rotation strategies (round-robin, weighted, retry on failure, thread-safe pools), see how to rotate proxies in Python. Pair that proxy logic with the identity logic here so an IP change and a user-agent change always happen at the same boundary.

For high concurrency, aiohttp lets you attach a fresh identity per request without blocking the event loop. Pass the profile headers and proxy on each session.get() call:

import asyncio
import random
import aiohttp

async def fetch(session: aiohttp.ClientSession, url: str) -> str:
    profile = random.choice(PROFILES)          # from the matching-headers section
    proxy = random.choice(PROXIES)
    async with session.get(
        url,
        headers=profile,
        proxy=proxy,
        timeout=aiohttp.ClientTimeout(total=20),
    ) as resp:
        data = await resp.json(content_type=None)
        return data["headers"]["User-Agent"]

async def main():
    urls = ["https://httpbin.org/anything"] * 20
    async with aiohttp.ClientSession() as session:
        results = await asyncio.gather(*(fetch(session, u) for u in urls))
    for ua in results:
        print(ua)

asyncio.run(main())

When your workload is stateless and per-request rotation is acceptable, this scales cleanly to hundreds of concurrent requests. When each task represents a logged-in session, build one ClientSession per identity instead of rotating inside a shared session. For the async patterns in depth, including connection limits and retries, see using proxies with Python requests, aiohttp, and async scraping.

Here is the hard truth that separates a scraper that survives from one that gets blocked in an hour: a serious anti-bot system does not need to read your user agent to know you are requests.

Before a single HTTP header is parsed, the server sees your TLS fingerprint. The order of cipher suites, the extensions, and the elliptic curves your client offers during the TLS handshake form a signature known as JA3 or JA4. Python's requests (through urllib3 and OpenSSL) produces a TLS fingerprint that is nothing like Chrome's. You can send a flawless Chrome 138 user agent with perfect client hints, and the handshake still says "this is a Python script." The user agent claims Chrome; the TLS says otherwise; the mismatch is decisive.

This is why user-agent rotation is necessary but not sufficient. To get past fingerprinting you need to match the TLS layer too. The common options:

• curl_cffi: a Python library that impersonates real browser TLS fingerprints. Use it as a drop-in for requests when a target fingerprints the handshake.
• Headless browsers (Playwright, Puppeteer, Selenium): they run a real browser engine, so the TLS, header order, and client hints are genuine. Heavier and slower, but the fingerprint surface is consistent by construction.
• A scraping API: offload the entire fingerprint problem, TLS included, to a service that keeps it aligned for you.

The decision between running this stack yourself and paying an API to run it is a real trade-off, covered in web scraping API vs self-managed proxies.

If maintaining a current UA pool, matching client hints, Accept-Language by region, header order, and TLS fingerprints sounds like a lot to keep aligned, that is the point. SparkProxy's Scraping API sets all of it together on the server side, so the layers never contradict each other.

The base endpoint is https://scrape.sparkproxy.io/api/v1, authenticated with an X-API-Key header. To rotate the user agent automatically, set device to random, and the API selects a real desktop, mobile, or tablet profile with its matching headers and fingerprint:

import requests

API = "https://scrape.sparkproxy.io/api/v1"

resp = requests.get(
    API,
    params={
        "url": "https://www.sparkproxy.io",
        "render_js": "true",     # real headless render, genuine header set + TLS
        "device": "random",      # rotates a real UA + matching Sec-CH-UA per request
        "country_code": "US",    # geo exit; keeps Accept-Language regionally sensible
    },
    headers={"X-API-Key": "sk-YOUR_KEY"},
    timeout=60,
)
print(resp.status_code)
print(resp.text[:500])

When you need to pin a specific identity instead of rotating, send it explicitly. custom_ua overrides the user-agent string, and forward_headers merges your own headers on top of the browser defaults:

import requests

resp = requests.post(
    "https://scrape.sparkproxy.io/api/v1",
    headers={"X-API-Key": "sk-YOUR_KEY", "Content-Type": "application/json"},
    json={
        "url": "https://www.sparkproxy.io/pricing",
        "render_js": True,
        "device": "desktop",
        "custom_ua": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
                     "(KHTML, like Gecko) Chrome/138.0.0.0 Safari/537.36",
        "forward_headers": {"Accept-Language": "en-US,en;q=0.9"},
        "stealth": True,         # enhanced anti-detection, aligned TLS fingerprint
    },
    timeout=60,
)
print(resp.json())

Because render_js runs a real browser engine and stealth aligns the fingerprint, the user agent, the client hints, and the TLS handshake all describe the same browser. That consistency, not the rotation by itself, is what gets the response back.