How to Detect and Filter Bot Traffic in Your Analytics

"Bot traffic" collapses four populations into one phrase, and only one of them is something you want gone.

The structural fact that shapes everything else: GA4 only sees clients that execute JavaScript. A curl-based scraper pulling 40,000 product URLs never appears in your analytics at all, because it never runs your tag. It is purely a server-log problem. In the other direction, a headless Chrome session leaving from a residential exit IP shows up in GA4 as a normal user with a normal session.

Your analytics and your access log are looking at almost disjoint sets of bots. Any plan that treats "clean up GA4" and "stop the scraping" as one project will fail at both.

GA4 gives you a fixed set of dimensions and no IP address, no ASN and no user agent string. Everything below works inside that constraint.

Two of those deserve expanding.

The city dimension is the one that pays. GA4 derives geography from the IP address at collection time and then discards the address, so traffic originating in a data centre reports the data centre's city. Ashburn, Virginia is AWS us-east-1. Council Bluffs, Iowa and The Dalles, Oregon are Google. Boardman, Oregon is AWS us-west-2. None of those belong in the top five cities of a business selling to humans.

Self-referrals are not a bot signal. They are a measurement bug roughly nine times out of ten, and treating them as bot traffic sends you chasing a fix in the wrong system. Check your cross-domain configuration and your unwanted-referral list first.

If you have the free BigQuery export enabled, this query finds the dead-session clusters directly instead of clicking through Explore:

-- GA4 BigQuery export: sessions with zero engagement, grouped by city and hostname.
-- A city with 300 sessions and a 95% dead rate is not a marketing channel.
WITH sessions AS (
  SELECT
    user_pseudo_id,
    (SELECT value.int_value FROM UNNEST(event_params)
       WHERE key = 'ga_session_id') AS session_id,
    ANY_VALUE(geo.city)                 AS city,
    ANY_VALUE(device.web_info.hostname) AS hostname,
    SUM((SELECT value.int_value FROM UNNEST(event_params)
           WHERE key = 'engagement_time_msec')) AS engagement_msec,
    COUNTIF(event_name = 'page_view')   AS pageviews
  FROM `your-project.analytics_123456789.events_*`
  WHERE _TABLE_SUFFIX BETWEEN '20260801' AND '20260831'
  GROUP BY user_pseudo_id, session_id
)
SELECT
  city,
  hostname,
  COUNT(*) AS sessions,
  COUNTIF(IFNULL(engagement_msec, 0) = 0 AND pageviews = 1) AS dead_sessions,
  ROUND(SAFE_DIVIDE(
    COUNTIF(IFNULL(engagement_msec, 0) = 0 AND pageviews = 1),
    COUNT(*)) * 100, 1) AS dead_pct
FROM sessions
GROUP BY city, hostname
HAVING sessions > 50
ORDER BY dead_pct DESC, sessions DESC
LIMIT 40;

Without BigQuery, build the same thing as a free-form Explore: City and Hostname as rows, Sessions plus Engagement rate plus Average engagement time per session as values, sorted ascending by engagement rate.

The tempting rule is "datacenter IP equals bot". It correlates. It also captures several large populations of real, paying humans, and the failure is silent because none of them will email you about it.

iCloud Private Relay. Apple routes Safari traffic for iCloud+ subscribers through two hops, and the second hop is operated by Cloudflare, Akamai or Fastly. The exit address belongs to a datacenter ASN and the person behind it is a customer on an iPhone. Apple publishes the egress ranges as a CSV with country, region and city columns, so there is no excuse for getting this one wrong:

# Apple publishes iCloud Private Relay egress ranges. These are real Safari
# users sitting on datacenter-owned IPs. Pull this before you write any ASN rule.
curl -s https://mask-api.icloud.com/egress-ip-ranges.csv \
  | awk -F, 'NF { print $1 }' \
  | sort -u > private-relay-allow.txt

wc -l private-relay-allow.txt

Corporate egress. Zscaler, Netskope, Cloudflare WARP and ordinary VPN concentrators put whole companies behind a handful of datacenter addresses. On a B2B site that is not an edge case, it is your buying committee arriving as one IP with a thousand sessions.

Cloud desktops. VDI, Citrix, Windows 365 and remote development environments all present as datacenter traffic with a software GPU.

Meanwhile plenty of traffic you actively want is datacenter by definition: your uptime monitor, your CDN health checks, Slack and Teams link previews, PageSpeed Insights, and the ad verification vendor an advertiser hired to check that their creative rendered on your page.

A disclosure, because it changes how you should read this section

SparkProxy sells datacenter proxies. Treat what follows as a disclosure rather than a neutral opinion. Our customers run price monitoring, ad verification, SEO rank tracking and brand protection. From your log's point of view, that traffic is indistinguishable from a scalper bot: same ASN class, same absence of browsing history, same clean-looking Chrome user agent.

We are not going to tell you proxy traffic is always benign, and anyone who does is selling you something. Some of it is abuse. What we will tell you is that intent does not live in the ASN field. An IP address describes a delivery path. It says nothing about whether the request came from a fraud farm, a competitor's price crawler, a compliance vendor you are contractually obliged to let through, or a customer with Private Relay switched on. That is exactly why single-signal blocking on ASN is the cheapest filter to build and the most expensive one to operate: the false positives are paying customers who never complain, they just leave. For the mechanics of how these ranges get allocated and registered, see our explainer on what a datacenter ASN is.

The known-bots setting is not a setting

In Universal Analytics, "Exclude all hits from known bots and spiders" was a checkbox. In GA4 it is always on and cannot be disabled. GA4 filters against the IAB/ABC International Spiders and Bots List. Two consequences that people consistently get backwards:

1. You cannot see what it removed. There is no report, no metric, no excluded-hits line. "Our GA4 shows almost no bot traffic" is not evidence of a clean property, it is evidence that the filter is doing its job invisibly.
2. It matches on user agent. Anything driving headless Chrome with a stock Chrome user agent walks straight through it.

Data filters: two types, and they are permanent

Admin > Data settings > Data filters offers exactly two filter types: Developer traffic, matching debug_mode, and Internal traffic, matching the traffic_type parameter. Each filter sits in one of three states. Testing marks the data without removing it and exposes it through the "Test data filter name" dimension. Active permanently drops matching events. Inactive does nothing.

Google's documentation is explicit that data filters are not applied retroactively. Switching one on today does nothing about last month's mess, and an event excluded by an active filter is gone: deactivating the filter does not bring it back and support cannot restore it. Leave every new filter in Testing for at least one full week and look at what it would have removed before you promote it.

The one IP-aware lever GA4 gives you

GA4 has no IP dimension and no ASN dimension, but it does have one IP-aware feature: internal traffic rules, under Admin > Data streams > your stream > Configure tag settings > Define internal traffic. Each rule matches an IP address or CIDR range and stamps a traffic_type value onto the event. The default value is internal, and the field is free text.

That is the loophole. Create a rule with traffic_type set to something like bot_dc, pointed at the two or three CIDR ranges causing measurable damage, then build a data filter on traffic_type equals bot_dc. The rule list per stream is short, so this is a scalpel for a handful of ranges, not somewhere to paste a four-thousand-line blocklist. It also evaluates at collection time, so it fixes nothing historical.

Stamp an edge verdict onto the session

The better pattern is to compute the verdict somewhere that can see the IP, the TLS handshake and the headers, then pass it into GA4 as a custom dimension:

<script>
  window.dataLayer = window.dataLayer || [];
  function gtag(){ dataLayer.push(arguments); }
  gtag('js', new Date());
  gtag('config', 'G-XXXXXXXXXX', {
    'bot_verdict': 'SERVER_INJECTED_VALUE',  // "human" | "verified_bot" | "suspect"
    'edge_score':  'SERVER_INJECTED_SCORE'   // e.g. Cloudflare bot score, 1-99
  });
</script>

Now you can segment on the verdict without deleting anything. A client that executes your JavaScript can in principle rewrite the value before gtag runs, so this is a reporting aid and not a security control. It still beats every client-side heuristic, because the input came from the connection rather than from the page.

Segments beat filters

Everything under Data filters destroys data. An Explore segment or a report comparison does not. Build a "suspected automated" segment out of the signals GA4 does expose (zero engagement time, single pageview, city in your datacenter list, hostname not equal to your domain) and keep both views side by side. When you are wrong, and you will be wrong, you edit the segment definition instead of losing a month.

Unwanted referrals are not a bot filter

Admin > Data streams > Configure tag settings > List unwanted referrals removes a domain from referral attribution. The session still counts. Use it for payment gateways and your own subdomains. It will not reduce your session total by one.

Your access log sees every request, including the large share of automated traffic that never runs a line of JavaScript and therefore never reaches GA4. The default combined log format throws away almost everything useful, so fix that first:

log_format botdetect escape=json
  '{"ts":"$time_iso8601",'
  '"ip":"$remote_addr",'
  '"host":"$host",'
  '"method":"$request_method",'
  '"uri":"$request_uri",'
  '"status":$status,'
  '"bytes":$body_bytes_sent,'
  '"rt":$request_time,'
  '"proto":"$server_protocol",'
  '"ua":"$http_user_agent",'
  '"ref":"$http_referer",'
  '"accept":"$http_accept",'
  '"accept_lang":"$http_accept_language",'
  '"accept_enc":"$http_accept_encoding",'
  '"sec_fetch_site":"$http_sec_fetch_site",'
  '"sec_fetch_mode":"$http_sec_fetch_mode",'
  '"sec_ch_ua":"$http_sec_ch_ua",'
  '"tls_ver":"$ssl_protocol",'
  '"tls_cipher":"$ssl_cipher"}';

access_log /var/log/nginx/access.json botdetect;

The single cheapest query against that log is the static-asset ratio. A browser rendering one page also pulls CSS, JavaScript, fonts and images. An HTML-only client does not:

# Lowest asset ratios first. A ratio near zero over hundreds of requests
# means something is reading your HTML without rendering it.
jq -r 'select(.status < 400)
       | [.ip, (.uri | test("\\.(css|js|png|jpe?g|gif|svg|woff2?|ico)$") | tostring)]
       | @tsv' /var/log/nginx/access.json \
  | awk -F'\t' '{ tot[$1]++; if ($2 == "true") asset[$1]++ }
      END { for (i in tot) if (tot[i] > 200)
              printf "%.3f\t%d\t%s\n", asset[i]/tot[i], tot[i], i }' \
  | sort -n | head -30

That query lies in three predictable ways. Your own mobile app hitting /api/* has an asset ratio of zero and is not a bot. Assets served from a CDN edge never reach origin, so a real browser can look asset-free in an origin-only log. Pre-rendered and AMP pages skew it further. Scope the query to HTML paths, or exclude your API prefix, before you believe the output.

That last row is worth building. Human inter-request gaps are long-tailed and messy. A script sitting on time.sleep(2) produces a spike at two seconds and almost no spread. Coefficient of variation captures it in one number:

# Inter-request timing regularity. High precision, low recall: it will not
# catch everything, but almost nothing it flags is a person.
import statistics
from collections import defaultdict

def timing_cv(events, min_requests=50):
    """events: iterable of (ip, epoch_seconds) in ascending time order."""
    by_ip = defaultdict(list)
    for ip, ts in events:
        by_ip[ip].append(ts)

    out = {}
    for ip, stamps in by_ip.items():
        if len(stamps) < min_requests:
            continue
        gaps = [b - a for a, b in zip(stamps, stamps[1:]) if b > a]
        if len(gaps) < 10:
            continue
        mean = statistics.fmean(gaps)
        if mean == 0:
            continue
        out[ip] = statistics.pstdev(gaps) / mean
    return dict(sorted(out.items(), key=lambda kv: kv[1]))

# CV below roughly 0.2 across 50+ requests: scripted on a fixed interval.
# CV above 1.0: normal human browsing, or several people behind one NAT.

A meaningful share of the "Googlebot" in your log is not Googlebot. The user agent field is free text and costs nothing to forge. Two verification methods exist, both documented by Google.

The first is reverse then forward DNS. Reverse-resolve the IP; the hostname must end in googlebot.com, google.com or googleusercontent.com; then forward-resolve that hostname and confirm it returns the original IP. The forward step is not optional, because reverse DNS records are set by whoever controls the address block:

import socket

GOOGLE_SUFFIXES = (".googlebot.com", ".google.com", ".googleusercontent.com")

def verify_googlebot(ip: str) -> bool:
    try:
        host, _, _ = socket.gethostbyaddr(ip)            # reverse
    except (socket.herror, socket.gaierror):
        return False
    if not host.rstrip(".").endswith(GOOGLE_SUFFIXES):
        return False
    try:
        _, _, forward = socket.gethostbyname_ex(host)    # forward
    except (socket.herror, socket.gaierror):
        return False
    return ip in forward

The second is faster and needs no DNS round trip. Google, Microsoft and OpenAI all publish their crawler ranges as JSON:

import ipaddress
import requests

RANGE_FILES = {
    "googlebot":      "https://developers.google.com/static/search/apis/ipranges/googlebot.json",
    "google-special": "https://developers.google.com/static/search/apis/ipranges/special-crawlers.json",
    "google-user":    "https://developers.google.com/static/search/apis/ipranges/user-triggered-fetchers.json",
    "bingbot":        "https://www.bing.com/toolbox/bingbot.json",
    "gptbot":         "https://openai.com/gptbot.json",
}

def load_networks(url):
    data = requests.get(url, timeout=10).json()
    nets = []
    for entry in data.get("prefixes", []):
        cidr = entry.get("ipv4Prefix") or entry.get("ipv6Prefix")
        if cidr:
            nets.append(ipaddress.ip_network(cidr))
    return nets

CACHE = {name: load_networks(url) for name, url in RANGE_FILES.items()}

def classify(ip: str):
    addr = ipaddress.ip_address(ip)
    for name, nets in CACHE.items():
        if any(addr in net for net in nets):
            return name
    return None

Refresh that cache daily. These files change, and a stale copy produces false negatives on newly allocated ranges, which is the worst kind of error here: you end up blocking the real crawler.

A request that claims to be Googlebot and fails verification is the highest-precision signal in this entire article. There is no legitimate reason for a browser to send a Googlebot user agent, and no meaningful false-positive rate to worry about. The motive is usually specific: plenty of sites serve a different version of a page to search crawlers, so a forged Googlebot UA is often somebody trying to get past a paywall or a registration gate. Track your Googlebot verification failure rate as a standing metric. It is one of the very few bot numbers you can state without hedging.

Stop asking "is this client a bot". Ask "does every layer of this connection agree about what client it is". A real browser is internally consistent across TLS, HTTP/2, header ordering and its JavaScript surface. Automation stacks usually are not, and the mismatch is far more informative than any single fingerprint.

TLS fingerprints, and why JA3 is past its expiry date

JA3 hashes the TLS ClientHello: version, cipher suites, extensions, elliptic curves and point formats, in the order sent. It worked well until Chrome 110 in February 2023 shipped ClientHello extension permutation, which randomises extension order per connection. Chrome's JA3 hash has not been stable since. A vendor still selling you a JA3 blocklist in 2026 is selling you false positives.

JA4, from FoxIO, sorts the cipher and extension lists before hashing, which survives the randomisation, and emits a readable string instead of an opaque MD5. Either way the hash alone is not the signal. The signal is the contradiction: a ClientHello whose fingerprint matches Python's urllib3, arriving with a user agent that claims Chrome 141, answers the question in one line. Our field-by-field breakdown of TLS fingerprinting covers what goes into the hash.

This cuts both ways, and you should know it. Libraries like curl_cffi and tls-client exist specifically to reproduce a browser ClientHello byte for byte, and they are good at it. TLS fingerprinting catches the lazy majority and misses anyone who spent an afternoon on the problem.

HTTP/2 settings and pseudo-header order

Before any request, an HTTP/2 client sends a SETTINGS frame and usually WINDOW_UPDATE and PRIORITY frames. The parameter values, the frame order and the order of the four pseudo-headers are all implementation-specific and stable per client. Akamai's HTTP/2 fingerprint format encodes exactly that.

Chrome sends its pseudo-headers as :method, :authority, :scheme, :path. Firefox and Safari each use a different fixed order, and most non-browser HTTP/2 clients use another one again. The test is not "is the order X", it is "does this order match the browser the user agent claims to be". This runs at the edge, needs no JavaScript, and works on your JSON API endpoints as well as your HTML pages.

Header presence and ordering

Browsers emit request headers in a fixed order per version. HTTP libraries emit them in dictionary order, which usually differs, and frequently omit headers a browser always sends. Each of the checks below needs its conditional clause or it will generate false positives:

• If the user agent claims Chromium 89 or newer over HTTPS, Sec-CH-UA, Sec-CH-UA-Mobile and Sec-CH-UA-Platform should be present. Safari and Firefox do not send client hints at all, so absence only matters when the UA claims Chrome or Edge.
• Fetch metadata headers (Sec-Fetch-Site, Sec-Fetch-Mode, Sec-Fetch-Dest) ship in Chromium 76 and later, Firefox 90 and later, and Safari 16.4 and later, on secure origins. A modern-browser UA over HTTPS with none of them is inconsistent.
• Accept: / on a request for an HTML document, or a missing Accept-Language entirely.
• Accept-Encoding advertising only gzip when the claimed browser has supported Brotli for years, and Zstandard since Chrome 123.

No single item there is decisive. Two or three together, plus a TLS contradiction, is about as close to certainty as this field gets.

Headless browser tells

Headless Chrome and Playwright still leave marks, though fewer with every release. Two pieces of advice you will find on the first page of search results are now actively wrong. "Check for HeadlessChrome in the user agent" is dead: Chrome's new headless mode, available as --headless=new from Chrome 112 and the default for plain --headless since Chrome 132, uses the ordinary Chrome UA string. "Check navigator.plugins.length === 0" is weak: modern Chrome reports a small fixed plugin list either way.

What still fires:

// Client-side headless signals. Post the result to your own endpoint and treat
// it as evidence, never as enforcement: anything running in the page can be
// patched by anything else running in the page.
function headlessSignals() {
  const s = {};
  s.webdriver = navigator.webdriver === true;

  const canvas = document.createElement('canvas');
  const gl = canvas.getContext('webgl') || canvas.getContext('experimental-webgl');
  if (gl) {
    const dbg = gl.getExtension('WEBGL_debug_renderer_info');
    s.renderer = dbg ? gl.getParameter(dbg.UNMASKED_RENDERER_WEBGL) : null;
    // SwiftShader, llvmpipe and ANGLE software backends mean there is no GPU.
    s.softwareGpu = /swiftshader|llvmpipe|software/i.test(s.renderer || '');
  }

  s.noChromeRuntime = /Chrome\//.test(navigator.userAgent) && !window.chrome;
  s.zeroOuterSize   = window.outerWidth === 0 || window.outerHeight === 0;
  s.emptyLanguages  = !navigator.languages || navigator.languages.length === 0;
  return s;
}

Read that list with the false positives attached. Software GPU is also what you get from a virtual machine, a remote desktop session, a Linux box with no drivers and plenty of locked-down corporate images, so it deserves almost no weight. navigator.webdriver is trivially patched and every stealth plugin patches it first, so its presence is meaningful while its absence proves nothing. For the wider surface, see headless browser detection and browser fingerprinting.

Cloudflare ships three different products under one banner and the difference decides what you can build.

Bot Fight Mode (Free) challenges traffic it judges automated, zone-wide, with no exceptions and no visible score. It will challenge your own API clients and your monitoring. Enable it only if there is nothing machine-readable on the domain.

Super Bot Fight Mode (Pro and Business) sorts traffic into definitely automated, likely automated and verified bots, each with its own allow, challenge or block action, plus a static-resource exemption.

Bot Management (Enterprise) is the real product. Every request gets a bot score from 1 to 99, where 1 is definitely automated and 99 is definitely human, and Cloudflare's guidance is that scores below 30 usually indicate automation. The score, the verified-bot flag, the verified-bot category and the JA3 and JA4 hashes are all available as WAF rule fields and can be forwarded to your origin.

The move most teams skip: log the score before you act on it.

# Cloudflare Transform Rule, Modify Request Header, applied to all requests.
# Forwards the edge verdict to your origin so it lands in the access log next
# to everything else, before you write a single blocking rule.
x-bot-score     = cf.bot_management.score
x-verified-bot  = cf.bot_management.verified_bot
x-bot-category  = cf.verified_bot_category
x-ja4           = cf.bot_management.ja4

Once you have a week of that distribution against your own conversion data, write a narrow rule rather than a zone-wide one:

# Challenge low-scoring traffic on expensive paths only. Never touch verified
# bots, and never touch your own monitoring.
(cf.bot_management.score lt 15
 and not cf.bot_management.verified_bot
 and http.request.uri.path in {"/search" "/api/quote" "/checkout"}
 and not ip.src in $monitoring_allowlist)

Cloudflare's verified bot categories cover search engine crawlers, monitoring and analytics, page preview, advertising and marketing, academic research, security, accessibility, feed fetchers and AI crawlers. Two things are worth knowing about that last one. Since 1 July 2025 Cloudflare blocks AI crawlers by default on newly onboarded domains and offers a pay-per-crawl option. If your crawl or referral traffic dropped in the second half of 2025 and nobody changed anything, check that toggle before you blame your content strategy.

The equivalents elsewhere have the same shape and different field names: AWS WAF Bot Control with its Common and Targeted rule groups (Targeted adds TLS fingerprinting and browser challenges), Fastly Next-Gen WAF, Akamai Bot Manager and HUMAN.

One correction that saves arguments: robots.txt is not enforcement. It is a polite request, honoured by exactly the crawlers you would not have blocked anyway. Use it to shape well-behaved crawling and put actual rules at the edge for everything else.

The advertising industry has its own vocabulary for this, defined by the Media Rating Council, and using it correctly is what gets you taken seriously in a conversation with a platform.

GIVT, General Invalid Traffic, is invalid traffic identifiable through routine, list-based filtration. Known data-centre traffic, crawlers on the IAB/ABC International Spiders and Bots List, non-browser user agents, browser pre-render and pre-fetch, and activity with irregular measurement patterns. Cheap to catch, and it catches the obvious.

SIVT, Sophisticated Invalid Traffic, requires advanced analytics, multi-point corroboration or human intervention. Hijacked devices and sessions, malware and adware, ad stacking, pixel stuffing, hidden ads, domain and app spoofing, falsified location or viewability, incentivised manipulation and cookie stuffing. This is where the money actually goes.

The practical conclusion is unglamorous. The lever with money attached is pre-bid filtering and the wording of your insertion order, not a post-campaign refund request. Chasing an after-the-fact claim on a self-serve platform burns a quarter and usually returns a templated reply. If you sell inventory rather than buy it, publish ads.txt and app-ads.txt; if you buy it, check sellers.json on the other side of every deal.

One reconciliation point saves a lot of pointless investigation. Google Ads clicks will never equal GA4 sessions, and part of that gap is invalid-click filtering happening at a different stage, on a different list, from GA4's bot filtering. A double-digit percentage gap is routine, and it has four boring causes before you reach fraud: users who leave during the redirect, tags that never fire on slow mobile connections, ad blockers, and consent denial. Driving that number to zero is not a project, it is a trap.

If you are on the buy side and want to see what your ads actually render as in each market, how brands use proxies for ad verification covers the same problem from the advertiser's chair.

Every signal in this article carries a false-positive population. Combining them with AND is so strict it catches nothing. Combining them with OR is a customer-service incident. Weight them, and act on the total:

# Weighted verdict. No single signal can block on its own, which is the point.
# Tune these weights against a labelled sample of your own traffic. Do not
# ship these numbers as they are.
WEIGHTS = {
    "claims_googlebot_fails_dns":   55,   # near-zero false positives
    "tls_ua_mismatch":              35,
    "h2_pseudo_header_mismatch":    30,
    "webdriver_true":               30,
    "timing_cv_below_0_2":          25,
    "missing_sec_ch_ua_on_chrome":  20,
    "zero_asset_ratio":             15,
    "datacenter_asn":               10,   # deliberately weak on its own
    "no_accept_language":           10,
    "software_gpu":                  5,   # VMs and remote desktops are real users
}

CREDITS = {
    "verified_search_crawler":    -100,
    "known_monitoring_vendor":    -100,
    "prior_completed_order":       -80,
    "authenticated_session":       -60,   # they logged in, they are a person
    "icloud_private_relay":        -40,
}

def verdict(signals):
    score = sum(WEIGHTS.get(s, 0) for s in signals)
    score += sum(CREDITS.get(s, 0) for s in signals)
    if score >= 70:
        return "block"
    if score >= 40:
        return "challenge"
    if score >= 20:
        return "log_and_segment"
    return "allow"

Three rules keep a model like that from hurting you.

Authenticated sessions are exempt. If somebody logged in and has an order history, the ASN is irrelevant. Anything that blocks a signed-in customer is a bug no matter what the score says.

Challenge before you block. A managed challenge or a Turnstile widget costs a real person about two seconds and costs a scripted client the session. A bare 403 costs a real person the entire visit, and teaches you nothing about whether you were right.

Run in shadow mode for two weeks. Log the verdict, act on none of it, then pull every session the rule would have blocked and join it against your orders table. If any of them bought something, your thresholds are wrong and you now know by exactly how much. That one query is the only honest accuracy test available to you, and it is the step almost everybody skips on the way to a rule that quietly costs more than the bots did.

Some plain statements to end on.

You cannot measure your own accuracy, because you have no labels. Every "we removed 22% bot traffic" claim, including any you make internally, is an estimate against an unmeasured ground truth. Treat the figure as a direction of travel, not a fact to report to a board.

Residential and mobile proxy traffic defeats every IP-based method by construction. The exit address belongs to a genuine consumer ISP, in a plausible city, with clean reputation history. No ASN list touches it.

Well-built automation defeats every fingerprint method. curl_cffi, patched Chromium builds and commercial antidetect browsers reproduce browser TLS, HTTP/2 and JavaScript surfaces closely enough that the mismatch signal disappears. If you are a target worth real effort, the effort gets spent, and your detection budget is running against somebody with a direct financial incentive.

The two error types are not symmetric. A bot you failed to block costs you some bandwidth and some skew in a report. A customer you blocked costs you the order, the lifetime value and possibly a review. Your thresholds should reflect that asymmetry rather than pretending the errors are equal.

And the goal is not zero automated traffic. The goal is traffic you can reason about. If you know that 12% of your sessions are automated, and you can segment them out whenever you look at conversion rate, you have solved the analytics problem completely without breaking anything. Blocking is a separate decision, with separate costs, made on a separate code path. Keeping those two decisions apart is the single most useful habit in this whole area.