Screaming Frog Proxy Setup: Auth, Rotation, Speed

Without a proxy, every request in a crawl comes from your office or home IP. The target's firewall sees one address making thousands of requests with the Screaming Frog SEO Spider user-agent, and many CDNs and bot managers treat that pattern as a scraper after a few hundred URLs. You also see the site as your own country sees it, which is useless when the audit is about a German storefront or a US-only landing page.

With a proxy, three things move. The source IP the server logs becomes the gateway's exit IP. The country the site detects becomes the exit's country. And the bottleneck shifts: the proxy's concurrency limit and the target's per-IP tolerance now cap your crawl speed, not your own connection.

Screaming Frog's own proxy guide states the constraints in three short lines, all read on 2026-10-30:

• "Only 1 proxy server can be configured."
• "You must restart for your changes to take effect."
• "No exceptions can be added, either all HTTP/HTTPS traffic goes via the proxy, or none of it does."

The first one shapes the whole post. You cannot hand the Spider a list of 50 proxies and let it round-robin. If you want many exit IPs, the rotation has to happen on the provider's side, behind one hostname and port. The third means rendered JavaScript, images, CSS and the robots.txt fetch all travel the same route as the HTML, so whatever you pay for in concurrency gets spent on assets too.

Many tutorials, and plenty of team runbooks, still say Configuration > System > Proxy. That was the location in older builds. The current user guide, which documents version 24.3 (listed on the release history page as released 29 June 2026), puts it here:

We have not pinned which release moved it, so if your menu has a Configuration > System entry, use that and the fields are the same. The move matters for one practical reason: proxy now sits with the machine-level options (storage mode, memory allocation) under File > Settings, not with the crawl options under Configuration. Treat it as a per-machine setting and do not count on a client's .seospiderconfig file to switch it on or off; the user guide does not say either way, so confirm on your install. Check this panel before every client crawl, because a proxy left ticked from yesterday's job is one of the causes the official FAQ lists for "Connection refused".

The panel itself is small. Tick Use Proxy Server, enter the address and port, supply credentials if you use them, click OK, then quit and relaunch. If you skip the restart, the Spider keeps crawling from your own IP and nothing in the interface tells you.

For SparkProxy, the values are:

Address:  gateway.sparkproxy.io
Port:     11000   (Random Rotation, new exit per request)
          11002   (Sticky Sessions, same exit for up to 5 minutes)
Username: your SparkProxy proxy username   (leave blank if you whitelist your IP)
Password: your SparkProxy proxy password

The Screaming Frog documentation only describes HTTP/HTTPS traffic for this panel. SparkProxy also offers SOCKS5 on port 13000, but we have not verified a SOCKS option in the Spider's proxy panel, so use the HTTP ports above.

SparkProxy supports both methods, and the choice depends on where the Spider runs rather than on preference.

User:pass works from any network. Enter the credentials in the panel and the Spider answers the gateway's 407 Proxy Authentication Required challenge on each connection. Use this on a laptop that moves between office, home and hotel Wi-Fi, because a whitelist would break every time your public IP changes.

IP whitelisting skips credentials entirely: the gateway accepts any connection from an address you have registered. It suits a fixed machine, such as a crawl server or a cloud VM that runs scheduled audits overnight. Plans include a fixed number of whitelist slots (5 on Starter, up to 25 on Plus, per the SparkProxy pricing table read 2026-10-30), so an agency with several crawl boxes should count them first. Our explainer on how IP whitelisting works for proxies covers the mechanics.

Test the credentials outside the Spider before you blame the Spider. One curl call tells you whether the gateway, the auth and the target all respond:

# Rotating port, user:pass auth. -I fetches headers only.
curl -sS -I -x "http://USER:PASS@gateway.sparkproxy.io:11000" https://www.sparkproxy.io/

# Same test with a whitelisted IP: no credentials in the proxy URL.
curl -sS -I -x "http://gateway.sparkproxy.io:11000" https://www.sparkproxy.io/

A good result starts with HTTP/1.1 200 Connection established (the tunnel) followed by the site's own status line. Two failures are worth recognising on sight:

• curl: (56) CONNECT tunnel failed, response 407 means wrong username or password, or an IP that is not on the whitelist.
• curl: (7) Failed to connect to gateway.sparkproxy.io port 11000 means the connection never reached the gateway: a typo in the host, a local firewall, or a corporate network blocking outbound ports.

If curl succeeds and the Spider fails, the problem is the panel or the missing restart.

The Spider has no indicator that says "proxy active". The reliable proof is the target server's access log, and on an SEO audit you often have it: it is your client's site, or your own staging copy.

Set a unique user-agent so your crawl is easy to find. Under Configuration > User-Agent, choose a custom string and append a token, for example Screaming Frog SEO Spider audit-2026-10-31. Crawl a few hundred URLs, then run this against the access log (combined log format, as nginx and Apache write by default):

import re
import sys
from collections import Counter, defaultdict

TOKEN = "audit-2026-10-31"
LINE = re.compile(r'^(\S+) \S+ \S+ \[([^\]]+)\] "[^"]*" (\d{3}) \S+ "[^"]*" "([^"]*)"')

def main(path):
    per_ip = Counter()
    statuses = defaultdict(Counter)
    with open(path, encoding="utf-8", errors="replace") as fh:
        for raw in fh:
            m = LINE.match(raw)
            if not m or TOKEN not in m.group(4):
                continue
            ip, status = m.group(1), m.group(3)
            per_ip[ip] += 1
            statuses[ip][status] += 1
    total = sum(per_ip.values())
    print(f"{total} requests from {len(per_ip)} distinct IPs")
    for ip, n in per_ip.most_common(10):
        print(f"{ip:<40} {n:>6}  {dict(statuses[ip])}")

if __name__ == "__main__":
    main(sys.argv[1])

Run it as python check_exits.py /var/log/nginx/access.log. If every line shows your office IP, the proxy is off or the Spider was not restarted. If you see gateway exits, the setup is live, and the distinct-IP count tells you how the rotation behaves with your thread count, which the next section explains.

No log access? Crawl a single page on a site you control and check its analytics real-time view for the country. It is cruder, but it catches the "forgot to restart" case.

Because the Spider accepts one proxy, a rotating gateway is the only way to spread a crawl across many addresses. Port 11000 on SparkProxy gives each request a fresh exit from the pool. In practice, the number of distinct IPs your log script reports will usually be well below the number of requests, and the reason is how HTTPS proxying works, not a fault in the pool.

For an https:// URL, the Spider opens a CONNECT tunnel through the gateway, then speaks TLS to the target inside it. The gateway cannot see the individual requests inside an encrypted tunnel, so it can only pick a new exit when a new tunnel opens. Like most HTTP clients, the Spider reuses open connections (keep-alive) to save handshakes. A thread that fetches 40 URLs over one kept-alive tunnel sends all 40 from one exit IP.

That gives you a rule of thumb you can verify in your own logs: on HTTPS sites, "per request" rotation behaves closer to "per connection" rotation. Two consequences follow.

1. Your thread count, not the pool size, sets how many exits are in use at any moment. Five threads means roughly five live tunnels and five active IPs, recycling as connections close.
2. A per-IP rate limit on the target is hit by one thread's tunnel, not by the whole crawl. If one thread runs fast on a long-lived connection, that one exit absorbs all its requests.

For a deeper look at choosing rotation timing for a given target, see our post on picking a proxy rotation interval. For Screaming Frog, the practical lever is crawl speed, covered below, because you cannot tell the Spider to close connections more often.

Random Rotation suits a plain HTML crawl of a public site: independent requests, no login, no state worth keeping. Three Spider features break that independence, and each one is a reason to switch the port to 11002 for Sticky Sessions, which hold one exit IP for up to 5 minutes. If the concept is new, our explainer on what a sticky session proxy is covers it in full.

JavaScript rendering. Under Configuration > Spider > Rendering, JavaScript mode loads each page in the Spider's Chromium engine, which then requests the page's scripts, styles, fonts and API calls. With rotation, those sub-requests can leave from different exits than the HTML did. Bot managers that issue a challenge cookie to one IP and then see it presented from another often answer with a 403 or a challenge page, and the Spider records the challenge page as the rendered content. Sticky Sessions keep the page and its assets on one IP.

Cookies. The Spider's default, under Configuration > Spider > Advanced > Cookie Storage, is "session only". The "Persistent" option stores cookies per crawl and shares them between threads. Either way, a cookie set by one exit is replayed from whatever exit the next connection gets. Consent banners, geo-redirect cookies and A/B bucketing cookies all tend to be tied to the first response. If the site keys behaviour off a cookie it set, keep the IP stable underneath it.

Forms-based authentication. Crawling a staging site or members area through Configuration > Authentication means the login cookie must stay valid for the crawl. Some applications bind a session to the client IP and log you out the moment it changes.

Sticky Sessions have a limit you should plan around: the 5-minute window ends and the IP changes, even mid-crawl. For rendering and consent cookies that is usually harmless, because each page completes well inside 5 minutes. For an IP-bound login it is not, and no rotating or sticky setting can fix it. Ask the site owner to allowlist your fixed IP, or crawl that section without the proxy.

Screaming Frog's user guide says "By default the SEO Spider crawls at 5 threads, to not overload servers." Under Configuration > Speed you can raise Max Threads or cap the crawl with Max URI/s, which the guide defines as "the maximum number of URL requests per second". The guide also notes Max Threads "can simply be left alone when you throttle speed via URLs per second", and that is the setting to use behind a proxy. URL/s is a ceiling the target actually experiences. Threads are only an indirect control.

Two separate limits apply, and people confuse them.

Your plan's thread allowance caps simultaneous connections through the gateway. SparkProxy Starter includes 100 threads, Core 250, Boost 500 and Plus 1,000 (pricing table read 2026-10-30). See our guide to concurrent connections in proxies for how that allowance is counted. JavaScript rendering multiplies connections, because Chromium fetches assets in parallel for each page, so a rendered crawl can hold several gateway connections per Spider thread.

The target's tolerance is the real ceiling. A site that returns 429 Too Many Requests at 10 URL/s will do so whether you have 100 threads or 1,000. The proxy spreads requests across IPs, which helps against per-IP limits, but it does nothing against limits keyed on user-agent, cookies or the request pattern of the crawl as a whole.

These are our starting points, not benchmarks. Raise them while the 4xx share in the Response Codes tab stays flat, and drop back the moment 403s or 429s start to climb:

The last row follows the official FAQ, which suggests trying "1 URL per second" when a site returns 503s. If a crawl starts clean and degrades after a few thousand URLs, that pattern points at an accumulating block on the exits or the fingerprint, and slowing down helps more than adding threads.

Two other Advanced settings matter behind a proxy. Response Timeout adds the proxy hop to every request; the user guide says the Spider "will wait 20 seconds to get any kind of HTTP response from a URL by default", while the FAQ still mentions 10 seconds, so check what your install shows. 5XX Response Retries re-requests server errors automatically, and on a rotating port each retry usually gets a fresh exit, which is often enough to clear a one-off upstream error.

For geo audits the proxy country matters as much as the IP count. If a retailer redirects non-German visitors from /de/ to a country picker, crawling it from a UK exit audits the picker, not the store. Run the exit IPs from the log script through a GeoIP lookup, or watch the site's analytics real-time country view, before you trust a single row.

SparkProxy covers 80+ countries from a pool of 1M+ premium IPs; the 24-hour free trial includes USA and worldwide rotating proxies. Use the trial to confirm the exits you get geolocate to the country the audit needs, with a short crawl of your own site and that GeoIP check, before you quote the job.

Before committing to a full crawl, spot-check a handful of key URLs from the target country with the SparkProxy Scraping API. A plain fetch with render_js=false costs 1 credit and country_code adds 5, so each check below is 6 credits, and the 1,000 free credits cover about 166 of them. Remember that render_js defaults to true (5 credits), so set it explicitly. The json_response=true envelope returns the body base64-encoded, so decode it before looking for markers:

import base64
import requests

API = "https://scrape.sparkproxy.io/api/v1"
KEY = "YOUR_API_KEY"
URLS = [
    "https://www.sparkproxy.io/de/",
    "https://www.sparkproxy.io/de/pricing",
]

def check(url, country):
    resp = requests.get(
        API,
        headers={"X-API-Key": KEY},
        params={
            "url": url,
            "render_js": "false",
            "country_code": country,
            "json_response": "true",
        },
        timeout=60,
    )
    resp.raise_for_status()
    env = resp.json()
    html = base64.b64decode(env["body"]).decode("utf-8", errors="replace")
    has_hreflang = 'hreflang="de' in html
    print(f"{url} status={env['status_code']} credits={env['credits_used']} "
          f"ms={env['duration_ms']} hreflang_de={has_hreflang} bytes={len(html)}")

for u in URLS:
    check(u, "de")

If the API returns the localised page and the Spider through the proxy returns a redirect or the picker, compare exit countries first. If both return the wrong version, the site is probably reading Accept-Language or a cookie rather than the IP. Set the language header under Configuration > HTTP Header in the Spider and crawl again.

Large audits rarely run on a laptop. The Spider's scheduler (File > Scheduling, listed in the user guide among licensed features) and its command-line interface both run headless. Since the proxy lives under File > Settings, configure it in the interface on the crawl machine itself, restart, run one short test crawl and check the log script output, then schedule. Do not assume a config file copied from another machine brings the proxy with it.

On Windows, the CLI executable is ScreamingFrogSEOSpiderCli.exe in the install folder. A nightly crawl that saves the crawl file and exports two tabs looks like this:

"C:\Program Files (x86)\Screaming Frog SEO Spider\ScreamingFrogSEOSpiderCli.exe" ^
  --crawl https://www.sparkproxy.io/ ^
  --headless ^
  --config "C:\crawls\configs\proxy-sticky-render.seospiderconfig" ^
  --save-crawl ^
  --output-folder "C:\crawls\output" ^
  --timestamped-output ^
  --export-tabs "Internal:All,Response Codes:Client Error (4xx)"

Register it as a scheduled task on a VM with a fixed public IP, and whitelist that IP in SparkProxy so no credentials live on the box:

$action  = New-ScheduledTaskAction -Execute "C:\crawls\nightly-crawl.bat"
$trigger = New-ScheduledTaskTrigger -Daily -At 1am
Register-ScheduledTask -TaskName "SF nightly audit" -Action $action -Trigger $trigger

--timestamped-output keeps one folder per run, which makes week-over-week diffs easy. The exported 4xx tab is your health check for the proxy: a sudden jump in 403s between runs means the exits or the speed need attention before the next crawl, not after.

Keep two config files per client, one for Text Only and one for JavaScript rendering. The port is set in the proxy panel, so switching from 11000 to 11002 means changing the panel and restarting, which is easy to forget at 1 a.m. A dedicated VM per proxy mode removes the problem.

Work through these in order, cheapest check first. The status text is what the Spider shows in its Status column, and the explanations quote the official FAQ where it has one.

The FAQ describes 403 as occurring "when a web server or security platform denies access to the SEO Spider's request". A proxy changes the IP the security platform sees. It does not change the default Screaming Frog SEO Spider user-agent, which some sites block by name, and it does not make Text Only mode pass a JavaScript challenge. If 403s persist from fresh exits at 1 URL/s, try a browser user-agent with JavaScript rendering on Sticky Sessions, and if that still fails, the block is not about your IP.

When the job is a client's site, the cleanest fix for a stubborn WAF is often a request to whitelist your crawl IP. A proxy is the right tool when you do not control the firewall, need another country's view, or cannot afford to get your office IP blocked.

For rank-tracking style SERP collection, which has its own block patterns and is a different job from a site crawl, see our post on datacenter proxies for SEO rank tracking.

Two line items. The Screaming Frog licence is £199 per licence, per year (GBP, pricing page read 2026-10-30); the free version stops at 500 URLs, so a large-site audit needs the licence anyway.

SparkProxy plans start from $75/month (USD, excluding VAT/GST, read 2026-10-30) and are sold as packs sized by threads, with unlimited bandwidth included. That billing model suits crawling in a way per-GB pricing does not. JavaScript rendering pulls every script, image and font through the proxy, which on a media-heavy site can multiply transfer many times over a Text Only crawl, yet the bill stays the same because you pay for concurrency, not gigabytes. Term discounts are 5% for 3 months, 10% for 6 and 15% for 12.

A single Spider at the speeds in the table above uses a small fraction of Starter's 100 threads. The larger plans make sense when an agency runs several crawl machines at once. The 24-hour free trial needs no credit card, which is enough to run the log check, one rendered sample crawl and the curl tests against a real client site before you commit.