Proxies for Threat Intelligence: Building SOC Infrastructure

A threat intelligence program is not one collection problem. It is four, and they want different products. Match the row.

The practical build for most teams: one flat datacenter plan carrying 80 to 90 percent of request volume, a small residential allocation in reserve for consumer platforms, and a separate isolated egress path for anything that touches live attacker infrastructure. Buy a single expensive tier for everything and you overpay by a large multiple while still getting filtered on the sources that matter, because IP type was never what failed.

The mechanics of running that collection, as opposed to buying it, are in the companion piece on datacenter proxies for cybersecurity threat intelligence. This post is the purchase and the architecture around it.

A scraping proxy has one job: get the page. A threat intelligence proxy has three, and the extra two should drive the purchase.

The adversary reads logs too. Commercial scraping targets rarely care who is crawling them. An attacker running a phishing kit absolutely does. Off-the-shelf kits ship with blocklists that filter visitors by ASN, reverse DNS, hostname pattern and user agent, specifically to hide the lure from security vendors, crawlers and sandbox infrastructure. Researchers have documented that filtering in kit source code for years. When your collector arrives from a range that resolves to your company, you do not get an error. You get a decoy page, a 404, or a redirect to a benign site, and your feed quietly records the wrong thing. That is worse than a block, because nothing alerts.

Compartmentalization is a requirement, not a nicety. If one exit IP fetches an open C2 panel on Tuesday and enriches a customer's indicator on Wednesday, you have linked two activities in a way that exists nowhere except inside your own infrastructure. An adversary correlating panel logs across campaigns can see it. So can anyone who later obtains those logs.

Repeatability under scrutiny. Intelligence products get challenged and timelines get rebuilt months later. You need to state what you fetched, when, from which country, and what came back, then fetch it again. That favors stable geo metadata and predictable sessions over the cheapest pool with the highest churn.

Model the program as tiers with different risk and economics. That structure is what makes the budget defensible.

Two things fall out of that table. The tier consuming almost all your request volume is also the cheapest to serve, so per-GB pricing fits this workload badly. And tier 3 is a security control rather than a scraping problem: provision it, review it and retire it like any other sensitive access path. For the underlying differences between IP types, see the comparison of residential, datacenter and mobile proxy types.

Most teams size on monthly request count and buy four times what they need. Size on peak concurrent connections instead. The arithmetic is short:

sustained_threads = (requests_per_day x avg_seconds_per_request) / 86400
plan_threads      = sustained_threads x peak_factor

A peak factor of 3 to 4 is realistic for a SOC, because enrichment queues drain in bursts after a feed ingest or when a hunt kicks off, not at a flat rate all day.

Worked example one. You enrich 200,000 indicators a day against four sources, so 800,000 requests, each plain HTTP call taking about 1.5 seconds. That is (800000 x 1.5) / 86400, roughly 14 sustained threads. At a peak factor of 4 you land near 56, and a 100-thread plan covers it with headroom for analyst lookups.

Worked example two. Continuous monitoring across a few thousand watched domains, 2,000,000 requests a day, with rendering pushing the average to 6 seconds. That is roughly 139 sustained threads, and at a peak factor of 2.5 you need about 350. Now you are shopping in the 500-thread bracket.

SparkProxy's published datacenter plans, all unlimited bandwidth with 30 days validity:

Larger Pro and Pro+ tiers, at 1500 and 2000 threads with 200 and 250 Mbps ceilings, exist under the Fair Usage Policy and are quoted rather than listed. Treat every speed number as a ceiling, not a guaranteed rate: real throughput depends on the target, the route, and how many threads you run at once. The network behind those plans covers over 1 million datacenter IPs across 80+ countries, including more than 50,000 US datacenter IPs, which is what makes per-country monitoring practical instead of theoretical.

One sizing trap is specific to this category: render budget moves the math more than volume does. Shifting 10 percent of your monitoring to a headless browser can double the concurrency you need. Decide the render share before you pick a plan, not after.

Every SOC with a competent platform team asks this. DIY wins for a handful of controlled static egress points and loses badly for everything else.

Start with the part most build plans miss. AWS, Microsoft Azure and Google Cloud all publish their public IP ranges as machine-readable files anyone can download: AWS as ip-ranges.json, Azure as Service Tags, Google as cloud.json. An adversary who wants to know whether a visitor came from a cloud provider needs a JSON file and a subnet match, not a threat feed. Default cloud VMs therefore give you an exit that is trivially classifiable as automation, which is exactly the property tier 3 collection cannot have.

Then price the addresses. AWS began charging for all public IPv4 addresses on 1 February 2024 at $0.005 per address-hour, roughly $3.60 per address per month before any compute, per Amazon's own published pricing. Fifty addresses is about $180 a month in address rent alone, plus instances, egress data charges, and the engineer who maintains it. Check current rates on each provider's pricing page before you model this, since cloud pricing moves.

The realistic architecture is both. Buy a commercial plan for tiers 0 through 2, where volume and country coverage decide the outcome, and keep a small owned egress path for work where you must control every packet and every log. Do not make either one do the other's job.

This is where the buying decision and the security architecture meet, and where the plan spec sheet starts to matter in a way nobody writes about.

Whitelist slots cap how many collector hosts you can authorize by IP. Starter includes 5, Core 10, Boost 15, Plus 25. Run twelve collectors across three regions and Starter cannot express that with IP authorization, regardless of thread count. Username and password authorization scales without slot limits and travels better through containers and ephemeral workers, so most SOC deployments should use credentials for collectors and reserve whitelist slots for fixed, long-lived hosts. The mechanics of both are in how proxy authentication works.

Issue credentials per workstream, not per team. Separate credentials for enrichment, brand monitoring, hunt support and sensitive collection buy you three things: attribution of spend, the ability to revoke one workstream without a global rotation, and a hard boundary that stops tier 3 traffic exiting on a tier 0 identity because somebody copied a config file.

Write a burn policy before you need one. Decide in advance what happens to an exit identity used against adversary-controlled infrastructure: retire it, never reuse it for benign enrichment, record which campaign it touched. Two lines of policy prevent a correlation problem you cannot fix afterwards.

Rotate on a schedule, not on incident. Credential rotation at case close or on a fixed cadence is cheap. Rotation performed in a panic during an active investigation is how collection stops mid-hunt.

The plumbing is short. SparkProxy routes through gateway.sparkproxy.io on three ports: 11000 for HTTP and HTTPS, 11002 for sticky sessions, 13000 for SOCKS5. Rotating exits come from 11000; use 11002 when a task needs one IP across several requests.

# Confirm what a target actually sees before you point this at anything
curl -x http://USER:PASS@gateway.sparkproxy.io:11000 \
     -s https://ipinfo.io/json

# Sticky session for a paginated source that must stay on one exit
curl -x http://USER-session-hunt2291:PASS@gateway.sparkproxy.io:11002 \
     -s https://ipinfo.io/json

Run the first command a dozen times before production and record the ASN and org values. That is your disclosure surface, and checking it costs two minutes. When sticky beats rotating is covered in what a sticky session proxy is.

For rendered pages and evidence capture without a browser fleet, the SparkProxy Scraping API at https://scrape.sparkproxy.io/api/v1 takes an X-API-Key header and handles rotation, rendering and country routing:

import requests

API = "https://scrape.sparkproxy.io/api/v1"
KEY = "YOUR_API_KEY"

def collect(url, country="US", render=False, shot=False):
    r = requests.get(
        API,
        headers={"X-API-Key": KEY},
        params={
            "url": url,
            "country_code": country,
            "render_js": "true" if render else "false",
            "screenshot": "true" if shot else "false",
            "format": "md",
        },
        timeout=90,
    )
    r.raise_for_status()
    return r.json()

# Enrichment: cheap, no browser
page = collect("https://indicator-under-review.test/robots.txt")

# Suspected lure: render it and keep an image for the report
lure = collect("https://suspect-lure.test/login", country="DE", render=True, shot=True)

Credit costs matter here, because SOC volume is lumpy. The Scraping API gives 1,000 free credits with no card, then Starter at $49 for 250,000 credits a month with 50 concurrent requests, Growth at $99 for 1,000,000, Pro at $249 for 3,000,000, and Scale at $599 for 8,000,000. A plain fetch costs 1 credit, a JavaScript render 5, and a screenshot or PDF 10, so capture images only on pages that may reach a report. Requesting format=md also returns clean Markdown instead of raw HTML, which makes diffing a watched page across two runs much easier. The wider trade-off between an API and self-managed exits is laid out in web scraping API vs self-managed proxies.

Send these to the vendor in writing. How plainly they answer is itself a signal.

1. What ASN and organization name do our requests exit under? Ask for live samples in the countries you need. Anything registered under a name containing "proxy" or "anonymizer" is a filtering risk for tier 3 work.
2. What connection metadata do you log, for how long, and who can access it? Get retention in days, in writing.
3. Where is the company incorporated, and where do those logs physically sit? This governs what a third party can compel.
4. How are residential IPs sourced and consented? A vendor that cannot answer clearly becomes a compliance finding at your next audit.
5. Can we isolate credentials per workstream, and revoke one without touching the others?
6. What is the acceptable-use policy on security research? Ask explicitly about fetching phishing pages and malware samples, and get the boundary in writing rather than discovering it during an incident.
7. What is the replacement policy and response time for a blocked or dead exit?
8. Is there a trial or refund window long enough to test against our own target list?

A broader evaluation checklist, including the tests worth running during a trial, is in what to evaluate when selecting a proxy service.

Two US cases set the useful legal boundaries. hiQ Labs v. LinkedIn (9th Cir., 2022) confirmed that scraping publicly accessible data does not violate the Computer Fraud and Abuse Act, and Van Buren v. United States (2021) narrowed CFAA liability to circumventing genuine access controls. Viewing an open phishing page sits on the safe side of that line. Logging into a captured panel with credentials you found does not. Personal data pulled from credential dumps also carries GDPR obligations wherever your SOC sits, so document the lawful basis before collection. None of this is legal advice, and cross-border work deserves counsel.

Fail closed, always. The worst outcome is a collector that falls back to direct egress when the proxy is unreachable, publishing your corporate IP to attacker infrastructure. Set the proxy in the HTTP client rather than OS-wide, assert the exit IP at the start of every batch, and abort the run if that assertion fails. Better still, run collectors in a container whose default route is denied, so a misconfiguration cannot reach the internet at all.

Watch DNS separately. Traffic can route through the proxy while name resolution goes out over your own resolver, telling the adversary's authoritative DNS server exactly who is looking. Test for that before go-live using the proxy DNS leak testing guide.

Instrument silent failure. Soft blocks matter more than hard blocks here, because a decoy page returns HTTP 200. Alert on status distribution, response-size distribution and content fingerprint drift per source, not only on error rates. The patterns are catalogued in how to detect when your scraper is blocked.

Rate-limit yourself. Hammering small research sites and community feeds is how a SOC loses access to sources it depends on. The ethical scraping and rate limiting guide covers sensible pacing.

Record the collection event. For anything that may reach a product, store the URL, UTC timestamp, exit IP and country, HTTP status, response headers, and a SHA-256 hash of the raw body taken before parsing. Normalize first and you have hashed your pipeline's output, not what the source served.

Four things a vendor is unlikely to raise.

A shared commercial pool is the wrong place for malware detonation. You would put attacker-facing traffic on exits other customers use, and most acceptable-use policies prohibit it. Detonation egress belongs on infrastructure you own and can burn.

Datacenter exits will fail on consumer social platforms. No amount of rotation defeats an ASN check. If tracking lures across social surfaces is core to your program, budget for residential from day one instead of discovering it in week three. Platform-specific patterns are in the notes on social media monitoring with proxies.

Unlimited bandwidth is not unlimited speed. Flat plans carry a speed ceiling and a fair usage policy. For bursty SOC workloads that ceiling rarely binds; for continuous multi-terabyte crawling it will.

Pool size is a vanity metric. A million addresses packed into a handful of subnets serves you worse than a smaller pool spread widely. Ask about subnet diversity in the countries you actually monitor.