Proxies for Web3 Data and NFT Marketplace Feeds

A normal scraping project has one adversary per target: the site's anti-bot layer. A Web3 dataset has four, and they behave nothing alike.

Chain state comes from an RPC node, which meters you by API key. Token metadata usually lives on IPFS behind a public HTTP gateway, which meters you by IP and often by concurrent connection count. Marketplace listings live behind a normal web application with a WAF in front of it, which fingerprints your client. Images and video live on a CDN that will happily serve you 10,000 files right up until it decides you are hotlinking.

The naive pipeline treats all four as "fetch a URL" and dies at different points for different reasons. A 429 from Alchemy and a 429 from ipfs.io mean opposite things about what you should change. Getting this taxonomy right before you write the fetch loop is the difference between a collector that scales linearly and one you rewrite twice.

There is a second structural difference. On-chain data is canonical and replayable, so a gap in your event index is repairable at any time. Off-chain data is not. A listing that existed at 14:02 and was cancelled at 14:07 leaves no trace you can go back and fetch. Floor price is a time series you either captured or lost. Spend your reliability budget accordingly: over-engineer the listing capture, relax about the log backfill.

Before choosing infrastructure, classify every endpoint you plan to hit by what the rate limiter is actually counting.

Two rows in that table are the ones people get wrong. Teams buy a residential pool to "fix" Alchemy throttling and see zero improvement, because compute units are debited from the key regardless of which IP presented it. Other teams hand-roll metadata collection from a single machine and get their office address blocked by a public gateway an hour into a 10,000-token pull.

The rule is short enough to remember: if you authenticate, you are metered by identity; if you do not, you are metered by address. Proxies change your address. They never change your identity.

Managed RPC providers price by request weight, not by connection. Alchemy meters compute units and enforces a compute-units-per-second ceiling against the key. Infura issues daily credits against a project ID. QuickNode bills API credits per method, with heavier calls like eth_getLogs and debug_traceTransaction costing multiples of a trivial eth_blockNumber. Exact allowances move with pricing changes, so read the current docs rather than trusting a number in any blog post, including this one.

What matters architecturally is that the meter is attached to the credential, and the credential is in the URL or the header:

import requests

RPC = "https://mainnet.infura.io/v3/YOUR_PROJECT_ID"

payload = {
    "jsonrpc": "2.0",
    "id": 1,
    "method": "eth_blockNumber",
    "params": [],
}

# Routing this through a different exit IP does NOT reset the quota.
# The project ID in the URL is the meter. Proxying it only adds latency.
r = requests.post(RPC, json=payload, timeout=15)
print(int(r.json()["result"], 16))

If you are hitting a keyed limit, every legitimate lever is on the request side, not the network side:

• Batch. JSON-RPC supports array batching, so one round trip can carry 50 eth_call items. You still pay per item on most plans, but you stop paying for 50 TLS handshakes.
• Pick cheaper methods. eth_getLogs over a tight block range beats pulling full blocks with transaction objects when all you need is events.
• Cache what cannot change. Finalized blocks are immutable, so a local store of decoded logs means you replay from disk instead of re-billing the provider every time you tweak a parser.
• Use multiple keys legitimately. Separate keys per environment or per service is ordinary engineering. Farming free accounts to dodge a paid tier is a terms-of-service violation, and providers detect it through payment and device signals regardless of your exit IP.
• Run your own node when read volume is large and steady. At sustained indexing volume, an archive node is frequently cheaper than the equivalent managed plan.

Mempool monitoring belongs here too. Pending transactions are not one global set: every node keeps its own view, and you observe that view through a provider subscription such as eth_subscribe with newPendingTransactions. The subscription is authenticated by your key over a persistent WebSocket, so proxies do nothing for it. Broader mempool coverage comes from adding peers or providers, not IP addresses.

Keyless public RPC endpoints are the mirror image. They have no credential to meter, so they meter the connection, usually with a low per-IP request rate and a short ban window on breach. They are genuinely useful for light, bursty, non-critical reads: confirming a contract exists, resolving a tokenURI, checking a balance during a backfill.

Here proxy distribution is the mechanism that makes them usable at all:

import itertools, random, requests

PUBLIC_RPCS = [
    "https://cloudflare-eth.com",
    "https://rpc.ankr.com/eth",
    "https://eth.llamarpc.com",
]

PROXIES = [
    "http://USER:PASS@gateway.sparkproxy.io:11000",
    "http://USER:PASS@gateway.sparkproxy.io:11001",
    "http://USER:PASS@gateway.sparkproxy.io:11002",
]
proxy_cycle = itertools.cycle(PROXIES)

def rpc_call(method, params, attempts=4):
    for _ in range(attempts):
        endpoint = random.choice(PUBLIC_RPCS)
        proxy = next(proxy_cycle)
        try:
            r = requests.post(
                endpoint,
                json={"jsonrpc": "2.0", "id": 1, "method": method, "params": params},
                proxies={"http": proxy, "https": proxy},
                timeout=20,
            )
            if r.status_code == 429:
                continue            # this exit IP is cooling off, take the next
            body = r.json()
            if "error" in body:
                continue            # endpoint-specific limitation, try another
            return body["result"]
        except requests.RequestException:
            continue
    raise RuntimeError(f"{method} failed on every endpoint and exit IP")

Two things make this work. You rotate the exit IP so no single address accumulates a rate-limit counter, and you rotate the endpoint so no single operator sees your whole workload. Treat public endpoints as best-effort: they change policy without notice, some silently serve stale state, and none of them promise archive access. Anything you must have on a schedule belongs on a paid provider or your own node. The retry discipline in our guide to retry and backoff strategies for web scraping transfers here without modification.

This is where NFT projects actually stall, and it is the most Web3-specific problem in the stack.

An ERC-721 contract returns a tokenURI per token. An ERC-1155 contract returns a single uri template containing {id}, which you substitute with the token ID as a 64-character lowercase hex string, zero-padded, without the 0x prefix. Either way the value is very often an ipfs:// URI pointing at a JSON document, and that document points at another CID for the image.

So a 10,000-item collection is not one fetch. It is 10,000 metadata fetches minimum, plus up to 10,000 media fetches if you want the art, and every one of them travels through an HTTP gateway that limits by IP.

Content addressing hands you an escape hatch with no equivalent in normal scraping. A CID names content, not a location. A bafybei... hash resolves to byte-identical data at ipfs.io, dweb.link, cloudflare-ipfs.com, or any other gateway that can find the block. Gateway diversity and IP diversity multiply. Six gateways behind twenty exit IPs give you 120 independent rate-limit buckets for the same corpus, and you can verify correctness for free by hashing the response and comparing it to the CID. No conventional scraping target lets you do that, because on a normal site the hostname is part of the data's identity.

The practical resolver looks like this:

import itertools, requests

GATEWAYS = [
    "https://ipfs.io/ipfs/",
    "https://dweb.link/ipfs/",
    "https://cloudflare-ipfs.com/ipfs/",
    "https://gateway.pinata.cloud/ipfs/",
]

PROXIES = [f"http://USER:PASS@gateway.sparkproxy.io:{p}" for p in range(11000, 11004)]
proxy_cycle = itertools.cycle(PROXIES)

def to_gateway_path(uri: str) -> str:
    """ipfs://bafy.../1.json  ->  bafy.../1.json"""
    if uri.startswith("ipfs://"):
        uri = uri[len("ipfs://"):]
    if uri.startswith("ipfs/"):
        uri = uri[len("ipfs/"):]
    return uri

def fetch_ipfs(uri: str, attempts: int = 8) -> bytes:
    path = to_gateway_path(uri)
    for i in range(attempts):
        gw = GATEWAYS[i % len(GATEWAYS)]
        proxy = next(proxy_cycle)
        try:
            r = requests.get(
                gw + path,
                proxies={"http": proxy, "https": proxy},
                timeout=45,
            )
            if r.status_code in (429, 502, 503, 504):
                continue          # throttled, or the block is cold on this gateway
            r.raise_for_status()
            return r.content
        except requests.RequestException:
            continue
    raise RuntimeError(f"could not resolve {uri} across {attempts} gateway and IP pairs")

Details that only surface at volume. A 504 usually means the gateway could not locate providers for a cold CID, which is not a rate-limit problem, and retrying the same gateway rarely helps: switch, because another may already hold the block in cache. Some gateways honor Retry-After on 429, and respecting it costs less than blind retries. Fetch metadata before media, since a JSON document is a few hundred bytes while an image can run several megabytes, and rarity work needs none of the images.

For any collection you track continuously, stop depending on gateways. Pull each CID once, verify it against the hash, and pin it yourself. After that your metadata reads are local, and the only IPFS traffic you generate is for tokens you have never seen.

Listings, bids, and floor prices are off-chain until someone fills them. They live in marketplace databases and reach you through one of two doors.

The front door is the official API. OpenSea's v2 API requires a key with per-key rate limits, and Reservoir and Magic Eden publish keyed APIs too. Use these first: they are stable, documented, and cheaper to maintain than a scraper. Proxies do not raise their limits, though geo-routing occasionally matters because some endpoints vary responses by region.

The side door is the web application, and that is ordinary scraping with ordinary defenses. Collection pages fetch their own JSON internally, sit behind Cloudflare or a comparable WAF, and check TLS and browser fingerprints before they check your IP. Two techniques carry most of the load: find the internal JSON endpoint the page calls rather than parsing rendered HTML, and present a coherent browser fingerprint when you call it. Our walkthrough on scraping hidden JSON API endpoints covers the first, and bypassing Cloudflare when web scraping covers the second.

A few marketplace-specific traps:

• Cursors expire. Pagination is cursor-based and a long page-through can die halfway. Checkpoint the cursor and the block height together so a resumed run stays comparable to the original.
• Listings are soft state. They expire, get cancelled, or get filled between your page 1 and your page 40, so snapshot pagination is never internally consistent for a fast-moving collection. Record a capture timestamp per row, not per run.
• Multi-chain collections share a name and nothing else. The same brand can exist on Ethereum, Polygon, and Solana with different supply and different floors. Key records on chain plus contract address, never on collection name.
• Currency is not uniform. ETH, WETH, pooled ETH, SOL, and stablecoin listings all appear as "the floor" somewhere. Store the raw amount and the currency, and convert at query time with a rate you also store.

Two derived metrics justify most NFT data pipelines: floor price and trait rarity. Both are harder than they look, for reasons that are about definitions rather than infrastructure.

Floor price is not one number

Ask three marketplaces for a collection's floor and you get three answers, all correct under their own definition.

Write your definition down before you write the query. A useful default: lowest publicly buyable single-item listing, denominated in the chain's native token, fees excluded but recorded, per chain and contract. Then apply it identically to every source, and store each source's own reported floor alongside yours so you can explain the gap when someone asks.

Rarity requires the whole collection

Trait rarity is a frequency computation over every token in the collection. You cannot sample it. If 137 of 10,000 tokens have a gold background, that trait is 1.37% frequent, and you only know the denominator by having all 10,000 metadata documents. That is why the IPFS section above matters more than it first appears: rarity is the workload that turns a gateway rate limit into a project blocker.

from collections import defaultdict

def trait_frequencies(tokens: list[dict]) -> dict:
    """tokens: parsed metadata docs with an 'attributes' list."""
    counts = defaultdict(lambda: defaultdict(int))
    total = len(tokens)
    for meta in tokens:
        for attr in meta.get("attributes", []):
            counts[attr["trait_type"]][str(attr["value"])] += 1
    return {
        trait: {value: n / total for value, n in values.items()}
        for trait, values in counts.items()
    }

def rarity_score(meta: dict, freqs: dict) -> float:
    """Classic sum-of-inverse-frequency score. Document your formula."""
    score = 0.0
    for attr in meta.get("attributes", []):
        f = freqs.get(attr["trait_type"], {}).get(str(attr["value"]))
        if f:
            score += 1 / f
    return score

Publish which formula you used. Sum-of-inverse-frequency, statistical rarity, and information-content rarity rank the same collection differently, so a rarity column with no stated method is not reusable.

For anything that settled on chain, logs are the truth. A Transfer event tells you ownership changed; marketplace contracts emit their own sale events with price and currency. Indexing them is a bounded, replayable job, which makes it the calmest part of the pipeline.

Two constraints shape the code. Providers cap how much a single eth_getLogs can return, and they surface it as an error rather than truncating, with strings like query returned more than 10000 results. And recent blocks can be reorganized, so a log you index at the chain tip may cease to exist.

def fetch_logs(rpc_call, address, topic0, from_block, to_block, span=2000):
    """Walk a block range in chunks, halving the span when the provider objects."""
    logs, start = [], from_block
    while start <= to_block:
        end = min(start + span - 1, to_block)
        try:
            chunk = rpc_call("eth_getLogs", [{
                "address": address,
                "topics": [topic0],
                "fromBlock": hex(start),
                "toBlock": hex(end),
            }])
        except RuntimeError as exc:
            if span > 1 and "more than" in str(exc):
                span = max(1, span // 2)     # dense range, shrink and retry
                continue
            raise
        logs.extend(chunk)
        start = end + 1
        if len(chunk) < 500 and span < 10000:
            span = min(10000, span * 2)      # sparse range, speed back up
    return logs

For reorg safety, index at the tip for latency but treat recent data as provisional. Ethereum finalizes after two epochs, which is 64 slots at 12 seconds each, roughly 12.8 minutes under normal conditions. A workable pattern is to write tip data into a "pending" table, promote rows to the canonical table once their block is finalized, and re-check block hashes on promotion so a reorged block gets dropped instead of double-counted. On chains with different finality models, adjust the confirmation depth rather than the pattern.

Everything above that is IP-metered can run on raw proxies, but the marketplace front-end half also needs JavaScript rendering, fingerprint handling, and retries. The SparkProxy Scraping API folds those into request parameters. The base URL is https://scrape.sparkproxy.io/api/v1 and authentication is the X-API-Key header.

A collection page that renders its listings client-side:

curl "https://scrape.sparkproxy.io/api/v1?url=https://example-marketplace.io/collection/some-collection&render_js=true" \
  -H "X-API-Key: YOUR_API_KEY"

For a marketplace that gates behind a WAF, request a residential exit and hold the response until the price node exists:

import requests

resp = requests.get(
    "https://scrape.sparkproxy.io/api/v1",
    headers={"X-API-Key": "YOUR_API_KEY"},
    params={
        "url": "https://example-marketplace.io/collection/some-collection",
        "render_js": "true",
        "premium_proxy": "true",     # residential pool for a defended front-end
        "wait_for": "[data-testid='floor-price']",
        "country_code": "us",
        "json_response": "true",
    },
    timeout=120,
)
resp.raise_for_status()
payload = resp.json()
print(payload["status_code"], payload["credits_used"])
html = payload["body"]

IPFS metadata is static JSON, so skip the browser entirely and keep the request cheap:

import json

def fetch_metadata(cid_path: str) -> dict:
    r = requests.get(
        "https://scrape.sparkproxy.io/api/v1",
        headers={"X-API-Key": "YOUR_API_KEY"},
        params={
            "url": f"https://ipfs.io/ipfs/{cid_path}",
            "render_js": "false",       # plain JSON, no rendering needed
            "json_response": "true",
        },
        timeout=60,
    )
    r.raise_for_status()
    body = r.json()
    if body["status_code"] != 200:
        raise RuntimeError(f"gateway returned {body['status_code']} for {cid_path}")
    return json.loads(body["body"])

The exit IP varies per request, so a 10,000-token metadata pull spreads across the pool without you managing rotation. Stamp every row at fetch time so staleness is measurable later:

import time
from concurrent.futures import ThreadPoolExecutor

def collect(token_ids, uri_for):
    def one(tid):
        meta = fetch_metadata(uri_for(tid))
        return {"token_id": tid, "fetched_at": time.time(), "meta": meta}
    with ThreadPoolExecutor(max_workers=8) as pool:
        return list(pool.map(one, token_ids))

Filters that only exist behind a click, such as a trait filter or a "buy now only" toggle, are handled with js_scenario, and oversized results come back from /api/v1/files/{job_id}. Both are in the Scraping API docs. If you would rather run your own pool for RPC and gateway traffic and reserve the API for defended front-ends, SparkProxy's datacenter and residential proxies drop into the same patterns, and our post on rotating proxy APIs covers the tradeoff.

Collecting the bytes is the easy part. These are the failures that make a Web3 dataset quietly wrong.

Metadata is mutable more often than people assume. A tokenURI pointing at an HTTPS endpoint can return different JSON tomorrow. Even an ipfs:// URI is only immutable if it names a CID directly: resolved through IPNS or a DNSLink, the content behind the name can change. Store the resolved CID with every metadata snapshot so you can prove what you saw.

Wash trading inflates volume. Self-dealing between wallets one person controls was a well-documented distortion in NFT volume figures through the 2021 and 2022 cycles, and reward programs made it worse. Any volume metric you publish should either apply a filter, such as excluding round-trip trades between the same address pair inside a short window, or state plainly that it does not.

Timestamps come from different clocks. Chain events carry block timestamps, marketplace listings carry server time. Mixing them in one series introduces skew that looks like signal. Keep both, and label which each column uses.

Decimals and address casing break joins quietly. Prices arrive as base units, 18 decimals for ether and 6 for USDC, so use integer arithmetic or Decimal and convert only at presentation time. Addresses arrive in EIP-55 mixed-case checksum form or lowercase depending on the source, so normalize to lowercase on write.

Delisted is not sold. A listing leaving the book means cancelled, expired, or filled, and only the on-chain sale event distinguishes them. Infer nothing from absence.

On-chain data is public by design. Blocks, logs, and balances are published to be read, and reading them is exactly what nodes exist for. That gives Web3 collection a cleaner baseline than most scraping work. The boundaries sit around it rather than through it.

• Prefer the official API. If a marketplace publishes a keyed API covering the data you need, use it and pay for the tier that fits. Scraping is the tool for data that has no API, is geo-restricted, or only exists in a rendered page.
• Do not use IP rotation to defeat an account-level quota. It does not work against a keyed limit, and where a provider binds free tiers to IPs, evading that is a terms violation. Buy the tier or run the node.
• Rate limit yourself on public infrastructure. Public gateways and RPC endpoints are donated capacity. Pulling 10,000 CIDs through ipfs.io at full concurrency degrades a shared resource. Pace it, cache it, pin what you reuse.
• Pseudonymous is not anonymous. Wallet addresses are public records, but deliberately linking them to real identities turns ledger data into personal data, with the legal obligations that implies.
• Stay off authenticated surfaces. Public collection pages are fair game under most terms. Logged-in areas, private order flows, and anything requiring someone's session are not.
• Analytics, not interference. Monitoring the mempool to measure inclusion latency or index pending activity is research. Building infrastructure to extract value from other users' pending transactions is a different activity, and this guide is not about it.

If you are collecting price and market data across the wider crypto stack rather than NFTs specifically, our companion piece on proxies for cryptocurrency data covers exchange feeds, geo-restricted order books, and DeFi dashboards.