How to Scrape Real Estate Listings with Proxies

Before any code, set the boundary. Scraping publicly displayed listing data that any anonymous visitor can see sits on defensible ground, and US courts have repeatedly declined to treat access to public pages as unauthorized access. That's not a license to do whatever you want. A few practical rules keep you on the right side:

• Public pages only. Don't scrape anything behind a login, a paywall, or an agent-only MLS portal. Member agreements bind those, and bypassing authentication is a different legal category.
• Respect robots.txt and ToS as signals. Most portals prohibit automated access in their terms. That rarely creates criminal exposure for public data, but it does mean the site will fight you technically, and it shapes how you should behave: slow, polite, low-footprint.
• Don't rebuild their product. Using collected data internally for valuation models, market research, or lead scoring is a different risk profile from republishing scraped listings as a competing portal, which invites licensing claims on MLS-sourced data.
• Rate-limit yourself. Hammering a server is both a detection signal and a bad-neighbor move. A polite crawl that spreads load is more sustainable than a fast one that gets you banned in an hour.

For the full legal framework (MLS licensing, RESO agreements, Fair Housing implications when data feeds automated decisions), read the compliance section of the real estate data aggregation guide. The rest of this article assumes you're collecting public listing data for internal analysis.

A listing detail page carries far more than a price. Decide your schema up front, because retrofitting a field after you've scraped 200,000 pages means re-scraping. Here's the field reference most property data pipelines converge on, with the data type and the gotcha that bites people.

Two of these are worth more than the rest for analysis. Days on market plus price history together tell you demand and seller motivation, which raw price alone never will. A 45-day listing with two price cuts is a different investment than a 4-day listing at asking, even at the same number.

The reason to route through a managed Scraping API rather than raw proxies is that real estate portals combine three problems at once: aggressive IP rate limits, JavaScript-rendered prices, and behavioral fingerprinting. The API handles proxy rotation, headless Chromium rendering, and anti-bot measures behind a single call, so you send a URL and get back rendered HTML or structured JSON. If you want the trade-off analysis, see web scraping API vs self-managed proxies.

The base endpoint is https://scrape.sparkproxy.io/api/v1 and it authenticates with an X-API-Key header. The simplest possible call, rendering a listing page through a US IP:

curl -G "https://scrape.sparkproxy.io/api/v1" \
  -H "X-API-Key: YOUR_API_KEY" \
  --data-urlencode "url=https://www.example-portal.com/homedetails/123-oak-st" \
  --data-urlencode "render_js=true" \
  --data-urlencode "country_code=US"

The same thing in Python, which is where the rest of this guide lives:

import requests

API = "https://scrape.sparkproxy.io/api/v1"
API_KEY = "YOUR_API_KEY"          # from your SparkProxy dashboard

resp = requests.get(
    API,
    headers={"X-API-Key": API_KEY},
    params={
        "url": "https://www.example-portal.com/homedetails/123-oak-st",
        "render_js": "true",       # portals render price/beds via JS
        "country_code": "US",      # match the portal's country
        "wait_for": "[data-testid='price']",  # don't capture until price exists
    },
    timeout=90,
)
resp.raise_for_status()
html = resp.text

The one parameter people forget is wait_for. Modern portals build the page shell first and hydrate the price a beat later. Without wait_for, you capture the shell and your price field comes back empty on a random fraction of pages. Give it the selector for the element you most care about and the API waits for it before capturing.

You can pull the full HTML and parse it yourself, but for scalar fields it's cleaner to let the API return JSON directly. Pass extract_rules as a JSON object mapping each output field to a CSS selector, and set format=json. You get back a dict keyed by your field names.

import requests, json

API = "https://scrape.sparkproxy.io/api/v1"
API_KEY = "YOUR_API_KEY"

# Map your schema field -> CSS selector on the target page.
# Selectors are portal-specific; inspect the page to confirm them.
extract_rules = {
    "price":          "[data-testid='price']",
    "address":        "h1[data-testid='address']",
    "beds":           "[data-testid='bed-bath-item']:nth-of-type(1)",
    "baths":          "[data-testid='bed-bath-item']:nth-of-type(2)",
    "sqft":           "[data-testid='bed-bath-item']:nth-of-type(3)",
    "days_on_market": "[data-testid='dom-value']",
    "listing_agent":  "[data-testid='listing-attribution'] .agent-name",
    "status":         "[data-testid='status-pill']",
}

resp = requests.get(
    API,
    headers={"X-API-Key": API_KEY},
    params={
        "url": "https://www.example-portal.com/homedetails/123-oak-st",
        "render_js": "true",
        "country_code": "US",
        "format": "json",
        "extract_rules": json.dumps(extract_rules),
        "wait_for": "[data-testid='price']",
    },
    timeout=90,
)
listing = resp.json()
print(listing)
# {"price": "$459,000", "address": "123 Oak St, Austin, TX 78704",
#  "beds": "3 bd", "baths": "2.5 ba", "sqft": "1,840 sqft", ...}

The values come back as the raw on-page text, so add a cleaning layer. Never store "$459,000" when you'll want to do arithmetic on it later.

import re

def to_int(text):
    """'$459,000' or '1,840 sqft' -> 459000 / 1840. None if no digits."""
    if not text:
        return None
    digits = re.sub(r"[^\d]", "", text)
    return int(digits) if digits else None

def to_float(text):
    """'2.5 ba' -> 2.5 . Handles 'Studio' as 0."""
    if not text:
        return None
    if "studio" in text.lower():
        return 0.0
    m = re.search(r"\d+(\.\d+)?", text)
    return float(m.group()) if m else None

clean = {
    "price":          to_int(listing.get("price")),
    "beds":           to_float(listing.get("beds")),
    "baths":          to_float(listing.get("baths")),
    "sqft":           to_int(listing.get("sqft")),
    "days_on_market": to_int(listing.get("days_on_market")),
    "address":        (listing.get("address") or "").strip(),
    "agent":          (listing.get("listing_agent") or "").strip(),
    "status":         (listing.get("status") or "").strip().lower(),
}

Keep the raw response too, at least during development. When a selector silently starts returning None after a portal redesign, having the raw HTML archived is the difference between a five-minute fix and a re-scrape.

Extracting one page is the easy 5%. The job is walking a search result set that runs to hundreds of pages and collecting every listing URL, then visiting each. Portals paginate three ways, and you handle each differently:

• Numbered pages (?page=2, /homes/2_p/): the friendliest case. Increment until you get an empty result set.
• Infinite scroll: no page param. You need the API to scroll the page so more cards load, which you drive with js_scenario.
• A capped result window: many portals hard-cap search results at roughly 20 pages regardless of how many matches exist. The fix is not more pages, it's narrower queries. Split a metro into price bands, zip codes, or bedroom counts so each query returns under the cap.

That last point is the one that trips up most people. If a metro has 8,000 listings but the portal only paginates 800 of them, you don't scrape harder, you slice the search. Query by zip, then by price band inside dense zips, and union the deduplicated results.

A numbered paginator that harvests listing URLs from each search page:

from bs4 import BeautifulSoup
from urllib.parse import urljoin
import time, random

def collect_listing_urls(search_url, session, max_pages=20):
    """Walk numbered search pages, return a deduped list of detail URLs."""
    found = []
    for page in range(1, max_pages + 1):
        page_url = f"{search_url}?page={page}"
        html = fetch_html(page_url, session)     # defined in the full script below
        if not html:
            break
        soup = BeautifulSoup(html, "html.parser")
        cards = soup.select("a[data-testid='property-card-link']")
        links = [urljoin(page_url, a["href"]) for a in cards if a.get("href")]
        if not links:
            break                                 # ran past the last page, stop
        found.extend(links)
        # polite, jittered delay between pages
        time.sleep(random.uniform(1.5, 4.0))
    # dedupe while preserving order
    return list(dict.fromkeys(found))

For infinite-scroll portals, drive the scroll with js_scenario so lazy-loaded cards render before capture:

scenario = {
    "instructions": [
        {"scroll_y": 4000}, {"wait": 1200},
        {"scroll_y": 8000}, {"wait": 1200},
        {"scroll_y": 12000}, {"wait": 1200},
    ]
}
params = {
    "url": search_url,
    "render_js": "true",
    "js_scenario": json.dumps(scenario),
    "country_code": "US",
}

Each scroll-and-wait pair triggers another batch of listing cards. Three or four cycles usually loads a full "page" worth of results on typical portals.

Two different geo problems live under one word. The first: some county assessor and regional MLS-adjacent portals only serve, or only serve correct pricing to, in-country IPs. The second: prices, currency, and availability on international portals depend on where the request appears to come from. Both are solved by the country_code parameter, an ISO 3166-1 alpha-2 code that pins the exit IP to that country.

Match the code to the portal's home market:

In code, geo-targeting is one parameter, so a multi-country crawl is just a loop over a config:

PORTALS = [
    {"name": "us_metro",  "search": "https://www.example-portal.com/austin-tx",  "country": "US"},
    {"name": "uk_london", "search": "https://www.example-uk.com/london",          "country": "GB"},
    {"name": "es_madrid", "search": "https://www.example-es.com/madrid",          "country": "ES", "premium": True},
]

for portal in PORTALS:
    urls = collect_listing_urls(portal["search"], session)
    for url in urls:
        row = scrape_listing(url, country=portal["country"],
                             premium=portal.get("premium", False))
        # ... persist row

If you're doing geo-targeted collection for market analysis rather than a single portal, the same pattern powers broader work like using proxies for market research and data collection.

Real estate portals run some of the strictest bot mitigation on the public web, and the highest-value pages (detail pages with price history and valuations) are guarded harder than search pages. The mistake is reaching for the most expensive proxy tier on every request. The right move is an escalation ladder: start cheap, escalate only the requests that get blocked.

The ladder, cheapest to strongest:

1. Rotating datacenter + JS render. Handles county portals, Redfin, Apartments.com, and most search pages. Set render_js=true.
2. Add stealth=true. Layers extra anti-detection (fingerprint hardening) for portals that sniff headless browsers.
3. Escalate to premium_proxy=true (residential exit IPs) for the sites and detail pages that hard-block datacenter ranges, like Zillow detail pages and Idealista.
4. Vary device (desktop, mobile, random) so repeated hits don't share one fingerprint.

The core discipline is rate, not just IP quality. Even with rotation, sending 200 requests a minute at one portal looks nothing like human browsing. Spread the load and add jitter. Our full playbook on this is how to avoid getting your proxy blocked; the general scraping foundations are in using datacenter proxies for web scraping.

Map symptoms to fixes so you're not guessing when a run degrades:

The one insight most guides skip: let the block response tell you when to escalate, instead of paying for residential up front. Send datacenter first, and only flip premium_proxy on for the exact URLs that return a 403. On a mixed target set, that typically keeps 70 to 90% of requests on the cheap tier and reserves the expensive tier for the pages that genuinely need it. The escalation logic in the next section does exactly this automatically.

Here's the whole thing wired together: a session, a request function with retry, backoff, and automatic datacenter-to-residential escalation, a detail-page extractor, and a runner that paginates, scrapes, cleans, and writes CSV.

import requests, json, time, csv, random, re
from bs4 import BeautifulSoup
from urllib.parse import urljoin

API = "https://scrape.sparkproxy.io/api/v1"
API_KEY = "YOUR_API_KEY"          # from your SparkProxy dashboard

session = requests.Session()
session.headers.update({"X-API-Key": API_KEY})

FIELDS = {
    "price":          "[data-testid='price']",
    "address":        "h1[data-testid='address']",
    "beds":           "[data-testid='bed-bath-item']:nth-of-type(1)",
    "baths":          "[data-testid='bed-bath-item']:nth-of-type(2)",
    "sqft":           "[data-testid='bed-bath-item']:nth-of-type(3)",
    "days_on_market": "[data-testid='dom-value']",
    "listing_agent":  "[data-testid='listing-attribution'] .agent-name",
    "status":         "[data-testid='status-pill']",
}

def to_int(t):
    d = re.sub(r"[^\d]", "", t or "")
    return int(d) if d else None

def to_float(t):
    if not t: return None
    if "studio" in t.lower(): return 0.0
    m = re.search(r"\d+(\.\d+)?", t)
    return float(m.group()) if m else None

def api_get(url, country="US", extract=None, premium=False):
    """Single call with retry, backoff, and datacenter->residential escalation."""
    params = {
        "url": url,
        "render_js": "true",
        "country_code": country,
        "stealth": "true",
        "wait_for": "[data-testid='price']",
    }
    if extract:
        params["format"] = "json"
        params["extract_rules"] = json.dumps(extract)
    if premium:
        params["premium_proxy"] = "true"

    for attempt in range(4):
        try:
            r = session.get(API, params=params, timeout=90)
        except requests.RequestException:
            time.sleep(2 ** attempt)
            continue
        if r.status_code == 200:
            return r.json() if extract else r.text
        if r.status_code == 403 and not params.get("premium_proxy"):
            params["premium_proxy"] = "true"     # escalate this URL to residential
        if r.status_code in (403, 429, 500, 502, 503):
            time.sleep((2 ** attempt) + random.uniform(0, 1.5))
            continue
        r.raise_for_status()
    return None

def fetch_html(url, sess, country="US"):
    return api_get(url, country=country)

def collect_listing_urls(search_url, country="US", max_pages=20):
    found = []
    for page in range(1, max_pages + 1):
        html = api_get(f"{search_url}?page={page}", country=country)
        if not html:
            break
        soup = BeautifulSoup(html, "html.parser")
        links = [urljoin(search_url, a["href"])
                 for a in soup.select("a[data-testid='property-card-link']")
                 if a.get("href")]
        if not links:
            break
        found.extend(links)
        time.sleep(random.uniform(1.5, 4.0))
    return list(dict.fromkeys(found))

def scrape_listing(url, country="US", premium=False):
    raw = api_get(url, country=country, extract=FIELDS, premium=premium)
    if not raw:
        return None
    return {
        "url":            url,
        "price":          to_int(raw.get("price")),
        "address":        (raw.get("address") or "").strip(),
        "beds":           to_float(raw.get("beds")),
        "baths":          to_float(raw.get("baths")),
        "sqft":           to_int(raw.get("sqft")),
        "days_on_market": to_int(raw.get("days_on_market")),
        "agent":          (raw.get("listing_agent") or "").strip(),
        "status":         (raw.get("status") or "").strip().lower(),
    }

def run(search_url, out="listings.csv", country="US"):
    urls = collect_listing_urls(search_url, country=country)
    print(f"Found {len(urls)} listings")
    cols = ["url", "price", "address", "beds", "baths",
            "sqft", "days_on_market", "agent", "status"]
    with open(out, "w", newline="", encoding="utf-8") as f:
        writer = csv.DictWriter(f, fieldnames=cols)
        writer.writeheader()
        for i, url in enumerate(urls, 1):
            row = scrape_listing(url, country=country)
            if row:
                writer.writerow(row)
            if i % 25 == 0:
                print(f"  {i}/{len(urls)} scraped")
            time.sleep(random.uniform(1.0, 3.0))   # polite pacing

if __name__ == "__main__":
    run("https://www.example-portal.com/austin-tx", out="austin.csv", country="US")

This survives the failure modes that stop naive scrapers: it waits for the price to render, backs off on 429s, escalates only blocked URLs to residential, dedupes listing links, and paces itself. Swap the selectors in FIELDS and the property-card-link selector for your target portal and it runs.

The Scraping API bills in credits, and the mode you pick per request is the biggest lever on cost. Rendering JavaScript and routing through residential IPs cost more, so match the mode to what the target actually needs rather than defaulting to the strongest option.

The math that matters at scale: a national run of 500,000 detail pages costs 2.5 million credits if you render JS on datacenter (5 each), but 12.5 million if you blindly use premium plus JS (25 each) on every page. The escalation ladder from the anti-bot section is what keeps you near the low end. Run datacenter first, escalate the blocked minority, and you pay the 25-credit rate only on the pages that truly won't yield any other way.