How to Scrape Craigslist Listings

Start here, because Craigslist is not eBay or Amazon. It has no public read API, its Terms of Use forbid automated access outright, and it has taken scrapers to court and won. In craigslist v. 3Taps (N.D. Cal., 2013), Craigslist sent a cease-and-desist, IP-blocked 3Taps, and 3Taps kept accessing the site through rotating addresses. The court let a Computer Fraud and Abuse Act claim proceed on the theory that access continued after authorization was expressly revoked. The case settled in 2015 with 3Taps shutting down and paying a reported $1 million. A parallel suit against PadMapper settled the same way.

That history sets the practical line. The risk on Craigslist is not scraping public HTML in the abstract, it is evading an explicit, individualized block after the site has told you to stop. Keep four habits and you stay on the defensible side:

• Do not push through a hard block. If Craigslist returns a 403 "IP has been automatically blocked" page or sends you a notice, backing off is both the technical and the legal answer. Punching through with fresh IPs is exactly the pattern that turned into a CFAA claim.
• Read robots.txt first. Fetch https://sfbay.craigslist.org/robots.txt for the city you target and honor the disallowed paths before you widen a crawl.
• Take facts, not people. Collect listing attributes (price, location, category, posted date). Do not scrape the reply or contact info behind the "reply" button, which is personal data and gated behind a separate step for a reason.
• Get counsel for commercial use. If you plan to republish or resell Craigslist data, talk to a lawyer before you build. The site's posture on that is settled and unfriendly.

None of this makes small-scale, throttled, facts-only collection reckless. It does mean Craigslist deserves more caution than a marketplace that ships an official API, and the sections below assume you are collecting responsibly.

There is no single craigslist.com/search. Craigslist runs on the order of 700 separate sites, one per metro area, each on its own subdomain: sfbay.craigslist.org, newyork.craigslist.org, losangeles.craigslist.org, london.craigslist.org, and so on. The subdomain, not your IP location, decides which city's inventory you see. Fetch the San Francisco subdomain from a German server and you still get San Francisco listings.

That changes the shape of the job. To cover a category nationwide you enumerate subdomains and loop, rather than paging one giant result set. The canonical list lives at https://www.craigslist.org/about/sites, where every city is an anchor to its subdomain:

import requests
from parsel import Selector

HEADERS = {
    "User-Agent": (
        "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
        "AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0 Safari/537.36"
    ),
    "Accept-Language": "en-US,en;q=0.9",
}

sites_html = requests.get(
    "https://www.craigslist.org/about/sites", headers=HEADERS, timeout=20
).text
sel = Selector(text=sites_html)

subdomains = []
for href in sel.css("a::attr(href)").getall():
    href = href.rstrip("/")
    if href.endswith(".craigslist.org") and "//" in href:
        sub = href.split("//", 1)[1].split(".")[0]
        if sub not in ("www", "geo", "reference"):
            subdomains.append(sub)

subdomains = sorted(set(subdomains))
print(len(subdomains), "Craigslist sites")   # ~700 worldwide

Craigslist also publishes a machine-readable reference of areas and categories at reference.craigslist.org, which its posting API consumes. It is handy when you want to map a subdomain to its region or resolve category names to codes without scraping the sites page.

Because content is keyed to the subdomain, you rotate IPs for one reason only: to spread request volume so no single address trips the rate limiter. If you are new to spreading load across a pool, what is proxy rotation and how does it work explains the mechanics that keep a 700-city sweep alive.

Here is the single most useful thing to know about scraping Craigslist search results, and the thing most tutorials skip. Append &format=rss to any Craigslist search URL and the site returns the results as an RSS/RDF feed instead of a web page:

https://sfbay.craigslist.org/search/sss?query=mountain+bike&format=rss

That one parameter sidesteps the biggest headache on the modern site. Since Craigslist rewrote its search around 2021, the search results page is rendered client-side in JavaScript, so a plain HTTP fetch of the HTML returns an app shell with no listings in it. The RSS feed is generated server-side and arrives as clean XML you can parse in three lines. Each item carries the posting title, its detail-page URL, a short description snippet, and a Dublin Core date. The feedparser library reads it directly:

import requests, feedparser

feed_url = "https://sfbay.craigslist.org/search/sss?query=mountain+bike&format=rss"
xml = requests.get(feed_url, headers=HEADERS, timeout=20).content
feed = feedparser.parse(xml)

for entry in feed.entries:
    print(entry.title, "|", entry.link, "|", entry.get("published"))

Turn the entries into records you can store, and derive the posting ID straight from the URL so you can de-duplicate later:

def parse_feed(feed):
    rows = []
    for e in feed.entries:
        rows.append({
            "title": e.title,
            "url": e.link,                 # the posting detail URL
            "posted": e.get("published"),  # dc:date, ISO 8601
            "summary": e.get("summary"),   # short HTML snippet
            "post_id": e.link.rsplit("/", 1)[-1].replace(".html", ""),
        })
    return rows

The catch that shapes your whole strategy: the Craigslist RSS feed returns only the newest slice of a search, historically about the first 25 items, not the full result set. So RSS is a discovery firehose, not an archive. You use it to catch new postings as they appear, which means you poll a tight query often rather than trying to page deep. Narrow each feed by category, area, and price band so the newest 25 actually cover what you care about, and run it on a schedule to build history over time.

Craigslist search URLs are built from a three-letter category code in the path, then query parameters after the ?. The codes are terse but stable. These are the common top-level section codes, each of which has finer subcategories under it:

The query parameters do the filtering, and they are the same on the HTML page and the RSS feed:

A helper keeps the URL building tidy so you can point the same code at any city and category:

from urllib.parse import urlencode

def search_url(city, category="sss", fmt="rss", **params):
    base = f"https://{city}.craigslist.org/search/{category}"
    query = {"format": fmt, **params} if fmt else dict(params)
    return f"{base}?{urlencode(query)}"

# Bikes under $500 within 25 miles of 94103, images only, newest first
url = search_url(
    "sfbay", "sss",
    query="mountain bike",
    max_price=500, postal=94103, search_distance=25,
    hasPic=1, sort="date",
)

Three libraries cover the whole flow: requests to fetch, feedparser to read the RSS search results, and parsel to pull fields out of the posting HTML. parsel is the same selector engine Scrapy uses, so its .css() and ::text syntax reads cleanly.

pip install requests parsel feedparser

Always send a real browser User-Agent and an Accept-Language header. Craigslist treats a bare Python client as a bot signal, and a missing language header is one of the first things a rate limiter notices. The HEADERS dictionary from the subdomains section above is what every request in this guide uses.

There are two kinds of page you fetch, and they behave differently. Search results come from the RSS feed, which is server-rendered XML. Individual posting pages are server-rendered HTML, so parsel selectors work on them directly. You never need a headless browser for either path, which is what keeps this cheap and fast.

The cleanest way to scrape Craigslist search results is to treat the RSS feed as your search endpoint. Given the search_url helper, one call per city returns the newest listings for a query as structured records:

def search_city(city, category="sss", **params):
    url = search_url(city, category, fmt="rss", **params)
    xml = requests.get(url, headers=HEADERS, timeout=20).content
    return parse_feed(feedparser.parse(xml))

rows = search_city("sfbay", "sss", query="mountain bike", max_price=500)
print(len(rows), "listings")

To sweep many metros, loop over the subdomains you enumerated and stay polite between requests. One city every few seconds keeps you well under Craigslist's threshold:

import time

def crawl_cities(cities, category="sss", delay=3, **params):
    all_rows = []
    for city in cities:
        rows = search_city(city, category, **params)
        for r in rows:
            r["city"] = city
        all_rows.extend(rows)
        time.sleep(delay)          # one city every few seconds, stay polite
    return all_rows

bikes = crawl_cities(subdomains[:20], query="mountain bike", max_price=500)

A warning about older tutorials. Nearly every "Craigslist scraper" written before the 2021 redesign parses the HTML page with selectors like .result-row, .result-title, and .result-price. Those classes no longer exist. The search HTML is now a JavaScript app that hydrates results client-side, so a requests.get on the page returns markup with zero listings in it, and a scraper built on the old selectors silently records empty pages as empty queries. If you must read the HTML view instead of RSS (for example to page past the newest 25 with the s offset), you either render the page with a headless browser or read the internal JSON endpoint the app calls, which you can find in the browser network tab. For most work the RSS feed is simpler and enough.

The RSS feed gives you the posting URL. The full detail (price, exact location, attributes, images, body text) lives on the posting page, and that page is server-rendered HTML you can parse. A posting URL looks like https://sfbay.craigslist.org/scz/bik/d/some-slug/7712345678.html, where the trailing number is the posting ID.

Craigslist plants a small anti-scrape trap in the body: a hidden span that reads "QR Code Link to This Post". Strip it so it does not pollute your text. Location is the other detail worth knowing, because even when a seller hides the street address, the latitude and longitude are sitting in the #map element's data attributes:

def fetch_posting(url):
    html = requests.get(url, headers=HEADERS, timeout=20).text
    p = Selector(text=html)

    # #postingbody carries a hidden "QR Code Link to This Post" span; drop it
    body = "".join(
        t for t in p.css("#postingbody::text").getall()
        if "QR Code Link to This Post" not in t
    ).strip()

    return {
        "title": p.css("#titletextonly::text").get(),
        "price": p.css(".price::text").get(),
        "posted": p.css(".postinginfos time::attr(datetime)").get(),
        "attributes": [
            t.strip() for t in p.css(".attrgroup span::text").getall() if t.strip()
        ],
        "latitude": p.css("#map::attr(data-latitude)").get(),
        "longitude": p.css("#map::attr(data-longitude)").get(),
        "images": p.css("figure a.thumb::attr(href)").getall(),
        "body": body,
        "url": url,
    }

A few field notes so the output is clean:

• Images resolve to images.craigslist.org URLs. The a.thumb href points at the full-size version; the thumbnail lives inside it.
• attributes is a flat list of the spec chips Craigslist shows (condition, make and model, size, and so on). Normalize the strings into your own keys before you compare across postings, because the labels vary by category.
• price can be missing entirely (free items, jobs, gigs), so treat None as valid rather than an error.

Craigslist rotates markup like every large site, so treat these selectors as a starting point. When a scrape starts returning empty fields, log the raw HTML and re-check the class names before you assume the page was blocked.

This is where Craigslist earns its reputation. Its rate limiting is IP-based and unforgiving, and it blocks entire datacenter address ranges at the network level before you send a single suspicious request. A cloud server's IP is often refused on the first hit, no matter how human your headers look. The failure modes are easy to recognize once you have seen them:

Detect the soft failures explicitly. A 200 response with an empty feed is not an empty market, it is a block wearing a disguise, and your pipeline will happily record zero results as if nobody listed a bike that day:

def is_blocked(status, text):
    markers = (
        "IP has been automatically blocked",
        "this ip has been blocked",
        "please verify you are a human",
    )
    low = text.lower()
    return status == 403 or any(m.lower() in low for m in markers)

resp = requests.get(url, headers=HEADERS, timeout=20)
if is_blocked(resp.status_code, resp.text):
    print("Blocked, rotate to a clean residential IP and slow down")

The durable fixes are pacing and IP quality, in that order. Pace requests like a human, one city every few seconds rather than a burst, and never retry a hard block by throwing new addresses at it. On IP quality, datacenter proxies are a poor fit here because Craigslist blocks their ranges wholesale, so residential exits are usually the only ones that stay up on the harder pages. For why address reputation decides everything on a site like this, see what is IP blacklisting and how to avoid it, and for the full anti-block playbook, how to avoid getting your proxy blocked.

Running your own residential pool, detecting blocks, and backing off correctly is real infrastructure to own. The SparkProxy Scraping API handles the exit rotation, the residential routing that Craigslist demands, and country pinning behind one request. You send a target URL and get the response back.

import requests, feedparser

resp = requests.get(
    "https://scrape.sparkproxy.io/api/v1",
    headers={"X-API-Key": "sk-xxxxxxxxxxxxxxxx"},   # key format: sk-...
    params={
        "url": "https://sfbay.craigslist.org/search/sss?query=mountain+bike&format=rss",
        "render_js": "false",     # RSS and posting pages are not JS, 1 credit base
        "premium_proxy": "true",  # residential exits: Craigslist blocks datacenter ranges
        "country_code": "us",     # a US exit for US city subdomains
    },
    timeout=60,
)
feed = feedparser.parse(resp.content)

Two settings matter for Craigslist specifically. Keep render_js off, because both the RSS feed and the posting pages are server-rendered, so a plain fetch is 1 credit instead of 5. Turn premium_proxy on, because this is one of the sites where residential IPs are not optional; a residential fetch without JS runs 10 credits, which is the price of not maintaining your own pool of clean addresses. You can also let the API return structured fields instead of raw HTML by passing extract_rules, which pushes the parsing server-side:

import json

resp = requests.get(
    "https://scrape.sparkproxy.io/api/v1",
    headers={"X-API-Key": "sk-xxxxxxxxxxxxxxxx"},
    params={
        "url": posting_url,
        "render_js": "false",
        "premium_proxy": "true",
        "extract_rules": json.dumps({
            "title": "#titletextonly",
            "price": ".price",
            "posted": {"selector": ".postinginfos time", "attr": "datetime"},
            "latitude": {"selector": "#map", "attr": "data-latitude"},
            "longitude": {"selector": "#map", "attr": "data-longitude"},
        }),
    },
    timeout=60,
)
data = resp.json()   # structured fields, no local parsing

Check the docs for the exact extract_rules schema. The build-versus-buy call comes down to whether maintaining a residential pool and a block detector is a good use of your time; web scraping API vs self-managed proxies lays out that trade-off in full.

Raw feed rows are not a dataset yet. Two cleanup passes make them usable. The important one is de-duplication by posting ID, because the same item is often cross-posted to several nearby metros, so a multi-city sweep sees it more than once:

import csv

# De-duplicate by Craigslist posting id (the same post lists in nearby areas)
unique = list({r["post_id"]: r for r in bikes}.values())

with open("craigslist_listings.csv", "w", newline="", encoding="utf-8") as f:
    writer = csv.DictWriter(
        f, fieldnames=["post_id", "city", "title", "url", "posted"]
    )
    writer.writeheader()
    for r in unique:
        writer.writerow({k: r.get(k) for k in writer.fieldnames})

From there the shape of the work depends on the goal. For price research, parse the price string into a number and take the median across a category and metro rather than trusting any single asking price. For arbitrage or sourcing, join the latitude and longitude from the posting pages onto the search rows so you can filter by distance. For monitoring, store the posted timestamp and the post_id and re-run the tight RSS queries on a schedule, keeping only IDs you have not seen. Because RSS only shows the newest listings, that scheduled poll is how you build history that the feed alone will never hand you in one shot.