How to Scrape Instagram Data (Public Data Only)

Instagram is not an e-commerce catalog. Almost every field you can pull describes a real, identifiable person, so the ethics here carry more weight than on any product-scraping job, and they decide the design before the code does.

Three separate legal questions matter, and "it's public" only answers the first one:

• Unauthorized access. In the US, the Ninth Circuit's ruling in hiQ Labs v. LinkedIn (2022) held that scraping data that's publicly accessible, meaning no login and no authentication wall, generally does not violate the Computer Fraud and Abuse Act. That is about access, not a license to do anything with what you collect.
• Contract. Instagram's Terms of Use prohibit automated collection outright. Scraping public pages can still breach that contract even where it isn't a CFAA violation, and Meta has pursued scrapers in court. These are two different questions.
• Data protection. This is the big one for Instagram, and it gets its own section below. Public personal data is still personal data under GDPR and similar laws, and "I found it in public" is not a lawful basis on its own.

Guardrails that keep an Instagram project defensible:

• Collect public data only. No logged-in sessions, no login walls, no private accounts, ever.
• Pull the minimum you need. Aggregate counts and public post metadata, not dossiers on individuals.
• Rate-limit yourself and back off on errors so you never degrade the service for real users.
• Don't republish media or full captions beyond what fair use allows, and never touch face or biometric data.
• Honor deletion. If someone removes a post or account, drop it from your store.
• Get a lawyer involved before anything commercial. This is engineering guidance, not legal advice.

The legitimate reasons to want public Instagram data are real: brand monitoring, influencer vetting, competitor benchmarking, trend research. If that's your use case, the business-side patterns live in Using Proxies for Social Media Monitoring. The point of this section is that the use case has to survive scrutiny before the pipeline is worth building.

Before you scrape anything, check whether the official API covers your need, because it is the only route Meta actually blesses, and it needs no proxies.

Meta's official Instagram Platform exposes two API surfaces: the Instagram Graph API (Instagram API with Facebook Login) and the newer Instagram API with Instagram Login. Both require an approved Meta app and both work against Instagram Professional accounts (Business and Creator). Through them you can read:

• Your own or a managed account's media, comments, mentions, and insights.
• Public media for a hashtag, through the Hashtag Search API, which a Business or Creator account can query for up to 30 unique hashtags in a rolling 7-day window.

One important date: the old Instagram Basic Display API was deprecated on December 4, 2024, so tutorials that lean on it are stale. Use the current Instagram Platform endpoints instead.

The official path looks like this. First resolve a hashtag to its ID, then read recent public media for it:

# Sanctioned route: Instagram Graph API, from a Business/Creator account you manage
curl -G "https://graph.facebook.com/v21.0/ig_hashtag_search" \
  --data-urlencode "user_id=<IG_BUSINESS_USER_ID>" \
  --data-urlencode "q=proxies" \
  --data-urlencode "access_token=<ACCESS_TOKEN>"

Where the Graph API falls short is arbitrary profiles you don't own. There is no official endpoint that hands you a stranger's follower count on demand. That gap is exactly why people reach for HTML scraping, and it's also where the data-protection weight lands hardest. If your need fits inside your own accounts, managed accounts, or hashtag search, stop here and use the API. Everything below is for public data the API genuinely does not reach.

A logged-out visit to a public profile or post still exposes a stable set of fields, mostly through Open Graph meta tags and a JSON-LD block. The markup drifts, but the fields are consistent. Here's the reference set worth pulling as of mid-2026.

The single most useful trick, and the one most tutorials miss, is the og:description string. Instagram packs the three headline counts into it as plain text, for example: 1,234 Followers, 567 Following, 89 Posts - See Instagram photos and videos from NASA (@nasa). One regex gets you followers, following, and posts without touching the private JSON endpoints that need an app-id header and a logged-in session. Counts can arrive abbreviated (1.2M), so normalize K, M, and B downstream.

Instagram scraping breaks naive scrapers faster than almost any other target. Four defenses do the damage:

The login wall. Instagram increasingly gates public content behind a login or consent modal, and the gate frequently returns HTTP 200 with a nearly empty page. If your code trusts the status code, response.ok is True, you save the "page", and you've stored a login prompt instead of a profile. You have to inspect the body.

Fingerprinting. Instagram reads TLS fingerprints (JA3/JA4), header order, and JavaScript signals. A plain Python urllib3 handshake looks nothing like a browser, so you get walled regardless of your User-Agent. Getting real content back means a genuine browser or a fingerprint-matching client.

IP reputation. Datacenter IP ranges are flagged almost immediately on Instagram. A single static datacenter proxy dies within a handful of requests. Residential and mobile IPs blend in and survive far longer.

Soft rate limits. Push too fast from one IP and Instagram serves Please wait a few minutes before you try again, then a temporary block. The thresholds are unpublished and they move.

A managed scraping API absorbs all four of these. The parsing is still yours, because the fields live in the HTML, but the anti-bot arms race moves off your plate. If you want the deeper theory on ban avoidance, How to Avoid Getting Your Proxy Blocked covers it end to end.

Proxy choice is the difference between a scraper that runs and one that gets a login wall on request two.

Datacenter proxies are the wrong tool here. Instagram flags their IP ranges quickly, so they burn out fast even with rotation. They're fine for easy targets, not for this one.

Residential proxies route through real consumer ISP connections, so their IPs carry the reputation of ordinary home users. Instagram trusts them far more, and they're the practical default for public Instagram collection. If you're fuzzy on the type, What Is a Residential Proxy: Types and Use Cases breaks down how they differ from datacenter and mobile.

Mobile proxies route through cellular carrier IPs (4G/5G). Because carriers rotate a small pool of IPs across thousands of subscribers behind CGNAT, a mobile IP is the hardest for Instagram to ban without collateral damage to real users. They're the strongest option for the toughest jobs, and also the priciest, so most public-profile work lands on residential.

With the SparkProxy Scraping API you don't manage any of this pool directly. Setting premium_proxy=true routes the request through the residential IP pool, and device=mobile makes the request present as a mobile client, which is the profile Instagram trusts most. That collapses the entire proxy decision into two parameters.

The SparkProxy Scraping API takes a target URL and handles the proxy, rotation, browser rendering, and anti-bot layer for you. You send one request and get the rendered HTML back. For Instagram, four parameters carry the weight:

• render_js=true: Instagram paints the profile in JavaScript, so a raw fetch returns a shell.
• premium_proxy=true: routes through residential IPs that survive Instagram's defenses.
• device=mobile: presents as a mobile client, the profile Instagram trusts most.
• country_code: the ISO alpha-2 exit country, useful when a profile or hashtag is geo-fenced.

The base endpoint is https://scrape.sparkproxy.io/api/v1 and auth is a single header, X-API-Key. A minimal request in cURL against a public organization account:

curl -G "https://scrape.sparkproxy.io/api/v1" \
  -H "X-API-Key: YOUR_API_KEY" \
  --data-urlencode "url=https://www.instagram.com/nasa/" \
  --data-urlencode "render_js=true" \
  --data-urlencode "premium_proxy=true" \
  --data-urlencode "device=mobile" \
  --data-urlencode "country_code=US"

The full parameter list and response fields live in the Scraping API docs. If you're weighing this against building your own rotation, browser farm, and fingerprint stack, Web Scraping API vs Self-Managed Proxies lays out the build-versus-buy math honestly.

Start with one profile. Wrap the request so every call carries the Instagram-specific parameters, and give it a generous timeout since a rendered request drives a real browser.

import requests

API = "https://scrape.sparkproxy.io/api/v1"
API_KEY = "YOUR_API_KEY"

def fetch_profile(username: str, country: str = "US") -> str:
    resp = requests.get(
        API,
        headers={"X-API-Key": API_KEY},
        params={
            "url": f"https://www.instagram.com/{username}/",
            "render_js": "true",       # Instagram renders the profile in JS
            "premium_proxy": "true",   # residential IPs survive the walls
            "device": "mobile",        # Instagram trusts mobile clients most
            "country_code": country,
        },
        timeout=90,
    )
    resp.raise_for_status()
    return resp.text

Before you trust the HTML, check whether Instagram handed you a login wall. Because that gate returns 200, raise_for_status() won't catch it. Scan the body for the telltale markers:

def is_login_wall(html: str) -> bool:
    """Instagram's login/consent gate returns HTTP 200, so the status code lies."""
    markers = (
        "require_login",
        "accounts/login",
        "loginForm",
        "Please wait a few minutes before you try again",
    )
    low = html.lower()
    return any(m.lower() in low for m in markers)

Now parse the public fields. The reliable source is the Open Graph meta tags, with the three counts pulled out of og:description by one regex. Use selectolax (a C-backed parser) for speed at volume; install it with pip install selectolax.

import re
from selectolax.parser import HTMLParser

COUNTS_RE = re.compile(
    r"([\d.,KMB]+)\s+Followers,\s+([\d.,KMB]+)\s+Following,\s+([\d.,KMB]+)\s+Posts",
    re.IGNORECASE,
)

def parse_profile(html: str) -> dict:
    tree = HTMLParser(html)

    def meta(prop: str):
        node = tree.css_first(f'meta[property="{prop}"]')
        return node.attributes.get("content") if node else None

    desc = meta("og:description") or ""
    m = COUNTS_RE.search(desc)
    url = (meta("og:url") or "").rstrip("/")
    return {
        "handle": url.split("/")[-1] or None,
        "name": meta("og:title"),
        "followers": m.group(1) if m else None,
        "following": m.group(2) if m else None,
        "posts": m.group(3) if m else None,
        "avatar": meta("og:image"),
        "verified": '"is_verified":true' in html,
    }

The og:image avatar URL is a signed, expiring CDN link, so if you keep it, rehost it immediately rather than storing the raw URL. Counts arrive human-formatted (1.2M, 12.3K), so write a small normalizer that turns K/M/B into integers before you store them for trend analysis.

A public post lives at /p// and exposes its caption and, when Instagram hasn't hidden them, engagement counts through the same meta and JSON-LD pattern.

import json

def parse_post(html: str) -> dict:
    tree = HTMLParser(html)
    ld = tree.css_first('script[type="application/ld+json"]')
    data = {}
    if ld and ld.text():
        try:
            data = json.loads(ld.text())
        except json.JSONDecodeError:
            data = {}

    def meta(prop):
        node = tree.css_first(f'meta[property="{prop}"]')
        return node.attributes.get("content") if node else None

    stats = {s.get("interactionType", ""): s.get("userInteractionCount")
             for s in data.get("interactionStatistic", [])} if data else {}

    return {
        "caption": meta("og:title"),
        "author": (data.get("author") or {}).get("alternateName"),
        "likes": stats.get("https://schema.org/LikeAction"),
        "comments": stats.get("https://schema.org/CommentAction"),
        "image": meta("og:image"),
    }

Treat likes and comments as optional. Instagram has hidden public like counts on and off since 2019, so your parser must not assume they exist. Store None and move on rather than throwing.

For hashtags, resist the urge to scrape /explore/tags//. That surface is gated behind login for logged-out clients most of the time, and hammering it is a fast route to a block. This is the one case where the official route is clearly better: the Hashtag Search API from the previous section returns public media for a hashtag with Meta's blessing and no proxy risk. Use it for anything hashtag-driven, and reserve HTML scraping for public profiles and individual public posts.

Instagram tracks pressure per IP, per fingerprint, and per account. The soft limit shows up as the Please wait a few minutes interstitial before it hardens into a temporary IP ban. Four habits keep an Instagram data scraper healthy at volume:

• Rotate the exit IP per request. With the Scraping API this is automatic; premium_proxy=true hands you a fresh residential IP each call.
• Keep concurrency low. Three to eight workers is plenty. Instagram punishes bursts harder than most targets.
• Back off with jitter. On a soft block, retry with exponential backoff plus a random delay so a batch of failures doesn't retry in lockstep.
• Cache aggressively. A follower count does not change minute to minute. Don't re-scrape a profile you pulled an hour ago; read from your own store.

import time, random

def fetch_with_retry(username: str, attempts: int = 3) -> str | None:
    for i in range(attempts):
        html = fetch_profile(username)
        if not is_login_wall(html) and 'og:description' in html:
            return html
        time.sleep(2 ** i + random.random())   # exponential backoff + jitter
    return None

Persist as you go rather than holding a run in memory, so a crash at profile 8,000 doesn't cost you the first 7,999. Stamp each row with a scraped_at timestamp so you can build clean time series and prove exactly when a data point was collected, which matters for the compliance record too.

This is where an Instagram project lives or dies, so read it before you scale anything.

Under GDPR, a person's name, handle, photo, and posts are personal data, and Article 4 defines that broadly. Crucially, data being publicly available is not an exemption. If any of your data subjects are in the EU or UK, collecting and storing their public Instagram data is processing, and processing needs a legal footing:

• A lawful basis (Article 6). Usually legitimate interest, which requires a documented balancing test weighing your purpose against the individual's rights. Consent is rarely workable at scrape scale.
• No special-category data (Article 9). If a post reveals health, religion, political views, or sexual orientation, that's a protected category you must not infer or store without a specific legal condition. Steer well clear.
• Transparency (Articles 13 and 14). You generally have to inform people you hold their data. Article 14(5) offers a narrow "disproportionate effort" exemption, but regulators read it strictly.
• Absolutely no biometrics. Do not run face recognition on scraped images. France's CNIL fined Clearview AI 20 million euros in 2022 for scraping public photos into a facial-recognition database, and other EU regulators followed with similar penalties. That is the cautionary tale for anyone thinking "public photos are fair game."

In August 2023, twelve data-protection authorities led by the UK's ICO issued a joint statement making clear that publicly accessible personal data is still protected, and that platforms and scrapers both carry obligations. Regulators are watching this space specifically.

Practical compliance that keeps you defensible: minimize to aggregate counts and public metadata, avoid building searchable profiles of named individuals, never touch face or biometric data, run a Data Protection Impact Assessment before a large collection, honor erasure requests, and delete anything a user removes. When in doubt, collect less. The safest Instagram dataset is one that studies trends and cohorts, not people.