How to Scrape App Store and Google Play Data

To scrape App Store and Google Play data for app store optimization, you have to treat the two stores as opposite problems. Apple hands you clean, official JSON: metadata, charts, and reviews all come from documented feeds with a country in the URL. Google Play gives you almost none of that, so its data lives in rendered HTML, an embedded JSON-LD block, and an undocumented RPC endpoint. Most guides use one generic HTML-parsing recipe for both and end up brittle on both. This one splits the pipeline the way the platforms actually work, shows the exact endpoints for rankings, ratings, reviews, and metadata, and explains why every request needs a country-matched IP if the ASO numbers are going to mean anything.

Get the framing right before you write a scraper, because "the data is public" only answers part of the question.

In the United States, the Ninth Circuit's ruling in hiQ Labs v. LinkedIn (2022) held that scraping data that is publicly accessible, meaning no login and no authentication wall, generally does not violate the Computer Fraud and Abuse Act. That is a statement about access, not a license to do anything with the data. Apple's iTunes feeds and Google Play both carry their own terms of service, and Google's in particular prohibit automated collection. Public access and contractual permission are two separate questions.

App reviews add a privacy dimension the store metadata does not. A review carries a display name and, on Google Play, sometimes a real name. That is personal data under GDPR and CCPA. Guardrails that keep an ASO project defensible:

• Collect public listing data only: rankings, ratings, review text, version notes, category. Never anything behind a sign-in.
• Treat reviewer identities as personal data. If you only need sentiment and star trends, drop the names at ingest and aggregate.
• Rate-limit yourself and back off on errors so you never degrade the service you are reading.
• If the data feeds a commercial ASO product, run it past a lawyer. This is engineering guidance, not legal advice.

For the wider picture on staying within limits, see our guide on ethical scraping and rate limiting.

App store optimization runs on four signal groups: where an app ranks, how it is rated, what users say, and how the listing is built. Each store exposes them through a different door. Here is the mapping, which is the single most useful thing to internalize before writing any code.

Two asymmetries jump out. Apple publishes ratings and reviews as official JSON but hides the star histogram; Google Play hides nothing structurally but wraps everything in obfuscated markup. Plan your extractor around that, not around one shared HTML parser.

Every request below goes through one helper. The SparkProxy Scraping API takes a target URL, an X-API-Key header, and a country_code, and returns the fetched content in a JSON envelope. The base URL is https://scrape.sparkproxy.io/api/v1.

import requests

API = "https://scrape.sparkproxy.io/api/v1"
KEY = "YOUR_API_KEY"

def fetch(target, country="US", render=False):
    r = requests.get(API, headers={"X-API-Key": KEY}, params={
        "url": target,
        "render_js": str(render).lower(),
        "country_code": country,
        "format": "json",
    }, timeout=60)
    r.raise_for_status()
    return r.json()   # envelope: {"status_code":..., "body": "...", ...}

The Apple JSON feeds need no browser, so keep render_js=false for them (1 credit each). Google Play's details page needs a rendered fetch (render_js=true). The country_code is an ISO 3166-1 alpha-2 code like US, GB, DE, or JP, and it does two jobs at once, which the country section covers.

Apple runs two public JSON endpoints that most app store scraping tutorials ignore in favor of parsing HTML. The iTunes Lookup API returns a full metadata record for a numeric track ID, and the Search API resolves keywords to apps.

import json
from urllib.parse import quote

def app_metadata(track_id, country="US"):
    target = f"https://itunes.apple.com/lookup?id={track_id}&country={country}"
    env = fetch(target, country=country, render=False)
    r = json.loads(env["body"])["results"][0]
    return {
        "name": r["trackName"],
        "seller": r["sellerName"],
        "category": r["primaryGenreName"],
        "rating_all": r.get("averageUserRating"),
        "rating_count": r.get("userRatingCount"),
        "rating_current": r.get("averageUserRatingForCurrentVersion"),
        "version": r["version"],
        "updated": r["currentVersionReleaseDate"],
        "price": r["formattedPrice"],
        "min_os": r["minimumOsVersion"],
        "release_notes": r.get("releaseNotes", ""),
        "url": r["trackViewUrl"],
    }

One field pair is easy to miss and matters for ASO tracking: averageUserRating is the lifetime score, while averageUserRatingForCurrentVersion resets when a developer prompts a fresh ratings cycle. If a competitor's score jumps overnight, compare the two before you assume they bought reviews.

For keyword and competitor discovery, the Search API is the itunes lookup api's companion. entity=software scopes results to iPhone apps (iPadSoftware and macSoftware exist too).

def search_apps(term, country="US", limit=50):
    target = (f"https://itunes.apple.com/search?term={quote(term)}"
              f"&country={country}&entity=software&limit={limit}")
    env = fetch(target, country=country, render=False)
    return json.loads(env["body"])["results"]

The Apple endpoints are reference tables in their own right:

App rankings on Apple come from RSS feeds, not a scraped chart page. The current generator lives at rss.applemarketingtools.com, and the array order in the response is the ranking.

def top_chart(country="us", feed="top-free", limit=50):
    target = f"https://rss.applemarketingtools.com/api/v2/{country}/apps/{feed}/{limit}/apps.json"
    env = fetch(target, country=country.upper(), render=False)
    results = json.loads(env["body"])["feed"]["results"]
    return [(rank + 1, a["id"], a["name"], a["artistName"])
            for rank, a in enumerate(results)]

The catch is that these feeds carry rank, ID, and name, but no rating. To build a full ranking snapshot you join the chart to the Lookup API by ID. That two-call pattern, RSS for position and Lookup for score, is the whole Apple app rankings scraper:

def ranked_with_ratings(country="US", feed="top-free", limit=25):
    snapshot = []
    for rank, app_id, name, artist in top_chart(country.lower(), feed, limit):
        meta = app_metadata(app_id, country=country)
        snapshot.append({"rank": rank, "id": app_id, "name": name,
                         "rating": meta["rating_all"], "count": meta["rating_count"]})
    return snapshot

For category-specific charts (Games, Finance, Health & Fitness), the current RSS generator is limited, but the legacy genre feed still responds. Genre 6014 is Games, 6015 Finance, 6013 Health & Fitness, 6023 Food & Drink. Treat that legacy endpoint as undocumented and expect it to change without notice.

Apple exposes reviews as a paginated RSS-to-JSON feed, and the country sits in the path. That is the part that makes it an ASO tool rather than a novelty: the same app returns entirely different reviews for us, gb, and jp.

def apple_reviews(track_id, country="us", max_pages=10):
    reviews = []
    for page in range(1, max_pages + 1):
        target = (f"https://itunes.apple.com/{country}/rss/customerreviews/"
                  f"page={page}/id={track_id}/sortby=mostrecent/json")
        env = fetch(target, country=country.upper(), render=False)
        entries = json.loads(env["body"]).get("feed", {}).get("entry", [])
        page_reviews = [e for e in entries if "im:rating" in e]
        if not page_reviews:
            break
        for e in page_reviews:
            reviews.append({
                "author": e["author"]["name"]["label"],
                "rating": int(e["im:rating"]["label"]),
                "title": e["title"]["label"],
                "body": e["content"]["label"],
                "version": e["im:version"]["label"],
                "updated": e["updated"]["label"],
            })
    return reviews

Two hard limits to design around. Apple caps this feed at 10 pages, roughly 500 reviews per country, so you cannot backfill a full history in one pass. And the first entry on page 1 is the app record itself, not a review, which is why the filter checks for im:rating. Because you cannot page infinitely, the right pattern is to poll sortby=mostrecent on a schedule and accumulate new reviews over time rather than trying to grab everything at once. If you already scrape review sites, the parsing shape will feel familiar from how to scrape Google reviews.

Google Play is where naive scrapers die. The visible DOM class names are randomized on every build, so a CSS selector that works today returns nothing next week. Do not target the rendered layout. Two stable structures live inside the page instead: a SoftwareApplication JSON-LD script and the AF_initDataCallback data blocks. The JSON-LD is the clean path for the headline metadata.

import re

def play_metadata(package, gl="US", hl="en"):
    target = f"https://play.google.com/store/apps/details?id={package}&gl={gl}&hl={hl}"
    env = fetch(target, country=gl, render=True)   # Google Play needs rendering
    html = env["body"]
    ld = {}
    for block in re.findall(r'

',
                            html, re.DOTALL):
        data = json.loads(block)
        if data.get("@type") == "SoftwareApplication":
            ld = data
            break
    agg = ld.get("aggregateRating", {})
    offers = ld.get("offers") or [{}]
    return {
        "name": ld.get("name"),
        "developer": (ld.get("author") or {}).get("name"),
        "category": ld.get("applicationCategory"),
        "rating": agg.get("ratingValue"),
        "rating_count": agg.get("ratingCount"),
        "price": offers[0].get("price"),
        "content_rating": ld.get("contentRating"),
    }

The gl parameter sets the country storefront and hl sets the language, the Google Play equivalents of Apple's country. For deeper fields the JSON-LD does not carry, the install count, the star histogram, the full localized description, and screenshot URLs, you parse the AF_initDataCallback blocks by their ds: key. Those index positions drift between releases, so pin them to a maintained parser and re-validate on each run rather than hard-coding offsets. The endpoints behind Play behave like classic hidden JSON APIs, which we break down in how to scrape hidden JSON API endpoints.

Google Play does not put reviews in the initial HTML beyond the first few. The store UI loads them from a batch RPC endpoint, https://play.google.com/_/PlayStoreUi/data/batchexecute, using the reverse-engineered RPC ID UsvDTd. You POST a form-encoded f.req payload and read back a )]}'-guarded JSON array that contains the reviews and a continuation token for the next page.

def build_reviews_req(package, sort=2, count=100, token=None):
    # sort: 1 = most relevant, 2 = newest, 3 = rating
    tok = "null" if token is None else f'\\"{token}\\"'
    inner = f'[null,null,[2,{sort},[{count},null,{tok}]],["{package}",7]]'
    return f'[[["UsvDTd","{inner}",null,"generic"]]]'

You send that payload through a country-matched residential exit, append hl and gl to the query string, strip the leading anti-hijack guard from the response, and pull the review array plus the nested pagination token. The RPC ID, the payload shape, and the token's position all change without notice, so treat this as maintenance-heavy and pin it to a tested library such as google-play-scraper rather than a snippet you never revisit. Unlike Apple's 10-page ceiling, the token paginates far deeper, but Google throttles it hard per IP, which is the next problem to solve. For turning that raw review stream into something useful, our note on proxies for review monitoring and sentiment analysis covers the downstream side.

ASO is a per-market discipline. Apple runs the App Store in over 170 country storefronts, and Google Play localizes by gl. The same app has a different rank, a different rating average, a different featured placement, and completely different reviews in each one. Passing a country parameter is necessary but not sufficient, because Apple and Google also read the request IP. A US IP asking for the Japan storefront can get a mismatched or partial response, and some endpoints geo-filter on IP regardless of the parameter.

Routing through country_code in the Scraping API solves both halves at once: the parameter and the exit IP agree, so the storefront you asked for is the storefront you get. There is a throughput reason too. Apple documents the Search API as limited to roughly 20 calls per minute, and Google Play's batch endpoint throttles aggressively, both per IP. Distributing requests across country-matched IPs localizes the data and lifts you past a single IP's ceiling in the same move. If geo-targeting is new to you, start with what geo-targeting means in proxies.

COUNTRIES = ["US", "GB", "DE", "JP", "BR"]

def track_ratings(track_id):
    for cc in COUNTRIES:
        meta = app_metadata(track_id, country=cc)   # each call exits from that country
        yield cc, meta["rating_all"], meta["rating_count"]

Once single fetches work, the failure modes are throttling and partial pages, not parsing. Wrap every call in backoff that respects the API's 429 response and its retry_after_seconds, and add jitter so retries do not synchronize into a new burst.

import time, random

def with_backoff(fn, *args, retries=4, **kwargs):
    for attempt in range(retries):
        try:
            return fn(*args, **kwargs)
        except requests.HTTPError as e:
            if e.response.status_code == 429:
                wait = int(e.response.headers.get("retry-after", 2 ** attempt))
                time.sleep(wait + random.random())
                continue
            raise
    raise RuntimeError("retries exhausted")

A few rules that keep a store-scraping run healthy at volume:

• Keep concurrency modest, 5 to 15 workers, and let country-matched IP rotation, not raw parallelism, carry throughput.
• Persist incrementally. Write each app or review page to storage as it lands, stamped with the country and a UTC timestamp, so a mid-run failure never costs the whole batch.
• Poll Apple reviews on a schedule because of the 10-page cap; do not try to backfill history you cannot reach.
• Cache metadata that changes slowly (category, developer) and re-fetch only the volatile fields (rank, rating, new reviews) on your tracking interval.
• Expect layout and RPC drift on Google Play. Validate a known app on every run and alert when a field goes null, which is your early warning that a parser needs an update.

If blocks start appearing despite country matching, the general playbook in how to avoid getting your proxy blocked applies directly.