Web Scraping with R and rvest

Most pages don't hand you a clean

Here's the gotcha the intro promised. read_html("https://...") has no proxy parameter, no header parameter, and no timeout. There is nowhere to hang them. The fix is to split the job: fetch with httr2 (which supports proxies), then parse the response body with rvest. resp_body_html() returns the same xml2 document that read_html() would, so every rvest selector works on it unchanged.

library(httr2)
library(rvest)

resp <- request("https://www.sparkproxy.io/") |>
  req_proxy(
    url      = "http://gate.sparkproxy.io",
    port     = 10000,
    username = "your-username",
    password = "your-password"
  ) |>
  req_user_agent("research-bot/1.0 (you@sparkproxy.io)") |>
  req_perform()

# Hand the fetched body to rvest
page   <- resp |> resp_body_html()
titles <- page |> html_elements("h2") |> html_text2()

Verify the proxy actually took effect before you trust a whole crawl to it. Route an IP echo endpoint through the same proxy and confirm the address is the proxy's, not your server's:

ip <- request("https://api.ipify.org") |>
  req_proxy("http://gate.sparkproxy.io", port = 10000,
            username = "your-username", password = "your-password") |>
  req_perform() |>
  resp_body_string()

cat("egress IP:", ip, "\n")

If that prints your real server IP, the proxy silently failed and every request is exposing your origin. Fix it before you scale.

Prefer the older httr stack? use_proxy() does the same job, and rvest can parse an httr response directly:

library(httr)
library(rvest)

resp <- GET(
  "https://www.sparkproxy.io/",
  use_proxy("gate.sparkproxy.io", port = 10000,
            username = "your-username", password = "your-password"),
  user_agent("research-bot/1.0")
)

page <- read_html(content(resp, as = "text"))

Use the http:// scheme for the proxy URL even when the target is HTTPS. The proxy opens a CONNECT tunnel and the tunnel carries the encrypted traffic. Rotating IPs, pool sizing, and per-IP request limits are their own topic; the guide on using datacenter proxies for web scraping covers the math.

Some data sits behind a login or a search form. rvest's session() holds a cookie jar across requests, so you authenticate once and reuse the session. The useful detail almost no tutorial mentions: session() passes its extra arguments straight to httr::GET(), which means you can inject a proxy and a user agent into the session the same way you would a plain request.

library(rvest)

# The proxy config rides along into every request this session makes
s <- session(
  "https://www.sparkproxy.io/login",
  httr::use_proxy("gate.sparkproxy.io", 10000,
                  username = "your-username", password = "your-password"),
  httr::user_agent("research-bot/1.0")
)

# Grab the login form, fill it, submit it
form <- html_form(s)[[1]]

filled <- html_form_set(form,
  email    = "you@sparkproxy.io",
  password = "your-password"
)

s <- session_submit(s, filled)

After the submit, the session carries the auth cookie. Now navigate authenticated pages with session_jump_to() and parse them with the same selectors:

s <- session_jump_to(s, "https://www.sparkproxy.io/dashboard/orders")

orders <- s |>
  html_element("table") |>
  html_table()

A form with more than one submit button needs a hint. Pass submit = to session_submit() with the name of the control you want, otherwise rvest uses the first one and you may trigger the wrong action. For plain pagination that uses URL query parameters (?page=2), skip the form machinery entirely and loop over the URLs as shown in the data frame section. Sessions are for state you can't express in a URL.

Hammering a site is how you get blocked and how you get your proxy pool burned. Two packages make good behavior the default.

polite bundles the whole etiquette layer: it reads robots.txt, declares a user agent, enforces a delay between requests, and caches responses so a re-run doesn't re-fetch. bow() introduces you to the host, scrape() fetches within the rules:

library(polite)

host <- bow(
  "https://www.sparkproxy.io/",
  user_agent = "research-bot (you@sparkproxy.io)",
  delay = 5          # seconds between requests
)

result <- scrape(host)   # obeys robots.txt and the delay

bow() returns a session object; use nod() to move to another path on the same host without re-reading robots.txt every time. If robots.txt disallows the path, scrape() warns and returns NULL instead of fetching, which is the behavior you want by default.

ratelimitr is lower level and composes with anything. It wraps a function so it can't be called more than N times per period, sleeping if you go over:

library(ratelimitr)
library(rvest)

# Cap our fetcher at 20 requests per minute
limited_fetch <- limit_rate(
  function(url) read_html(url),
  rate(n = 20, period = 60)
)

pages <- lapply(urls, limited_fetch)

Reach for polite when you want the full courteous default on a single host. Reach for ratelimitr when you need a hard calls-per-second ceiling across a custom fetch function, for example one that already routes through httr2 and a proxy. If blocks persist even with sane delays and rotation, the cause is usually headers or fingerprint, not rate; how to avoid getting your proxy blocked walks through the fixes.

rvest reads the HTML the server sends. If a page builds its content with JavaScript after load, that content is not in the HTML rvest sees, and your selectors come back empty. You can drive a headless browser from R, but it's heavy. The lighter path is to hand the URL to a scraping API that renders the page server-side, solves the proxy and fingerprint problem, and returns finished HTML that drops straight into rvest.

The SparkProxy Scraping API is an ordinary httr2 request. The base endpoint is https://scrape.sparkproxy.io/api/v1, you authenticate with the X-API-Key header, and you pass the target URL and options as query parameters. resp_body_html() parses the rendered result for rvest:

library(httr2)
library(rvest)

scrape_via_api <- function(target,
                           api_key = Sys.getenv("SPARKPROXY_API_KEY")) {
  request("https://scrape.sparkproxy.io/api/v1") |>
    req_headers(`X-API-Key` = api_key) |>
    req_url_query(
      url           = target,
      render_js     = "true",   # headless Chromium renders the page
      premium_proxy = "true",   # route through residential IPs
      country_code  = "us"      # geo-target the request
    ) |>
    req_perform() |>
    resp_body_html()
}

page   <- scrape_via_api("https://www.sparkproxy.io/")
prices <- page |> html_elements(".price") |> html_text2()

The parameters map to the controls you'd otherwise build yourself: