Ruby Web Scraping With Proxies: A Practical Guide

Every request you send carries your IP address. Send too many from one address and the server does the obvious thing: it rate-limits you, then blocks you. A proxy sits between your Ruby process and the target, swapping your real IP for one from a pool. Rotate that pool and the request pattern that used to look like a bot now looks like ordinary traffic.

Three problems push most people toward proxies:

• Rate limits and bans. Sites cap requests per IP per minute. Once you cross the line, you get 429 or 403 responses no matter how polite your code is.
• Geo-restricted content. Prices, search results, and availability change by country. To read what a user in Germany sees, you need a German exit IP.
• Fingerprinting. Modern anti-bot systems score the IP's reputation before they even look at your headers. Datacenter ranges with a clean history pass more often than a flagged one.

If you are new to the mechanics behind all this, what is web scraping covers the fundamentals, and using datacenter proxies for web scraping explains why proxy type matters for the sites you target. For the failure modes specifically, how to avoid getting your proxy blocked is worth a read before you scale up.

You need Ruby 3.3 or newer (Ruby 3.4 is the current stable line) and three gems: Nokogiri for parsing, plus HTTParty and Faraday for the HTTP clients you will compare below. Net::HTTP ships with the standard library, so nothing to install there.

Create a Gemfile:

# Gemfile
source 'https://rubygems.org'

gem 'nokogiri', '~> 1.18'   # HTML/XML parsing
gem 'httparty', '~> 0.22'   # ergonomic HTTP client
gem 'faraday',  '~> 2.12'   # HTTP client with middleware

Then install and check your versions:

bundle install
ruby -v            # ruby 3.4.x
bundle exec ruby -e "require 'nokogiri'; puts Nokogiri::VERSION"

Nokogiri ships precompiled native gems for common platforms, so bundle install no longer forces a libxml2 build on most machines. If you are on an unusual platform and the build fails, install the system libxml2 and libxslt headers first.

One note on credentials: never hardcode a proxy password or API key in a committed file. Read them from the environment (ENV.fetch('SPARKPROXY_API_KEY')) so a leaked repo does not leak your account.

Net::HTTP is verbose but it is always there, with no dependency to add. Its constructor takes the proxy details as positional arguments after the host and port:

require 'net/http'
require 'uri'

uri = URI('https://www.sparkproxy.io/pricing')

proxy_host = 'gate.sparkproxy.io'
proxy_port = 8000
proxy_user = 'you@sparkproxy.io'
proxy_pass = ENV.fetch('PROXY_PASS')

http = Net::HTTP.new(uri.host, uri.port, proxy_host, proxy_port, proxy_user, proxy_pass)
http.use_ssl = (uri.scheme == 'https')
http.read_timeout = 20
http.open_timeout = 10

request = Net::HTTP::Get.new(uri)
request['User-Agent'] = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 ' \
                        '(KHTML, like Gecko) Chrome/128.0 Safari/537.36'

response = http.request(request)
puts response.code          # "200"
puts response.body[0, 300]  # first 300 chars of HTML

When the target URL is HTTPS and the proxy is plain HTTP, Net::HTTP handles the CONNECT tunnel for you once use_ssl = true is set. You do not issue the CONNECT by hand.

Here is the detail that trips people up. The third argument to Net::HTTP.new defaults to :ENV, not nil. Leave it out and Ruby silently reads the http_proxy (and https_proxy) environment variables and routes through whatever it finds there. That is the usual cause of "my scraper is using the wrong IP" bugs. Pass the proxy host explicitly, or clear the env vars, so you stay in control of the exit IP.

A cleaner variant uses Net::HTTP::Proxy, which returns a subclass preconfigured with your proxy:

proxy = Net::HTTP::Proxy(proxy_host, proxy_port, proxy_user, proxy_pass)

proxy.start(uri.host, uri.port, use_ssl: true) do |http|
  response = http.get(uri.request_uri)
  puts response.code
end

Fetching gives you a string of HTML. Nokogiri turns that string into a document you can query with CSS selectors or XPath. This is the core of Nokogiri scraping in Ruby:

require 'nokogiri'

doc = Nokogiri::HTML(response.body)

# CSS selectors, the common case
doc.css('div.product-card').each do |card|
  name  = card.at_css('h2.title')&.text&.strip
  price = card.at_css('span.price')&.text&.strip
  link  = card.at_css('a')&.[]('href')
  puts "#{name} | #{price} | #{link}"
end

# XPath when CSS is not enough
first_heading = doc.xpath('//h1').first&.text

Two habits save you hours:

• Use at_css when you want one node and css when you want the whole set. Calling .text on a missing node raises, so pair at_css with the safe navigation operator (&.) as shown.
• Watch encoding. If a page declares one charset in its headers and another in a meta tag, mojibake creeps in. Force it when you know the answer: Nokogiri::HTML(response.body, nil, 'UTF-8').

Nokogiri does not fetch anything on its own. It only parses. That separation is deliberate: you control the request (and therefore the proxy, headers, and retries), and Nokogiri handles the DOM.

HTTParty trims the boilerplate. To scrape with Ruby through a proxy, set four options. The names are specific, and getting them wrong is the most common HTTParty proxy mistake:

require 'httparty'
require 'nokogiri'

response = HTTParty.get(
  'https://www.sparkproxy.io/pricing',
  http_proxyaddr: 'gate.sparkproxy.io',
  http_proxyport: 8000,
  http_proxyuser: 'you@sparkproxy.io',
  http_proxypass: ENV.fetch('PROXY_PASS'),
  headers: {
    'User-Agent' => 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) ' \
                    'AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.0 Safari/605.1.15'
  },
  timeout: 20
)

puts response.code
doc = Nokogiri::HTML(response.body)
puts doc.at_css('title')&.text

The options are http_proxyaddr, http_proxyport, http_proxyuser, and http_proxypass. There is no proxy: key in HTTParty, so if you copied that from a Faraday example, it will be ignored and your real IP goes out. Check response.request.options if you are unsure which address the request actually used.

For a reusable client, subclass and set the proxy once:

class Scraper
  include HTTParty
  http_proxy 'gate.sparkproxy.io', 8000, 'you@sparkproxy.io', ENV.fetch('PROXY_PASS')
  default_options.update(timeout: 20)
end

Scraper.get('https://www.sparkproxy.io/pricing')

Faraday is the choice when you want middleware: retries, logging, JSON parsing, and connection reuse. In Faraday 2.x the proxy is a single URL string, credentials included:

require 'faraday'
require 'nokogiri'

conn = Faraday.new(
  url: 'https://www.sparkproxy.io',
  proxy: 'http://you%40sparkproxy.io:PASS@gate.sparkproxy.io:8000',
  headers: { 'User-Agent' => 'Mozilla/5.0 (X11; Linux x86_64) Chrome/128.0 Safari/537.36' }
) do |f|
  f.options.timeout = 20
  f.adapter Faraday.default_adapter   # Net::HTTP under the hood
end

response = conn.get('/pricing')
puts response.status
doc = Nokogiri::HTML(response.body)

Note the %40 in the proxy URL. The username is an email address, and the @ inside it has to be percent-encoded so Faraday does not read it as the separator before the host. Build the string with URI.encode_www_form_component if your username or password contains reserved characters:

user = URI.encode_www_form_component('you@sparkproxy.io')
pass = URI.encode_www_form_component(ENV.fetch('PROXY_PASS'))
proxy_url = "http://#{user}:#{pass}@gate.sparkproxy.io:8000"

Providers authenticate you one of two ways, and your Ruby code changes with each.

Username and password. You pass credentials with every request, as in all the examples above. This works from any machine and any IP, which makes it the right choice for scrapers that run on ephemeral cloud workers whose IPs change.

IP whitelisting. You register your server's outbound IP in the provider dashboard, then send requests with no credentials at all. It is simpler in code and avoids leaking a password, but it breaks the moment your server IP changes, so it suits fixed infrastructure.

With whitelisting, the same Net::HTTP call drops the user and pass arguments:

http = Net::HTTP.new(uri.host, uri.port, 'gate.sparkproxy.io', 8000)
http.use_ssl = true

If you want the deeper mechanics of both schemes, using datacenter proxies for web scraping walks through where each one fits.

Rotation is what actually keeps you unblocked. There are two ways to do it in Ruby.

Option A: a rotating gateway. Point every request at one endpoint and let the provider assign a fresh exit IP per request. Your code stays identical to the single-proxy examples above; the rotation happens server-side. This is the least code and the most reliable, because the pool is managed for you.

Option B: rotate a list yourself. If you hold a list of endpoints, cycle through them. A round-robin Enumerator beats Array#sample because it guarantees even use instead of random clustering:

PROXIES = [
  { host: 'gate1.sparkproxy.io', port: 8000 },
  { host: 'gate2.sparkproxy.io', port: 8000 },
  { host: 'gate3.sparkproxy.io', port: 8000 }
].freeze

proxy_cycle = PROXIES.cycle   # infinite round-robin enumerator

def fetch(url, proxy)
  uri  = URI(url)
  http = Net::HTTP.new(uri.host, uri.port, proxy[:host], proxy[:port],
                       'you@sparkproxy.io', ENV.fetch('PROXY_PASS'))
  http.use_ssl = true
  http.request(Net::HTTP::Get.new(uri))
end

urls.each do |url|
  response = fetch(url, proxy_cycle.next)
  # parse response.body with Nokogiri here
  sleep(rand(1.0..3.0))   # jittered delay so the pattern is not robotic
end

The sleep(rand(1.0..3.0)) matters more than people expect. Perfectly even timing is itself a bot signal. A little jitter between requests makes the traffic look human. For the full set of anti-block tactics, how to avoid getting your proxy blocked goes further than rotation alone.

Even with rotation, requests fail. Connections reset, proxies time out, and sites return a 429 that a retry on a fresh IP will clear. Wrap your fetch in retry logic with exponential backoff, and treat status codes as signals rather than just success or failure:

def fetch_with_retry(url, proxy_cycle, max_retries: 3)
  attempts = 0
  begin
    attempts += 1
    response = fetch(url, proxy_cycle.next)

    case response.code.to_i
    when 200
      response
    when 403, 429, 503
      raise "blocked with #{response.code}"   # retry on a new proxy
    else
      response                                 # 404 etc, do not retry
    end
  rescue Net::OpenTimeout, Net::ReadTimeout, Errno::ECONNRESET, RuntimeError => e
    if attempts < max_retries
      sleep(2**attempts)   # 2s, 4s, 8s
      retry
    end
    warn "giving up on #{url}: #{e.message}"
    nil
  end
end

Two more fixes stop most blocks:

• Send a real User-Agent. Ruby's default agent for Net::HTTP and HTTParty is a dead giveaway. Set a current browser string, and vary it if you rotate heavily.
• Match the IP to the content. If a page returns a CAPTCHA on a datacenter IP, retrying on another datacenter IP rarely helps. That is the point to switch to residential IPs, which is exactly what premium_proxy does in the API below.

When JavaScript renders the data you need, none of this reaches it. Net::HTTP and HTTParty fetch raw HTML; they do not run scripts. For single-page apps you either drive a headless browser (Ferrum or Selenium in Ruby) or use an API that renders for you.

Managing pools, headless browsers, and CAPTCHA logic is a project of its own. The SparkProxy Scraping API collapses that into one HTTP call: you send a URL, it handles the proxy rotation, JavaScript rendering, and anti-bot handling, and returns the HTML. From Ruby it is a plain GET with an X-API-Key header.

The endpoint is https://scrape.sparkproxy.io/api/v1. The parameters you will reach for most:

With Net::HTTP and the standard library only:

require 'net/http'
require 'uri'
require 'nokogiri'

uri = URI('https://scrape.sparkproxy.io/api/v1')
uri.query = URI.encode_www_form(
  url:           'https://www.sparkproxy.io/pricing',
  render_js:     'true',
  premium_proxy: 'true',
  country_code:  'US'
)

request = Net::HTTP::Get.new(uri)
request['X-API-Key'] = ENV.fetch('SPARKPROXY_API_KEY')

response = Net::HTTP.start(uri.host, uri.port, use_ssl: true) { |http| http.request(request) }

doc = Nokogiri::HTML(response.body)
puts doc.at_css('h1')&.text

The same call reads cleaner in HTTParty, where query and headers map straight onto the request:

require 'httparty'
require 'nokogiri'

response = HTTParty.get(
  'https://scrape.sparkproxy.io/api/v1',
  headers: { 'X-API-Key' => ENV.fetch('SPARKPROXY_API_KEY') },
  query: {
    url:           'https://www.sparkproxy.io/pricing',
    render_js:     true,
    premium_proxy: true,
    country_code:  'US'
  }
)

doc = Nokogiri::HTML(response.body)
doc.css('div.plan').each { |plan| puts plan.at_css('.name')&.text&.strip }

Set json_response: true when you want the metadata alongside the page. The envelope carries status_code, duration_ms, and credits_used, with the page itself base64-encoded in body:

require 'base64'
require 'json'

response = HTTParty.get(
  'https://scrape.sparkproxy.io/api/v1',
  headers: { 'X-API-Key' => ENV.fetch('SPARKPROXY_API_KEY') },
  query: { url: 'https://www.sparkproxy.io/pricing', json_response: true }
)

payload = JSON.parse(response.body)
puts payload['status_code']         # 200
puts payload['credits_used']
html = Base64.decode64(payload['body'])
doc  = Nokogiri::HTML(html)

When does this beat raw proxies? When the target renders with JavaScript, throws CAPTCHAs, or fingerprints aggressively, the API is usually cheaper than the engineering time you would spend fighting it. When you scrape simple, static HTML at high volume, raw datacenter proxies are more economical. Web scraping API vs self-managed proxies breaks the trade-off down with numbers.

Here is the whole thing wired together: fetch through the API with rendering and geo-targeting, parse with Nokogiri, and write rows to CSV. Swap the selectors for your target's markup.

require 'httparty'
require 'nokogiri'
require 'csv'

API = 'https://scrape.sparkproxy.io/api/v1'
KEY = ENV.fetch('SPARKPROXY_API_KEY')

def scrape(target_url, country: 'US')
  response = HTTParty.get(
    API,
    headers: { 'X-API-Key' => KEY },
    query: {
      url:           target_url,
      render_js:     true,
      premium_proxy: true,
      country_code:  country
    },
    timeout: 60
  )
  raise "API error #{response.code}" unless response.code == 200

  Nokogiri::HTML(response.body)
end

def extract(doc)
  doc.css('div.product-card').map do |card|
    {
      name:  card.at_css('h2.title')&.text&.strip,
      price: card.at_css('span.price')&.text&.strip,
      url:   card.at_css('a')&.[]('href')
    }
  end
end

pages = %w[
  https://www.sparkproxy.io/products?page=1
  https://www.sparkproxy.io/products?page=2
]

CSV.open('results.csv', 'w') do |csv|
  csv << %w[name price url]
  pages.each do |page|
    products = extract(scrape(page))
    products.each { |p| csv << [p[:name], p[:price], p[:url]] }
    puts "#{page}: #{products.size} rows"
    sleep(rand(1.0..2.0))
  end
end

That is a working scraper in under 40 lines. The API absorbs the proxy rotation and rendering, Nokogiri does the extraction, and Ruby's standard CSV library handles output. Start from this, adjust the selectors, and you have a scraper you can actually maintain.