Elixir Web Scraping With Proxies: A Practical Guide

Every request you send carries your IP address. Send too many from one address and the server does the obvious thing: it rate-limits you, then blocks you. A proxy sits between your Elixir process and the target and swaps your real IP for one from a pool. Rotate that pool and the request pattern that used to look like a bot starts to look like ordinary traffic.

Three problems push most teams toward proxies:

• Rate limits and bans. Sites cap requests per IP per minute. Cross the line and you get 429 or 403 responses no matter how polite your code is.
• Geo-restricted content. Prices, search results, and availability change by country. To read what a user in Germany sees, you need a German exit IP.
• Fingerprinting. Anti-bot systems score the IP's reputation before they even read your headers. A clean datacenter range passes more often than a flagged one.

Elixir brings something most scraping stacks lack: the BEAM runs millions of cheap processes, so firing 500 concurrent requests costs almost nothing in memory. That makes the proxy pool, not the runtime, your real bottleneck. Keep that in mind as you read on, because it changes how you size concurrency later.

If you are new to the mechanics, what is web scraping covers the fundamentals, and using datacenter proxies for web scraping explains why proxy type matters for the sites you target. For the failure modes specifically, how to avoid getting your proxy blocked is worth reading before you scale up.

You need Elixir 1.18 or newer on Erlang/OTP 27, plus a handful of libraries. Req is the modern HTTP client (it wraps Finch), Floki parses the HTML, and HTTPoison covers the case where you want hackney's proxy convenience.

Create a project and add the dependencies to mix.exs:

# mix.exs
defp deps do
  [
    {:req, "~> 0.5.0"},        # high-level HTTP client, built on Finch
    {:finch, "~> 0.19.0"},     # the pool underneath, for direct control
    {:floki, "~> 0.36.0"},     # HTML parsing with CSS selectors
    {:httpoison, "~> 2.2"},    # hackney-based client with simple proxy_auth
    {:html5ever, "~> 0.16.0"}  # optional fast parser backend for Floki
  ]
end

Fetch them and confirm your versions:

mix deps.get
elixir -v            # Elixir 1.18.x (compiled with Erlang/OTP 27)

One habit up front: never hardcode a proxy password or API key in a committed file. Read them from the environment with System.fetch_env!/1, which returns the value or raises if it is missing. That keeps a leaked repo from leaking your account.

One subtle trap: System.fetch_env!/1 inside a module attribute runs at compile time, not runtime, so @key System.fetch_env!("SPARKPROXY_API_KEY") bakes whatever value existed when you compiled. Read secrets inside a function instead, so the running node picks up the real environment.

Req is the client to reach for first. It handles redirects, retries, and JSON decoding out of the box, and it exposes the proxy through :connect_options, which Req passes down to Finch and Mint.

The proxy is a {scheme, host, port, opts} tuple. Authentication is the part people miss: Mint has no proxy_auth shortcut, so you build the proxy-authorization header yourself as HTTP Basic and pass it in :proxy_headers.

basic =
  "Basic " <>
    Base.encode64("you@sparkproxy.io:" <> System.fetch_env!("PROXY_PASS"))

resp =
  Req.get!("https://www.sparkproxy.io/pricing",
    connect_options: [
      proxy: {:http, "gate.sparkproxy.io", 8000, []},
      proxy_headers: [{"proxy-authorization", basic}]
    ],
    headers: [
      {"user-agent",
       "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 " <>
         "(KHTML, like Gecko) Chrome/128.0 Safari/537.36"}
    ],
    receive_timeout: 20_000
  )

IO.puts(resp.status)              # 200
IO.puts(String.slice(resp.body, 0, 300))

When the target is HTTPS and the proxy scheme is :http, Mint opens the CONNECT tunnel for you. You do not issue the CONNECT by hand. The proxy-authorization header rides on that CONNECT request, which is exactly why it belongs in :proxy_headers and not in the normal :headers list.

Reach for Finch directly when you want explicit pool control: a fixed number of connections, a shared proxy across every request, and no per-request overhead. Finch is a supervised process, so you start it in your application's supervision tree and set the proxy once at the pool level through :conn_opts.

# in your application.ex children list
{Finch,
 name: SparkFinch,
 pools: %{
   default: [
     size: 10,
     count: 1,
     conn_opts: [
       proxy: {:http, "gate.sparkproxy.io", 8000, []},
       proxy_headers: [
         {"proxy-authorization",
          "Basic " <>
            Base.encode64("you@sparkproxy.io:" <> System.fetch_env!("PROXY_PASS"))}
       ]
     ]
   ]
 }}

Then every request through SparkFinch uses that proxy:

{:ok, resp} =
  Finch.build(:get, "https://www.sparkproxy.io/pricing")
  |> Finch.request(SparkFinch)

IO.puts(resp.status)

The key detail is that the proxy lives on the pool, not the request. Finch reuses connections, so a pool of size: 10 holds ten keep-alive connections through the proxy and hands them out as requests come in. That reuse is what makes Finch fast at volume, and it is why you configure the proxy once rather than on every call.

HTTPoison wraps hackney, and hackney gives you the one thing Mint does not: a plain proxy_auth tuple, no manual Base64 required. If you would rather not build the authorization header yourself, this is the shortest path.

{:ok, resp} =
  HTTPoison.get(
    "https://www.sparkproxy.io/pricing",
    [{"User-Agent", "Mozilla/5.0 (X11; Linux x86_64) Chrome/128.0 Safari/537.36"}],
    proxy: {"gate.sparkproxy.io", 8000},
    proxy_auth: {"you@sparkproxy.io", System.fetch_env!("PROXY_PASS")},
    timeout: 10_000,
    recv_timeout: 20_000
  )

IO.puts(resp.status_code)

Two timeouts trip people up here. :timeout is the connection timeout and :recv_timeout is how long hackney waits for the response body. Proxies add a hop, so bump :recv_timeout above hackney's 5-second default or slow targets will look like failures when they are just slow.

HTTPoison matters for one more reason: Crawly uses it as the default fetcher, so the proxy options you learn here carry straight into the crawling framework below.

Fetching gives you a string of HTML. Floki turns that string into a document you can query with CSS selectors. This is the core of Floki parsing in Elixir, and it stays out of your fetch logic entirely.

{:ok, document} = Floki.parse_document(resp.body)

rows =
  document
  |> Floki.find("div.product-card")
  |> Enum.map(fn card ->
    name  = card |> Floki.find("h2.title") |> Floki.text() |> String.trim()
    price = card |> Floki.find("span.price") |> Floki.text() |> String.trim()
    href  = card |> Floki.find("a") |> Floki.attribute("href") |> List.first()
    %{name: name, price: price, url: href}
  end)

A few habits save you time:

• Floki.find/2 always returns a list, even for one match. Pipe it through Floki.text/1 for the visible text or Floki.attribute/2 for an attribute, and remember Floki.attribute/2 also returns a list, so List.first/1 pulls the single value.
• Floki ships with a built-in parser, but for large pages the html5ever backend (a Rust NIF) parses noticeably faster. Enable it once in config/config.exs:

config :floki, :html_parser, Floki.HTMLParser.Html5ever

Floki fetches nothing on its own. It only parses. That separation is deliberate: you own the request, and therefore the proxy, headers, and retries, while Floki owns the DOM.

Rotation is what actually keeps you unblocked, and providers authenticate you one of two ways that change your code.

Username and password. You pass credentials with every request, as in the examples above. This works from any machine and any IP, which suits scrapers on ephemeral cloud workers whose IPs change.

IP whitelisting. You register your server's outbound IP in the dashboard, then send requests with no credentials at all. Simpler in code, no password to leak, but it breaks the moment your server IP changes, so it fits fixed infrastructure. With whitelisting you drop :proxy_headers and :proxy_auth entirely.

For the rotation itself, there are two approaches:

Option A: a rotating gateway. Point every request at one endpoint and let the provider assign a fresh exit IP per request. Your code is identical to the single-proxy examples above, the rotation happens server-side, and it is the least code and the most reliable because the pool is managed for you.

Option B: rotate a list yourself. If you hold a list of endpoints, cycle through them. Because scraping runs concurrently on the BEAM, a plain module-level list is not enough; you need shared, concurrency-safe state. A small Agent does the job with an even round-robin instead of random clustering:

defmodule ProxyPool do
  use Agent

  @proxies [
    {"gate1.sparkproxy.io", 8000},
    {"gate2.sparkproxy.io", 8000},
    {"gate3.sparkproxy.io", 8000}
  ]

  def start_link(_), do: Agent.start_link(fn -> 0 end, name: __MODULE__)

  def next do
    index =
      Agent.get_and_update(__MODULE__, fn i ->
        {i, rem(i + 1, length(@proxies))}
      end)

    Enum.at(@proxies, index)
  end
end

Each call to ProxyPool.next/0 hands out the next proxy in order, and the Agent serializes access so two concurrent scrapers never collide on the counter. A rotating gateway avoids all of this, but the Agent pattern is useful when you hold specific endpoints and want deterministic spread. For the full set of anti-block tactics beyond rotation, how to avoid getting your proxy blocked goes deeper.

This is where Elixir pulls ahead. Task.async_stream/3 runs a function over a collection across many processes at once, with built-in back-pressure and a concurrency ceiling. You get bounded parallelism for free, no thread pool to manage.

urls
|> Task.async_stream(
  fn url -> scrape(url) end,
  max_concurrency: 10,
  ordered: false,
  timeout: 30_000,
  on_timeout: :kill_task
)
|> Enum.reduce([], fn
  {:ok, rows}, acc -> rows ++ acc
  {:exit, _reason}, acc -> acc
end)

Each option earns its place:

• max_concurrency defaults to System.schedulers_online/0, which is your CPU core count. That default is wrong for scraping. Scraping is I/O-bound, not CPU-bound, and the real limit is your proxy pool, not your cores. Set max_concurrency to match your pool size so you never send more parallel requests than you have clean exit IPs. This is the single most important knob, and copying a CPU-bound example silently caps you at 4 or 8.
• ordered: false releases each result the moment it finishes instead of waiting to preserve input order. For scraping, where you just collect rows, that raises throughput.
• on_timeout: :kill_task kills a hung request at the :timeout boundary rather than crashing the whole stream, so one dead proxy does not stall the batch.

Pairing max_concurrency with your proxy count is the concrete win: if the rotating gateway holds 50 IPs, max_concurrency: 50 keeps every request on its own fresh address, and no single IP absorbs enough traffic to trip a rate limit.

When you outgrow a script, Crawly gives you a full crawling framework: request scheduling, deduplication, item pipelines, and middlewares. It is the closest thing Elixir has to Scrapy. You define a spider, and Crawly drives it.

defmodule ProductSpider do
  use Crawly.Spider

  @impl Crawly.Spider
  def base_url(), do: "https://www.sparkproxy.io"

  @impl Crawly.Spider
  def init(), do: [start_urls: ["https://www.sparkproxy.io/products"]]

  @impl Crawly.Spider
  def parse_item(response) do
    {:ok, document} = Floki.parse_document(response.body)

    items =
      document
      |> Floki.find("div.product-card")
      |> Enum.map(fn card ->
        %{
          name:  card |> Floki.find("h2.title") |> Floki.text() |> String.trim(),
          price: card |> Floki.find("span.price") |> Floki.text() |> String.trim()
        }
      end)

    next_requests =
      document
      |> Floki.find("a.next")
      |> Floki.attribute("href")
      |> Enum.map(&Crawly.Utils.request_from_url/1)

    %Crawly.ParsedItem{items: items, requests: next_requests}
  end
end

Crawly fetches through HTTPoison by default, so the proxy options from earlier drop straight into its config. Point the fetcher at your rotating gateway and let Crawly handle concurrency:

# config/config.exs
config :crawly,
  concurrent_requests_per_domain: 8,
  closespider_itemcount: 500,
  fetcher:
    {Crawly.Fetchers.HTTPoisonFetcher,
     [
       proxy: {"gate.sparkproxy.io", 8000},
       proxy_auth: {"you@sparkproxy.io", System.get_env("PROXY_PASS")}
     ]},
  middlewares: [
    Crawly.Middlewares.DomainFilter,
    Crawly.Middlewares.UniqueRequest,
    {Crawly.Middlewares.UserAgent,
     user_agents: ["Mozilla/5.0 (X11; Linux x86_64) Chrome/128.0 Safari/537.36"]}
  ],
  pipelines: [
    Crawly.Pipelines.Validate,
    Crawly.Pipelines.DuplicatesFilter,
    {Crawly.Pipelines.WriteToFile, extension: "jl", folder: "./output"}
  ]

Start the crawl with Crawly.Engine.start_spider(ProductSpider). The concurrent_requests_per_domain setting is your rotation partner here: with a rotating gateway, each of those 8 in-flight requests lands on a different exit IP, so concurrency and IP spread scale together.

Managing pools, headless browsers, and CAPTCHA logic is a project of its own. The SparkProxy Scraping API collapses that into one HTTP call: you send a URL, it handles proxy rotation, JavaScript rendering, and anti-bot handling, and it returns the HTML. From Elixir it is a plain GET with an X-API-Key header.

The endpoint is https://scrape.sparkproxy.io/api/v1. The parameters you reach for most:

With Req it is a few lines, and Req builds the query string for you:

resp =
  Req.get!("https://scrape.sparkproxy.io/api/v1",
    headers: [{"x-api-key", System.fetch_env!("SPARKPROXY_API_KEY")}],
    params: [
      url: "https://www.sparkproxy.io/pricing",
      render_js: true,
      premium_proxy: true,
      country_code: "US"
    ],
    receive_timeout: 60_000
  )

{:ok, document} = Floki.parse_document(resp.body)
IO.puts(Floki.find(document, "h1") |> Floki.text())

Without json_response, the API returns raw HTML that you pass straight to Floki. Set json_response: true when you want metadata alongside the result, and Req decodes the JSON envelope into a map automatically:

resp =
  Req.get!("https://scrape.sparkproxy.io/api/v1",
    headers: [{"x-api-key", System.fetch_env!("SPARKPROXY_API_KEY")}],
    params: [url: "https://www.sparkproxy.io/pricing", json_response: true]
  )

IO.inspect(resp.body["status_code"])   # 200
IO.inspect(resp.body["duration_ms"])
IO.inspect(resp.body["credits_used"])

When does this beat raw proxies? When the target renders with JavaScript, throws CAPTCHAs, or fingerprints aggressively, one call with render_js and premium_proxy replaces a headless-browser and proxy-pool project, and it is usually cheaper than the engineering time you would spend fighting it. When you scrape simple, static HTML at high volume, raw datacenter proxies are more economical. Web scraping API vs self-managed proxies breaks the trade-off down with numbers.

Here is the whole thing wired together: fetch through the API with rendering and geo-targeting, run the pages concurrently with Task.async_stream, parse with Floki, and write rows to CSV. Swap the selectors for your target's markup.

defmodule ProductScraper do
  @api "https://scrape.sparkproxy.io/api/v1"

  def run(pages) do
    pages
    |> Task.async_stream(&scrape/1,
      max_concurrency: 8,
      ordered: false,
      timeout: 60_000,
      on_timeout: :kill_task
    )
    |> Enum.flat_map(fn
      {:ok, rows} -> rows
      {:exit, reason} ->
        IO.warn("task failed: #{inspect(reason)}")
        []
    end)
    |> write_csv("products.csv")
  end

  defp scrape(url) do
    resp =
      Req.get!(@api,
        headers: [{"x-api-key", System.fetch_env!("SPARKPROXY_API_KEY")}],
        params: [url: url, render_js: true, premium_proxy: true, country_code: "US"],
        receive_timeout: 60_000
      )

    if resp.status == 200 do
      extract(resp.body)
    else
      IO.warn("#{url} returned #{resp.status}")
      []
    end
  end

  defp extract(html) do
    {:ok, doc} = Floki.parse_document(html)

    doc
    |> Floki.find("div.product-card")
    |> Enum.map(fn card ->
      %{
        name:  card |> Floki.find("h2.title") |> Floki.text() |> String.trim(),
        price: card |> Floki.find("span.price") |> Floki.text() |> String.trim(),
        url:   card |> Floki.find("a") |> Floki.attribute("href") |> List.first()
      }
    end)
  end

  defp write_csv(rows, path) do
    lines =
      Enum.map_join(rows, "", fn r ->
        ~s("#{r.name}","#{r.price}","#{r.url}"\n)
      end)

    File.write!(path, "name,price,url\n" <> lines)
    IO.puts("wrote #{length(rows)} rows to #{path}")
  end
end

ProductScraper.run([
  "https://www.sparkproxy.io/products?page=1",
  "https://www.sparkproxy.io/products?page=2"
])

That is a working scraper in under 50 lines. The API absorbs proxy rotation and rendering, Task.async_stream runs the pages in parallel with a bounded ceiling, and Floki does the extraction. Start from this, adjust the selectors, and you have a scraper you can actually maintain.