Kotlin Web Scraping with Proxies

Kotlin runs on the JVM, so it inherits mature HTTP clients (OkHttp, Ktor), the fastest HTML parser on the platform (Jsoup), and a coroutine model that makes thousands of concurrent requests cheap without a callback mess. You get null safety, data classes for your scraped records, and full access to the Java ecosystem. If you are brand new to the topic, the short version of what web scraping is covers the ground rules before you write a line of code.

The one thing that trips people coming from Python or Node.js: Kotlin coroutines are multi-threaded by default. Dispatchers.IO is a pool of many threads, so shared mutable state (a rotation counter, a results list) has real data races. That single fact changes how you rotate proxies and collect results, and it is the mistake this guide spends the most time preventing.

Here is what each library is actually for, so you pick the right tool per job:

You will usually pair one HTTP client (Ktor or OkHttp) with one parser (Jsoup or skrape{it}). This guide shows all four so you can mix them to taste.

Start with a Gradle project using the Kotlin DSL (build.gradle.kts). These are the versions current as of early 2026; bump them to the latest patch when you build.

dependencies {
    // HTTP clients
    implementation("io.ktor:ktor-client-core:3.1.3")
    implementation("io.ktor:ktor-client-cio:3.1.3")
    implementation("io.ktor:ktor-client-okhttp:3.1.3")
    implementation("com.squareup.okhttp3:okhttp:4.12.0")

    // Parsing
    implementation("org.jsoup:jsoup:1.18.3")
    implementation("it.skrape:skrapeit:1.2.2")

    // Concurrency
    implementation("org.jetbrains.kotlinx:kotlinx-coroutines-core:1.10.1")
}

One note on skrape{it}: version 1.2.2 is the current release and it wraps Jsoup, so anything Jsoup parses, skrape{it} parses. It has not seen frequent updates, so if you need the newest CSS selector behavior, call Jsoup directly. Both are shown below.

A data class holds each scraped record. Kotlin makes this a one-liner:

data class Product(val name: String, val price: String, val url: String)

Ktor's client is coroutine-native, so every request is a suspend function. Create one client for the whole app and reuse it; building a client per request wastes its thread and connection pools.

import io.ktor.client.*
import io.ktor.client.engine.cio.*
import io.ktor.client.request.*
import io.ktor.client.statement.*

val client = HttpClient(CIO)

suspend fun fetch(url: String): String =
    client.get(url) {
        header("User-Agent", "Mozilla/5.0 (compatible; SparkBot/1.0)")
    }.bodyAsText()

bodyAsText() reads the response as a string you can hand straight to a parser. Set a real User-Agent; the default Ktor agent string is an easy fingerprint for anti-bot systems to flag. Close the client on shutdown with client.close() so its background threads exit cleanly.

Scraping from one IP gets that IP throttled or blocked fast. Sending traffic through proxies is what keeps a job alive, and datacenter proxies are the usual starting point for high-volume, public targets (see using datacenter proxies for web scraping for the why).

Ktor's CIO engine takes a proxy through ProxyBuilder:

import io.ktor.client.*
import io.ktor.client.engine.cio.*
import io.ktor.client.plugins.*

val client = HttpClient(CIO) {
    engine {
        // HTTP proxy; use ProxyBuilder.socks(host, port) for SOCKS5
        proxy = ProxyBuilder.http("http://proxy-1.sparkproxy.io:10000")
    }
}

OkHttp uses java.net.Proxy directly, and you reuse a single base client by cloning it with newBuilder(). This is the OkHttp equivalent of caching an agent: the clone shares the connection pool, dispatcher, and thread pools with the base client instead of allocating fresh ones.

import okhttp3.OkHttpClient
import okhttp3.Request
import java.net.InetSocketAddress
import java.net.Proxy
import java.util.concurrent.TimeUnit

val base = OkHttpClient.Builder()
    .connectTimeout(15, TimeUnit.SECONDS)
    .callTimeout(30, TimeUnit.SECONDS)
    .build()

fun clientThrough(host: String, port: Int): OkHttpClient =
    base.newBuilder()
        .proxy(Proxy(Proxy.Type.HTTP, InetSocketAddress(host, port)))
        .build()

fun get(client: OkHttpClient, url: String): String {
    val request = Request.Builder()
        .url(url)
        .header("User-Agent", "Mozilla/5.0 (compatible; SparkBot/1.0)")
        .build()
    client.newCall(request).execute().use { response ->
        return response.body?.string().orEmpty()
    }
}

The .use { } block is doing quiet but critical work: OkHttp responses hold a live socket, and forgetting to close the body leaks connections until the pool starves. use closes it for you even if string() throws.

Most paid proxies need a username and password. Here Ktor and OkHttp diverge, and the difference surprises people.

Ktor's ProxyBuilder does not attach Proxy-Authorization for credentials embedded in the URL on the CIO or Java engines as of Ktor 3.x. If you point CIO at http://user:pass@host:port, the userinfo is ignored and the proxy answers with 407 Proxy Authentication Required. Two ways out: whitelist your server's IP in the dashboard so no credentials are needed, or run Ktor on the OkHttp engine and set a proxyAuthenticator. The OkHttp engine is the cleaner path because it works the same everywhere.

import io.ktor.client.*
import io.ktor.client.engine.okhttp.*
import okhttp3.Credentials
import java.net.InetSocketAddress
import java.net.Proxy

val client = HttpClient(OkHttp) {
    engine {
        config {
            proxy(Proxy(Proxy.Type.HTTP, InetSocketAddress("proxy-1.sparkproxy.io", 10000)))
            proxyAuthenticator { _, response ->
                response.request.newBuilder()
                    .header("Proxy-Authorization", Credentials.basic("USERNAME", "PASSWORD"))
                    .build()
            }
        }
    }
}

Plain OkHttp uses the same proxyAuthenticator:

val authed = base.newBuilder()
    .proxy(Proxy(Proxy.Type.HTTP, InetSocketAddress("proxy-1.sparkproxy.io", 10000)))
    .proxyAuthenticator { _, response ->
        response.request.newBuilder()
            .header("Proxy-Authorization", Credentials.basic("USERNAME", "PASSWORD"))
            .build()
    }
    .build()

Credentials.basic() builds the base64 Basic ... value for you. Keep the real username and password in environment variables, not in source, so they never land in a commit or a stack trace.

Once you have the HTML string, parsing is a separate concern from fetching. That split matters: you fetch through the proxy, then parse the returned string locally, so the parser never touches the network.

Jsoup is the direct route. CSS selectors, null-safe accessors, done:

import org.jsoup.Jsoup

fun parseProducts(html: String): List<Product> =
    Jsoup.parse(html).select("div.product-card").map { card ->
        Product(
            name = card.selectFirst("h2.title")?.text().orEmpty(),
            price = card.selectFirst("span.price")?.text().orEmpty(),
            url = card.selectFirst("a")?.absUrl("href").orEmpty(),
        )
    }

selectFirst returns a nullable element, so the ?. and orEmpty() keep a missing field from crashing the run. absUrl("href") resolves relative links to absolute ones, which saves a class of bugs when a site uses /product/123 instead of a full URL.

skrape{it} wraps the same engine in a Kotlin DSL that reads a little cleaner if you like typed blocks:

import it.skrape.core.htmlDocument

fun parseWithSkrape(html: String): List<Product> = htmlDocument(html) {
    findAll("div.product-card").map { card ->
        Product(
            name = card.findFirst("h2.title").text,
            price = card.findFirst("span.price").text,
            url = card.findFirst("a").attribute("href"),
        )
    }
}

Note the trade-off: skrape{it}'s findFirst throws if the selector matches nothing, while Jsoup's selectFirst returns null. On messy real-world pages where fields come and go, Jsoup's nullable result is often safer to work with. Use skrape{it} when the markup is consistent and you want the tighter DSL.

This is where Kotlin pulls ahead. Coroutines are cheap, so you can launch thousands, but that is exactly the danger: fire 5,000 requests at once and you exhaust file descriptors, hammer the target, and get every IP blocked in seconds. The fix is a Semaphore that caps how many requests run at the same time.

import kotlinx.coroutines.*
import kotlinx.coroutines.sync.Semaphore
import kotlinx.coroutines.sync.withPermit

suspend fun scrapeAll(urls: List<String>, concurrency: Int = 10): List<String> =
    coroutineScope {
        val gate = Semaphore(concurrency)
        urls.map { url ->
            async(Dispatchers.IO) {
                gate.withPermit { fetch(url) }
            }
        }.awaitAll()
    }

Semaphore(10) means at most 10 requests are in flight no matter how many URLs you pass. withPermit acquires a slot, runs the block, and releases the slot even if fetch throws. Ten to twenty concurrent requests per domain is a sane starting band; push higher only after you confirm the target and your proxy pool tolerate it. Going wide without a cap is one of the fastest ways to get blocked, and staying polite is half of avoiding proxy blocks.

Collecting results is the other coroutine trap. Do not have each coroutine .add() into a shared mutableListOf(); ArrayList is not thread-safe and concurrent adds will drop or corrupt entries. Return values from async and let awaitAll() assemble the list, as above. That pattern has no shared mutable state, so there is no race to lose.

Round-robin rotation in Kotlin is where the multi-threaded coroutine model bites hardest. A plain var index = 0; index++ read-modify-write is not atomic, and on Dispatchers.IO two coroutines on two threads will read the same index, hand out the same proxy, and skip another. Use AtomicInteger.

import java.util.concurrent.atomic.AtomicInteger

data class ProxySpec(val host: String, val port: Int, val user: String, val pass: String)

class ProxyPool(private val proxies: List<ProxySpec>) {
    private val counter = AtomicInteger(0)

    fun next(): ProxySpec {
        val raw = counter.getAndIncrement()
        // getAndIncrement eventually overflows Int.MAX_VALUE into negatives,
        // so normalize before the modulo or you get IndexOutOfBounds.
        val index = (raw % proxies.size + proxies.size) % proxies.size
        return proxies[index]
    }
}

The double-modulo on the index line is not decoration. AtomicInteger.getAndIncrement() wraps from Int.MAX_VALUE to Int.MIN_VALUE after about two billion calls, and a raw negative % in Kotlin returns a negative remainder, which throws IndexOutOfBoundsException on a long-running job. (raw % size + size) % size maps any int, negative included, back into 0 until size. Most copy-paste rotation snippets skip this and crash after a few days of uptime.

Cache one OkHttp client per proxy so you are not rebuilding clients on the hot path:

import java.util.concurrent.ConcurrentHashMap
import okhttp3.Credentials

class ProxyClients(private val base: OkHttpClient) {
    private val cache = ConcurrentHashMap<String, OkHttpClient>()

    fun clientFor(p: ProxySpec): OkHttpClient =
        cache.getOrPut("${p.host}:${p.port}") {
            base.newBuilder()
                .proxy(Proxy(Proxy.Type.HTTP, InetSocketAddress(p.host, p.port)))
                .proxyAuthenticator { _, response ->
                    response.request.newBuilder()
                        .header("Proxy-Authorization", Credentials.basic(p.user, p.pass))
                        .build()
                }
                .build()
        }
}

ConcurrentHashMap plus getOrPut keeps the cache thread-safe, and each cached client still shares the base client's dispatcher and connection pool. That is the combination that lets one process rotate through hundreds of proxies without leaking thread pools.

Proxies fail. A connection times out, a target returns 429, an IP gets benched. The reliable pattern: catch the failure, rotate to a fresh proxy, and back off before retrying so you do not re-trigger the rate limit. Coroutines make the wait a non-blocking delay.

import kotlinx.coroutines.delay
import java.io.IOException
import kotlin.random.Random

val RETRYABLE = setOf(408, 429, 500, 502, 503, 504)

suspend fun fetchWithRetry(
    url: String,
    pool: ProxyPool,
    clients: ProxyClients,
    maxRetries: Int = 4,
): String {
    var last: Exception? = null
    repeat(maxRetries + 1) { attempt ->
        val proxy = pool.next() // fresh proxy on every attempt
        try {
            val client = clients.clientFor(proxy)
            val request = Request.Builder().url(url).build()
            client.newCall(request).execute().use { response ->
                if (response.isSuccessful) return response.body?.string().orEmpty()
                if (response.code !in RETRYABLE) error("HTTP ${response.code}")
            }
        } catch (e: Exception) {
            last = e
        }
        val backoff = (1000L shl attempt).coerceAtMost(15_000L)
        delay(backoff + Random.nextLong(250)) // jitter spreads concurrent retries
    }
    throw last ?: IllegalStateException("all $maxRetries retries failed for $url")
}

Two decisions make this hold up. Rotate the proxy on every attempt, not just the first, so a dead IP never gets a second try in the same call. Add jitter to the backoff so that when 20 concurrent requests all hit a 429 at once, they do not retry in lockstep and re-trip the limit. Doubling delays via 1000L shl attempt gives 1s, 2s, 4s, 8s, capped at 15 seconds, which stays polite under load.

Managing your own pool gives full control. Sometimes a target is defended hard enough (JavaScript rendering, aggressive fingerprinting, CAPTCHAs) that maintaining rotation, retries, and headless browsers yourself stops being worth it. The SparkProxy Scraping API handles the exit IP, rotation, and optional browser render server-side. You send a target URL, it returns the response. The trade-off between the two approaches is laid out in web scraping API vs self-managed proxies.

The base endpoint is https://scrape.sparkproxy.io/api/v1 and auth is the X-API-Key header. Here it is with OkHttp, building the query string with HttpUrl:

import okhttp3.HttpUrl.Companion.toHttpUrl
import okhttp3.OkHttpClient
import okhttp3.Request

fun scrapeViaApi(target: String): String {
    val endpoint = "https://scrape.sparkproxy.io/api/v1".toHttpUrl().newBuilder()
        .addQueryParameter("url", target)
        .addQueryParameter("render_js", "true")      // run a real browser (5 credits)
        .addQueryParameter("premium_proxy", "true")  // residential exit IPs
        .addQueryParameter("country_code", "us")     // geo-target the exit
        .build()

    val request = Request.Builder()
        .url(endpoint)
        .header("X-API-Key", System.getenv("SPARKPROXY_API_KEY"))
        .build()

    OkHttpClient().newCall(request).execute().use { response ->
        return response.body?.string().orEmpty()
    }
}

The same call with the Ktor client:

suspend fun scrapeViaApi(client: HttpClient, target: String): String =
    client.get("https://scrape.sparkproxy.io/api/v1") {
        header("X-API-Key", System.getenv("SPARKPROXY_API_KEY"))
        url {
            parameters.append("url", target)
            parameters.append("render_js", "true")
            parameters.append("premium_proxy", "true")
            parameters.append("country_code", "us")
        }
    }.bodyAsText()

The response body is the target page's HTML, so you feed it straight into parseProducts() from the parsing section. render_js=true runs a real browser for JavaScript-heavy pages, premium_proxy=true upgrades to residential IPs for tougher targets, and country_code picks the exit geography. A practical split: run your own ProxyPool and OkHttp clients for high-volume, simple, static targets where per-request cost matters, and send the hard, JavaScript-rendered, or heavily defended pages to the Scraping API so you are not maintaining a headless browser fleet in Kotlin.