Golang Web Scraping Proxy Setup: net/http and Colly

Go is a strong fit for scraping: real threads through goroutines, a fast net/http stack, and single-binary deployment. That same speed is what gets you blocked. Fire a few hundred concurrent requests from one IP and the target rate-limits you, serves a CAPTCHA, or bans the address. A proxy pool spreads that traffic across many IPs so no single address crosses the site's detection threshold. If you want the ground-level concept first, see what web scraping is.

You have three practical ways to attach proxies to a Go scraper, and they suit different jobs:

Most real projects mix them. Raw net/http for high-volume JSON and simple pages, Colly when you are crawling and parsing HTML across a site, and a managed scraping API versus self-managed proxies for the hard targets. The sections below cover all three.

Every Go HTTP client routes through a proxy by way of an http.Transport. The Proxy field takes a function that returns a proxy URL for a given request, and http.ProxyURL wraps a static URL into that function.

package main

import (
	"fmt"
	"io"
	"log"
	"net/http"
	"net/url"
	"time"
)

func main() {
	proxyURL, err := url.Parse("http://proxy-1.sparkproxy.io:10000")
	if err != nil {
		log.Fatal(err)
	}

	client := &http.Client{
		Transport: &http.Transport{
			Proxy: http.ProxyURL(proxyURL),
		},
		Timeout: 30 * time.Second,
	}

	resp, err := client.Get("https://www.sparkproxy.io/ip")
	if err != nil {
		log.Fatal(err)
	}
	defer resp.Body.Close()

	body, _ := io.ReadAll(resp.Body)
	fmt.Printf("status=%d body=%s\n", resp.StatusCode, body)
}

Two rules save you from the most common mistakes. First, always set a Timeout on the http.Client. The zero value is no timeout at all, so a dead proxy hangs your goroutine forever. Second, and this catches almost everyone: http.Get, http.Post, and the package-level helpers use http.DefaultClient, which has no proxy. If you build a configured client and then call http.Get(...), your request goes out on your real IP with no error. Always call your own client.Do or client.Get.

For a http:// proxy that carries https:// traffic, Go opens a CONNECT tunnel automatically. You do not configure anything extra for the target scheme. The http:// in the proxy URL describes how you reach the proxy, not how the destination is fetched.

Most commercial proxies need credentials. Put them in the proxy URL as userinfo and Go handles the Proxy-Authorization header for you, on both plain HTTP requests and the CONNECT tunnel used for HTTPS.

proxyURL, _ := url.Parse("http://user:pass@proxy-1.sparkproxy.io:10000")

client := &http.Client{
	Transport: &http.Transport{Proxy: http.ProxyURL(proxyURL)},
	Timeout:   30 * time.Second,
}

If your password contains characters like @, :, or /, encode them first with url.QueryEscape, otherwise url.Parse splits the URL in the wrong place and you get a 407 Proxy Authentication Required.

net/http speaks HTTP and HTTPS proxies out of the box but not SOCKS5. For a SOCKS5 proxy you need golang.org/x/net/proxy, which gives you a dialer you plug into the Transport.

import (
	"net/http"
	"golang.org/x/net/proxy"
)

func socks5Client() (*http.Client, error) {
	dialer, err := proxy.SOCKS5("tcp", "proxy-1.sparkproxy.io:1080",
		&proxy.Auth{User: "user", Password: "pass"}, proxy.Direct)
	if err != nil {
		return nil, err
	}

	tr := &http.Transport{}
	// Prefer DialContext so request cancellation and timeouts propagate.
	if cd, ok := dialer.(proxy.ContextDialer); ok {
		tr.DialContext = cd.DialContext
	} else {
		tr.Dial = dialer.Dial
	}
	return &http.Client{Transport: tr, Timeout: 30 * time.Second}, nil
}

The ContextDialer assertion matters. The plain Dial method ignores context, so a context.WithTimeout on your request will not cancel a stuck SOCKS5 connection. DialContext respects it. Install the dependency with go get golang.org/x/net/proxy.

Here is the detail most Go tutorials get wrong. The Transport's Proxy function is called once per request, receiving the *http.Request. That is the correct place to rotate: return a different proxy each call and the pool spreads automatically. But the same Transport is shared by every concurrent goroutine, so a naive counter++ inside that function is a data race. Use sync/atomic.

package main

import (
	"net/http"
	"net/url"
	"sync/atomic"
	"time"
)

var proxyStrings = []string{
	"http://user:pass@proxy-1.sparkproxy.io:10000",
	"http://user:pass@proxy-2.sparkproxy.io:10000",
	"http://user:pass@proxy-3.sparkproxy.io:10000",
}

// rotatingProxy returns a Proxy func that round-robins the pool, safe under
// concurrent goroutines. URLs are parsed once, not on every request.
func rotatingProxy(raw []string) func(*http.Request) (*url.URL, error) {
	parsed := make([]*url.URL, len(raw))
	for i, p := range raw {
		u, err := url.Parse(p)
		if err != nil {
			panic(err)
		}
		parsed[i] = u
	}
	var counter uint64
	return func(_ *http.Request) (*url.URL, error) {
		n := atomic.AddUint64(&counter, 1)
		return parsed[n%uint64(len(parsed))], nil
	}
}

func newClient() *http.Client {
	return &http.Client{
		Transport: &http.Transport{
			Proxy:               rotatingProxy(proxyStrings),
			MaxIdleConns:        100,
			MaxIdleConnsPerHost: 100, // default is 2; raise it for single-host scraping
			IdleConnTimeout:     90 * time.Second,
		},
		Timeout: 30 * time.Second,
	}
}

Parse the URLs once when you build the function, not on every request. Parsing per call is wasted CPU on your hottest path. Build one client and reuse it for the whole job. A fresh http.Transport per request throws away the connection pool and leaks idle connections and goroutines. The four rotation strategies (round-robin, random, weighted, sticky) map almost one to one across languages; the Python versions in how to rotate proxies in Python use the same math, only the concurrency primitives differ.

One more thing worth knowing: Go's Transport pools connections keyed by the pair (proxy, target host). Each proxy in your rotation keeps its own idle connection set, so a 50-proxy pool hitting one domain can hold 50 separate keep-alive pools. That is why the MaxIdleConnsPerHost value below matters so much.

The default Transport is tuned for a browser making a handful of requests, not a scraper hammering one host. Two defaults will silently cap your throughput and one will leak sockets. Fix all three.

MaxIdleConnsPerHost is the one nobody mentions. http.DefaultMaxIdleConnsPerHost is 2. Scrape a single domain with 100 goroutines and only two connections get reused; every other request pays a full handshake, then its connection is thrown away instead of pooled. Your scraper looks CPU-bound and slow for no obvious reason. Raise the value to match your per-host concurrency.

The second footgun is the response body. If you do not read it to EOF and close it, the connection cannot be returned to the pool, so keep-alive is defeated and you burn through ephemeral ports under load. Always drain and close, even when you do not care about the content.

import (
	"io"
	"net/http"
)

// drain fully reads and closes the body so the connection returns to the pool.
func drain(resp *http.Response) {
	io.Copy(io.Discard, resp.Body)
	resp.Body.Close()
}

func fetch(client *http.Client, target string) ([]byte, error) {
	resp, err := client.Get(target)
	if err != nil {
		return nil, err
	}
	defer drain(resp)
	return io.ReadAll(resp.Body)
}

io.ReadAll followed by the deferred drain is belt and suspenders: ReadAll reaches EOF, and the defer guarantees the close even on an early error path. Skip this and you will see connection counts climb and latency creep up as the run goes on.

For crawling and HTML extraction, Colly (github.com/gocolly/colly/v2) is the standard Go framework. It handles link following, HTML parsing through goquery, and request throttling, and it has a built-in colly proxy switcher.

The simplest form sets a single proxy:

package main

import (
	"fmt"
	"log"

	"github.com/gocolly/colly/v2"
)

func main() {
	c := colly.NewCollector(
		colly.UserAgent("Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36"),
	)

	if err := c.SetProxy("http://user:pass@proxy-1.sparkproxy.io:10000"); err != nil {
		log.Fatal(err)
	}

	c.OnHTML("h1", func(e *colly.HTMLElement) {
		fmt.Println("title:", e.Text)
	})

	c.Visit("https://www.sparkproxy.io")
}

For rotation, Colly ships proxy.RoundRobinProxySwitcher, which cycles a list of proxies and accepts http, https, and socks5 URLs in the same call.

import (
	"log"

	"github.com/gocolly/colly/v2"
	collyproxy "github.com/gocolly/colly/v2/proxy"
)

func newCollector() *colly.Collector {
	c := colly.NewCollector(colly.Async(true))

	rp, err := collyproxy.RoundRobinProxySwitcher(
		"http://user:pass@proxy-1.sparkproxy.io:10000",
		"http://user:pass@proxy-2.sparkproxy.io:10000",
		"socks5://user:pass@proxy-3.sparkproxy.io:1080",
	)
	if err != nil {
		log.Fatal(err)
	}
	c.SetProxyFunc(rp)
	return c
}

SetProxyFunc takes any colly.ProxyFunc, so if you need random or weighted selection rather than round-robin, write your own function with the same signature and pass it here. The switcher rotates on every request, which is what you want for stateless page fetching.

Go makes concurrency cheap, which is exactly why unbounded goroutines get you banned. Ten thousand simultaneous requests to one host looks nothing like a human. Cap concurrency and add delay.

With Colly, use colly.Async(true) plus a LimitRule, and always call c.Wait():

import (
	"time"

	"github.com/gocolly/colly/v2"
)

func crawl(c *colly.Collector, urls []string) {
	c.Limit(&colly.LimitRule{
		DomainGlob:  "*",
		Parallelism: 4,               // at most 4 in-flight requests per matching domain
		RandomDelay: 2 * time.Second, // random 0-2s gap to look less robotic
	})

	for _, u := range urls {
		c.Visit(u)
	}
	c.Wait() // Async collectors need this or main() exits before requests finish
}

Forgetting c.Wait() is the classic Colly bug: with Async(true), Visit returns immediately, so without Wait the program exits before a single response lands.

For raw net/http work, bound the goroutines with a worker pool. golang.org/x/sync/errgroup with SetLimit is the cleanest way, and it collects the first error for you.

import (
	"net/http"

	"golang.org/x/sync/errgroup"
)

func scrapeAll(client *http.Client, urls []string) ([]string, error) {
	g := new(errgroup.Group)
	g.SetLimit(8) // never more than 8 concurrent requests

	results := make([]string, len(urls))
	for i, u := range urls {
		i, u := i, u // capture loop vars (not needed on Go 1.22+, harmless)
		g.Go(func() error {
			body, err := fetch(client, u)
			if err != nil {
				return err
			}
			results[i] = string(body)
			return nil
		})
	}
	if err := g.Wait(); err != nil {
		return nil, err
	}
	return results, nil
}

SetLimit(8) is the whole game. It caps concurrency so you protect the target, your own file descriptors, and your proxy budget at the same time. Writing to results[i] from many goroutines is safe here because each goroutine owns a distinct index. Pick the limit to match your pool: pushing 100 goroutines through 3 proxies just piles requests onto three IPs and gets all three flagged. The async I/O trade-offs are the same ones covered for Python in async scraping with requests and aiohttp; Go swaps the event loop for goroutines but the pacing math is identical.

Proxies fail. They time out, return 407, or the target answers 429. Catch it, back off, and retry. Because the Transport's Proxy func rotates per request, each retry through the same client automatically lands on the next proxy.

import (
	"fmt"
	"io"
	"math/rand"
	"net/http"
	"time"
)

func fetchWithRetry(client *http.Client, target string, maxRetries int) ([]byte, error) {
	var lastErr error
	for attempt := 0; attempt <= maxRetries; attempt++ {
		resp, err := client.Get(target)
		if err == nil && resp.StatusCode < 400 {
			defer resp.Body.Close()
			return io.ReadAll(resp.Body)
		}
		if resp != nil {
			io.Copy(io.Discard, resp.Body) // drain so the connection can be reused
			resp.Body.Close()
			lastErr = fmt.Errorf("status %d", resp.StatusCode)
		} else {
			lastErr = err
		}
		if attempt == maxRetries {
			break
		}
		backoff := time.Duration(1<<attempt) * time.Second // 1s, 2s, 4s, 8s
		jitter := time.Duration(rand.Intn(250)) * time.Millisecond
		time.Sleep(backoff + jitter)
	}
	return nil, lastErr
}

Two decisions make this reliable. Double the delay each attempt (1s, 2s, 4s, 8s) so you do not hammer a rate limit, and add jitter so a batch of goroutines that all hit 429 at once do not retry in lockstep and re-trigger it. Drain the failed response before retrying, or you leak the connection you just decided to abandon. For the wider set of signals that get a request blocked in the first place, see how to avoid getting your proxy blocked.

Running your own pool gives you full control. Sometimes the target fights back with JavaScript rendering, fingerprinting, or aggressive anti-bot defenses, and maintaining that yourself stops being worth it. The SparkProxy Scraping API rotates the exit IP server-side on every request and can render JavaScript, so there is no pool, no agent, and no retry loop to keep alive. You send a target URL, it returns the response.

package main

import (
	"fmt"
	"io"
	"log"
	"net/http"
	"net/url"
	"os"
	"time"
)

func main() {
	endpoint, _ := url.Parse("https://scrape.sparkproxy.io/api/v1")

	q := endpoint.Query()
	q.Set("url", "https://example.com/products")
	q.Set("render_js", "true")     // run a real browser for JS-heavy pages
	q.Set("premium_proxy", "true") // route through residential IPs
	q.Set("country_code", "us")    // geo-target the exit IP (ISO 3166-1 alpha-2)
	endpoint.RawQuery = q.Encode()

	req, err := http.NewRequest(http.MethodGet, endpoint.String(), nil)
	if err != nil {
		log.Fatal(err)
	}
	req.Header.Set("X-API-Key", os.Getenv("SPARKPROXY_API_KEY"))

	client := &http.Client{Timeout: 90 * time.Second}
	resp, err := client.Do(req)
	if err != nil {
		log.Fatal(err)
	}
	defer resp.Body.Close()

	body, _ := io.ReadAll(resp.Body)
	fmt.Printf("status=%d bytes=%d\n", resp.StatusCode, len(body))
}

Authentication is the X-API-Key header carrying the key from your dashboard. country_code sets the exit geography, render_js runs a headless browser when the page needs JavaScript, and premium_proxy upgrades to residential IPs for tougher targets. Because a fresh IP is assigned per call, the rotation, retries, and transport tuning above are handled for you. Use a longer client timeout here (90 seconds is sensible) since a rendered page takes longer than a raw fetch. A pragmatic split: run your own net/http pool for high-volume simple pages where per-request cost matters, and send the JavaScript-heavy or heavily defended pages to the API.