Web Scraping With Go and Colly: Proxies and Rotation

Colly (github.com/gocolly/colly/v2) is the most widely used scraping framework in Go. You register callbacks that fire on HTML elements, and Colly manages the request queue, connection pool, cookie jar, and parsing through goquery underneath. It gives you crawling and extraction without hand-rolling a worker pool.

There is one limitation to understand before you write a line of code: Colly fetches raw HTML over HTTP and does not run JavaScript. If a page renders its content client-side, OnHTML sees the empty shell the server sent, not the DOM a browser would build. That single fact decides most of your architecture. Static and server-rendered pages are Colly's home turf; heavily client-rendered or bot-defended pages need a headless browser or a rendering API instead.

If you want the lower-level proxy plumbing that sits under Colly, the Go net/http proxy guide covers transports and dialers in detail. For the build-versus-buy call on the hard targets, see scraping API versus self-managed proxies. New to the concept entirely? Start with what web scraping is. This guide stays focused on Colly.

Colly needs Go 1.20 or newer. Add the v2 module to your project:

go get github.com/gocolly/colly/v2

The whole framework revolves around one type, the Collector. You create one with colly.NewCollector, attach callbacks, then call Visit. Here is a complete scraper that prints every H2 on a page:

package main

import (
	"fmt"
	"log"

	"github.com/gocolly/colly/v2"
)

func main() {
	c := colly.NewCollector(
		colly.UserAgent("Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36"),
	)

	c.OnHTML("h2", func(e *colly.HTMLElement) {
		fmt.Println("heading:", e.Text)
	})

	if err := c.Visit("https://www.sparkproxy.io/blog"); err != nil {
		log.Fatal(err)
	}
}

colly.NewCollector takes functional options. The most useful ones early on are colly.UserAgent(string) so you do not announce yourself as Go's default agent, colly.AllowedDomains(...) to keep a crawl on the sites you meant, and colly.MaxDepth(n) to cap how deep link following goes. You can also set fields directly, for example c.UserAgent = "...", but the options are cleaner.

One rule that trips up newcomers: register every callback before you call Visit. Colly runs callbacks as the response is parsed, so a callback added after the visit has already returned never fires.

Colly is event driven. Each request passes through a fixed sequence of callbacks, and knowing the order tells you where to put each piece of logic.

This scraper wires up the full chain so you can watch the flow in your logs:

c := colly.NewCollector()

c.OnRequest(func(r *colly.Request) {
	r.Headers.Set("Accept-Language", "en-US,en;q=0.9")
	fmt.Println("visiting", r.URL)
})

c.OnResponse(func(r *colly.Response) {
	fmt.Printf("got %d bytes, status %d\n", len(r.Body), r.StatusCode)
})

c.OnHTML("title", func(e *colly.HTMLElement) {
	fmt.Println("title:", e.Text)
})

c.OnError(func(r *colly.Response, err error) {
	fmt.Printf("request to %s failed: %v\n", r.Request.URL, err)
})

c.OnScraped(func(r *colly.Response) {
	fmt.Println("done with", r.Request.URL)
})

c.Visit("https://www.sparkproxy.io")

If your OnHTML callback never runs, work down this short list before anything else. Check that your selector actually matches the served markup. Check that the content is present in the raw HTML and not injected by JavaScript, because Colly will not run that script. Confirm the request did not fail (put a print in OnError). Those three cover almost every "my callback does nothing" report.

Real scraping means pulling fields out of a repeating structure and turning them into records. Inside an OnHTML handler, the *colly.HTMLElement gives you the matched node, and helper methods read its children by selector.

type Product struct {
	Name  string
	Price string
	URL   string
}

func main() {
	c := colly.NewCollector(colly.AllowedDomains("sparkproxy.io"))

	var products []Product

	// Fire once per product card, then read fields from inside it.
	c.OnHTML("div.product-card", func(e *colly.HTMLElement) {
		p := Product{
			Name:  e.ChildText("h3.product-title"),
			Price: e.ChildText("span.price"),
			URL:   e.Request.AbsoluteURL(e.ChildAttr("a", "href")),
		}
		products = append(products, p)
	})

	c.Visit("https://www.sparkproxy.io/products")

	fmt.Printf("scraped %d products\n", len(products))
}

The methods you will reach for constantly:

• e.Text and e.Attr("href") read the matched element itself.
• e.ChildText(selector) and e.ChildAttr(selector, attr) read a descendant.
• e.ForEach(selector, func(i int, el *colly.HTMLElement)) loops over repeated children when one card holds a list.
• e.Request.AbsoluteURL(link) resolves a relative href against the current page, so you never store a broken /path with no host.
• e.DOM drops you into the underlying *goquery.Selection when you need selector logic Colly's helpers do not cover.

Use ForEach when a single matched element contains a variable number of sub-items, such as a table of specs or a list of tags:

c.OnHTML("table.specs", func(e *colly.HTMLElement) {
	specs := map[string]string{}
	e.ForEach("tr", func(_ int, row *colly.HTMLElement) {
		key := row.ChildText("td:first-child")
		val := row.ChildText("td:last-child")
		if key != "" {
			specs[key] = val
		}
	})
	fmt.Println(specs)
})

A scraper that reads one page is a fetch. A crawler follows links. In Colly you queue a new request from inside a callback by calling e.Request.Visit on a resolved URL. Constrain it with AllowedDomains and MaxDepth or a runaway crawl will happily wander off to the whole internet.

c := colly.NewCollector(
	colly.AllowedDomains("sparkproxy.io", "www.sparkproxy.io"),
	colly.MaxDepth(2),
)

// Follow every in-domain link.
c.OnHTML("a[href]", func(e *colly.HTMLElement) {
	link := e.Request.AbsoluteURL(e.Attr("href"))
	e.Request.Visit(link) // Colly skips already-visited URLs automatically
}) 

c.Visit("https://www.sparkproxy.io/blog")

Colly dedupes URLs it has already queued, so you will not revisit the same page unless you opt in with colly.AllowURLRevisit(). That dedupe is per collector and lives in memory for the run.

Production crawls usually split into two collectors: one walks list or category pages and finds detail links, the other scrapes each detail page. It keeps concerns separate and lets you give the detail collector its own limits and proxies. Pass data from the list page to the detail page through a colly.Context.

list := colly.NewCollector(colly.AllowedDomains("sparkproxy.io"))
detail := list.Clone() // shares config, gets its own callbacks

list.OnHTML("a.product-link", func(e *colly.HTMLElement) {
	link := e.Request.AbsoluteURL(e.Attr("href"))

	ctx := colly.NewContext()
	ctx.Put("listPrice", e.ChildText(".price")) // carry the list-page price forward
	detail.Request("GET", link, nil, ctx, nil)
})

detail.OnHTML("body", func(e *colly.HTMLElement) {
	fmt.Printf("%s | list price was %s\n",
		e.ChildText("h1"),
		e.Request.Ctx.Get("listPrice"),
	)
})

list.Visit("https://www.sparkproxy.io/products")

list.Clone() copies the collector's settings without its callbacks, so both share allowed domains and transport tuning. detail.Request("GET", link, nil, ctx, nil) is the form of Visit that carries a context, which is how the list price reaches the detail handler. Most single-collector tutorials skip this, and it is exactly the pattern you need once a scrape spans more than one page type.

Crawl one site hard enough from one IP and it rate-limits you, serves a CAPTCHA, or bans the address. Routing through proxies spreads the traffic. For a single static proxy, SetProxy is all you need:

c := colly.NewCollector()

if err := c.SetProxy("http://user:pass@proxy-1.sparkproxy.io:10000"); err != nil {
	log.Fatal(err)
}

c.OnHTML("h1", func(e *colly.HTMLElement) {
	fmt.Println(e.Text)
})

c.Visit("https://www.sparkproxy.io/ip")

Credentials go in the URL as user:pass@host:port and Colly passes them to the proxy for you. If the password contains characters like @, :, or /, percent-encode them first, otherwise URL parsing splits in the wrong place and you get a 407 Proxy Authentication Required. Colly also honors the standard HTTP_PROXY and HTTPS_PROXY environment variables when you set no proxy in code, which is handy for a quick test but too blunt for rotation. For how the credential handshake works under the hood, see how proxy authentication works.

A single proxy still funnels everything through one IP. For anything at scale, rotate.

Colly ships a proxy switcher in github.com/gocolly/colly/v2/proxy. RoundRobinProxySwitcher takes a list of proxy URLs and cycles a different one on every request. It accepts http, https, and socks5 schemes in the same call.

package main

import (
	"log"

	"github.com/gocolly/colly/v2"
	collyproxy "github.com/gocolly/colly/v2/proxy"
)

func newCollector() *colly.Collector {
	c := colly.NewCollector(colly.Async(true))

	rp, err := collyproxy.RoundRobinProxySwitcher(
		"http://user:pass@proxy-1.sparkproxy.io:10000",
		"http://user:pass@proxy-2.sparkproxy.io:10000",
		"socks5://user:pass@proxy-3.sparkproxy.io:1080",
	)
	if err != nil {
		log.Fatal(err)
	}
	c.SetProxyFunc(rp)
	return c
}

RoundRobinProxySwitcher is safe to use with colly.Async(true) because it increments its index with an atomic counter internally, so concurrent goroutines never hand out the same proxy by racing. That detail matters the moment you write your own switcher.

SetProxyFunc accepts any colly.ProxyFunc, which is just func(http.Request) (url.URL, error). Round-robin is often not what you want. If some IPs are faster or you want random selection to look less mechanical, supply your own function, and make it concurrency-safe yourself:

import (
	"math/rand"
	"net/http"
	"net/url"
	"sync"
)

// randomSwitcher picks a random proxy per request, safe across goroutines.
func randomSwitcher(raw []string) colly.ProxyFunc {
	parsed := make([]*url.URL, len(raw))
	for i, p := range raw {
		u, err := url.Parse(p)
		if err != nil {
			panic(err)
		}
		parsed[i] = u
	}
	var mu sync.Mutex
	return func(_ *http.Request) (*url.URL, error) {
		mu.Lock()
		defer mu.Unlock()
		return parsed[rand.Intn(len(parsed))], nil
	}
}

Parse the URLs once when you build the function, not on every call, and guard the shared state (math/rand's global source is not safe for concurrent use without a lock). The four rotation strategies, round-robin, random, weighted, and sticky, are covered generally in what proxy rotation is and how it works; with Colly you plug any of them in through this one function.

By default a Colly collector is synchronous: each Visit blocks until the response is handled. To fetch pages in parallel, create the collector with colly.Async(true), then cap the parallelism with a LimitRule and wait for the queue to drain with c.Wait().

import (
	"time"

	"github.com/gocolly/colly/v2"
)

func crawl(c *colly.Collector, urls []string) {
	c.Limit(&colly.LimitRule{
		DomainGlob:  "*",             // must match the target host or the rule is ignored
		Parallelism: 4,               // at most 4 in-flight requests per matching domain
		RandomDelay: 2 * time.Second, // random 0 to 2s gap between requests
	})

	for _, u := range urls {
		c.Visit(u)
	}
	c.Wait() // async Visit returns immediately; without Wait, main exits first
}

Two mistakes account for nearly every broken Colly concurrency setup:

Forgetting c.Wait(). With Async(true), Visit queues the request and returns at once. If main reaches its end before the goroutines finish, the program exits and you scrape nothing. Always call Wait after queuing.

A DomainGlob that does not match. The limit only applies to requests whose host matches the glob. Set DomainGlob: "example.com" while scraping www.example.com and the rule is silently skipped, so your "throttled" crawl runs wide open. Use "*" unless you deliberately want per-domain rules.

Match the parallelism to your proxy pool, not to what Go can spawn. Pushing 100 goroutines through 3 proxies just piles requests onto three IPs and gets all three flagged. A starting point of 2 to 4 requests per domain per proxy is sane, then raise it while watching your block rate. For the wider set of signals that get a request flagged, see how to avoid getting your proxy blocked.

During development, refetching the same pages on every run wastes time and burns proxy budget. Set colly.CacheDir and Colly stores each response on disk, serving repeats from cache:

c := colly.NewCollector(colly.CacheDir("./colly_cache"))

Delete the folder to force fresh fetches. Cached responses are read from disk and do not touch a proxy, which is the behavior you want while iterating on selectors.

Colly does not retry failed requests on its own. Handle failures in OnError, and bound the retries so a permanently dead target does not loop forever. Store the attempt count in the request's Context:

c.OnError(func(r *colly.Response, err error) {
	const maxRetries = 3

	attempts := 0
	if v, ok := r.Request.Ctx.GetAny("attempts").(int); ok {
		attempts = v
	}

	if attempts >= maxRetries {
		log.Printf("giving up on %s after %d attempts: %v", r.Request.URL, attempts, err)
		return
	}

	r.Request.Ctx.Put("attempts", attempts+1)
	backoff := time.Duration(1<<attempts) * time.Second // 1s, 2s, 4s
	time.Sleep(backoff)
	r.Request.Retry() // re-submits the same request; the switcher gives it a new proxy
}) 

r.Request.Retry() re-queues the identical request, and because your proxy switcher rotates per request, the retry lands on the next IP automatically. Exponential backoff (double the wait each attempt) keeps you from hammering a rate limit while it cools off.

Here are the Colly-specific failures worth memorizing:

Colly is excellent until a target renders its data with JavaScript or fights back with fingerprinting and anti-bot walls. Colly cannot run the JavaScript, and defeating a serious bot manager from scratch is a project of its own. When you hit that wall, route those specific pages through the SparkProxy Scraping API. It renders JavaScript in a real browser, rotates the exit IP server-side on every request, and returns the finished HTML, so there is no pool, no retry loop, and no headless browser for you to maintain.

The base endpoint is https://scrape.sparkproxy.io/api/v1 and authentication is the X-API-Key header. Call it from Go with a plain http.Client:

package main

import (
	"fmt"
	"io"
	"log"
	"net/http"
	"net/url"
	"os"
	"time"
)

func main() {
	endpoint, _ := url.Parse("https://scrape.sparkproxy.io/api/v1")

	q := endpoint.Query()
	q.Set("url", "https://example.com/products")
	q.Set("render_js", "true")     // run a real browser for JS-heavy pages
	q.Set("premium_proxy", "true") // route through residential IPs for tough targets
	q.Set("country_code", "us")    // geo-target the exit IP (ISO 3166-1 alpha-2)
	endpoint.RawQuery = q.Encode()

	req, _ := http.NewRequest(http.MethodGet, endpoint.String(), nil)
	req.Header.Set("X-API-Key", os.Getenv("SPARKPROXY_API_KEY"))

	client := &http.Client{Timeout: 90 * time.Second}
	resp, err := client.Do(req)
	if err != nil {
		log.Fatal(err)
	}
	defer resp.Body.Close()

	body, _ := io.ReadAll(resp.Body)
	fmt.Printf("status=%d bytes=%d\n", resp.StatusCode, len(body))
}

render_js runs the headless browser, premium_proxy upgrades to residential exit IPs, and country_code sets the exit geography. Because a fresh IP is assigned per call, the rotation and retry logic from earlier is handled for you. Use a longer client timeout, around 90 seconds, since a rendered page takes longer than a raw fetch. The API can also return parsed output directly: set format to md for clean Markdown, or pass extract_rules with your CSS selectors to get structured JSON back, which skips the goquery step entirely for the pages you send it.

A pragmatic split keeps costs down: let Colly crawl and parse the static, server-rendered pages where per-request cost matters, and forward only the JavaScript-heavy or heavily defended pages to the API. The two work well side by side.