Building a Scraping Team vs Buying the Dataset

Buy the dataset when the data is a commodity, the schema is stable, your competitors can buy the identical file, and nothing about your product depends on collecting it differently. Public company filings, standard ecommerce catalogues, job postings, app store listings. Somebody already maintains a pipeline for all of these and sells the output at a price you cannot match on your own, because their cost is amortised across every customer.

Build when the specificity is the product. A non-obvious source list, a field nobody else extracts, a refresh cadence tighter than the market offers, or a definition of "the same product across three retailers" that only your domain knowledge can encode. These are not available for purchase at any price, because the thing you need does not exist until you define it.

Everything in between is arithmetic, and the arithmetic is in the cost model section. Run it before the meeting, because the version of this conversation without numbers always resolves in favour of whoever speaks most confidently.

"Buy" covers three different products, and mixing them up is the most common error in this decision.

Finished datasets. Somebody else's pipeline, their schema, their coverage, delivered as files or through an API. You are buying records. Bright Data's datasets page, read today, prices these at "Up to $0.0025 per record" with a "$250" minimum order, in size tiers of 100K, 500K, 1M, 5M, 20M and a complete-dataset option, with marketplace datasets described as "Refreshed monthly".

Scraping platforms. Somebody else's infrastructure, your logic. You still write or configure the extraction, you still own the schema, and you still get paged when a site changes. Apify's published plans run Free at $0 with $5 of included usage, Starter at $19 with $19 included, Scale at $199 with $199 included and Business at $999 with $999 included, with compute billed at $0.2 per compute unit falling to $0.13 at the top tier.

Proxies or a scraping API. Somebody else's network, everything else yours. SparkProxy's Scraping API sits here: 1,000 free credits with no card, then Starter at $49 for 250,000 credits, Growth at $99 for 1,000,000, Pro at $249 for 3,000,000 and Scale at $599 for 8,000,000, where a plain fetch costs 1 credit, a JavaScript render 5 and a screenshot or PDF 10.

Notice what changes as you move down that list: you buy less finished work and more raw capability, and the per-unit price collapses. Notice also what does not change. In options two and three you still own the parser, the schema, the QA and the pager. That ownership, not the infrastructure, is the real cost of building. Our comparison of scraping APIs against self-managed proxies covers the boundary between the bottom two rungs.

One comparison in that table is worth doing carefully, because it is the number everyone quotes and almost everyone misreads.

A plain fetch on SparkProxy's Starter tier works out at roughly $0.0002 per request ($49 divided by 250,000 credits). Bright Data's dataset rate is up to $0.0025 per record. That looks like collecting it yourself is about twelve times cheaper per unit.

It is not, and the reason is the word "unit". A record is not a page. One product listing page may yield fifty records, in which case buying is cheaper per record than fetching. One record may require four page loads plus a JavaScript render, in which case it costs you 21 credits rather than 1. Before you quote any per-unit comparison in a business case, work out your own pages-per-record ratio on a sample of a hundred records. It is the single most load-bearing number in this decision and almost nobody measures it.

Four cost lines. They are not close to equal, and the one buyers focus on is the smallest.

Proxies and network. Published, predictable, and the cheapest line on the list. A flat concurrency plan or a credit-based API prices out at tens to a few hundred dollars a month for most workloads. See how much proxies cost for the full ladder by type.

Infrastructure. Schedulers, queues, workers, storage, monitoring. Real, bounded, and boring. Most teams over-engineer this and under-budget the next one.

Engineering time to build. A first working scraper for one site is a day or two. This is the number that appears in the business case, and it is the least relevant one, because a scraper's cost is not its build cost.

Engineering time to keep it working. This is the whole game. Sites change on their schedule, not yours. Selectors rot. Anti-bot layers get upgraded on a Tuesday. A pipeline covering twenty sources will have something broken most weeks, and the fix arrives through an on-call rotation you now have to staff. Our guide on detecting when a scraper is blocked exists because silent breakage is the expensive kind: data that keeps flowing while being quietly wrong costs more than data that stops.

The honest formulation of the whole decision is one line:

buy if   D  <  P + I + E

D = dataset or subscription price per month
P = proxy or scraping API spend per month
I = infrastructure per month
E = fully loaded engineering cost per month, build amortised plus maintenance

P and I are published numbers you can look up in ten minutes. D is a quote. E is the term everyone guesses at, and it is usually larger than P and I combined by an order of magnitude. Which means the build-versus-buy decision is essentially a question about headcount wearing a costume made of infrastructure.

An illustrative model, not a measurement from any test we ran. The assumptions are stated so you can replace them with your own.

The workload: 2,000,000 product records across five retail sites, refreshed monthly, two pages per record, no JavaScript rendering needed on three of the five sites.

The build side, using published SparkProxy rates:

That first row is instructive on its own. Rendering is what destroys a credit budget: 40% of pages consuming 77% of the credits. Before assuming you need a headless browser, test whether the data is in the initial HTML or in a JSON endpoint the page calls. Our notes on scraping API credit pricing go through where credits actually go.

The buy side, using Bright Data's published dataset rate:

2,000,000 records at up to $0.0025 each is up to $5,000 per delivery at the headline rate, before the refresh discounts shown on their pricing table and before any negotiation at that volume. Subscription tiers on the same page are labelled one-time, biannual (Save 25%), quarterly (Save 50%) and monthly (Save 80%), so the effective rate on a monthly refresh is materially lower than the one-time rate. Read the actual figure for your own size tier on their page rather than applying those percentages blind.

Now the decision, expressed as the only question that matters:

Two conclusions fall straight out, and they are not the ones people expect.

The proxy line never decides it. At $240 a month it is under 2% of the build total once a single part-time engineer is attached. Teams spend weeks comparing proxy vendors and minutes estimating E, which is precisely backwards.

The crossover is a headcount threshold, not a volume threshold. Somewhere around a quarter to a half of one engineer's time, buying wins on this workload. Doubling the record count moves the buy side linearly and the build side barely at all, which is why building gets better at scale and worse at small scale. The instinct that "we are too small to buy" has it exactly upside down.

Bright Data's dataset pricing table offers one-time purchase, biannual "Save 25%", quarterly "Save 50%" and monthly "Save 80%". Read that ladder as a message rather than a promotion.

The vendor is telling you, in the only language a pricing page speaks, that one-time dataset purchases are the worst deal they sell and they would rather you subscribed. They are right, and not only for their revenue. Web data decays. A product catalogue pulled in January describes a store that no longer exists by June. Teams that buy a one-time file almost always come back within two quarters, having paid the highest per-record rate on the page for data they then had to re-buy.

The practical guidance: if you catch yourself specifying a one-time purchase, interrogate the assumption. Either the data genuinely is static, in which case say so out loud and check it, or you are about to pay a premium for a decision you will reverse. The same logic applies in reverse to the build side. A scraper you plan to run once is a scraper you will run forever, and it should be budgeted as an ongoing service from day one.

Five failure modes, none of which show up in a sales demo.

Coverage is close but not right. The dataset has 90% of your source list. The missing 10% is the part your analysts care about, and it is missing because it is hard, which is why nobody sells it.

The schema is theirs. Their field definitions, their normalisation rules, their notion of what counts as one product. If your business logic disagrees, you inherit a translation layer that has to be maintained forever, which is a piece of engineering nobody costed.

Freshness is a range, not a promise. "Refreshed monthly" says when the pipeline runs, not when any given record was last verified. Ask for the distribution of record ages in a sample, not the headline cadence.

Your competitors have the same file. Commodity data creates commodity insight. If your product's edge is supposed to come from the data itself, buying it from a marketplace is a strange way to get an edge.

Minimums and tiers shape your question. A $250 minimum and jumps between 100K, 500K, 1M, 5M and 20M tiers mean you buy at the tier boundary rather than at your actual need. That is fine at scale and wasteful at small volumes.

Four, and they are all people problems wearing technical clothes.

Bus factor of one. The pipeline works because one person understands it. They go on holiday, a site changes, and nobody else can read the selector logic. This is the single most common way an in-house scraping effort dies, and it dies quietly, as a slow decay in coverage nobody is watching.

The maintenance tax is invisible until it is not. Nobody budgets for it because nobody bills for it. It shows up as an engineer's calendar filling with small fixes, and by the time it is visible as a line item it has been running for a year.

Legal and compliance review arrives late. Someone eventually asks what you are collecting, from where, under which terms, and whether personal data is involved. That review is cheaper to schedule than to receive. Our notes on ethical scraping and rate limiting cover the technical half of behaving well; the legal half needs your own counsel.

Scaling changes the architecture, not the budget line. A pipeline that works at five sources and 2M records is a different system at fifty sources and 200M. The rewrite lands as a surprise because the original estimate priced the first version. Building a distributed web scraper is the shape of the thing you eventually need.

Almost nobody stays at either pole, and the middle is not a compromise. It is usually the correct answer.

Buy the commodity layer. Company registries, standard catalogues, public filings, anything where the schema is settled and every buyer wants the same fields. Paying a marketplace rate for these is cheaper than any internal alternative and frees your engineers from maintaining scrapers that create no advantage.

Build the differentiated layer. The twelve sources nobody sells, the field your model needs that nobody extracts, the hourly refresh on the forty SKUs that matter. Run that on a scraping API or a flat proxy plan so the infrastructure decision stays boring, and spend your engineering attention on the extraction logic rather than on the network.

Then keep the two layers separable. The most expensive mistake in a hybrid is welding bought data and collected data into one pipeline with no seam, so that changing vendors means rewriting everything. Keep a normalisation boundary with a schema you own, and both sides become replaceable. The same argument, applied one layer down, is in proxy pool service build vs buy.

Week one: measure pages per record. Pull a hundred records by hand or with a throwaway script and count the page loads each one took, including the ones that needed rendering. This single ratio converts every per-request price into a per-record price and makes the whole comparison possible. Without it you are comparing two numbers that are not the same kind of thing.

curl -G "https://scrape.sparkproxy.io/api/v1" \
  -H "X-API-Key: YOUR_API_KEY" \
  --data-urlencode "url=https://example.com/category/widgets?page=1" \
  --data-urlencode "render_js=false" \
  --data-urlencode "premium_proxy=true" \
  --data-urlencode "country_code=US" \
  -D headers.txt -o page1.html

The X-Credits-Used header in headers.txt gives the real cost of that fetch, and the 1,000 free credits are enough to price a hundred records before you talk to anyone. If render_js=false returns the data you need, your per-record cost just dropped by 80%.

Week two: get two dataset quotes with your own source list. Not the marketplace listing, your list. The gap between the marketplace dataset and a custom collection quote is where the real decision lives, and vendors will tell you which of your sources are hard if you ask.

Week three: write down E honestly. Who maintains this, what fraction of their time, at what fully loaded cost, and who covers them when they are away. If the answer to the last question is "nobody", the build option is not actually available yet and the business case is fiction.

Week four: decide the layers, not the whole. Split your source list into commodity and differentiated. Buy the first, build the second, and set a review date six months out. The ratio will move, and a decision made once is a decision that will be wrong by next year.