How to Allocate Proxy Costs Across Teams and Clients

Allocate along whichever resource is actually scarce, and split the rest evenly. If concurrency is your binding constraint, charge for reserved threads. If you are on a credit plan, charge for credits consumed. If nothing is scarce because the plan is comfortably oversized, stop building a cost model and divide by headcount, because the allocation will cost more in argument than it recovers in accuracy.

Then add two rules that matter more than the model you chose. Attribute usage at the moment of the request, not from a spreadsheet at month end. And publish the numbers for at least one quarter before anybody is billed on them.

Every proxy plan sells one of four units, and none of them is the unit your finance system wants. The mechanics of each are in datacenter proxy pricing models and the current market rates in how much proxies cost.

The mismatch is the whole problem. On a flat concurrency plan the marginal cost of one more request is zero, so a usage-based allocation is charging for something that did not cost anything. On a per-GB plan the marginal cost is real and page weight drives it, so a per-request allocation charges the team scraping lightweight JSON the same as the team pulling image-heavy product pages. If that is your situation, allocating by bytes and pairing it with reducing proxy bandwidth costs will recover more than any billing policy.

The honest framing for internal conversations: you are not recovering a marginal cost, you are apportioning a shared fixed cost. Say that out loud early. It changes the argument from "prove I caused this" to "agree a fair share", which is a much shorter meeting.

The fourth is what most organisations converge on, and the reason is political rather than technical. A pure metered model makes every month's number a surprise, which finance dislikes. A pure even split makes the largest consumer invisible, which everybody else dislikes. A fixed base that covers the plan's floor plus a variable component that tracks consumption satisfies both, and the split between them is a policy decision rather than a calculation.

One model to avoid outright: allocating by "value delivered" or by revenue attributable to the data. It sounds fair for about a week and then becomes a negotiation with no ground truth.

Here is why the choice matters more than the arithmetic. One agency, one SparkProxy Boost plan at $240 a month for 500 threads with unlimited bandwidth, six clients. The figures below are an illustrative construction, not measurements from any real account.

Under reserved capacity, each client is allocated a share of the 500 threads, and the plan cost divides at $240 divided by 500, which is $0.48 per thread per month.

Under metered usage, the same clients consumed 3,000,000 requests between them, so the plan cost divides at $240 divided by 3,000,000, which is $0.08 per thousand requests.

Same invoice, same month, same total. Foxtrot pays $19.20 under one model and $4.00 under the other, a factor of 4.8. Nobody is cheating and no arithmetic is wrong. The two models simply answer different questions: reserved capacity asks what you made us hold available, metered usage asks what you actually took.

Which one is right depends on why you are on Boost rather than Core. If you upgraded because peak concurrency demanded it, the cost was caused by whoever needed the peak, and reserved capacity is correct. If you upgraded because total volume grew, metered usage is correct. Working out which constraint you are actually on is the subject of concurrent connections in proxies and how many proxies you need.

Notice the second-order effect too. Under reserved capacity Foxtrot has an incentive to hand back threads it does not use, which frees real capacity. Under metered usage Foxtrot has an incentive to collect less data, which may be exactly the wrong outcome if that data is valuable.

Attribution built at month end from logs is always incomplete, and the gaps always get assigned to whoever complains least. Build it at the point of the request instead.

On the SparkProxy Scraping API there is a parameter for exactly this. tag takes a string of up to 128 characters using alphanumerics plus _ - . : @ /, attaches it to the request, and echoes it back in the response so you can group and filter by project, client or feature.

curl -G "https://scrape.sparkproxy.io/api/v1" \
  -H "X-API-Key: YOUR_API_KEY" \
  --data-urlencode "url=https://example.com/catalog/page/1" \
  --data-urlencode "render_js=false" \
  --data-urlencode "json_response=true" \
  --data-urlencode "tag=client/foxtrot:catalog-sync"

With json_response=true the envelope carries job_id, status_code, duration_ms, credits_used and the tag you sent, which is everything an allocation needs in one record. Write those five fields to your own store as each job completes and the month-end report becomes a group-by rather than an investigation.

Design the tag taxonomy before you use it, because you will not be able to fix history. A workable shape is three segments: who it is for, what it is, and which environment it ran in, for instance client/foxtrot:catalog-sync@prod. Keep the separator characters consistent, never put anything confidential in a tag, and never reuse a tag value for a different purpose after a client leaves.

On a proxy plan rather than the API there is no per-request envelope, so the boundary you get is the credential. Give each team or client its own sub-user and its own whitelist slot so that traffic is separable at the source, which is covered in sub-users and credential rotation and IP whitelisting. Whitelist slots are capped per tier, 5 on Starter, 10 on Core, 15 on Boost and 25 on Plus, so the plan you buy also sets a ceiling on how many parties you can cleanly separate that way. Six clients on Boost fit comfortably. Twenty do not.

Whatever you do, keep an explicit unattributed bucket and charge it to the platform, never to the largest consumer. If unattributed usage passes roughly a tenth of the total, you have an instrumentation defect rather than a billing disagreement, and no allocation policy will fix it.

Every well-run plan carries capacity nobody is using, on purpose, so that a burst does not queue behind another team's backfill. That headroom is real and it costs real money, and pretending otherwise is how allocation models lose credibility.

Take the same Boost plan. Allocate all 500 threads and you have no reserve. Allocate 400 and hold 100 back, and the cost per allocated thread rises from $0.48 to $0.60, a 25% increase, because the same $240 is now spread across fewer billable units.

That 25% is not overhead in the pejorative sense. It is the price of every team's jobs finishing on time, and the small consumers benefit from it as much as the large ones, arguably more, since they are the ones who would get queued behind somebody else's backfill. Two defensible ways to handle it:

Load it into the unit rate, as above, so every allocated thread carries its share silently. Simple, and invisible to the teams, which is both the advantage and the disadvantage.

Bill it as a separate platform line, split evenly, so the headroom shows up as its own number. Harder to explain the first time, easier to defend every time after that, and it makes the cost of reliability an explicit decision rather than a hidden markup.

Either is fine. Smearing it proportionally onto the largest consumer is not, because it makes growth look more expensive than it is and quietly discourages the teams doing the most valuable work.

If you are reselling or rebilling a Scraping API plan, this one is worth a note in the policy document.

SparkProxy's credit tiers are Starter at $49 for 250,000 credits, Growth at $99 for 1,000,000, Pro at $249 for 3,000,000 and Scale at $599 for 8,000,000. Divide through and the blended cost per credit is:

• Starter: $0.000196
• Growth: $0.000099
• Pro: $0.000083
• Scale: $0.0000749

That is a fall of about 62% from the entry tier to the largest one. In practical terms, a thousand JavaScript-rendered fetches costs 5,000 credits, which is about $0.50 on Growth and about $0.37 on Scale, roughly 24% cheaper for an identical job.

The implication for rebilling: if you set a client-facing rate card on the tier you were using when you wrote it, your margin grows every time you upgrade, without anyone deciding that it should. That is not dishonest, but it should be a choice. Either pass the tier improvements through at review time, or state plainly that your rate is a service price rather than a cost pass-through. Do not describe it as "at cost" and then keep the tier discount. The mechanics of what each request type consumes are in scraping API credit pricing.

The single most effective thing you can do before billing anybody is to not bill them for a quarter.

Publish a monthly report showing each team or client what they consumed and what it would have cost under the proposed model. Change nothing about the money. Three things happen, reliably:

The instrumentation gaps surface while they are still free to fix. Somebody always discovers that a scheduled job has been tagged to the wrong project since it was written.

Teams self-correct before any invoice exists. A team that sees it is generating 40% of the requests usually finds a cache, a conditional request or a JSON endpoint within a fortnight, which is a better outcome than collecting money from them.

The model gets argued about in the abstract rather than in a dispute over a specific invoice, which is a far more productive conversation and produces a policy people will actually defend afterwards.

Run showback for a quarter, then switch on chargeback with the model that survived the argument. If nothing changed during showback, that is useful evidence that the allocation is not worth doing at all.

Allocation exists because pooling is cheaper. Once it stops being cheaper, stop allocating.

The test is not "does this client's share exceed the cheapest plan". It is: does their allocated share exceed the price of the smallest standalone plan that meets their peak concurrency? Those are different questions and the second one is the correct one.

Run it on Alpha from the worked example. Alpha's metered share is $96 a month and it reserves 150 threads. The smallest standalone SparkProxy plan that covers 150 threads is Core at $140 a month, since Starter carries 100. So Alpha costs $96 inside the pool and $140 outside it, and pooling saves $44 a month. Alpha stays.

Now imagine a client needing 600 threads whose allocated share came to $500. The smallest plan covering 600 threads is Plus at $440. That client is now paying more for a share of the pool than a dedicated plan would cost, so they get their own plan and everybody's arithmetic gets simpler.

Three non-financial reasons override the arithmetic entirely:

• Isolation requirements. If a client contractually requires that their traffic never shares infrastructure with another client, the decision is made.
• Invoice requirements. Some clients need the provider's own invoice in their name. A share of yours will not satisfy procurement.
• Blast radius. If one client's behaviour can get an address range flagged in a way that breaks everyone else, separating them is a reliability decision, not a billing one.

For very small standalone work, proxies for freelance scraping projects covers the one-person version of this decision.

Four practical rules, learned the expensive way by a lot of agencies.

Do not invent a unit you do not buy. If your plan is unlimited bandwidth, do not bill a per-gigabyte rate. The number is fabricated, a client who learns your plan terms will notice, and you have created an argument you cannot win. Bill for the service, or bill a share of a real plan cost. Background on why that unit does not exist on a flat plan is in whether unlimited bandwidth is worth it.

Decide pass-through or marked-up, and say which. Both are legitimate. Presenting a marked-up figure as "at cost" is not, and it is the kind of thing that surfaces during a contract review at the worst possible time.

Keep the line stable enough to be boring. A charge that moves by a few dollars every month generates more finance questions than it recovers in revenue. Banding clients into a small number of tiers and reviewing quarterly beats a precise figure that changes constantly.

Show the basis, not the vendor's pricing. A client is entitled to know what they are paying for and how it was calculated. They are not automatically entitled to your supplier's rate card, and publishing it constrains your own negotiating position at renewal.

Six steps, thirty minutes, once a month.

1. Pull the plan invoice total. One number, from the provider.
2. Pull your own usage totals by tag or sub-user. Requests, credits, or threads reserved, whichever your model uses.
3. Compute the unattributed residual. Total consumption minus the sum of attributed consumption. Log it every month so you can see the trend rather than one month's figure.
4. Apply the model. Fixed component split by policy, variable component by share.
5. Check three sanity conditions. The allocations sum to the invoice. No allocation is negative. No client moved more than a set threshold, say 30%, against last month without a known cause.
6. File the working, not just the answer. Keep the inputs alongside the output. When somebody queries a figure six months later, the question is always "what was the usage", and a number with no inputs behind it cannot be defended.

Step five is the one that earns its keep. A client whose allocation jumps 80% in a month has either changed their workload, which you should know about before they do, or exposed a bug in your attribution, which you want to find before they do.