JwumaHow to price an annotation project without being surprised
· 2 min read · procurement, commercial
Cost is driven by rubric complexity and annotator scarcity far more than by volume. Per-unit pricing looks comparable across vendors and hides both. The lines most first budgets omit are rubric development, calibration, adjudication of genuine disagreement and the rework caused by specification changes.
The short version
- Rubric complexity affects cost more than raw volume.
- Annotator scarcity is the dominant driver on specialist work.
- Per-unit pricing hides complexity and encourages silent shortcuts.
- Calibration and adjudication are real cost lines, not overhead.
- Mid-project specification changes are the largest common overrun.
Scoping something like this?
We would rather talk about the problem than quote against a volume figure.
See capabilitiesWhy does per-unit pricing mislead?
Because it makes two suppliers look comparable when the underlying work is not. A price per label says nothing about whether the label was produced by a calibrated annotator against a frozen rubric with adjudication on disagreement, or by whoever was available.
It also creates the wrong incentive. A supplier paid per unit under a fixed price has a direct interest in resolving ambiguity quickly and silently, which is precisely the behaviour that ruins datasets.
What are the missing budget lines?
Rubric development, which is real work and is usually done badly when unfunded. Calibration before production. Adjudication of genuine disagreement, which on subjective tasks is a meaningful proportion of items. Gold item creation and maintenance.
And rework. Specification changes mid-programme are the single largest common overrun, and budgeting zero for them assumes a first specification that is complete, which never happens.
- Rubric development and iteration
- Annotator calibration before production
- Adjudication of genuine disagreement
- Gold item creation and refresh
- Rework from specification change
How should scarcity change your model?
On expert work, the cost is practitioner time and there is no sourcing arrangement that makes a specialist clinician cheap. Budget from the professional rate upward rather than from a data-labelling rate.
The lever that does exist is throughput per hour of expert time, which comes from clear specifications and good tooling. That is worth investing in precisely because the underlying hourly cost is fixed.
What should you agree commercially?
How change is priced, before you need it. A programme with no agreed mechanism for specification change either freezes a specification everyone knows is wrong, or renegotiates under pressure.
Corpshore prices per project, per accepted task, per hour and with setup fees depending on the work, and the mix is settled with staff rather than forced into one model. What matters is that the mix matches the shape of the work rather than the shape of the invoice.
People also ask
What drives the cost of an annotation project?
Rubric complexity and annotator scarcity, far more than raw volume. Specialist work is priced by practitioner time, which no sourcing model reduces.
What do first-time data budgets usually miss?
Rubric development, annotator calibration, adjudication of genuine disagreement, gold item maintenance and rework caused by mid-programme specification changes.
Is per-unit pricing a good way to compare vendors?
It makes vendors look comparable while hiding whether labels were produced by calibrated annotators against a frozen rubric, and it rewards resolving ambiguity silently under fixed price.