JwumaInter-annotator agreement, what number is actually good enough
· 2 min read · quality, specification
There is no universal threshold. High agreement on an easy task means nothing, and low agreement on a genuinely contested task may be the finding rather than a defect. Ask for agreement broken down by slice and by annotator rather than one aggregate number, because the aggregate reliably hides the part that is broken.
The short version
- A single aggregate agreement score hides slice-level failures.
- High agreement on a trivial task carries no information.
- On subjective tasks, systematic disagreement is often the signal.
- Agreement should be reported per slice and per annotator.
- Chance-corrected measures matter when class distribution is skewed.
Scoping something like this?
We would rather talk about the problem than quote against a volume figure.
See capabilitiesWhy is one number not enough?
Because datasets are not uniform. Agreement of 0.85 overall can comfortably contain a slice at 0.4 that happens to be the slice you actually care about, and nothing in the headline number tells you it is there.
This is the single most common way a dataset passes acceptance and then fails in training. Ask for the breakdown before you accept, not after.
When is disagreement actually the answer?
On genuinely subjective judgements. If half of qualified annotators consider a response acceptable and half do not, forcing a single label destroys the most informative thing in the batch.
For alignment and safety work particularly, a well-run supplier records the split and routes it to adjudication with the trail preserved, rather than resolving it by majority and reporting clean agreement.
What should you ask a supplier to report?
Agreement per slice, agreement per annotator and a chance-corrected measure where class distribution is skewed. Raw percentage agreement on a task where ninety percent of items share one label tells you almost nothing.
Also ask how reviewers themselves are measured. A supplier that measures annotators but not reviewers has an unexamined single point of failure sitting above every label in the dataset.
- Agreement per data slice
- Agreement per annotator
- A chance-corrected measure on skewed distributions
- How reviewers are themselves measured
- How genuine splits are recorded and adjudicated
People also ask
What inter-annotator agreement score should I require?
There is no universal number. What matters is agreement broken down by slice and by annotator, and whether the task is one where disagreement is legitimate signal rather than error.
Is high agreement always good?
No. High agreement on an easy task carries no information, and can also indicate annotators converging on a shared misreading of the rubric rather than genuinely agreeing.
Should disagreement always be resolved?
Not on subjective tasks. Systematic disagreement between qualified annotators is often the most useful information in a batch, and flattening it into one label discards it.