JwumaRLHF preference data, how to specify it so it is worth having
· 2 min read · specification, quality
Useful preference data requires a written rationale with every judgement, not just a ranking. Rankings alone tell a model very little; the reason is what carries the lesson. Specify the rubric, freeze it per batch, calibrate annotators against it before production and record genuine disagreement rather than resolving it by majority.
The short version
- A ranking without a written rationale carries little training signal.
- Annotators should be calibrated against the rubric before production work.
- Rubrics are frozen per batch so judgements are comparable.
- Genuine disagreement is recorded and adjudicated, not averaged away.
- Rationales that merely restate the ranking should be rejected in QA.
Scoping something like this?
We would rather talk about the problem than quote against a volume figure.
See capabilitiesWhy insist on written rationale?
Because a preference on its own is nearly information-free. Ten annotators preferring response B tells a model that B was better in some unspecified way. Ten annotators preferring B because it declined to speculate about a medication dose is a lesson that generalises.
It also makes quality checkable. A ranking cannot be audited; a rationale can be read, and a rationale that just restates the ranking in other words is visibly empty.
What does calibration involve?
Running annotators through a set of items with known correct judgements under the rubric, before they touch production data and correcting the misreadings that surface.
Skipping this is a false economy that shows up two weeks later as unusable agreement scores. The misreadings are always there; calibration determines whether you find them before or after paying for the batch.
How should disagreement be handled?
Recorded. On alignment work particularly, a genuine split among calibrated annotators is a finding about the prompt, not an error to be smoothed out.
Ask your supplier what they do with splits. Resolving by majority and reporting clean agreement is common and it destroys the most valuable rows in the dataset.
What should QA actually reject?
Rationales that restate the ranking, rationales that cite a criterion not in the rubric, and rankings inconsistent with the rationale attached to them. That last one is more common than it sounds and is invisible to any check that only looks at rankings.
It is worth specifying these rejection criteria explicitly rather than assuming a supplier applies them.
People also ask
What is RLHF preference data?
Human judgements about which of two or more model outputs is better, used to train a model toward preferred behaviour. Useful preference data includes a written rationale for each judgement.
Why does written rationale matter in RLHF?
Because a bare ranking carries almost no signal. The reason a response was preferred is what generalises, and it is also the only part of the judgement that can be audited.
Should annotators be calibrated before production RLHF work?
Yes. Calibration against the frozen rubric surfaces misreadings before you pay for a batch rather than after, and it is the main determinant of whether agreement scores end up usable.