JwumaBuilding a domain expert panel for medical, legal and financial evaluation
· 2 min read · expert data, evaluation
An expert panel needs three things specified before recruitment: the exact qualification required, how that qualification will be verified and the calibration set the panel will be aligned against. The verification question is the one most buyers skip, and it is where the difference between suppliers actually lies.
The short version
- Specify the exact qualification, not a general seniority level.
- Ask whether credentials are register-checked or document-checked.
- Panels need calibration against a shared set before production work.
- Practising status matters as much as the original qualification.
- Panel size should follow the disagreement rate, not a fixed number.
Scoping something like this?
We would rather talk about the problem than quote against a volume figure.
See capabilitiesWhat should you specify before recruitment starts?
The precise qualification, including whether currently practising is required. A retired specialist and a practising one are different things, and which you need depends on whether current practice norms matter to your task.
Also specify the calibration set. A panel of individually excellent practitioners with no shared calibration produces high-quality individual judgements that do not aggregate into a usable metric.
What is the verification question nobody asks?
How the credential was checked. There is a real difference between someone reading a certificate and someone confirming the person appears on the issuing body's public register, and many suppliers describe both as verified.
Ask specifically what happens for professions where no public register exists. The honest answer is that the check is weaker there, and a supplier who says so is more trustworthy than one who claims uniform verification.
- Is the credential register-checked or document-checked?
- Is the check dated and recorded?
- What happens where no register is published?
- Is current practising status confirmed?
- Is the distinction visible to you per panellist?
How big should a panel be?
Follow the disagreement rate rather than picking a number. Start small, measure how often qualified panellists disagree on your actual task, and size up only on the axes where disagreement is high.
Uniform panel sizes across all task types waste expert time on the questions where three practitioners always agree, and under-sample the ones where they do not.
People also ask
How do I verify an expert panel's credentials?
Ask whether credentials are checked against the issuing body's public register or only read as documents, whether the check is dated and what happens for professions with no published register.
Does a domain expert panel need calibration?
Yes. Without a shared calibration set, individually sound expert judgements do not aggregate into a consistent metric.
How many experts should be on a panel?
Size it from the measured disagreement rate on your actual task rather than a fixed number, and add panellists only on the axes where qualified experts genuinely disagree.