JwumaMultilingual evaluation, why English-only testing misleads
· 2 min read · evaluation, languages
Model behaviour is frequently not consistent across languages, including safety behaviour. A guardrail that holds in English can fail entirely in a lower-resource language. Evaluating by translating an English test set does not detect this, because it tests translated English rather than how people actually write in that language.
The short version
- Safety guardrails can behave differently across languages.
- Translated test sets measure translated English, not native usage.
- Evaluation should be done by native speakers of the target language.
- Cultural appropriateness is separate from linguistic correctness.
- Lower-resource languages typically show the largest behavioural gaps.
Scoping something like this?
We would rather talk about the problem than quote against a volume figure.
See capabilitiesWhat actually differs between languages?
More than fluency. Refusal behaviour, hedging, willingness to speculate and safety guardrails are all learned largely from English-dominant data, and their transfer to other languages is uneven and untested.
The practical consequence is that a model certified safe on an English evaluation may not be safe in deployment for speakers of another language, and nothing in the English result indicates this.
Why does translating your test set not work?
Because translated prompts are not how native speakers write. They carry English syntax and framing, so they exercise the model in a way real users do not, and they systematically miss the phrasings that actually trigger failures.
They also cannot test cultural appropriateness at all, because the cultural content of the original is English.
What does good multilingual evaluation require?
Native speakers writing prompts natively, and native speakers judging the output. Both halves matter. A native prompt judged by a translation is only marginally better than no evaluation.
Judgement should cover cultural appropriateness separately from linguistic correctness. An answer can be grammatically perfect and locally inappropriate, and collapsing the two into one score hides it.
- Prompts written natively, not translated
- Judgement by native speakers of the language
- Cultural appropriateness scored separately
- Safety behaviour tested per language, not assumed
Where should you look first?
The lowest-resource language in your deployment footprint, because that is where the behavioural gap is largest and where the least prior evaluation exists.
It is also where finding qualified evaluators is hardest, which is why this tends to be deferred and why the gap persists.
People also ask
Do AI safety guardrails work the same in every language?
Often not. Guardrails learned largely from English-dominant data transfer unevenly, and a guardrail that holds in English can fail in a lower-resource language.
Can I evaluate a model by translating my English test set?
Not reliably. Translated prompts carry English syntax and framing, so they exercise the model differently from real native usage and miss the phrasings that actually cause failures.
Who should judge multilingual model output?
Native speakers of the target language, judging cultural appropriateness separately from linguistic correctness so that a fluent but locally inappropriate answer is not scored as correct.