JwumaWhy your speech model fails on real multilingual speech
· 2 min read · speech, evaluation
Most speech systems assume one language per utterance. Real multilingual speakers change language mid-sentence, which produces silent failure: the system transcribes the dominant language and drops or garbles the rest. It is usually under-measured because monolingual test sets do not contain the phenomenon.
The short version
- Code-switching means changing language within a single utterance.
- It is normal speech across much of Africa, South Asia and Southeast Asia.
- Monolingual test sets cannot detect code-switch failure.
- Tagging requires annotators fluent in both languages, not one.
- Conventions for borrowed single words must be fixed explicitly.
Scoping something like this?
We would rather talk about the problem than quote against a volume figure.
See capabilitiesWhat does the failure actually look like?
Not an error message. The system transcribes the parts in the language it expects and silently mangles or omits the rest, producing output that looks plausible and is missing content.
Because the output is well-formed, downstream quality checks pass. This is why the problem is usually discovered by users rather than by evaluation.
Why is it under-measured?
Because the test sets do not contain it. A dataset assembled as clean single-language recordings, then split into train and test, cannot reveal a failure mode that requires mixed-language input to appear.
The first step is almost always building an evaluation set with genuine code-switched speech, before commissioning any training data. Measuring the gap is cheaper than assuming its size.
Who can actually annotate it?
Someone genuinely fluent in both languages. Not a fluent speaker of one with textbook knowledge of the other, because the boundaries are frequently ambiguous and resolving them requires native intuition in both.
This is the constraint that determines project feasibility. For a language pair where few people are fluent in both, the annotator pool is the binding limit, not budget.
What convention decisions matter most?
How to treat a single borrowed word inside an otherwise monolingual sentence. Whether an established loanword counts as a switch. How to mark morphological blending where a stem from one language takes an affix from another.
Each has a defensible answer and no universal one. What matters is choosing explicitly and freezing it, because annotators applying different answers across a corpus produce data that teaches nothing.
- Single borrowed words: switch or not?
- Established loanwords: switch or not?
- Morphological blends: how marked?
- Proper nouns: excluded from switching?
People also ask
What is code-switching in speech recognition?
Changing language within a single utterance. It is normal multilingual speech and breaks systems that assume one language per utterance, usually by silently dropping or garbling the unexpected language.
Why do standard test sets miss code-switch failures?
Because they contain single-language recordings. A failure that only appears with mixed-language input cannot be detected by a test set that has none.
Who can annotate code-switched speech?
Only annotators genuinely fluent in both languages, because boundary cases require native intuition in each. This annotator pool, not budget, is usually the binding constraint on such projects.