Score a multilingual model per language, or the headline will mislead you

A single error rate for a model that handles twelve languages is a weighted average whose weights nobody chose. The weights come from how many words each language contributes to the test set, which usually reflects the budget and the availability of data rather than the importance of the language.

The consequence is predictable. A language that is nine percent of the words can be thirty points worse than the rest and move the headline by less than a point. The model has a real defect, the number says the model is fine, and the defect is discovered in production.

Two aggregates, and they will disagree

Pool every word across all languages and divide: that is the micro average, and it answers the question of how many words would come out wrong on a stream of audio with this language mix. Compute a score per language and take the mean of those scores: that is the macro average, and it answers how the model does on an average language.

A worked set of five languages makes the gap concrete. Word shares of 60, 15, 12, 8 and 5 percent, with per-language error rates of 0.11, 0.12, 0.12, 0.14 and 0.45. The micro average is 0.13. The macro average is 0.19. Same five numbers, six points apart, and the entire difference is the weighting.

The sharper version of that fact: the language holding five percent of the words contributes 48 percent of the macro average. Neither aggregate is the truth. Micro tells you what a user of the current traffic mix experiences; macro tells you how the model performs per language. Publish both and put the word share on every row so the reader can see which one they are looking at.

The columns that earn a place in the scorecard

One row per language, plus a total row, with the following fields. Each one answers a question that the error rate alone cannot.

• Words and utterances — the weight of the row, and the size of the sample behind it.
• Distinct speakers — a language with 20,000 words from three speakers is a speaker study, not a language result.
• Error rate with an interval — the point estimate and the uncertainty, computed per slice rather than borrowed from the total.
• Substitutions, deletions and insertions as shares — a deletion-heavy slice usually means the model is giving up or being cut off by a length limit.
• Character error rate — for scripts where word boundaries are not stable, this is the number that is comparable to another system.

Forced language versus detected language

If the system identifies the language on its own, run the set twice: once letting it detect, and once forcing the reference language. The difference between the two runs is the cost of language identification errors, and it is not spread evenly. It concentrates in the smaller languages and in short utterances, where there is not enough audio for a reliable decision.

Reporting only the detected-language number mixes two failure modes into one figure. Reporting only the forced number hides a failure that real users will hit on every short utterance in a language the system does not expect. The pair is the useful unit, and the gap between them is worth a line in the release notes.

One pattern to watch: if the detected-language score degrades sharply in the under-two-second bucket while the forced score holds, the model is fine and the identifier is not. That is a different team, a different fix, and a different conversation than an acoustic accuracy problem.

Which slice does an utterance belong to

The language of the audio, the language of the reference text, and the first language of the speaker are three different things, and a manifest with a single language column mixes them. An English utterance spoken by a Spanish-first-language speaker is English audio with an English reference, so it belongs in the English slice, and if accent matters it also belongs in an accent breakdown. If that column is populated by the speaker first language, the English slice silently becomes an accent slice.

Utterances that switch language inside the sentence deserve their own row rather than an assignment to the majority language. Split them across two slices and the boundary errors disappear into two rows that both look acceptable, which is the opposite of what a scorecard is for.

Whichever convention you choose, write it above the table. One sentence prevents the argument that otherwise happens when two teams compare two scorecards that use the column differently.

Compare slices with the interval, not the point estimate

Smaller languages have fewer words and fewer speakers, so their intervals are wider. A ten-point gap between a well-resourced and a low-resource language may be entirely inside the interval, or entirely real, and the only way to know is to compute the interval per slice.

When errors cluster by speaker, resample speakers rather than utterances to build that interval. Resampling utterances treats every word as independent evidence and produces an interval that is too narrow, which turns noise into a finding.

And do not read an error rate across languages as if the unit meant the same thing in each. Word error rates are not comparable across writing systems, so a per-language trend over time is a legitimate comparison while a per-language ranking at one point in time is not.

Gate the release on the worst slice

The headline is a convenience for the reader who will not scroll. The scorecard is what the model is, and the release decision should hinge on the worst slice that matters commercially rather than on the average.

A gate of that kind needs one more decision made in advance: which slices are protected. A language that is nine percent of the test words can be a hundred percent of the revenue, and only the business can say so. Once those slices are named, the scorecard turns a single number into a policy.