Linguacorpus

Multilingual Speech Recognition

Short answer. A pooled multilingual WER weights every word equally regardless of language, so a language with ten times the word count dominates the headline. In a five-language set where English holds 59% of the reference words, a language failing at 65% WER moves the pooled number by less than one point. Report per-language WER next to the pooled figure, and run a leave-one-language-out pass to see which language is actually setting it.

How to do it

How a language disappears inside a pooled number

Pooled WER is total edits divided by total reference words. The denominator is the sum across languages, so each language contributes in proportion to its word count rather than to its importance to you.

The run below is a five-language set where English holds 48,000 reference words and Tagalog holds 1,200. Tagalog fails at 65.0% and Swahili at 66.7%. The pooled headline is 12.3%, and the unweighted mean of the five per-language scores is 35.8%. Both are arithmetically correct and only one of them describes the product.

Leave-one-out tells you what sets the headline

Remove a language from the pool, recompute, and see how far the headline moves. Dropping English moves it 7.0 points. Dropping Tagalog moves it 0.8 points. That asymmetry is a direct measure of how much each language controls the number you report.

It cuts the other way too. A language can be failing so badly that it is the only thing holding the headline up, and removing it makes the system look better without anything having improved.

Equal weighting per language is also wrong

Averaging the five per-language WERs gives 35.8% here, which weights a 900-word language the same as a 48,000-word one. That over-corrects rather than fixing anything.

Neither number is the truth. The truth is a set of per-language numbers plus a decision about which languages your product is judged on. Report that decision explicitly instead of encoding it by accident in an aggregation rule.

The failure modes an aggregate cannot show

A multilingual model usually fails in one of three ways: uniformly mediocre across languages; good on high-resource languages and bad on low-resource ones; or good at everything except languages that are acoustically or orthographically close, which it confuses with each other.

  • Only the per-language breakdown separates these three.
  • The third case is invisible in any aggregate: a model that transcribes Hindi audio as Urdu produces a bad WER for both and no distinctive signal in the pooled number.
  • If the model also does language identification, score that separately. A language-ID error shows up as a catastrophic WER in one language and an artificially good one in another.

Code

Five languages with realistic word counts and error rates, showing how far the pooled WER sits from the per-language picture.

import numpy as np

LANGUAGES = {
    # language: (reference words, substitutions, deletions, insertions)
    "English": (48000, 2100, 600, 900),
    "Spanish": (22000, 1500, 400, 500),
    "Hindi":   (9000, 1400, 500, 700),
    "Tagalog": (1200, 360, 180, 240),
    "Swahili": (900, 270, 150, 180),
}

total_ref = sum(v[0] for v in LANGUAGES.values())
total_err = sum(sum(v[1:]) for v in LANGUAGES.values())

print(f"{'language':<10}{'ref words':>11}{'errors':>9}{'WER':>8}{'share of pool':>15}")
for lang, (n, s, d, i) in LANGUAGES.items():
    print(f"{lang:<10}{n:>11,}{s + d + i:>9,}{(s + d + i) / n:>8.1%}{n / total_ref:>15.1%}")

per_lang = [sum(v[1:]) / v[0] for v in LANGUAGES.values()]
headline = total_err / total_ref
print(f"\npooled WER, every word weighted equally  {headline:.1%}")
print(f"unweighted mean of the five languages    {np.mean(per_lang):.1%}")

print("\nleave-one-out pooled WER, to see which language sets the headline:")
for lang in LANGUAGES:
    n = total_ref - LANGUAGES[lang][0]
    e = total_err - sum(LANGUAGES[lang][1:])
    print(f"  without {lang:<8}{e / n:>7.1%}   (headline moves {(e / n - headline) * 100:>+5.1f} points)")
  • numpy only. The counts are literals at the top so the arithmetic can be checked by hand.
  • Per-language WER runs from 7.5% for English to 66.7% for Swahili. The pooled figure is 12.3% and the unweighted mean is 35.8% — a 23-point spread between two defensible aggregations.
  • English is 59.2% of the pool, so removing it moves the headline 7.0 points. Tagalog is 1.5% of the pool and moves it 0.8 points despite failing at 65%.

Where this goes wrong

Reporting one WER for a multilingual model

A single number over a mixed-language test set is dominated by whichever language has the most words, which is usually the one you are least worried about. In the example on this page the pooled figure is 12.3% while four of the five languages sit above it and one is at 66.7%. Per-language numbers are not optional here.

Assuming a large pooled denominator means a stable metric

A large denominator reduces sampling noise and does nothing about the weighting problem. A 12.3% pooled WER over 81,100 words can hide a language at 65%.

Not measuring language identification separately

If the model detects the language as well as transcribing it, a language-ID error shows up as a catastrophic WER for one language and an artificially good one for another. Score language identification on its own, as its own accuracy figure, before you look at any WER.

Using one test set translated across languages

The same sentences read in five languages are not five test sets. They share content and often share readers, so the errors correlate and the per-language numbers are not independent evidence.

Comparing a low-resource WER to a high-resource one

A 25% WER on a language with 20 hours of training data and a 25% WER on a language with 5,000 hours are not the same result. The comparison only means something next to the training data volume.

When to buy the data instead of building the pipeline

Buy when the language coverage is the problem: a language with no public corpus, or a set of languages that has to be evaluated under the same test conditions so the per-language numbers are comparable. Buy when the speakers have to be native and recruitment is the bottleneck, which for low-resource languages it usually is. Build it yourself when you are adding a well-resourced language to an existing pipeline and can source the audio from your own traffic. Ask for the per-language word counts with the delivery.

How buying training data works →

More technical reference

All technical reference →

Need data for Multilingual Speech Recognition?

Tell us the language, the hours, and what the data needs to look like. You will get a real number and a real timeline — not a range. If we cannot source it well, we will tell you that instead.

  • Pilot batch before the full run, so problems surface early.
  • Consent documentation delivered with the data.
  • No medical or clinical data. No recorded telephone calls.

We reply within two business days. Your details are used only to answer this request. See our privacy policy.

Contact

Talk to a human

Send a specification and we will come back with a real number and timeline.

Submit a sourcing request

Or email hello@linguacorpus.com