Accented Speech Recognition

Short answer. Accented speech recognition is the same modeling problem as any other ASR, but the number that matters is not the pooled WER — it is the spread across accent groups. A system at 12% pooled WER can sit at 4% on the majority accent and 38% on the worst group, and the pooled figure hides that because the majority group contributes most of the words. Always report per-group WER with the reference word count for each group, and check whether the test set accent mix resembles the deployment population before you quote anything.

How to do it

You need an accent label per speaker, not per utterance

Accent is a property of a speaker, so the label belongs on the speaker record and propagates to their utterances. Two label sources are in common use. Self-reported background (the speaker states where they learned the language and where they live now) is cheap, scales to any corpus, and is noisy — people under-report or over-report depending on the context of the recording. Perceptual labels from trained listeners are stable and expensive, and they are what you want if the label is going to carry weight in a product decision.

The scheme has to be fixed before annotation starts. L1 background, region of origin, and current region are three different variables and they correlate imperfectly. "Indian English" as a single bucket spans speakers whose L1 is Hindi, Tamil, Bengali or Marathi, and those do not produce the same errors.

Compute per-group WER first, and the pooled number only as a summary

Pooled WER is a word-count-weighted average of the group WERs. That is a mechanical fact, and it has a consequence people miss: the pooled number moves when the test set composition moves, even if the system does not change at all. Two teams can evaluate the identical model on the identical audio and report pooled WERs three points apart because one of them sampled more L2 speakers.

So print the table. One row per group, with the group WER, the reference word count, and the utterance count. The pooled number goes at the bottom, labeled as weighted. If a group has fewer than about 1,000 reference words, the group WER is too noisy to act on, and the word count column is what tells you that.

Report the worst group, and the gap, as a first-class result

The pair (pooled WER, worst-group WER) is more informative than either number alone, and the difference between them is the fairness gap. A model that improves the pooled WER by 1.5 points while the worst group stays flat has improved the product for the users who were already served and not for the ones who were not. That is a legitimate engineering outcome, but it should be a decision rather than an accident, and it only becomes visible when the worst group is on the report.

Weighting the groups equally instead of by word count gives a third number. It answers a different question — "how does this system do across accent diversity" rather than "how will it do on this traffic mix" — and both are worth having. Publish which one you are quoting.

Re-weight the test set to the deployment mix before you decide

If the deployment population is 30% L2 speakers and the test set is 8%, the pooled number overstates performance for the actual users. Re-weighting is arithmetic: take the per-group WERs and apply the deployment proportions. The result is an estimate, not a measurement, but it is a much better estimate than the raw pooled figure, and the difference between the two is often larger than the difference between the two systems you are choosing between.

The complementary check is whether the accent groups in your test set are the ones in your deployment. A test set with four well-sampled L1 backgrounds says nothing about the fifth, and the fifth is usually the one with the fewest public hours.

Code

Per-group WER with word counts, the pooled and balanced summaries, and the re-weighted estimate under a realistic deployment mix.

import jiwer

DATA = {
    'US English': [
        ('turn the lights off', 'turn the lights off'),
        ('call me back tomorrow', 'call me back tomorrow'),
        ('what time does the store open', 'what time does the store open'),
        ('set an alarm for seven', 'set an alarm for seven'),
        ('play the news briefing', 'play the news briefing'),
        ('remind me to buy milk', 'remind me to buy milk please'),
    ],
    'Indian English': [
        ('turn the lights off', 'turn lights off'),
        ('call me back tomorrow', 'call me back tomorrow'),
        ('what time does the store open', 'what time the store open'),
    ],
    'Nigerian English': [
        ('turn the lights off', 'turn lights'),
        ('call me back tomorrow', 'call back tomorrow'),
    ],
}

stats = {}
for group, pairs in DATA.items():
    refs = [p[0] for p in pairs]
    hyps = [p[1] for p in pairs]
    words = sum(len(r.split()) for r in refs)
    stats[group] = (jiwer.wer(refs, hyps), words, len(pairs))
    print('%-18s WER %.3f   %3d reference words   %d utterances'
          % (group, stats[group][0], words, len(pairs)))

total_words = sum(v[1] for v in stats.values())
pooled = sum(v[0] * v[1] for v in stats.values()) / total_words
balanced = sum(v[0] for v in stats.values()) / len(stats)
worst = max(stats, key=lambda g: stats[g][0])

print('pooled WER (word-weighted)      : %.3f' % pooled)
print('balanced WER (equal group weight): %.3f' % balanced)
print('worst group: %s at %.3f  ->  gap %.3f over pooled'
      % (worst, stats[worst][0], stats[worst][0] - pooled))

mix = {worst: 0.40}
for g in stats:
    if g != worst:
        mix[g] = 0.60 / (len(stats) - 1)
print('WER under a 40%% worst-group deployment mix: %.3f'
      % sum(stats[g][0] * w for g, w in mix.items()))

pip install jiwer. No audio files are needed: the pairs are written out so the arithmetic is visible.
Expected output: US English 0.036 on 28 words, Indian English 0.143 on 14 words, Nigerian English 0.375 on 8 words. Pooled comes out around 0.120 because the US group contributes more than half the words; the worst group is 3x the pooled figure.
The re-weighted line is the one to read carefully. Moving the worst group from 16% of the words to 40% takes the estimate from 0.120 to about 0.20, without changing the system.
The word counts are deliberately small so the script stays short. In a real evaluation, a group with 8 reference words has a 95% interval wider than plus or minus 20 points, which is why the word count column belongs on the report.

Where this goes wrong

Quoting a group WER computed on a few hundred words

At 20% WER on 300 reference words, the 95% confidence interval is roughly 15% to 25% — wide enough that the "worst group" you identified may not be the worst group. Either report the interval, or pool enough data that the interval is narrower than the differences you are making decisions about. A common failure is ranking three accents by a few points on 200 words each and concluding something about the model.

Letting the majority accent decide the pooled number

A test set that is 85% one accent produces a pooled WER that is essentially that accent's WER with a small correction. Improving the minority accents by 10 points each moves the pooled number by less than a point, so the pooled metric cannot guide the work. Track the per-group numbers over time as separate series.

Using self-reported accent labels as ground truth for a product decision

Self-reported background answers "where did you learn the language," which correlates with but does not equal how a listener would classify the speech. Two speakers with the same answer can be perceptually different, and a speaker who moved at 12 and has lived in the target country for 20 years will often be labeled by their origin rather than by their speech. If the label is going into a fairness report, use perceptual labels from multiple listeners and report the listener agreement.

Assuming accent is the variable when it is the channel

In many corpora the L2 speakers were recorded on different equipment, in different rooms, or through a different codec than the L1 speakers, because they were collected in a different country at a different time. The measured "accent gap" is then partly a channel gap. The check is to compare accents within a single recording setup, or to compare L1 speakers recorded on the same equipment as the L2 speakers. If the gap shrinks, part of it was never about accent.

Treating accent as a nuisance to be normalized away

A model fine-tuned to erase accent variation can lose the phonetic detail that a downstream pronunciation or intelligibility product needs, and it can degrade for speakers whose accent was well handled already. The measurable version of this mistake is a model that improves worst-group WER while making the pooled number worse — which is sometimes the right trade and should be a stated decision.

When to buy the data instead of building the pipeline

Buy when your deployment population includes accents that your recording team cannot supply, which is the normal case for any product shipping outside one country. Buy when you need the accent label itself, with a stated labeling scheme and inter-annotator agreement, because that is a data product and not something you can derive from audio afterward. Buy when the evaluation has to be defensible — a per-group table with word counts and a documented labeling protocol is what a reviewer will ask for, and it cannot be reconstructed from an unlabeled corpus. If you are testing one specific accent that you have native speakers of on staff, record it yourself; a few hundred utterances from ten speakers is a day of work and gives you cleaner control over the channel.

How buying training data works →

More technical reference

Noise-Robust Speech Recognition

noise robust speech recognition
SpecAugment

specaugment
Room Impulse Response

room impulse response
Dereverberation

dereverberation
ASR Evaluation Metrics

asr evaluation metrics
Zero-Crossing Rate

zero crossing rate

All technical reference →

Need data for Accented Speech Recognition?

Tell us the language, the hours, and what the data needs to look like. You will get a real number and a real timeline — not a range. If we cannot source it well, we will tell you that instead.

Pilot batch before the full run, so problems surface early.
Consent documentation delivered with the data.
No medical or clinical data. No recorded telephone calls.