Linguacorpus

Phoneme Recognition

Short answer. Phoneme recognition converts audio into a sequence of phoneme symbols — usually the 39-phone ARPAbet set for English — and it is scored with phoneme error rate (PER), the edit distance between the reference and hypothesis phone strings divided by the reference phone count. PER is not WER on a smaller alphabet: English speech carries roughly 3 to 5 phones per word, so one word error usually costs several phone errors, and a single phone substitution costs one ninth of a word. Stress marks are a second axis of error and must be scored by a stated convention.

How to do it

The label set has to be fixed before anything else

English phone inventories in common use: 39-phone ARPAbet (the TIMIT set), 48-phone IPA-derived sets, and 61-phone sets that keep the closure and release phases of stops separate. The 39-phone set is 15 vowels with three stress levels each (AA0, AA1, AA2) plus 24 consonants. CMUdict uses the same symbols, which is why it is the default for English.

Two conventions have to be declared. First, whether stress is scored: collapsing AH0, AH1 and AH2 to AH removes up to 20% of the reference labels in some sentences and typically improves PER by 3 to 8 points, so a comparison between a stress-scored and a stress-free number is meaningless. Second, whether the silence and closure symbols (SIL, CL, EPI) are counted in the denominator. Most published numbers exclude them, which is why your first run will look worse than the paper.

Where the reference comes from, and why it matters more than the model

A phoneme recognition evaluation needs a phone-level reference. There are two ways to get one. Forced alignment against a known transcript gives you boundaries and labels derived from the audio under a pronunciation dictionary, which is accurate but inherits the dictionary: if the speaker said "gonna" and the dictionary says "going to", the reference is wrong and the recognizer is punished for being right. Free phonetic transcription by a trained annotator avoids the dictionary but costs far more and produces lower inter-annotator agreement, typically 85-92% phone-level agreement between two trained transcribers on spontaneous speech.

The reference construction choice sets the ceiling on your measured accuracy. Report which one you used, and if you use forced alignment, report the dictionary.

Score with edit distance over phones, not letters or words

PER is Levenshtein distance with substitutions, insertions and deletions each costing 1, divided by the number of reference phones. The implementation is identical to WER; only the token unit changes, which is why you can reuse a WER library by joining the phone strings with spaces.

Frame-level accuracy is a different metric and is always higher, because a frame error requires the boundary to be wrong as well as the label. A model reporting 85% frame accuracy often reports 20-30% PER on the same data. Never compare the two, and never let a vendor quote frame accuracy when you asked for PER.

Read the error pattern, not just the number

A PER of 15% means nothing on its own; which 15% matters. Substitutions between vowels that differ only in stress are a different problem from substitutions between /s/ and /f/, and insertions in silence are a different problem from deletions of word-final consonants. Break the confusion down into a phone confusion matrix and look at the top ten cells.

The usual findings: vowel errors concentrate in unstressed syllables, where the acoustic difference is genuinely small; consonant errors concentrate in the same places a human listener would struggle (fast speech, low SNR); and insertions concentrate at word boundaries where a stop closure is released. If your test set is read speech at 16 kHz, expect the numbers to be much better than on spontaneous telephone speech, by 10 points or more.

Code

A small lexicon mapped to ARPAbet, then PER computed with strict stress scoring and with stress collapsed, side by side on the same four edits.

import jiwer

LEXICON = {
    'the':   ['DH', 'AH0'],
    'quick': ['K', 'W', 'IH1', 'K'],
    'brown': ['B', 'R', 'AW1', 'N'],
    'fox':   ['F', 'AA1', 'K', 'S'],
    'jumps': ['JH', 'AH1', 'M', 'P', 'S'],
    'over':  ['OW1', 'V', 'ER0'],
    'lazy':  ['L', 'EY1', 'Z', 'IY0'],
    'dog':   ['D', 'AO1', 'G'],
}

def to_phones(text):
    return [p for w in text.split() for p in LEXICON[w]]

def no_stress(tokens):
    return [t.rstrip('012') for t in tokens]

def per(ref, hyp):
    return jiwer.wer(' '.join(ref), ' '.join(hyp))

SENTENCE = 'the quick brown fox jumps over the lazy dog'
ref = to_phones(SENTENCE)
print('reference: %d phones over %d words' % (len(ref), len(SENTENCE.split())))

def edit(tokens, i, new):
    out = list(tokens)
    out[i] = new
    return out

cases = {
    'substitution IH1 -> EH1 (wrong vowel)': edit(ref, 4, 'EH1'),
    'stress error IH1 -> IH0 only':          edit(ref, 4, 'IH0'),
    'deletion: drop the final G':            ref[:-1],
    'insertion: extra AH0 after "the"':      ref[:2] + ['AH0'] + ref[2:],
}

for name, hyp in cases.items():
    strict = per(ref, hyp)
    loose = per(no_stress(ref), no_stress(hyp))
    print('%-38s strict PER %.3f  stress-free PER %.3f' % (name, strict, loose))

print('the same four edits as word-level WER: %.3f each (1 of %d words)'
      % (1.0 / len(SENTENCE.split()), len(SENTENCE.split())))
  • pip install jiwer. Everything else is the standard library.
  • Expected output: three of the four edits score 0.032 strict PER (1 error over 31 reference phones) and the stress-only edit scores 0.032 strict but 0.000 stress-free. That zero is the whole argument for declaring your stress convention before you quote a number.
  • The last line is the unit comparison: each edit costs 0.032 in phone terms and 0.111 in word terms, a factor of 3.4 on this sentence. On longer words the factor is larger.
  • Reference length is printed from the lexicon lookup, not hard-coded, so adding a word to LEXICON and to the sentence keeps the counts honest.

Where this goes wrong

Comparing PER across papers with different phone sets

A 39-phone ARPAbet result and a 61-phone TIMIT result are not comparable: the larger set splits stops into closure and release, which turns one error into two, and it changes the denominator. A paper reporting 18% PER on 61 phones is often equivalent to 13% on 39. Before you benchmark anything, match the inventory, the stress convention, and the silence policy of the number you are comparing against.

Using forced alignment as the reference for spontaneous speech

Forced alignment needs a transcript and a pronunciation dictionary, and spontaneous speech violates both. Disfluencies, reductions ("gonna", "kinda"), and mispronunciations are absent from the dictionary, so the aligner places the nearest legal phone sequence and the reference stops describing the audio. On conversational data this alone can add 8-12 points of measured PER that is a reference artifact, not a model failure.

Reporting frame accuracy as if it were PER

Frame accuracy counts each 10 ms frame and gives partial credit for near-misses at boundaries; PER counts each phone and gives none. The gap is routinely 50 points. If a number is above 80% and described as accuracy on a phoneme task, it is almost certainly frame-level, and it tells you nothing about how many phones were wrong.

Ignoring the denominator when silence is included

If your reference includes SIL tokens at every pause, the denominator grows and PER falls, without the recognizer improving. Worse, the effect scales with how much silence the corpus has, so a corpus of read sentences and a corpus of conversational speech will produce PER numbers that cannot be compared. Decide explicitly and state it.

Treating all phone errors as equally costly

Substituting AH0 for AH1 is inaudible; substituting S for F changes the word. An unweighted PER treats them identically, which is fine for comparing models on the same test set and misleading for predicting intelligibility. For a pronunciation-scoring or TTS product, weight the confusion matrix by perceptual distance, or evaluate on the downstream task instead.

When to buy the data instead of building the pipeline

Buy when you need phone-level annotation at a scale and agreement level that a small team cannot reach: word-level timestamps plus phone boundaries plus stress marks, on speech that forced alignment cannot handle. Buy when the language lacks a pronunciation dictionary you trust, because building the lexicon is usually the larger half of the project and it is not a one-time cost — it has to be validated against real speech. Building it yourself is correct when you have a transcript and read speech in a well-resourced language, since forced alignment is free and good. The purchase starts paying when the audio is spontaneous, the language is under-documented, or the evaluation has to survive an external review.

How buying training data works →

More technical reference

All technical reference →

Need data for Phoneme Recognition?

Tell us the language, the hours, and what the data needs to look like. You will get a real number and a real timeline — not a range. If we cannot source it well, we will tell you that instead.

  • Pilot batch before the full run, so problems surface early.
  • Consent documentation delivered with the data.
  • No medical or clinical data. No recorded telephone calls.

We reply within two business days. Your details are used only to answer this request. See our privacy policy.

Contact

Talk to a human

Send a specification and we will come back with a real number and timeline.

Submit a sourcing request

Or email hello@linguacorpus.com