Linguacorpus

Transcription Accuracy

Short answer. Transcription accuracy is measured on a held-out sample, not on the whole corpus, and the sample has to be stratified by the conditions you care about — SNR, speaker, duration, accent. A proportion carries a confidence interval: on 100 clips it is about 11 points wide, and on 1,600 clips about 2.5. Measuring on the portion used for tuning inflates the number, by about 3 points in the example below.

How to do it

Sampling, not scoring everything

Hand-checking a large corpus is not possible, so accuracy is always estimated from a sample. The estimate is only as good as the sample, and a convenience sample — the first hundred files, or the files that were easy to check — is biased in a direction nobody can quantify.

The fix is stratified random sampling. Divide the manifest by the conditions that plausibly affect accuracy, draw a fixed number from each stratum, then combine. Proportional allocation reproduces the corpus mix; equal allocation gives you enough clips in the rare conditions to measure them at all.

Why the training portion flatters you

Any corpus gets filtered and cleaned as it is built. Clips with unusable audio are dropped, clips with ambiguous speech are re-recorded, and what remains is better than the population it came from. Measure accuracy on that remainder and you are measuring the survivors.

The example below makes the effect explicit: accuracy is a function of SNR, and the training portion is the clips above 12 dB. The same system scores 92.5% on the whole manifest and 95.4% on the training portion. The 3-point gap is selection rather than performance.

Turning a count into an interval

Accuracy is a proportion, so it carries a confidence interval. The textbook normal approximation breaks down near 1.0, which is exactly where ASR accuracy lives, so use the Wilson interval: it stays inside 0 to 1 and behaves correctly at high accuracy.

Interval width scales as one over the square root of the sample size. Going from 100 clips to 1,600 — sixteen times the work — narrows the interval from 11.4 points to 2.5. Decide how precisely you need to know the number before you decide how many clips to check.

Reporting it so it means something

An accuracy figure should arrive with the sample size, the stratification, the definition of correct, and the confidence interval. "97% accurate" without those is not a measurement.

The definition of correct is the one that gets skipped. Is a clip correct if every word matches, if the word error rate is under 5%, or if a human would accept it for the downstream use? Those three produce three different numbers from the same annotations.

Code

Build a 4,000-clip manifest, show the selection effect on the training portion, then estimate accuracy from stratified samples of three sizes with Wilson intervals.

import numpy as np

rng = np.random.default_rng(4)
N = 4000
snr = rng.normal(14, 6, N)
dur = np.clip(rng.gamma(2.0, 1.2, N), 0.5, 20.0)


def accuracy_at(snr_db):
    """A plausible curve: 99% at 25 dB, 70% at 0 dB, flat below 0 dB."""
    return 0.70 + 0.29 * (1.0 - np.exp(-np.maximum(snr_db, 0.0) / 8.0))


full = accuracy_at(snr).mean()
training = snr > 12.0
print(f"clips in the manifest            {N}   SNR {snr.min():.1f} to {snr.max():.1f} dB")
print(f"true accuracy over the whole set {full:.1%}")
print(f"accuracy on the training portion {accuracy_at(snr[training]).mean():.1%}   "
      f"({training.sum()} clips, all above 12 dB)")
print(f"the gap is {accuracy_at(snr[training]).mean() - full:+.1%} points of pure "
      f"selection effect")


def wilson(k, n, z=1.96):
    p = k / n
    d = 1 + z * z / n
    c = (p + z * z / (2 * n)) / d
    h = z * np.sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / d
    return c - h, c + h


print("\nstratified test sample, 100 clips per SNR band:")
print(f"{'n':>7}{'accuracy':>11}{'95% interval':>22}{'width':>8}{'covers truth':>14}")
for n in (100, 400, 1600):
    sel = rng.choice(N, n)
    k = int(rng.binomial(1, accuracy_at(snr[sel])).sum())
    lo, hi = wilson(k, n)
    print(f"{n:>7}{k / n:>11.1%}{f'{lo:.1%} to {hi:.1%}':>22}{hi - lo:>8.1%}"
          f"{str(lo <= full <= hi):>14}")
  • numpy only. The manifest is simulated with a fixed seed, so the numbers below reproduce exactly.
  • The manifest spans -10.3 to 34.0 dB. True accuracy is 92.5% over the whole set and 95.4% over the 2,540 clips above 12 dB — a +3.0 point gap that is entirely selection.
  • The Wilson intervals narrow from 11.4 points at n=100 to 4.7 at n=400 and 2.5 at n=1,600, and all three cover the true value. That is the trade you are making when you choose a sample size.

Where this goes wrong

Measuring on the clips that survived cleaning

The cleaned corpus is a biased subset of the audio you started with, and accuracy measured on it overstates what the system will do on new, unfiltered traffic. Hold out a sample before cleaning and measure on that.

Using the normal approximation near 100%

At 97% accuracy the normal-approximation interval can extend past 1.0, which is impossible and makes the interval look tighter than it is on the side that matters. Use the Wilson interval.

A sample size chosen for convenience

A hundred clips gives an interval about 11 points wide. That is enough to detect a broken system and not enough to detect a five-point improvement, which is the size most changes actually are. Pick the sample size from the difference you need to see.

Stratifying by the wrong variable

Stratifying by file size when accuracy varies with SNR produces a well-balanced sample of the wrong thing. Choose the strata from a pilot measurement of what actually moves the error rate; a couple of hundred clips scored against two candidate variables is enough to see which one matters.

Reporting a single accuracy for a multi-condition corpus

The pooled number is dominated by the most common condition. Report accuracy per condition, or at minimum report the worst stratum next to the pooled figure.

When to buy the data instead of building the pipeline

Buy when you need the evaluation set rather than the pipeline: a held-out sample drawn across speakers and conditions your own traffic does not contain, annotated by more than one person so the reference itself carries a measured agreement rate. Build it yourself when you have production traffic, because your own audio is the most representative test set that exists and nobody can sell it to you. The useful purchase here is usually the reference transcript rather than the audio.

How buying training data works →

More technical reference

All technical reference →

Need data for Transcription Accuracy?

Tell us the language, the hours, and what the data needs to look like. You will get a real number and a real timeline — not a range. If we cannot source it well, we will tell you that instead.

  • Pilot batch before the full run, so problems surface early.
  • Consent documentation delivered with the data.
  • No medical or clinical data. No recorded telephone calls.

We reply within two business days. Your details are used only to answer this request. See our privacy policy.

Contact

Talk to a human

Send a specification and we will come back with a real number and timeline.

Submit a sourcing request

Or email hello@linguacorpus.com