Linguacorpus

Mean Opinion Score (MOS)

Short answer. MOS is the mean of listener ratings on a 1-to-5 scale, and on its own it is not evidence of anything. A MOS of 4.0 from 8 listeners carries a 95% confidence interval roughly 1.5 points wide, and repeated 8-listener studies of the same system land about 1.0 point apart. From 40 listeners that spread narrows to 0.45. Report the listener count and the interval next to the number, or it is not comparable to anything.

How to do it

What the number is made of

A MOS is an average of human judgments, so it inherits two kinds of noise: variation between listeners, and variation within one listener across trials. The first is what makes small panels unreliable, and it is larger than most people expect.

Listeners also disagree systematically. A panel of 8 recruited from one team rates differently from a panel of 8 recruited from a crowd platform, and that gap can exceed the difference between the two systems you are comparing.

The interval is the number you should quote

For a panel of n listeners, the standard error of the mean is the sample standard deviation divided by the square root of n. For a small panel use the t distribution rather than the normal: at 7 degrees of freedom the 95% multiplier is 2.365, not 1.96.

The run below shows the consequence. Eight listeners produce a MOS that moves by about 1.0 point across repeated studies. Forty listeners produce one that moves by about 0.45. The estimate improves as the square root of the panel size, which is a slow curve: going from 8 to 40 listeners, five times the cost, narrows the interval by only 2.24x.

When two MOS numbers are actually different

Two systems scored 4.1 and 3.9 by the same panel of 8 listeners are not distinguishable. Their intervals overlap almost completely. This is the most common error in TTS and voice-conversion evaluation: reporting a small difference from a small panel as an improvement.

The practical rule is to compare intervals, not means. If the two 95% intervals overlap, run more listeners or switch to a paired comparison — asking each listener which of two samples is better removes the between-listener variance entirely and is far more sensitive than two separate MOS scores.

The limits of the scale itself

The 1-to-5 scale is not linear. The gap between 3 and 4 is not the same perceptual distance as the gap between 4 and 5, and listeners avoid the extremes. A MOS of 4.5 on a synthetic voice does not mean it is 90% of a human.

Absolute MOS also drifts with the rating interface, the instructions, and whether listeners are paid per sample. It is comparable within one study. Across studies, treat it as an ordinal signal at best.

Code

Simulate the same listener population rated by panels of different sizes, then compute the t-based confidence interval for one 8-listener study.

import numpy as np

rng = np.random.default_rng(11)
opinions = np.clip(rng.normal(4.1, 0.8, 200000), 1, 5)

print("How much a MOS moves just from who happened to be in the room:")
print(f"{'listeners':>10}{'mean MOS':>11}{'95% of studies land in':>26}{'spread':>9}")
for n in (8, 15, 40, 120):
    means = np.array([rng.choice(opinions, n).mean() for _ in range(4000)])
    lo, hi = np.percentile(means, [2.5, 97.5])
    print(f"{n:>10}{means.mean():>11.2f}{f'{lo:.2f} to {hi:.2f}':>26}{hi - lo:>9.2f}")

sample = rng.choice(opinions, 8)
se = sample.std(ddof=1) / np.sqrt(len(sample))
t7 = 2.365                      # t value for 95% confidence at 7 degrees of freedom
print(f"\none 8-listener study: MOS {sample.mean():.2f} +- {t7 * se:.2f} "
      f"(95% CI {sample.mean() - t7 * se:.2f} to {sample.mean() + t7 * se:.2f})")
print(f"standard error {se:.3f}. Going from 8 to 40 listeners narrows the interval by "
      f"sqrt(40 / 8) = {np.sqrt(5):.2f}x, not 5x.")
  • numpy only. The listener population is simulated with a fixed seed, so the arithmetic is reproducible.
  • The observed MOS moves by about 0.98 across repeated 8-listener studies and about 0.45 across 40-listener studies — the square-root relationship in practice.
  • The last line uses 2.365 rather than 1.96 because an 8-listener panel has 7 degrees of freedom. Using 1.96 would make the interval about 17% narrower than it should be.

Where this goes wrong

Reporting a MOS without the listener count

A MOS of 4.2 means nothing on its own. The same underlying quality can produce 4.2 from a panel of 8 and 4.0 from a panel of 60, and only the second is evidence. The count and the interval belong in the same sentence as the number.

Using 1.96 for a small panel

The normal-approximation multiplier understates the interval whenever the panel is small. At 8 listeners the correct 95% multiplier is 2.365. Using 1.96 makes the interval about 17% narrower than it should be.

Comparing MOS across studies

Absolute MOS depends on the instructions, the interface, the listeners' language, and the sample mix. A system at 4.3 in one paper is not better than one at 4.1 in another. Compare within a study, or use a paired design.

Averaging away a bimodal panel

A MOS of 3.0 can come from a panel that uniformly said 3, or from half saying 5 and half saying 1. Those are different products. Look at the distribution before you report the mean.

Letting listeners rate the same system repeatedly

If a listener hears five samples from system A and one from system B, their internal scale shifts. A listener who has just rated four poor samples will rate the fifth differently from one who has rated four good ones, so the order you present samples in decides which listener they are. Balance sample order across listeners and check for per-listener bias before averaging.

When to buy the data instead of building the pipeline

Buy when you need the listening panel rather than the arithmetic: native speakers of a language you cannot staff, a panel large enough that the confidence interval is smaller than the difference you are trying to detect, or a blind comparison run by someone with no stake in the outcome. Build the analysis yourself — it is the code above — but treat the panel as the product, because it determines whether the number means anything. Budget for 40 listeners or more if you intend to claim an improvement.

How buying training data works →

More technical reference

All technical reference →

Need data for Mean Opinion Score (MOS)?

Tell us the language, the hours, and what the data needs to look like. You will get a real number and a real timeline — not a range. If we cannot source it well, we will tell you that instead.

  • Pilot batch before the full run, so problems surface early.
  • Consent documentation delivered with the data.
  • No medical or clinical data. No recorded telephone calls.

We reply within two business days. Your details are used only to answer this request. See our privacy policy.

Contact

Talk to a human

Send a specification and we will come back with a real number and timeline.

Submit a sourcing request

Or email hello@linguacorpus.com