Measuring dataset diversity beyond the total hour count

Totals hide distributions

A dataset is sold by its total, and the total is the one number that cannot describe it. Two thousand-hour corpora — one from twelve speakers recorded in a single studio over a month, one from six hundred speakers across nine countries — are not the same product, are not interchangeable in training, and are routinely offered under the same headline figure.

Diversity is not a feeling about a dataset. It is a set of distributions, one per axis that matters for the intended use, and each one can be counted. The work is deciding which axes matter and reporting all of them, including the ones that look unflattering.

The axes worth counting

The axis list below is written for speech corpora; the same shape applies to text and video, with the modalities swapped.

• Speakers: the distinct count, and the distribution of hours across them — the largest share held by any one speaker, or a concentration figure. Twelve speakers where one holds half the hours is a different asset from twelve speakers at roughly even shares.
• Sessions and recordings: how many separate recording events, and the length distribution. One hundred ten-minute sessions and one hundred ten-hour sessions produce the same hours and very different training behavior.
• Acoustic conditions: hours per environment type, per noise band, per microphone or device. A corpus claiming noisy coverage whose noisy hours sit 90% in one cafe has exactly one noise profile.
• Speaker background: accent or dialect region, age band, gender, and any other axis the deployment cares about, reported as a table rather than as prose.
• Speaking style: read, scripted, spontaneous, conversational. A corpus whose spontaneous hours are actually scripted sentences read with feeling has no spontaneous hours.
• Text-side axes, for corpora with transcripts: topic, domain, and vocabulary coverage, which for language tasks often matter more than acoustic variety.

Concentration, collapsed into one number

Distributions are hard to compare across datasets, so it helps to collapse each axis into a single concentration figure. The effective number of categories is one divided by the sum of the squared shares, and it is the honest answer to "how many speakers is this really."

A worked case: a corpus with 60% of its hours on one speaker and 10% each on four others. The squared shares are 0.36 for the dominant speaker plus 0.01 for each of the four, summing to 0.40, and one divided by 0.40 gives an effective count of about 2.5 — five speakers on paper, two and a half in the data. By contrast, six hundred speakers with roughly even shares come out close to six hundred. The figure is useful precisely because it refuses to be flattered by a long tail of contributors who appear for a few minutes each.

Coverage is not the same as usable volume

A diversity report has a second job: separating categories that exist from categories that can train anything. A language or accent with two hours of material is covered on paper and absent in practice; a model will not learn it from that, and an evaluation slice cut from it will be noise.

A workable convention is to report every axis twice — the category list, and the hours per category — and to state a minimum for a category to count as supported. The minimum depends on the task; the point is to state one and apply it, so that a claim of broad coverage does not quietly mean four categories with real volume and thirty-six with tokens.

The same logic runs in reverse for evaluation. If the dataset will be sliced by accent to compare performance across groups, each slice needs enough items for the metric to be stable, which is a much higher bar than the training minimum.

Match the profile to the deployment

Diversity is not a virtue in the abstract; it is a match between the corpus and the place the model will run. A read-speech corpus with impeccable demographic balance is the wrong asset for a call center deployment, and a call center corpus with one accent and one channel is the wrong asset for a consumer product.

The practical move is to write the target profile first — the speakers, conditions, and styles the deployed system will meet — and then measure the candidate dataset on the same axes. The gap between the two tables is the shopping list: more hours in some buckets, re-weighting in training, or a different dataset entirely.

One honest note for anyone publishing a diversity report: an axis left unmeasured does not read as unknown. It reads as uniform, and every reader assumes whatever is convenient. Count it, or say plainly that it was not counted.