Reading a diarization error rate: what the three components tell you

Diarization error rate is the sum of three error types divided by the total duration of reference speech: missed speech, false alarm, and speaker confusion. A collar of typically 0.25 seconds around each reference boundary is excluded from scoring to absorb annotation disagreement.

The single number is a summary. The components are the diagnosis, and two systems at the same DER frequently need entirely different work.

Collapse the labels to isolate detection

The fastest way to find out whether you have a detection problem or a clustering problem is to score the same output twice more, once with every reference speaker collapsed to a single label, which makes confusion impossible, and once with the hypothesis collapsed.

If detection-only DER is 4 percent and full DER is 25 percent, the segmenter is fine and the clustering is where the error lives. If detection-only DER is already 20 percent, no amount of clustering work will help.

With pyannote.metrics this is a few lines: build the collapsed copy by writing every turn into a fresh Annotation under one label, then call DiarizationErrorRate(collar=0.25, skip_overlap=False) on each version. Passing detailed=True returns the components as separate keys — missed detection, false alarm, confusion, and total — rather than a single figure.

Misses: speech you did not assign to anyone

Misses usually come from a VAD that is too strict, a segmentation threshold set too high, or a minimum-duration rule that deletes short utterances. They also come from speech the model simply does not represent well: a quiet voice, a strong accent, speech under music.

Dump the missed regions as a list of start and end times and listen to the five longest. If they are backchannels and short interjections, the fix is a parameter. If they are one speaker's quiet turns spread through the file, the fix is a different model or better gain staging, and no threshold will recover them.

False alarms: time you claimed and should not have

False alarms come from a permissive VAD firing on speech-like non-speech, from microphone bleed into a neighbouring room, and from a segmenter that keeps a speaker alive through a silence.

In a transcript product a false alarm is usually cheaper than a miss. A wrong speaker label on a stretch of silence is invisible to a reader, while a missing sentence is not. The DER formula weights them equally, so if you tune against it you will spend effort on the error your users do not notice. Weight the components according to what the output is for, and say which weighting you used.

Confusion: right detection, wrong speaker

Confusion means the system found speech and attributed it to the wrong person. The usual cause is the speaker count: too few clusters merges two people into one, too many splits one person across two labels. Short turns produce noisy embeddings, and a speaker who changes microphone or moves between recordings produces embeddings that do not look like themselves.

One diagnostic separates these. Plot the confused regions and check whether they concentrate in particular files or particular speakers. Concentrated by file points at a channel difference. Spread evenly across all files and speakers points at the clustering.

Establish a floor before you chase a target

Score a trivial hypothesis against your reference: one speaker talking for the whole file, or the VAD output with a single label. That number is the floor, and it is usually higher than people expect. If the best system you have is only a couple of points above the floor, the model is not doing the work.

Then pick a target from the application rather than from a leaderboard. If the deliverable is a speaker-attributed transcript, the metric that matters is how many words carry the wrong speaker, and it is not proportional to DER: a system can lose a full point of DER on silence and change no word labels at all. Report both, and let the word-level number decide.