Diarization Error Rate (DER)
Short answer. DER is the fraction of reference speech time that is wrong, broken into three components: missed speech (reference speech the system called silence), false alarm speech (non-speech the system called speech), and speaker confusion (speech attributed to the wrong speaker). DER = (missed + false alarm + confusion) / total reference speech time. The standard convention ignores 0.25 s around every reference speaker change, because human annotators cannot place boundaries more precisely than that.
How to do it
Three components, three different bugs
A single DER number tells you the system is wrong. The breakdown tells you how. Missed speech means the voice activity decision is too conservative. False alarm means it is too aggressive. Speaker confusion means the segmentation is fine but the clustering is wrong — the system knows a speaker changed and attached the wrong identity.
Those have nothing in common as engineering problems. A DER of 0.20 made entirely of confusion is a clustering bug; the same 0.20 made of missed speech is a threshold bug. Report the three numbers separately, always.
The collar, and why it exists
A reference marks a speaker change at 4.00 s. The true change is somewhere within a few tens of milliseconds of that, but the annotator had to choose a point and a second annotator would choose a different one. Scoring a system against a boundary that is itself uncertain punishes it for being right.
The convention is to ignore a 0.25 s window on each side of every reference boundary. Errors inside the collar are not counted. In the example below the collar removes 1.5 s of scored speech and drops DER from 0.274 to 0.248, with most of the change coming out of the missed-speech and confusion components.
The denominator is reference speech, not file length
DER divides by total speech time in the reference, not by the duration of the recording. A file that is 30% speech and 70% silence has a smaller denominator than its length suggests, so the same absolute error seconds produce a larger DER.
This is why DER is not comparable across corpora with different speech ratios. A 0.15 on a meeting corpus where people talk most of the time is a different achievement from a 0.15 on sparse call recordings.
Collars change the number, so state yours
0.25 s is the convention behind most published numbers, but not all of them. Some evaluations use zero collar, some collar only boundaries where both speakers are active, and some apply a forgiveness window to the hypothesis rather than the reference.
A DER quoted without its collar is not a number you can compare. When you read a paper, find the collar before you read the score.
Code
Frame-level DER with the three components reported separately, computed once with no collar and once with the standard 0.25 s collar.
import numpy as np
HOP = 0.01
DUR = 14.0
COLLAR = 0.25
ref = [(0.0, 4.0, 'A'), (4.0, 8.0, 'B'), (8.0, 12.0, 'A')]
hyp = [(0.0, 3.9, 'A'), (4.1, 8.0, 'B'), (8.0, 10.0, 'B'),
(10.0, 11.6, 'A'), (12.3, 13.0, 'X')]
n = int(DUR / HOP)
ref_lab = np.array(['-'] * n, dtype=object)
hyp_lab = np.array(['-'] * n, dtype=object)
for t0, t1, s in ref:
ref_lab[int(t0 / HOP):int(t1 / HOP)] = s
for t0, t1, s in hyp:
hyp_lab[int(t0 / HOP):int(t1 / HOP)] = s
bounds = sorted({0.0, DUR} | {b for seg in ref for b in seg[:2]})
keep = np.ones(n, dtype=bool)
for b in bounds:
lo, hi = max(0, int((b - COLLAR) / HOP)), min(n, int((b + COLLAR) / HOP))
keep[lo:hi] = False
def components(mask):
r, h = ref_lab[mask], hyp_lab[mask]
speech = r != '-'
missed = int((speech & (h == '-')).sum())
false_alarm = int(((r == '-') & (h != '-')).sum())
confusion = int((speech & (h != '-') & (h != r)).sum())
return missed, false_alarm, confusion, int(speech.sum())
for label, mask in [("no collar", np.ones(n, dtype=bool)), (f"collar {COLLAR}s", keep)]:
miss, fa, conf, total = components(mask)
der = (miss + fa + conf) / total
print(f"{label:>12} missed {miss * HOP:>5.2f}s false alarm {fa * HOP:>5.2f}s "
f"confusion {conf * HOP:>5.2f}s scored speech {total * HOP:>5.2f}s DER {der:.3f}")- numpy only. The reference and hypothesis are the five segment tuples at the top.
- Without a collar: missed 0.59 s, false alarm 0.70 s, confusion 2.00 s over 12.00 s of reference speech, giving DER 0.274.
- With the 0.25 s collar: missed drops to 0.15 s and confusion to 1.75 s while false alarm is unchanged at 0.70 s, because the false alarm sits away from any reference boundary. Scored speech falls to 10.50 s and DER to 0.248.
Where this goes wrong
Comparing DERs computed with different collars
A 0.25 s collar routinely removes 10 to 20% of the scored speech on conversational data, and it removes exactly the regions where systems make most of their errors. A collar-free DER and a collared DER can differ by five points on the same system.
Reporting DER without the breakdown
A system at 0.15 made of false alarm is failing on silence detection; at 0.15 made of confusion it is failing on clustering. Fixing the wrong one costs a month.
Forgetting that the denominator excludes silence
If the evaluation corpus has a low speech ratio, DER looks worse than the absolute error suggests. A corpus that is 30% speech has a denominator less than a third of its total duration, so the same error seconds produce a DER roughly three times larger than a length-normalized figure would. Report reference speech time next to the DER so a reader can convert between the two.
Letting the hypothesis label speakers arbitrarily
DER requires hypothesis labels to be mapped onto reference labels first. An unmapped hypothesis scores as pure speaker confusion, which looks like a clustering failure and is actually a scoring bug.
Scoring overlap as confusion
If the reference allows two speakers at once and the hypothesis does not, every overlap frame counts as speaker confusion, and overlap is common in real conversation. Decide whether overlap is scored, excluded, or its own class, and put that in the specification.
When to buy the data instead of building the pipeline
Build the scorer — it is arithmetic over segment lists, and owning it is what lets you vary the collar and see what it does. Buy when you need the reference segments themselves, because DER is only as good as the human annotation it is scored against, and a reference without collar-quality boundary annotation produces a DER that measures annotator disagreement. For conversational audio with overlap, the reference is the expensive part and the part that cannot be generated. The reference is the product; the scorer is thirty lines.
More technical reference
-
Sample Rate for Speech Recognition
sample rate speech recognition
-
MFCC vs Mel Spectrogram
mfcc vs mel spectrogram
-
Audio Augmentation
audio augmentation
-
Mean Opinion Score (MOS)
mean opinion score
-
Far-Field Speech Recognition
far field speech recognition
-
PII Redaction in Speech Transcripts
pii redaction
Need data for Diarization Error Rate (DER)?
Tell us the language, the hours, and what the data needs to look like. You will get a real number and a real timeline — not a range. If we cannot source it well, we will tell you that instead.
- Pilot batch before the full run, so problems surface early.
- Consent documentation delivered with the data.
- No medical or clinical data. No recorded telephone calls.