Speaker Diarization
Short answer. Speaker diarization answers "who spoke when": it segments audio into speaker turns and labels each turn. The labels are arbitrary, so a diarization system is scored on two separate things — whether the turn boundaries are in the right place, and whether the turns group correctly by speaker. Comparing labels directly gives 0% agreement on a perfect system. Scoring requires mapping hypothesis labels onto reference labels first, usually by majority overlap.
How to do it
Boundaries are half the problem, and the harder half
A diarization output is a list of segments, each with a start time, an end time, and a speaker label. The label part is easy to get approximately right. The boundary part is where systems fail.
When two people talk back and forth quickly, turn boundaries land inside words. A system that is 200 ms late on every boundary is still right about who is speaking almost all of the time, but it breaks any downstream step that needs to attribute a specific word to a specific person. Boundary offset has to be measured, not assumed.
The label permutation problem
Nothing in the audio says which speaker should be called "spk1". The reference might call them A and B; the system might call them spk2 and spk1, or 0 and 1. Raw label comparison on a perfect system scores 0%, because every frame is "wrong".
The fix is a mapping step: build the frame-level confusion matrix between reference and hypothesis labels, then assign each hypothesis label to the reference label it overlaps most. The code below goes from 0.0% raw agreement to 94.7% after mapping, with no change to the underlying segmentation.
With more than a handful of speakers, greedy majority overlap is not always optimal and the standard approach is a Hungarian assignment over the confusion matrix. For two or three speakers, majority overlap is enough.
Overlapping speech is where scoring gets contentious
Real conversation has people talking at once. A reference with two simultaneous speakers has two active labels on the same frame, and a hypothesis that allows one label per frame is wrong on one of them by construction.
Decide up front whether overlap is scored, ignored, or counted as its own category, and write it into the specification. Two teams scoring the same system under different overlap conventions will report numbers that differ by several points.
What the training data has to contain
A diarization model learns speaker change, so the data needs recordings where the speaker actually changes and the change is annotated at the word level rather than per file. A corpus of single-speaker files has nothing to teach it.
The other requirement is recurrence. If every session introduces a new pair of strangers, the model sees no consistent voice to track across the recording, and the clustering step has nothing to cluster on.
Code
Reference and hypothesis segment lists scored three ways: raw label agreement, agreement after mapping, and turn-boundary offset.
import numpy as np
HOP = 0.01
ref = [(0.0, 2.1, 'A'), (2.1, 5.0, 'B'), (5.0, 7.4, 'A')]
hyp = [(0.0, 2.3, 'spk1'), (2.3, 4.8, 'spk2'), (4.8, 7.4, 'spk1')]
n = int(7.4 / HOP)
ref_lab = np.array(['-'] * n, dtype=object)
hyp_lab = np.array(['-'] * n, dtype=object)
for t0, t1, s in ref:
ref_lab[int(t0 / HOP):int(t1 / HOP)] = s
for t0, t1, s in hyp:
hyp_lab[int(t0 / HOP):int(t1 / HOP)] = s
print(f"frame agreement with labels as given {(ref_lab == hyp_lab).mean():.1%}")
ref_labels = sorted({s for _, _, s in ref})
hyp_labels = sorted({s for _, _, s in hyp})
conf = np.zeros((len(ref_labels), len(hyp_labels)), int)
for r, h in zip(ref_lab, hyp_lab):
if r != '-' and h != '-':
conf[ref_labels.index(r), hyp_labels.index(h)] += 1
mapping = {h: ref_labels[int(conf[:, j].argmax())] for j, h in enumerate(hyp_labels)}
print(f"best label mapping: {mapping}")
mapped = np.array([mapping.get(h, h) for h in hyp_lab], dtype=object)
print(f"frame agreement after mapping {(ref_lab == mapped).mean():.1%}")
print("turn boundaries (a diarization system has to get these right, not just the labels):")
for (r0, r1, _), (h0, h1, _) in zip(ref, hyp):
print(f" ref turn {r0:>4.2f}-{r1:>4.2f} found {h0:>4.2f}-{h1:>4.2f} "
f"start offset {(h0 - r0) * 1000:>+5.0f} ms end offset {(h1 - r1) * 1000:>+5.0f} ms")- numpy only. Nothing is read from disk, so the whole example is the segment lists at the top.
- Prints 0.0% frame agreement before the label mapping and 94.7% after it — the mapping changes nothing about the segmentation and everything about the score.
- The boundary table shows every turn off by 200 ms in one direction or the other. That defect is invisible in the frame-agreement number and fatal for word-level speaker attribution.
Where this goes wrong
Reporting label accuracy without a mapping step
Score diarization by comparing label strings and a perfect system scores near zero, while a system that happens to guess your naming convention scores well. On a two-speaker file where the system calls its speakers 0 and 1 and the reference calls them A and B, the unmapped score is 0.0%, not 50%. Map first, always, and publish the mapping alongside the number.
Scoring only speech frames
A system that labels every frame as speaker A and never marks silence scores well on speaker attribution while being useless, because it never says when nobody is talking. Score silence as a class, or use a metric that counts false alarm speech separately.
Ignoring overlap
If the reference marks overlapping speech and your hypothesis allows one speaker per frame, the achievable score is capped before you start. Either allow overlap in the hypothesis format or exclude overlap frames explicitly and say so.
Training on one speaker per file
A diarization model trained only on single-speaker recordings learns to output one speaker for everything, and it will label a two-hour meeting as one voice with full confidence. The data has to contain the phenomenon: multiple speakers per recording with annotated turn boundaries.
When to buy the data instead of building the pipeline
Buy when the recordings are conversations and you need turn boundaries annotated at the word level — that is annotation work rather than engineering work, and it is the part teams underestimate. Buy when speaker overlap is present, because overlapping speech is expensive to annotate and easy to get wrong. Build it yourself when your audio is already one speaker per file and you only need to group files by voice; that is speaker identification, not diarization, and it is a much smaller problem.
More technical reference
-
Forced Alignment
forced alignment
-
Voice Activity Detection (VAD)
voice activity detection
-
Diarization Error Rate (DER)
diarization error rate
-
Sample Rate for Speech Recognition
sample rate speech recognition
-
MFCC vs Mel Spectrogram
mfcc vs mel spectrogram
-
Audio Augmentation
audio augmentation
Need data for Speaker Diarization?
Tell us the language, the hours, and what the data needs to look like. You will get a real number and a real timeline — not a range. If we cannot source it well, we will tell you that instead.
- Pilot batch before the full run, so problems surface early.
- Consent documentation delivered with the data.
- No medical or clinical data. No recorded telephone calls.