Overlapping speech: why diarization fails there and how to record it

In spontaneous conversation a meaningful share of speech time has more than one person talking: backchannels, interruptions, agreement, the listener finishing a sentence. A system that emits one label per frame has no way to represent that, so it chooses, and the choice is usually to keep the louder or the earlier speaker. The result is not random noise. It is a systematic bias that lands on exactly the moments a transcript is most useful.

Why it is hard at every stage

The difficulty is not in one component, and fixing one does not fix the others.

• Segmentation: the frame contains a mixture, and the model has to decide whether to emit one speaker or two. Its training data decides how often it dares to emit two.
• Embedding: a vector extracted over a region with two voices is a blend, which lands between the two speakers in embedding space and can attach to either cluster.
• Clustering: those blended vectors pull two clusters toward each other, which is one mechanism behind two speakers being reported as one in meeting audio.
• Scoring: if overlap is not marked in the reference, a correct two-speaker detection is counted as confusion against a reference that says only one person was talking.

What the tools actually do

The pyannote segmentation model is trained to predict combinations of active speakers, so the pipeline can in principle emit two concurrent turns. In practice that output is unstable and the 3.1 pipeline mostly reports a single turn per frame. Measure it rather than trusting the architecture: count the files in your run that contain any two turns with a nonzero intersection.

SpeechBrain's separation models, such as speechbrain/sepformer-whamr16k, take a mixture and return one waveform per estimated source. They were trained on two-speaker mixtures, so they handle exactly that case and degrade as soon as a third voice appears. NeMo is configured through a YAML file rather than call arguments, which makes swapping one component easy and a quick experiment slow.

Commercial APIs generally return one speaker per segment. Ask the vendor what the output does in overlap and whether concurrent turns are representable at all. A direct answer to that question tells you more about the system than a benchmark does, and a vague one is also an answer.

Separate to detect, not always to transcribe

Separation is genuinely useful for locating overlap and for attributing it. Feeding the separated streams to a recognizer is a separate decision with its own answer. Separation introduces artifacts and removes the natural acoustics, and a recognizer trained on real mixtures can score worse on cleaned-up sources.

Test it rather than assuming: transcribe the mixture, transcribe the separated streams, and compare word error rate on the overlapping regions against the same reference. If separation does not win on that subset, use it only for detection and attribution.

Annotate overlap as concurrent turns, not as a flag

RTTM has no dedicated overlap column, so the convention is to write two turns with overlapping time ranges and different speaker names, and derive overlap by intersecting them. Put the rule in the guideline in one sentence: if two speakers are audible at the same instant, both turns are written, and the shorter one is not absorbed into the longer one.

Then measure the result on your own reference. Overlap is commonly reported in the range of 5 to 20 percent of speech time for spontaneous two-person conversation, and much lower for interviews and scripted speech. A reference that reports well under 5 percent on conversational audio usually means the annotators are writing only the dominant speaker, and such a reference cannot score the behaviour it is meant to measure.

Compute the overlap ratio yourself

The measurement is a sweep over turn boundaries: collect every start as a plus one event and every end as a minus one, sort by time, and accumulate the duration where two or more turns are active. Divide by the span covered by turns.

Apply the same function to the reference and to the hypothesis. The gap between the two numbers is the overlap problem reduced to a single figure, and it is usually larger than the DER gap on the same file, because DER counts an unmarked overlap region as confusion rather than as the specific failure it is.

Record the flag in the data even when nothing acts on it yet. Keeping the concurrent turns costs nothing at annotation time, and re-deriving them later means re-annotating the file.