Running pyannote diarization: the parameters that matter and the merge with ASR

The pyannote pipeline looks like one function call, but it is three stages stacked: a segmentation model that decides who is talking frame by frame, an embedding model that turns speech into vectors, and a clustering step that groups the vectors into speakers. Every parameter you set acts on one of those stages, and knowing which one is what makes tuning tractable.

Give it the speaker count if you know it

Loading is two lines: pipeline = Pipeline.from_pretrained("pyannote/speaker-diarization-3.1", use_auth_token=TOKEN) followed by pipeline.to(torch.device("cuda")). The keyword is use_auth_token in older releases and token in newer ones, and either way the Hugging Face account has to have accepted the model conditions.

Then the run: pipeline("meeting.wav", num_speakers=4). If the count is genuinely known, pass it. The clustering stage then has nothing to estimate, and the most common diarization error — the wrong number of speakers — is removed before it can happen.

If you only have a bound, pass min_speakers and max_speakers instead. Do not pass a count and bounds together, and be aware that a wrong exact count is worse than a range: it forces the clustering to split one person into two rather than admit the estimate was off.

The setting that changes the most for the least effort

In the 3.1 configuration, min_duration_off defaults to 0.0. Raising it merges short gaps into a single turn: pipeline.instantiate({"segmentation": {"min_duration_off": 0.5}}) returns a pipeline with the override applied.

The direction of the effect depends on your audio. On read or scripted speech, where one speaker pauses mid-turn, raising it removes spurious speaker changes. On fast conversational turn-taking it merges two people into one, which converts a boundary error into a confusion error, and confusion is the more expensive of the two because it is harder to spot. Test 0.0 against 0.5 and 1.0 on your own files and watch the turn count and the confusion component, not just the overall score.

Merging diarization with a transcript

If you have word-level timings, assign at the word level. If you only have segments, use a dominant-speaker rule and flag the segments where it is weak. With pyannote.core.Segment, the rule is short: crop the diarization to the segment window with diarization.crop(window), iterate with itertracks(yield_label=True), and accumulate turn.end - turn.start per speaker. The speaker with the largest total wins.

Then compute the share: the winner's total divided by the segment length. Any segment below about 0.6 is a segment that spans a speaker change, and it needs to be split rather than labelled. Split it at the last word boundary before the change, never mid-word, because a word belongs to exactly one speaker.

Track the number of segments you flagged. If it is more than a few percent of the file, the diarization is wrong, not the merge rule.

What to check before trusting the output

Five checks, all cheap, and each one catches a different failure.

• Speaker count per file against what the recording actually contains. A two-person interview that returns eleven speakers has a channel or noise problem.
• The distribution of turn durations. A cluster of turns around 100 ms means the segmenter is chattering at boundaries.
• Speech time per speaker. One speaker holding 95 percent of a two-person conversation is a red flag even when the score looks fine.
• Words that fall in no turn at all. These are VAD gaps, and they are the words most likely to be dropped from a transcript.
• Whether any file reports overlap. On spontaneous conversation, a pipeline that reports zero overlap everywhere is collapsing it, not observing its absence.

Where this pipeline stops being enough

Three conditions degrade it: telephone-band audio, heavy overlap, and single-channel recordings with eight or more speakers. The first is a bandwidth problem that upsampling does not fix, because the information was never in the recording.

The honest caveat is that this is a general-purpose model. If your domain is one accent on one channel, a model fine-tuned on that channel can beat it — but measure before assuming, because on small labelled sets fine-tuning often loses to a well-configured general model. What you can change without any training is the input: an accurate speaker count is worth more than most parameter sweeps.