Choosing a diarization toolkit: pyannote, NeMo, or a commercial API

Every comparison you can find online was run on a benchmark corpus with its own recording conditions, speaker counts, and annotation conventions. None of that is your audio. The protocol is reusable. The numbers are not, and producing your own takes about a day.

What each family is good at

The three families differ less in accuracy than in where they put the difficulty.

• pyannote.audio: one pipeline, one call, sensible general-purpose defaults. Best when you want a working baseline today and your audio is meetings or broadcast. The tuning surface is small, which is a feature until it is a limitation.
• NeMo: config-file driven, with swappable detection, segmentation, embedding, and clustering components. Best when you need to replace one stage or already run NVIDIA tooling. The cost is setup time and a combination space large enough to absorb a week.
• Commercial API: returns diarized text, absorbs codec and scale variety, and removes the infrastructure. The costs are per-minute pricing, no visibility into the model, no way to rescore the output when your recognizer changes, and audio leaving your network.
• End-to-end models that emit speaker labels directly, such as the Sortformer family, trade flexibility for speed and a fixed maximum speaker count. Useful when the count is bounded and latency matters.

Build a reference you can score against

Twenty to thirty minutes of your own audio in three to five files, covering the conditions that matter: channel, speaker count, language, noise. Annotate speaker turns only, because DER does not need words.

Two rules keep it honest. Annotate to a fixed convention and write it down, especially for overlap and for what counts as a turn boundary. And have a second person annotate five minutes of the same file, because the disagreement between your two annotators sets a floor on everything you measure afterwards. If they differ by three points of DER, a two-point difference between two tools is not a finding.

Score every candidate the same way

The protocol is short, and the discipline is in not skipping steps.

• One command per tool at its defaults, on the same files, before any tuning.
• Score with the same collar against the same reference, and report the three components separately rather than one number.
• Score a trivial hypothesis so the numbers have a floor to sit above.
• Run each tool twice, once with the exact speaker count and once with a range, and record the difference. Sensitivity to that input is a property of the tool, not of your data.
• Sort the files by error rate and read the worst one for each tool. A tool that is two points better overall but fails differently on your hardest condition is not automatically the better choice.

The checks that are not about accuracy

These decide as many procurements as the scores do, and they are cheaper to check.

• Output format: can it write RTTM or per-turn JSON with concurrent turns, or only a diarized transcript. If the output cannot be rescored, you cannot change the recognizer without re-running the diarization.
• Licensing: model weights and source code are frequently under different terms. A permissive code license says nothing about commercial use of the weights.
• Data handling: whether the audio leaves your network, whether it is retained, and whether it trains anything. For a hosted API this is answered in a contract, not a README, and it is usually the deciding question.
• Speaker count limits: some models cap at four or eight speakers. A cap that is fine today is a rewrite later.
• Runtime on your own hardware: time the longest file you own and record peak memory. A tool that needs a large GPU for a three-hour file is a constraint on the pipeline, not a preference.

A defensible default

With no strong reason to choose, start with the open pipeline that writes RTTM in one call, validate it against your own reference, and treat the result as a baseline rather than an answer. Then read the errors by component: concentrated in overlap, look at an overlap-aware model; concentrated in one channel, the problem is the audio; concentrated in the speaker count, the fix is the input you pass, not the model.

State the ceiling honestly as well. A general-purpose model on your data will not match one fine-tuned on it, and fine-tuning needs labelled turns, which is the reference you just built, scaled up by an order of magnitude. Knowing that up front is what keeps a day-long evaluation from turning into a quarter-long project.