Building a speaker-labeled transcript you can ship
Merging diarization with ASR is a data-model decision before it is a code decision. The format, the boundary cases, and a review loop that stays small.
The deliverable is a document where every word carries a speaker. The naive merge labels each ASR segment with its dominant speaker, and it breaks exactly where the transcript is most valuable: at speaker changes.
Assign at the word level, keep timings in the master file, and generate the display formats from it.
Pick a master format and generate the rest
Use JSONL, one record per run of same-speaker text, with word entries inside. RTTM holds turns but no words, SRT has no speaker field at all, and WebVTT has a real voice tag (<v Alice>) but no word timings. None of them can hold the full record, so none of them should be the master.
One convention to state explicitly, because it causes silent bugs later: speaker labels are local to a file. SPEAKER_00 in one recording has no relationship to SPEAKER_00 in another. Cross-file identity is a separate clustering pass over embeddings, and if you need it, do it once and store the mapping rather than deriving it per consumer.
The merge, word by word
The rule that holds up is simple: assign each word to the turn containing its midpoint. For each word, compute the overlap with every turn, keep the largest, and record the speaker. The midpoint version and the maximum-overlap version agree on almost everything, and the overlap version behaves better for unusually long words.
Words with no timestamp get no speaker, plus an estimated flag. Fill them from the neighbouring word in the same segment so the display format has something to print, but keep the flag, and count them per file. If more than a couple of percent of words are estimated, the problem is upstream in the diarization or the VAD, and the merge rule is not what needs attention.
The boundary cases that decide the format
These are the cases that make one team's transcript unusable to another, so each one needs a written rule rather than a judgement call at merge time.
- A word that straddles a turn boundary belongs to one speaker. Assign by midpoint, never split a word into two speakers.
- A segment spanning a speaker change is split at the word boundary that precedes it, not at the change itself.
- A word inside a VAD gap keeps an empty speaker rather than a guess, and the count of those words is reported.
- A backchannel under the main speaker will be attributed to the main speaker by any pipeline that does not report overlap. Flag it or accept it, but decide which.
- A turn with no words at all, such as a laugh or a breath, stays in the turn file and out of the transcript, so the display format never emits an empty caption.
Review the flagged material, not the file
A review of everything does not survive contact with a deadline. Build a flag list instead: dominant-speaker share below 0.6, segments with more than two speaker changes, words with estimated timings, turns shorter than 300 ms, and any file where estimated words exceed 2 percent.
Give the reviewer those regions with two seconds of lead-in and three verdicts to choose from — correct, wrong speaker, wrong words. Free-text notes do not aggregate, and the whole value of the loop is that the numbers add up.
Then measure the correction rate by flag type. If material that was not flagged is also getting corrected, the flags are missing a failure mode. If flagged material is almost never wrong, the flags are too broad and the reviewer is burning time on noise.
What to keep alongside the transcript
Keep the raw ASR output before corrections, so the correction rate is a measurable number instead of an impression. Keep the diarization turns in RTTM, the parameters used for both stages, and the flag counts per file.
Six months later, when someone asks why one speaker in the corpus sounds like two different people, that bundle is the only thing that answers the question. The transcript on its own will not.