Forced alignment with WhisperX: handling words the aligner cannot place
WhisperX aligns the transcript that Whisper produced, so transcription errors become alignment errors. The per-word scores are how you find them.
WhisperX chains a recognizer to a phoneme-based aligner. That order matters: the aligner is not an independent measurement of where the words are, it is a placement of the text the recognizer already wrote. When the two seem to disagree, the transcript is usually the thing that is wrong.
The pipeline exposes a per-word score, which is the only cheap way to find the words where that happened without listening to the file.
The pipeline, with the parts that matter
The whole run is four calls: audio = whisperx.load_audio("interview.wav"), asr = whisperx.load_model("large-v2", device, compute_type="float16"), result = asr.transcribe(audio, batch_size=16, language="en"), then align_model, meta = whisperx.load_align_model(language_code="en", device=device) and aligned = whisperx.align(result["segments"], align_model, meta, audio, device).
The align model is a wav2vec2 phoneme model, one per language. Coverage is uneven, and for languages without a listed model you have to pass model_name explicitly rather than relying on a default. WhisperX also resamples to 16 kHz internally, so feeding it a 48 kHz file is safe and feeding it a telephone-band 8 kHz file is not.
Reading the per-word score
Each word in aligned["segments"][i]["words"] has word, start, end, and score. Two failure signatures matter more than the score itself.
The first is start and end being None. The aligner could not place the word at all, which happens with words produced by a recognizer repetition loop, with words the phoneme model cannot cover, and with segments where the audio does not contain the text.
The second is a low but present score, meaning the word was placed somewhere, possibly the wrong somewhere. Do not use a fixed cutoff as a rule. Compute the distribution per file and look at the bottom two percent: a threshold of 0.3 is reasonable on clean read speech and floods you with false alarms on noisy meeting audio.
Add one arithmetic check that catches misplacements a score will not: characters per second for each word. English conversational speech runs around 10 to 15 characters per second. A word claiming eight characters in 0.1 seconds is not fast speech, it is a wrong timestamp.
What to do with the words that fail
Fix the transcription first when the failure is systematic. A recognizer that repeats one phrase across a long silence produces a segment full of unplaceable words, and no alignment setting repairs it. Re-run the transcription with condition_on_previous_text=False, and enable the VAD path with vad_filter=True and vad_parameters=dict(min_silence_duration_ms=500) so silence is not fed to the model at all.
When the failure is isolated, drop the timestamp instead of accepting it. A word marked as unverified is a usable record. A word with a confidently wrong timestamp is not, because it teaches whatever trains on it a boundary that does not exist.
When a whole segment fails, listen to it. In practice the cause is a transcription error rather than an alignment error, and the aligner is telling you the audio does not say what the text says.
A check to run on every file
Three numbers, computed from the aligned output: the share of words with no timestamp, the second percentile of the scores, and the ratio of total aligned duration to audio duration. The last one is the one people skip. If the words cover 70 percent of a file that is nearly all speech, whole regions were never covered, and the cause is upstream in the VAD or the transcription.
In code: words = [w for s in aligned["segments"] for w in s["words"]], missing = sum(w.get("start") is None for w in words), scores = np.array([w["score"] for w in words if w.get("score") is not None]), then np.percentile(scores, 2). For duration, info = torchaudio.info("interview.wav") gives num_frames and sample_rate, so the ratio is the sum of placed word durations divided by num_frames / sample_rate.
Limits worth knowing before you build on it
The align model was trained on read speech in its target language. On heavy accents, code-switching, or non-native speech the word boundaries stretch, and the scores get less informative at the same time.
WhisperX returns word timings, not phoneme timings. If you need phones, this is not the tool. It will also place a hallucinated word inside silence without complaint, so cross-check the timestamps against a VAD pass and treat any word landing entirely in non-speech as unverified.