Forced Alignment
Short answer. Forced alignment takes audio and its transcript — both already known — and finds the start and end time of every word or phoneme. It is not recognition: the text is an input, not an output. Production systems score every frame against every phoneme with an acoustic model, then run a Viterbi search for the best path through the known text. A simple energy-based segmentation works only when words are separated by silence, which makes it a demonstration rather than a solution.
How to do it
The input is audio plus text, and the text is trusted
This is what separates alignment from recognition. A recognizer is handed audio and must produce text. An aligner is handed both and must produce timings. Because the text is trusted, the search space is a fraction of the recognizer's, and the resulting timings are far more accurate than an ASR transcript would allow.
That matters for data work: given a studio recording and a verified transcript, alignment yields word-level timestamps you can trust, without introducing the errors a recognizer would.
What the energy-based version does, and where it stops working
The code below synthesizes four words as four voiced bursts separated by silence, frames the signal at 10 ms, thresholds on frame energy, and assigns one word per detected region in order. It recovers every boundary to within 20 ms.
It works because silence is doing the segmentation. Run the same code on connected speech and it fails immediately: there are no gaps between words in normal speech, so the energy gate returns one long region and the word-to-region assignment has nothing to work with. Narrow the gap in the script from 0.30 s to 0.02 s and the regions merge.
What production alignment actually does
A real aligner has three parts. A pronunciation lexicon maps each word to its phoneme sequence. An acoustic model — usually a neural network trained on hundreds of hours — scores how well each frame of audio matches each phoneme. A Viterbi search finds the phoneme sequence that best explains the audio while matching the known transcript.
The lexicon is the part people forget. A word that is not in it cannot be placed, which is why alignment on names, numbers, and code-switched terms produces the worst timings.
Granularity is a choice you have to make
Word-level timestamps are enough for most data work. Phoneme-level timestamps are what pronunciation scoring, accent analysis, and phone-duration modelling need.
The cost difference is real and the accuracy difference is larger than it looks. A word boundary sits at a silence or a phoneme transition, both of which are acoustically clear. A phoneme boundary inside a word is a judgment call that two trained annotators will place tens of milliseconds apart.
Code
Synthesize four words separated by silence, recover their boundaries from frame energy alone, and compare the result with the ground truth.
import numpy as np
SR = 16000
WORDS = ["record", "the", "following", "sentence"]
def synth(words, sr=SR, seed=0):
"""One voiced burst per word, separated by 0.30 s of silence."""
rng = np.random.default_rng(seed)
out, spans, t = [], [], 0.0
for i in range(len(words)):
dur = 0.28 + 0.06 * i
n = int(dur * sr)
tt = np.arange(n) / sr
burst = np.sin(2 * np.pi * (120 + 15 * i) * tt)
burst += 0.4 * np.sin(2 * np.pi * (900 + 150 * i) * tt)
burst *= np.hanning(n)
burst += 0.005 * rng.standard_normal(n)
out.append(burst)
spans.append((round(t, 3), round(t + dur, 3)))
out.append(np.zeros(int(0.30 * sr)))
t += dur + 0.30
return np.concatenate(out), spans
audio, truth = synth(WORDS)
win, hop = int(0.025 * SR), int(0.01 * SR)
frames = np.lib.stride_tricks.sliding_window_view(audio, win)[::hop]
db = 10 * np.log10((frames ** 2).mean(axis=1) + 1e-12)
db -= db.max()
active = db > -30
edges = np.flatnonzero(np.diff(np.r_[0, active.view(np.int8), 0]) != 0)
regions = [(edges[k] * hop / SR, edges[k + 1] * hop / SR) for k in range(0, len(edges), 2)]
print(f"{len(regions)} speech regions found for {len(WORDS)} words\n")
print(f"{'word':<11}{'true start':>11}{'found start':>13}{'error ms':>10}{'duration':>10}")
for w, (t0, t1), (f0, f1) in zip(WORDS, truth, regions):
print(f"{w:<11}{t0:>11.3f}{f0:>13.3f}{(f0 - t0) * 1000:>10.1f}{f1 - f0:>10.3f}")- numpy only. The audio is synthesized, so there is nothing to download and no file path to fix.
- Prints four regions for four words with start errors of 0 to 20 ms, which is the 10 ms frame hop showing up as quantization.
- Change the gap in synth from 0.30 to 0.02 and the regions merge into one, which is exactly what happens on connected speech.
Where this goes wrong
Trusting an aligner on words outside its lexicon
Names, product codes and numbers are usually missing from pronunciation dictionaries, and aligners produce poor timings for them silently rather than flagging them. An aligner given a word it cannot find will stretch the surrounding phonemes to cover the gap, which moves the boundaries on the words either side of it. If your data has many proper nouns, check a sample of those alignments by hand.
Aligning against a transcript that is not verbatim
If the transcript omits a filler word, or writes a number as digits where the speaker said it as words, the aligner has to absorb a mismatch and will smear the timings around it. The transcript has to match the audio word for word before the output means anything.
Assuming phoneme boundaries are as reliable as word boundaries
Word boundaries sit at acoustic discontinuities and are stable. Phoneme boundaries inside a word are a judgment call. A phoneme-duration model built on single-annotator timings will encode that annotator rather than the language. Report phoneme timings with a tolerance, or annotate the same audio twice and measure the disagreement.
Aligning audio that was resampled or trimmed afterwards
If the audio was normalized or cut after the transcript was made, every timestamp shifts by the size of the edit. Keep the alignment and the audio in the same file lineage, or record the offset explicitly.
When to buy the data instead of building the pipeline
Buy when the alignment itself is the product: phoneme-level timestamps for a pronunciation model, or word-level timings across hundreds of hours where running an aligner and cleaning its output would take longer than sourcing the annotations. Buy when the vocabulary falls outside a standard lexicon — brand names, drug names, technical jargon — because that is where you need a human rather than a better aligner. Build it yourself when you have a clean transcript, a standard lexicon, and an open-source aligner that already covers your language.
More technical reference
-
Voice Activity Detection (VAD)
voice activity detection
-
Diarization Error Rate (DER)
diarization error rate
-
Sample Rate for Speech Recognition
sample rate speech recognition
-
MFCC vs Mel Spectrogram
mfcc vs mel spectrogram
-
Audio Augmentation
audio augmentation
-
Mean Opinion Score (MOS)
mean opinion score
Need data for Forced Alignment?
Tell us the language, the hours, and what the data needs to look like. You will get a real number and a real timeline — not a range. If we cannot source it well, we will tell you that instead.
- Pilot batch before the full run, so problems surface early.
- Consent documentation delivered with the data.
- No medical or clinical data. No recorded telephone calls.