Punctuation restoration for ASR output: what the training data has to be

Speech recognizers output a stream of words. Readers, translators, and most downstream systems need sentences. Punctuation restoration is the separate model that inserts commas, periods, and question marks after recognition, and it is unusual among speech tasks in one respect: it needs no audio at all.

That makes it cheap to start and easy to get subtly wrong, because the decisions that matter are conventions rather than acoustics.

The task is token classification, and that decides the data

The standard formulation runs a tokenizer over unpunctuated text and, for every token, predicts the mark that follows it: nothing, comma, period, or question mark. Sometimes a colon or a dash is added, and quotation marks almost never. Because the input is text and the label is a property of the text, the training data is any large corpus with reliable punctuation - the marks are stripped to form the input and kept as the labels. No audio is required, and that is the part that surprises people. A punctuation model can be trained on material that has nothing to do with your acoustic domain, because deciding where a clause ends depends on syntax, not on the recording.

The convention, however, has to come from somewhere, and that is where the domain does matter. Written text punctuates differently from speech. Broadcast captions, meeting transcripts, and subtitles each have their own style, and a model trained on newswire will punctuate conversational speech in a way that reads oddly even when every mark is defensible. If the product needs caption-style output, the model has to be trained on caption-style text.

Fix the convention before anything is annotated

• Sentence-final or clause-internal. A comma placed where the speaker paused and a comma placed where the grammar requires one disagree often, and the disagreement is invisible in aggregate metrics.
• Trailing-off clauses. Whether an unfinished sentence gets a period, an ellipsis, or a dash.
• Questions without question words. A flat "you are coming" can end in a period or a question mark, and the only reliable signal is intonation, which a text model does not have. Decide which way to be wrong, and be consistent about it.
• Quotation marks and dashes. Most punctuation models skip them entirely. A consumer that expects them will not get them, and a consumer that expects plain text will be confused if you add them.
• Capitalization. Usually bundled with punctuation and better treated as a separate decision, because it has its own failure modes.
• Fillers and false starts. Verbatim transcripts contain fragments. Whether a run of fillers gets a comma after each fragment is a convention decision, and it has to match the convention used for the recognition labels, or the two layers will disagree at every fragment.

The segment problem

Recognition output arrives in segments of a few seconds, and a punctuator that sees one segment at a time puts a period at the end of every segment. This is the most common failure in the whole pipeline, and it is invisible in a per-segment metric, because a period at the end of a segment is often correct.

Three fixes, usually combined. Feed the model a window of several consecutive segments so it can see where the sentence continues, and mark the segment boundaries as a feature rather than as sentence boundaries. Train with segments concatenated exactly the way they will be concatenated at inference, so the boundary feature is not a surprise at run time. And evaluate on multi-segment output rather than on isolated sentences.

If the recognizer produces word timestamps, pause duration is a strong additional signal: a long silence between two segments is evidence for a sentence boundary and a short one is evidence against it. That feature is available at inference and is worth feeding in when the timestamps are reliable enough to trust.

Evaluate on the marks, not on accuracy

Most tokens are followed by nothing, so a model that predicts no punctuation anywhere scores well on raw accuracy and is useless. The metrics that mean something are precision, recall, and F1 per mark type, plus two composites: sentence-boundary F1, which asks whether the model finds the ends of sentences at all, and a segment-final period rate, which measures how often it ends a segment that should have continued.

Two more checks belong in the evaluation. Use held-out text from the same genre as the deployment text, because a general newswire set will flatter the model. And evaluate on real recognizer output rather than on reference transcripts, since the punctuator sees the recognizer errors and not the truth. A model that is excellent on clean text and mediocre on the actual output of the recognition system is the normal case, and the second number is the one that describes the product.

Where it sits in the pipeline

The order of the post-processing steps changes the output. Casing and inverse text normalization can run before punctuation or after it, and the two orders produce different results on the same input. Pick one order, write it down, and evaluate the pipeline in that order, because a per-component evaluation will not catch the interaction.

For latency, a small text transformer runs far faster than real time on a CPU, but punctuation applied per segment cannot see sentence context. A short buffer of a few segments is the usual compromise, and the buffer size is a product decision: more context means better punctuation and more delay. Finally, keep punctuation out of the word error rate unless the metric was designed for punctuated output, since scoring punctuated text against an unpunctuated reference measures the punctuator rather than the recognizer.