Speech translation data: decide what artifact you are buying

Three pipelines, three artifacts

Speech translation is not one product. In the common case, speech-to-text translation, the artifacts are the source audio, its transcript, and the target-language text, aligned at the segment level. In a cascaded system, recognition and translation are trained separately, so the artifacts are recognition data (audio with transcript) and translation data (source text against target text), which can come from different projects and be combined later, cheaper to source, with errors compounding where the two meet. In speech-to-speech translation, the target side is audio.

That last case is the expensive one and the one most often mis-scoped. A direct speech-to-speech model wants the same content spoken natively in the target language, which cannot come from the original speaker, or synthesized target audio from a text-to-speech system, in which case the voice, the model and its version are part of the data provenance. Deciding which pipeline is being built before any recording starts is what prevents buying the wrong artifact: every pipeline needs the transcript, and only one of them needs target audio.

Transcribe, verify, translate — in that order

A translation produced directly from audio by a bilingual annotator reads fluently and drifts. Omissions appear, additions appear, and the hardest items — numbers, names, hedges, units — get smoothed into something plausible. Translating from a verified transcript removes that class of error, because the source is fixed in text before the translation starts and any reviewer can compare the two.

The workflow that holds up has four steps. Transcribe the audio. Verify the transcript against the audio, as a second pass or a checked sample. Translate from the verified transcript. Then verify the translation, with a back-translation by a different translator to catch omissions and a terminology check against the glossary. Each step leaves an artifact that can be checked by someone who cannot perform the other step, which is what makes the process auditable rather than a matter of trust.

What the translation guideline has to settle

The guideline is what keeps a hundred translators producing one dataset. These are the decisions that come up in every project, and that vary per translator when they are not written down.

• Disfluencies. The transcript is verbatim; the translation is clean, because fillers and false starts do not carry across languages and a literal rendering of them reads as an error.
• Names and entities. A do-not-translate list for product names, and one rule for people and place names: keep the source form, or transliterate into the target script, applied everywhere.
• Numbers and dates. Converted to target-language conventions and checked separately, because a number error is the most expensive kind of translation error and the least visible in an averaged metric.
• Culture-specific terms. A written policy per class: translate, transliterate with a gloss, or keep the original. The classes are enumerated in advance, not decided clip by clip.
• Register and politeness. Languages with a formal and an informal second person, or with honorific levels, need one register per domain, chosen in the guideline. A dataset that switches register between segments teaches the model to switch, and the switch reads as a defect to any native speaker.
• Embedded foreign words. If the source speaker switches languages mid-sentence, the guideline says whether the translation keeps the embedded term as spoken, which is what an interpreter would do, or translates it.

Segmentation and alignment are part of the spec

Segment-level pairing is the default: the source segment and its translation must cover the same span of speech, which means the segmentation is decided once and used on both sides. Translating text that was segmented differently breaks the pair, and the break stays invisible until someone tries to train on it. The segment length is a real choice: very short utterances are ambiguous to translate without context, and very long segments hide omissions and make review expensive. A sentence or two of speech is the usual working range.

Word-level alignment is a different and more expensive annotation. It is what interpreting research and word-order evaluation need, and it should be scoped as its own line item with its own quality checks rather than assumed as part of a translation delivery. A supplier who has not been asked for word alignment will not produce it, and a pair of segment-level files cannot be converted into word alignments afterwards without a new annotation pass.

Evaluate with more than one number

Automatic metrics over the translated text — the BLEU family and its successors such as chrF — are the standard baseline and are weak on speech. They penalize valid paraphrases, they are insensitive to the errors that matter most, and a single averaged number hides where the system fails.

• Human adequacy and fluency on a sample, scored separately. Meaning preserved and target-language quality fail independently, and one number for both hides which problem exists.
• Entity and number accuracy, extracted and checked on their own, because a metric average buries exactly the errors a user notices.
• For speech-to-speech, a listening test on the generated audio, covering intelligibility and naturalness, since a correct translation in an unintelligible voice is not a working product.
• Reference translations from at least two independent translators on the evaluation set. A single-translator reference measures that translator, and reporting the agreement between the two is what makes the score interpretable.