Fine-tuning a speech model on domain jargon: legal, financial, industrial

A general recognizer is trained on the most common words of a language. It is good at them and mediocre at the words that carry the meaning in a specialized setting: the names of statutes and clauses, financial instruments and tickers, part numbers and machine codes, and the product names a support team says a hundred times a day. These words are rare in pretraining data by construction, and they are the words the buyer actually needs.

The failure is a frequency problem rather than an acoustic one, and that changes what fixes it.

Why more general audio will not fix it

Adding more hours of general speech does not change the ratio that matters. A term that appeared a handful of times in a very large pretraining corpus is effectively unknown, and a corpus ten times larger still contains it a handful of times per unit of speech. What does change the ratio is data where the term appears repeatedly in the conditions you deploy in: the same channel, the same accent region, the same speaking style. That is a data problem with a specific shape, and it is a different shape from "more hours".

Build the term list before recording anything

The term list is the specification. Sources that work: product catalogs, support ticket logs, contract and filing templates, transcripts of real calls, the list of statutes or instruments the team actually cites, and the internal glossary that someone already maintains for humans. Rank by frequency in that material and keep the top few thousand terms. The return falls off quickly below that, and a list of ten thousand terms is usually a list of ten thousand terms nobody will record properly.

• How it is spoken. The written form and the spoken form are different strings. An alphanumeric code can be read as letters and digits, as a spelled sequence, or with the separator named, and all three occur in the wild. Pick the convention that matches deployment and write it down, because the label has to match what the speaker actually says.
• Whether it is read as a word or spelled out. Some acronyms are pronounced as words and some are spelled letter by letter, and the same string can go both ways in different organizations. No rule gets this right; a maintained list is the only source.
• Whether it has more than one accepted pronunciation. If it does, either pick one and tell the speakers, or label both, but do not leave the decision to the recording session.

The recording script: context and speakers are the point

The rule that decides whether the collection works: a term seen once in the fine-tuning set is memorized as an acoustic template tied to one voice, not learned as a word. The model needs the term in different sentence positions, at different speaking rates, and from different speakers before the mapping generalizes. A term that appears in exactly one utterance is a term the model will get right on that utterance and wrong everywhere else.

A practical shape for the script: several sentences per term, spread across the speaker pool, written so the term does not always sit in the same slot of the sentence. The sentences should be natural, because the model also learns the words around the term, and a script of unnatural sentences teaches an unnatural context.

Keep a general-speech portion in the same batch. A fine-tuning set that is nothing but jargon will make the model worse at ordinary sentences, and the ratio between the two is a knob worth tuning on a validation set that contains both kinds of speech. Spelled-letter sequences deserve their own block of the script: the confusable pairs, the letters that share a vowel sound, the nasal pair, and the letter that is also a word, are the highest-error class in most domain transcripts, and they improve only with dedicated examples.

Decoder biasing is cheaper, and it has limits

Before collecting anything, try biasing the decoder with the term list. Several toolkits accept a word list, a context prompt, or a set of preferred words that are favored during decoding. It costs nothing to test and it can fix a large share of close calls.

Two limits make it a first step rather than the answer. The bias acts at decoding, so it can break a tie between two similar candidates but cannot repair an acoustic confusion severe enough that the correct word never appears in the candidate list. And a long list causes insertion errors: the model starts hearing terms from the list in audio that does not contain them, trading substitutions for insertions, which reads to a user as hallucination. Biasing is usually enough for a few hundred terms and stops being enough once the list reaches the thousands.

Evaluate at the term level, or the gain is invisible

The arithmetic that hides the improvement: if jargon is a small share of the tokens in a test set, cutting its error rate in half moves the overall word error rate by about a point, which is indistinguishable from run-to-run variation. A team that reports only the overall number will conclude the data collection did nothing.

• Term error rate: error rate computed only over utterances that contain at least one term from the list. This is the number the collection was paid for.
• Per-term recall for the most frequent terms. It shows which terms are still failing, which is what a second collection round is built from.
• A held-out term set: terms that appear in no training data at all, to distinguish a model that learned the domain from one that memorized the list. Hold out speakers as well as terms, since a term recorded from a single voice has the same problem in the test set as in the training set.
• The general test set, always. A jargon model that has lost its accuracy on ordinary speech is not a better product, and the rehearsal ratio is the knob that controls the trade.
• One more check before delivery: confirm that none of the terms in the evaluation set appear in the training transcripts in any form, including as part of a longer word. A substring match is easy to miss and it invalidates the number.