Fine-tuning Whisper on your own audio: what the data has to look like

Whisper is a general model trained on a very large and very varied pile of audio. That is exactly why it is strong on clean read speech and uneven on everything else: a heavy accent, a channel it has rarely seen, or a vocabulary that almost never appeared in its training data. Fine-tuning is the standard fix, and it is mostly a data-format problem rather than a training problem.

The failure mode is not a crash. It is a model that gets better on your audio and quietly worse everywhere else, which nobody notices until a different kind of audio arrives.

What one training example has to contain

A Whisper training example is one audio array and one target string. The feature extractor converts the audio into log-mel frames over a fixed 30-second window, so every example is padded or truncated to 30 seconds before the encoder sees it. That one fact drives the segmenting decision: a 3-second clip and a 28-second clip cost the same compute, and a corpus of 1-second clips spends most of its budget on padding.

A practical target is segments between roughly 5 and 30 seconds. Below a few seconds the padding dominates. Above 30 seconds the audio is truncated and the transcript no longer matches the input, which teaches the model to drop words.

• Keep case and punctuation in the labels. The tokenizer covers them and the pretrained model already produces them. Stripping them to match an older pipeline removes a capability you did not have to pay to train.
• The target string is what was said, in the same language, with the same words. No translation, no paraphrase, no tidying.
• Keep a set of clips that contain only silence or background noise, labeled with an empty string. Whisper was trained on weakly labeled web audio and will happily emit a phrase or a subtitle credit on silence. A few hundred empty targets are the cheapest fix available.
• Normalize text the same way across train and test. If the labels say "twenty three" and the test references say "23", every score computed afterwards is measuring the mismatch rather than the model.

How much audio, and which audio

There is no threshold that turns a fine-tune from useless to useful, and anyone quoting a single number is describing their own corpus. What determines the amount is how far your domain sits from general web speech. A single channel, one accent region, and a narrow vocabulary can move with a modest number of hours. A new language with several dialects and noisy field recordings needs far more, and even then the honest comparison is against the unmodified pretrained model on your own test set.

Two properties of the corpus matter more than its size. Distinct speaker count decides whether the model adapts to the domain or to the voices in your recordings. Condition coverage decides whether the improvement transfers at all: if every file comes from one microphone in one room, expect the gain to vanish when the microphone changes.

Build two test sets before training starts. One is from your domain, made the same way as the training data but with different speakers. The other is general speech that has nothing to do with the project. The first tells you whether fine-tuning worked; the second tells you what it cost.

The settings that decide the outcome

• Learning rate. Fine-tuning runs at a small rate with a warmup, one to two orders of magnitude below what pretraining used. A rate high enough to converge quickly is also high enough to overwrite the pretrained representations.
• Encoder freezing. Training the decoder first and then unfreezing the encoder at a lower rate is the usual recipe on a small corpus, because it limits how far the acoustic model can drift.
• Evaluation by generation. Word error rate has to be computed on generated text, not on the teacher-forced loss. The two move apart quickly once labels are noisy, and the loss is the more optimistic of the two.
• Parameter-efficient methods. Adapters or low-rank updates keep the base weights fixed and train a small number of parameters. That is the right first attempt when memory is the constraint or when one base model has to serve several domains.
• The decoder prompt. Whisper can be conditioned on previous text. Whatever conditioning you train with is the conditioning you have to serve with, because a model trained with prompts and deployed without them behaves like a different model.

The failures that show up early

• Loss falls while WER rises. Usually a learning rate that is too high, or labels that disagree with the audio in a systematic way.
• Repetition loops on long audio. Fine-tuning on short segments and then running on long recordings is a mismatch the model never saw, and the loop is what running out of the trained pattern looks like.
• Silence hallucination gets worse. Usually because the training set contains no examples of non-speech, so the model never learns that the correct output can be nothing at all.
• General speech degrades. Usually too many epochs, or a fine-tuning set that contains none of the general conditions. Mixing a fraction of general data back in is the standard mitigation, and the ratio is worth tuning against the general test set rather than guessing.

What to report when it is done

One number is not a result. Report word error rate per condition slice - accent, channel, noise level, utterance length - alongside the general test set, and report the hallucination rate measured as insertions on clips that contain no speech.

Then report the comparison against the unmodified pretrained model on the same sets. The interesting outcome is not always "fine-tuning won". A fine-tune that gains two points on your domain and loses four on general speech is a different product from one that gains one and loses nothing, and that choice belongs to whoever is going to deploy it.