Fine-tuning wav2vec2 for a low-resource language without wrecking it

Self-supervised models such as wav2vec2 are pretrained on unlabeled audio, which means the expensive part - learning what speech sounds like - is already finished before a single minute is transcribed. The fine-tuning stage only has to attach characters to those sounds, and with a few hours of labeled audio that difference is the whole reason the approach works for languages that will never have a thousand hours.

Most of the outcome is decided before training starts: which checkpoint you begin from, which characters the model is allowed to output, and which layers are allowed to move.

Choose the checkpoint, then accept its mismatch

The multilingual checkpoints in the XLSR and XLS-R families were pretrained on dozens to a hundred-plus languages, and their published language lists are worth reading before anything else. If your language is on the list, start there. If it is not, a multilingual checkpoint still transfers, because the early layers encode general phonetic structure rather than language-specific words. The gap between being on the list and not being on it is real, and it is smaller than the gap between a monolingual checkpoint and anything else.

The mismatch that hurts more is channel and speaking style. A checkpoint pretrained mostly on read audiobook speech carries assumptions about recording quality, and telephone-band or field-recorded audio violates them. That does not make fine-tuning fail, but it does mean the early layers have to move, which is an argument against freezing everything.

The character set is a decision you cannot quietly change later

The output layer has one class per vocabulary item plus the blank, so the vocabulary is baked into the checkpoint. Build it from the union of every character in the training, validation, and test transcripts, and check that union explicitly. Any character in the test set that is missing from the vocabulary gets mapped to an unknown token at inference, and the model can never produce it. That is a silent ceiling on accuracy which no amount of training removes.

• Case. Folding to lowercase is the common choice and roughly halves the effective vocabulary. It also means the model can never output a capital, so case has to be restored downstream if the product needs it.
• Diacritics and tone marks. In Arabic script, Vietnamese, Yoruba and others, these marks distinguish words. Folding them shrinks the vocabulary and merges distinct words into one class; keeping them preserves the distinction but splits the training signal across more classes. The test that decides it: count the word pairs in your transcripts that differ only by a mark. If that count is high, keep them.
• Punctuation. Usually stripped for CTC models, which then need a separate punctuation model if the product needs sentences.
• The word delimiter. A distinct symbol between words is the convention. Without it the model has no way to know where words end, and every word boundary has to be recovered by a language model afterwards.

What to freeze

Freeze the convolutional feature extractor. It is small, it is the least task-specific part of the model, and with a few hours of audio it overfits long before the transformer does.

Above that, the useful rule is that the smaller the corpus, the more of the transformer to freeze. Under roughly ten hours, freezing the bottom half of the encoder and training the top layers plus the output head usually converges to a better model than full fine-tuning, and it is faster. A two-stage version, where the head trains first with the encoder frozen and the top layers are unfrozen afterwards at a lower rate, is more work for a small gain. Layer-wise learning rate decay, where deeper layers train at a lower rate, is the smooth version of the same idea.

Two defaults that quietly hurt

• Feature masking. The fine-tuning configuration usually ships with time masking enabled at a low probability and a fixed mask length of a handful of frames. It was a pretraining device, and it does help small corpora, but it is not neutral: it changes the effective difficulty of every batch and interacts with whatever waveform augmentation you add. Decide whether you want it instead of inheriting it, and if you use it, do not stack a second masking mechanism on top.
• The data collator. A widely copied collator truncates labels when it pads inputs to a multiple of eight, which silently drops characters from the longest transcripts in each batch. Check the label length after collation against the label length before, once, and it will never surprise you again.

The loss that goes to infinity, and the metric that matters

CTC requires the output sequence to be at least as long as the target, and the feature extractor downsamples the audio by a fixed factor. A very short clip, a fraction of a second, can therefore produce fewer output frames than the transcript has characters, and the loss for that example is infinite. It arrives as a NaN several hundred steps into a run, caused by one file. Filter for it explicitly: drop utterances whose label length exceeds half the expected frame count, and keep the filter in data preparation so the same file cannot come back.

For scoring, decode by taking the most likely class per frame, collapsing repeats, and removing the delimiter, then compute word error rate on the decoded strings with case folded and punctuation stripped, using the same normalizer on both sides. Report the number per test set rather than per training run. And if a language model is added to the decoder, report the no-language-model number beside it, because the gap between the two is what the language model is buying, and it is usually smaller than expected.