Training an ASR model with NeMo: manifests, tokenizers, and the config that ties them

NeMo is a training and deployment stack rather than a box of parts: the training loop, the tokenizers, the data pipeline, and the export path are already wired together. The price is that almost everything is configured through YAML and manifests. That trade is worth it once a project has more than one model in it, and it also means a wrong field in a manifest does not raise an error. It just trains a slightly worse model, which is the failure that costs weeks.

Three things have to be right before the first run: the manifest, the tokenizer, and the handful of config keys that change the outcome. The rest is tuning.

The manifest is the dataset

A manifest is JSON Lines: one object per utterance, one line each. Three keys are required, and one of them is quietly load-bearing.

• audio_filepath: the path to the audio. Relative paths resolve from the directory the job runs in, not from the manifest, so either write absolute paths or always launch from the same place. A manifest that works on one machine and fails on another is almost always this.
• duration: the length in seconds. It is used to sort and bucket the data and to report progress, so an estimate is worse than useless. A wrong duration produces batches whose lengths are ragged, and memory spikes that look like a batch-size problem are usually a duration problem.
• text: the transcript, in the convention the tokenizer expects. This is where most of the real work lives.
• offset plus duration let one entry point at a slice of a longer file, so a corpus stored as hour-long recordings does not have to be cut into files. And a config entry can carry a list of manifests with per-dataset weights, which is how a general corpus is mixed into a domain corpus for rehearsal.

Validate the manifest before every run

Four checks, all a few lines each, and each one catches a class of bug that otherwise shows up as a mysterious word error rate: every audio file exists and opens; no transcript is empty; the declared duration matches the file within a small tolerance; and no audio_filepath appears in more than one split. Run them as a build step rather than by hand, because a manifest that is regenerated from a spreadsheet will break one of the four eventually.

Pick the tokenizer before you build the model

The tokenizer decides the size of the output layer, so changing it later means rebuilding the decoder and re-exporting the model. Three options cover most cases, plus two rules that apply whichever one you pick.

• Character-level: one token per character plus the word delimiter. Nothing to train, works for any language with a script, and the only cost is that output sequences are long, which makes every step slower per utterance.
• Byte-pair encoding: train a BPE vocabulary on the transcripts. It shortens sequences and helps most where the character inventory is large - Chinese, Japanese, languages written in Devanagari or Arabic script, and anything with heavy morphology. It costs one extra artifact to version, and the vocabulary has to be regenerated whenever the transcript convention changes.
• Word-piece and the other subword variants: the same idea with different merging rules. Pick one, record why, and move on. The difference between them is smaller than the difference between either one and a bad transcript convention.
• Rule one: train the tokenizer on the training split only. Including the validation and test transcripts leaks their vocabulary into the model, and the leak is invisible afterwards because it makes the numbers look better rather than worse.
• Rule two: keep the tokenizer artifact with the checkpoint. A model whose tokenizer has been regenerated does not fail loudly, it decodes into plausible-looking garbage, and that takes far longer to diagnose than a crash.

CTC or transducer, and the keys that matter

CTC models are simpler and cheaper: one network, a per-frame output, and a loss that assumes the alignment is monotonic. Transducer models add a prediction network and a joint network, usually score better on the same data, and cost considerably more memory because the joint network is evaluated over every combination of output frames and target tokens in the batch. For a first run on a new corpus, CTC is the honest default. Move to a transducer when the accuracy gap is worth the GPU time.

• The train and validation manifest paths, and the batch size per device. With multiple devices the effective batch is the batch size multiplied by the device count, so the learning rate has to scale with it.
• Bucketing. Length-aware bucketing with a sensible bucket size is the largest single throughput change for variable-length utterances, and it only works if the durations in the manifest are accurate.
• The optimizer schedule: warmup steps and peak learning rate. Transducer training is more sensitive to warmup than CTC, and a transducer that diverges in the first few hundred steps is usually missing it.
• Precision. Brain-float rather than half precision on hardware that supports it. Half precision needs loss scaling, and a NaN in the middle of a long run is expensive to track down.

Many small files, and the first thousand steps

A corpus of a hundred thousand short clips is a filesystem problem before it is a training problem. Reading them one at a time from network storage, with several worker processes per device, spends more time in the file layer than in the model. Packing a manifest into tarred shards plus an index turns that into sequential reads. The tradeoffs: the tarred set is a copy that has to be regenerated when the corpus changes, shuffling becomes approximate because the reader shuffles a buffer of shards rather than the whole corpus, and debugging gets harder because no single file can be opened and listened to. Keep the original manifest and audio around for as long as the model is in service.

• The training loss should fall. A flat loss is usually a learning rate that is too low or a tokenizer that does not match the transcripts, in which case the model is learning to emit the blank token everywhere.
• Validation WER should fall too, computed greedily with the same decoding that will run in production.
• Transcribe a fixed sample of validation utterances after the first epoch and read them. A manifest whose text came from a different list than the audio produces output that is fluent and unrelated to the recording, and no loss curve reveals that.
• Watch for loss spikes at a specific step. They are usually one corrupt file, and the toolkit will happily train on a truncated file that decodes to noise.
• Keep the experiment directory, the config, and a hash of the manifest together. A checkpoint without the config that produced it is a binary nobody can reproduce.