Choosing a delivery format for a dataset: audio, labels, and structure

Three decisions, not one

Delivery format is usually treated as a single question answered with a container name. It is three questions with different reversal costs, and treating them as one is how teams end up with audio they can re-encode and labels they cannot re-derive.

The audio container is the most reversible, because a lossless master converts to anything later. The label format is the least reversible, because the file format decides what structure can be represented, and structure that was never recorded cannot be recovered from a flat file. The directory and identifier scheme sits between them: cheap to change once, expensive to change after a split has been built on it.

Audio: ask for a master and a working copy

The clean specification asks for two audio deliverables rather than one. A lossless master at the rate and bit depth the material was captured at, with no processing beyond trimming. And a derived working copy in whatever the training pipeline prefers.

The reason for two is that the uses have opposite requirements. The master exists to be re-derived from, so it must not have been through a lossy encoder and must keep the original rate. The working copy exists to be read on every epoch, so it should be one standard format with no decode ambiguity.

If storage forces a single copy, the master is the one to keep, and it should be lossless. Compressed audio is the wrong place to save space, because the decoder differences that surface later are invisible in the file size.

• Files above four gigabytes. A long session crosses the standard container size limit, and the producer may switch to an extended header without mentioning it. Ask which variant, and whether long sessions arrive as one file or several.
• One file per utterance or one file per session. Both are legitimate. The specification should say which, and when both are delivered, the naming has to make the relationship between an utterance and its session obvious.
• Channel layout. Mono is the safe default. If the material is stereo, say whether the channels are two separate microphones, a stereo pair, or a duplicated mono signal, because the three are handled differently downstream.
• Whether utterance cuts include padding. A cut on the exact word boundary clips fricatives. A cut with padding needs the padding length stated, and it has to be the same length across the corpus.

Labels: choose the format from the structure

The label format should follow the shape of the annotation rather than habit, and the deciding question is whether the annotation is flat or nested.

• Flat, one row per utterance: a delimited file works, and a JSON Lines file works better, because it survives a field that contains the delimiter and it streams without loading everything into memory.
• Nested, such as tokens with per-token attributes or several aligned tiers: JSON Lines, or a tier-based format. A delimited file cannot represent nesting without inventing an escaping convention that every reader will implement differently.
• Segment times with text: a subtitle-style text file is readable by people and adequate for machines, but it is a poor container for anything beyond start, end, and text.
• Word or phoneme alignment: TextGrid is the de facto interchange for alignment output and is read by Praat, by praatio, and by the aligners themselves. It is plain text, so it diffs and versions well.
• Multi-tier annotation with overlapping speakers: ELAN's EAF format, or JSON with an explicit tier model. Overlap is where flat formats fail completely, because one row cannot hold two simultaneous labels.
• Speaker turns: RTTM for the turn boundaries, alongside the per-utterance labels in the main annotation file. RTTM is small and well defined, and there is no reason to invent a replacement.

Conventions that cause silent corruption

These are the fields where two producers can both claim compliance and still deliver files that behave differently.

• Character encoding, stated as UTF-8 without a byte order mark, plus the normalization form. For languages written with combining marks, an unnormalized corpus contains strings that look identical and compare unequal.
• Line endings, and whether the final line is terminated. Small details that break naive parsers.
• Timestamp units and origin: seconds with a fixed number of decimals, or sample indices; absolute from the session start, or relative to the utterance; and whether the end time is inclusive. Each of these has caused an off-by-one that nobody noticed for a month.
• Speaker and file identifiers: opaque, ASCII, no spaces, fixed field order, and consistent with the manifest. Identifiers must not be names, and they must be stable, because a split built on them becomes invalid the moment they change.
• Null conventions: how an unintelligible stretch is marked, how a missing value is written, and how a participant who is present but silent is recorded.

Structure, and the manifest that holds it together

A layout that has held up: audio in one directory tree, annotations in another, metadata in a third, and a manifest plus checksums at the root. Splits belong in the manifest as a field rather than as directories, so that re-splitting is a change to one file instead of a move of thousands.

The manifest is the piece worth arguing about. One row per delivered unit, carrying the identifier, the speaker, the session, the duration, the split, the annotation status, and a checksum. With that file, any question about the corpus is a query. Without it, every question is an email.

Ship a README with the layout, the field definitions, and the version of the guideline the labels were produced under. That document is the difference between a delivery and a usable delivery.

Give the producer a validator, not a paragraph

The item worth writing first is a script. A small validator that checks what is mechanically checkable: file presence against the manifest, header parameters on every audio file, encoding and line endings on every label file, identifier format, and timestamp monotonicity.

Then require one example file, in the exact final format, passing that script, before production starts. This turns a class of arguments into a pass or a fail, and it moves the discovery of format problems from delivery week to week one, where they cost almost nothing.