Documenting a dataset for internal handover

Write for the person who was not there

An internal handover is a transfer between two people who will never speak. The receiver may be a new hire, a team that inherited the project, or the original requester six months later with no memory of the details. The test is simple: can they use the data correctly without asking a question?

Most handovers fail that test for a predictable reason. They document what the data is, and leave out what is wrong with it and how to load it. Those two omissions cause more misuse than any other gap, and both are cheap to fill at handover time and nearly impossible to reconstruct afterwards.

Five documents, not one

The package that survives a handover is five short documents rather than one long one. They are read by different people for different reasons, and they change at different rates, which is exactly why merging them into a single file makes all five worse.

• The dataset card: what the corpus is, what it contains, and what it is for.
• The field reference: every file, every column, every key, with its type, units, and allowed values.
• The manifest: one row per delivered unit, with identifiers, speaker, session, duration, split, and checksum.
• The known-issues list: what is wrong, how to detect it, how much of the corpus it affects, and the workaround.
• The loading example: a short working snippet that reads the data, builds a split, and iterates one batch. It is the highest-value item in the package, because it is the only one that fails loudly when it is wrong.

The card fields that carry the weight

A dataset card has become a recognizable genre, and the fields that matter most are not the descriptive ones.

• Identity: name, version, date, owner, contact, licence, and any use restrictions, written as restrictions rather than as encouragement.
• Provenance: who collected it, where, when, from what population, under which consent version, and any jurisdiction-specific conditions. The consent version determines what the data may be used for, and it changes over time.
• Composition: languages and varieties, speaker count, demographics as ranges rather than exact distributions, hours by slice, recording conditions and devices, and annotation depth.
• Structure: directory layout, naming convention, formats, and the split rule — random, by speaker, or by session. State the rule, because a receiver who re-splits with a different rule creates a leak that stays invisible in the aggregate metric.
• Labels: the annotation schema, the guideline version the labels were produced under, and the agreement figures with the method used to compute them. An agreement number without its method is not a fact.
• Quality: the acceptance rate, the defect mix, the sampling method used at acceptance, and the known failure modes.

The two fields that prevent silent errors

Two properties deserve their own line on the card, because getting either wrong produces no error message at all.

The first is leakage. State explicitly whether any speaker, session, or recording site appears in more than one split. In a corpus built from repeated sessions with the same people, a random split by utterance puts the same speaker in training and evaluation, and the resulting evaluation number will be excellent and meaningless. It is the most common serious defect in internally reused data, and it is checkable in one query against the manifest.

The second is out-of-scope uses. Write down what the data cannot support. A corpus recorded as read speech from a fixed script is a poor fit for conversational modelling. A corpus with variable recording conditions is a poor basis for speaker verification. A corpus with no negative examples cannot train a rejection threshold. Stating limitations is not a confession of weakness; it is the field that stops a colleague from spending a month on a use the data cannot serve.

Known issues, written as properties

The known-issues list is where most handovers are thinnest and where the most value sits. The format that works has four fields per entry: what the issue is, how to detect it, how much of the corpus it affects, and what to do about it.

• What: background speech is present and untranscribed in part of the corpus. Detect: a flag in the manifest. Extent: a stated fraction of files. Workaround: filter on the flag, or keep it when the model is meant to be robust to background speech.
• What: a subset of sessions has a higher noise floor. Detect: the signal-to-noise field in the manifest. Extent: the listed session identifiers. Workaround: use for robustness training, exclude from a clean evaluation set.
• What: one participant was recorded under two identifiers. Detect: two identifiers sharing session dates and site in the manifest. Extent: the affected rows. Workaround: treat as one speaker when splitting, and fix the identifiers in the next version.
• What: one section of the guideline was applied inconsistently. Detect: a batch flag marking material produced before the guideline revision. Extent: the batches in question. Workaround: use for training only, not for evaluation.

Version the card with the corpus

The card describes a version, and the version has to be a field on the card itself. A card describing a corpus version that no longer exists is worse than no card, because it is confidently wrong.

Adopt one rule: any change to the data bumps the corpus version, and any change to the card bumps the card version and records what changed. Keep the old cards. When a model was trained on version three and the corpus now sits at version five, the old card is the only record of what was actually used.

Finish with a walkthrough rather than an email. Hand the package over in a session where the receiver restates what the data is, how they will split it, and what they will not use it for. Every correction in that conversation is a sentence the documents were missing.