TTS style and prosody data: the recording matrix is the dataset

Define the style set before the studio is booked

A style collection has two common designs. Discrete categories — neutral, cheerful, serious, sad, angry, whisper — optionally with a strength level, are what most projects use, because a director can ask for them and a model can be conditioned on them. Dimensional ratings on axes such as valence and arousal carry more information, but they are harder for speakers to hit on purpose and harder for raters to agree on, and a model conditioned on two continuous numbers needs far more data to learn the axes.

The taxonomy mistake to avoid is mixing three different things in one flat list. Emotion (angry, sad), register (news, audiobook, conversational assistant) and delivery (whisper, projected) are separate axes. A dataset that lists them together produces a label set where the categories overlap and a model that cannot separate them. Each style also needs a one-line definition and an audible example, or excited means something different to every speaker who walks into the booth.

The recording matrix: the same text across styles and takes

Style is only learnable if the same words appear in different styles, so the shape of the collection is a matrix: sentences, times styles, times takes. A single take per sentence per style teaches the model that each style has exactly one prosody. Several takes of the same sentence in the same style give the model the distribution — what varies between renditions and what stays constant — which is what makes synthesized speech sound alive rather than stamped.

The confound to control is who recorded what, and when. The same speaker should record every style, because a design where one voice covers the happy set and another covers the sad set teaches the model that style and speaker are the same variable. Within a session, the order of styles should be balanced or randomized: if the energetic style always lands in the first hour and the quiet style in the last, the model learns the difference between a fresh voice and a tired one. A rejected-take log, with a reason for each rejection, keeps the yield visible, because style collections reject more takes than neutral ones and that number belongs in the plan.

Scripts that carry prosody

Two properties make a script useful for prosody work. Phonetic balance, so every phone appears in several contexts and the acoustic side of the model is not skewed toward one vocabulary. And sentence shapes that exercise the prosodic system: statements, yes-or-no questions, questions that open with a question word, lists, sentences with emphatic stress, and long sentences with internal clause boundaries that force the speaker to plan intonation across a span.

The sharpest training signal is a set of minimal pairs in which the words are identical and the meaning is carried by stress or intonation. Written down, the two members of a pair are the same sentence; spoken, they are different utterances, and only the prosody separates them. These pairs are rare in off-the-shelf scripts and are exactly what a style-conditioned model needs. The transcript must also be exact, including punctuation, because punctuation is the only text-side cue to phrase boundaries and a missing comma is a lost label.

Record for pitch and timing, not for loudness

Prosody lives in the pitch track and the timing, and the standard cleanup chain damages both. Aggressive noise reduction smears the pitch contour it is trying to clean. Lossy codecs at low bitrates flatten the fine structure of the pitch. Automatic gain control removes the intensity differences that carry emphasis, which is the signal itself. A de-esser or a compressor in the chain does the same thing more gently.

The recording spec follows from that: capture at a lossless or high-bitrate setting, keep automatic gain control off, hold a constant distance from the microphone because proximity changes the spectrum, and prefer a room with a short reverberation tail, since a long tail blurs the timing cues that prosody models depend on. One check matters more than the rest: clipping. The emphatic takes are the loudest ones and they clip first, and clipped audio cannot be recovered by any later step. A raw, single-channel file per take, kept alongside any processed version, lets a later pipeline make its own decisions.

Verify the style labels with listeners, and keep take-level metadata

The speaker was asked to sound sad, and that is not evidence that the audio sounds sad. The verification is a listening check on a sample: raters hear a clip and pick the intended style from the set, and the confusion matrix shows which styles sit too close together. Sad, tired and calm sit close together on the perceptual map, and when two styles are confused at a high rate the fix is to merge them or to sharpen their definitions with new examples, not to ship the labels as recorded.

The metadata that makes the collection usable sits at the take level: the style label, the intended strength, the take index, the session, the speaker, the text identifier, and whether the take was rejected and why. That table lets a buyer train on a subset, re-balance the styles, or commission a matching extension later. Without it, a style dataset is one undifferentiated block of audio, and the matrix that was so carefully recorded cannot be recovered from the files.