What is a voice agent, and the data a demo never has

A voice assistant matches what it heard to an intent and answers. A voice agent holds a conversation: it decides when to respond and when to wait, calls tools in the middle of a turn, recovers when it mishears, and finishes a task rather than a sentence.

That difference is not an architecture detail a buyer can leave to the engineering team. It changes what has to be in the data, and most conversation corpora on the market were built for the assistant problem: clean turns, one intent each, no interruption, no outcome label.

The hard part is timing, not transcription

Recognition accuracy is necessary and not sufficient. The failure a user notices first is rarely a wrong word. It is the agent talking over them, or pausing long enough that they repeat themselves, or answering an acknowledgment as if it were a request.

This is why the channel structure of the audio matters more here than in any other speech project. Two speakers captured on separate channels preserve exactly who spoke when, including the overlap. A mixed-down recording cannot teach a model when to yield, because the information needed to learn that was destroyed by the mixing. Ask for dual-channel capture, sample-accurate, with the overlap kept rather than cleaned up.

The labels are the second half. Backchannels — the short acknowledgments people make while listening — have to be marked as such, because they look like short turns and are not. Pauses need a rough classification: the pause that means the speaker has finished, the pause that means they are thinking, and the pause that means they were interrupted. None of that can be recovered afterwards from a transcript.

The coverage grid, and why the average is not the number

A voice agent is deployed to whoever calls, and the error rate that matters is the worst slice rather than the mean. Coverage therefore becomes a design decision: which accents, which devices, which environments, in what proportions.

The grid is larger than it looks. Accent, device path — handset, speakerphone, headset, car — and acoustic environment are three axes, and performance varies along all three. A collection that is well balanced on accent and entirely recorded in a quiet room has not covered the car. A collection recorded through a single device path cannot tell you what happens on any other.

Choosing the cells to cover deliberately, and stating which ones you are not covering, is more useful than collecting a broad sample and hoping the proportions work out. The cells you skip are the ones that generate support tickets later.

Outcome labels are what make the data about the task

The metric for an agent is whether the task got done, and that cannot be computed from a transcript. It needs an episode-level label: what the user was trying to accomplish, whether it succeeded, and if not, where it broke — misheard, routed to the wrong action, asked for information the user had already given, or simply never finished.

Those labels are expensive in a specific way. They require listening to a whole episode rather than reading segments, and the judgment is often about intent rather than wording. In exchange, they are what turns a pile of recordings into something you can train and evaluate against, because the interesting failures are not transcription errors.

Record the agent's own actions on the same timeline — which tool was called, when, and with what arguments — or the failure analysis cannot separate a hearing problem from a decision problem.

What you can buy, and what you have to build

Conversation corpora, intent and slot sets, interruption-heavy material, and accent coverage are all purchasable, and they are worth buying because collecting them takes longer than building the rest of the system.

The evaluation set is not. Your tool inventory, your users' phrasing, your device mix, and your failure modes are yours, and a test set drawn from someone else's product measures someone else's product. Build the first version from your own pilot calls even if it is small, and then freeze it.

There is one more thing worth buying deliberately: the negative space. Recordings of requests the agent should hand off to a person, should refuse, or should ask a clarifying question about. Those cases are cheap to capture during a pilot and nearly impossible to reconstruct afterwards.

What to put in the specification

The recognizer-facing details belong on the list even though they look like the engineer's problem, because the agent inherits the recognizer's requirements and cannot outperform them.

• Dual-channel or mixed, and whether overlap is preserved in the delivery.
• How backchannels and thinking pauses are labeled, and by what rule.
• The coverage grid: which accent, device, and environment cells are included, with counts per cell.
• Whether each episode carries an outcome label and a timeline of the agent's actions.
• Sample rate, noise floor, and whether levels are normalized across sessions.