Voice conversion versus voice cloning: two problems that look alike

Both produce audio that sounds like a particular person, and both get described with the same words in a sales conversation. Underneath, they are different tasks with different inputs, and that difference decides what data the project needs and who has to sign off on it.

The shortest version: cloning turns text into speech using a target voice. Conversion takes speech that already exists — a specific performance, with its timing, emphasis, and emotion — and re-renders it in a target voice. One needs text-audio pairs; the other needs audio-audio data and a much smaller reference set for the target.

The input difference drives the whole data spec

For cloning, the training material is recordings of the target speaker paired with their transcripts, and the volume question is really a coverage question.

For conversion, the target speaker's contribution can be small. A few minutes of clean, consistent speech is often enough to model timbre, because the system is not learning what to say — only how this person sounds. The expensive part sits on the source side. If the goal is supervised training, the useful material is parallel: the same content spoken by the source speaker and by the target speaker. Parallel speech is costly to produce, because the second speaker has to match the first speaker's content and ideally their timing, and it is the largest single cost driver in a supervised conversion project.

Non-parallel methods remove that requirement at a price. Output tends to carry artifacts, and the prosody is inherited from the source, which can sound subtly wrong when the target speaker's own habits differ. Real projects often mix the two: a small parallel set to anchor quality, a larger non-parallel set for robustness.

What conversion buys, and where it breaks

Conversion preserves the performance, and that is the entire point. Delivery, emphasis, timing, and emotion come from the source recording, which means a performance can be re-voiced without being re-performed. That is what makes the technique useful for dubbing, for accessibility work, and for reusing an existing recording in a market where the original performer is not available.

Its failure modes are inherited from the source. Noise, clipping, breaths, and room tone come through in the output, because the system is asked to preserve everything except identity. Large pitch or speaking-rate differences between source and target produce artifacts, and a very deep source voice mapped onto a high target voice tends to sound strained. Because prosody is copied rather than regenerated, an output can sound like the target person without sounding like them — a specific kind of uncanny that a similarity score will not catch.

That last point is the reason to insist on listening tests rather than similarity metrics alone. Similarity and naturalness are perceptual properties. A mean opinion score protocol, run on your own material and on both candidate systems, is the only comparison that travels between teams.

The rights chain has two people in it, not one

This is where the two tasks diverge most sharply, and it is the part that gets skipped.

A cloning project touches one person: the speaker whose voice is being modeled, with their agreement covering the specific use. A conversion project touches two. There is the target voice, the person whose identity the output carries, and there is the source performer, whose delivery is being reused. If the source is a real person's recording, that recording is also personal data about them, and a re-voiced version of their speech is a strong candidate for being treated as identifiable under the same reasoning that treats voice as biometric data. Agreement from the target does not cover the source, and permission to use a recording does not automatically cover re-voicing it.

There is also a category of case that no consent fixes. Using conversion to make a recording sound like someone who has not agreed to it — a public figure, a colleague, a customer whose call was recorded — is out of bounds regardless of how the source material was obtained.

So the first document a conversion project needs is not a recording plan. It is a list of whose rights are involved, with permission for each one written down before any audio is processed.

Choosing between them, in practice

One question settles most of these quickly. Ask who the person in the output is, and where their performance came from. If both answers point at the same consented speaker, the project is cloning. If the performance came from someone else, it is conversion, and the work starts with the source's paperwork.

• To say new things in a specific voice: cloning, and the recording plan is the project.
• To say the same things in a different voice — localization, dubbing, re-voicing an archive — conversion is usually the cheaper route, because the performance already exists and does not have to be paid for twice.
• To get an expressive character voice for arbitrary text: cloning plus a style-coverage recording plan, since conversion cannot invent a performance it was never given.
• When the source material is one real person's recording and the target is another real person, treat the project as a rights project first and a technical one second.

What the data spec looks like for each

Open-source toolkits such as ESPnet and Kaldi ship recipes for both directions, which makes it easy to prototype either one before committing to a data collection. What they do not settle is the specification, which is where the two projects stop resembling each other.

• Cloning: one speaker, a coverage list, consistency controls, and a consent scope that survives the model being trained.
• Conversion: a target reference set that is clean and consistent, a source set whose rights are documented, and a parallel subset if supervised training is the plan.
• Both: a listening-test protocol agreed before delivery, because perceptual scores produced by a supplier are not comparable to scores produced by you.