Linguacorpus

Speaker Embedding

Short answer. A speaker embedding is a fixed-length vector — typically 128 to 512 floats — that represents the voice in an utterance, so two utterances can be compared with one cosine similarity. The classical recipe pools statistics over frames (mean and standard deviation of MFCCs, concatenated); the modern one comes from a trained network (x-vector at 512 dimensions, ECAPA-TDNN at 192) and separates speakers far better at fewer dimensions. The vector is not interpretable dimension by dimension, and it carries channel and phonetic content along with identity.

How to do it

What the vector has to do

Three requirements, and they conflict. The output length must not depend on how long the utterance is, so a 1-second and a 30-second recording produce the same number of dimensions. Comparison must be cheap, so it has to be a geometric operation rather than a second model call. And the representation must be more sensitive to who is speaking than to what they are saying, which is the hard part.

That last requirement is what separates a usable embedding from a useless one. Content variation between two utterances from the same speaker is often larger than identity variation between two speakers, especially when the utterances come from different recording sessions.

The classical recipe: statistics pooling over MFCC frames

Extract 20 to 40 MFCCs per frame at a 10 ms hop, then compute the mean and standard deviation of each coefficient over the frames and concatenate. Twenty coefficients give a 40-dimensional embedding; forty give 80. It takes ten lines of code and it works well enough to demonstrate the geometry.

  • The mean captures the average vocal tract shape, which is the largest identity cue.
  • The standard deviation captures how much the tract moves during speech, which separates speakers who differ in articulation rate and clarity.
  • Frame count is discarded, so the vector is the same size for any utterance length.
  • What is missing is any training signal that says "make same-speaker pairs close and different-speaker pairs far apart." That is the whole difference from a modern embedding.

Training is what makes the modern embedding work

An x-vector is the same idea with a trained network in front of it: several time-delay layers over the frame features, then a statistics pooling layer, then a bottleneck that produces 512 dimensions. ECAPA-TDNN does the same with attention-based pooling and squeeze-excitation, and gets better results at 192 dimensions. The dimension count is a design parameter, not a quality measure.

The training objective is what changes the numbers. A softmax classifier over thousands of training speakers, plus a margin penalty (AAM-softmax) that forces same-speaker vectors to be tighter, produces a distribution where same-speaker cosine similarities cluster near 0.7-0.9 and different-speaker similarities near 0.0-0.2. The untrained MFCC statistics produce overlapping distributions, which is exactly what the code below shows.

The `d vector` name is historical. It comes from the 2014 Deep Speaker work and refers to the same object: a fixed-length vector from a deep network, with cosine or PLDA scoring on top.

How the vector gets compared, and what it leaks

The standard is L2-normalize, then cosine similarity. PLDA is an alternative scoring backend that models the within-speaker and between-speaker distributions explicitly; it usually beats raw cosine by a few points of EER when the training data is large enough to estimate those distributions, and it is worth the extra complexity for identification at scale.

The vector also encodes channel, language, and speaking style, because those factors are present in the training data and no objective removes them completely. Two recordings of the same person on a laptop microphone and a phone can be farther apart than two different people on the same headset. That is the failure mode behind almost every "the system works in the lab" story, and it is why domain-matched enrollment audio matters more than a bigger model.

Code

Build a 40-dimensional embedding from MFCC mean and standard deviation over frames, then compare same-speaker and different-speaker pairs with cosine similarity.

import numpy as np
import librosa

SR = 16000

def voice(f0, formants, seed, dur=2.0):
    """Harmonic stack with two fixed formant peaks: a crude but deterministic voice."""
    rng = np.random.default_rng(seed)
    t = np.arange(int(SR * dur)) / SR
    vib = 1.0 + 0.02 * np.sin(2 * np.pi * 5.0 * t)
    sig = np.zeros_like(t)
    for k in range(1, 60):
        f = k * f0
        if f > SR / 2:
            break
        env = (np.exp(-((f - formants[0]) / 400.0) ** 2)
               + 0.7 * np.exp(-((f - formants[1]) / 700.0) ** 2))
        sig += (env / k) * np.sin(2 * np.pi * f * vib * t + rng.uniform(0, 2 * np.pi))
    sig += 0.005 * rng.normal(size=t.size)
    return (sig / np.max(np.abs(sig))).astype(np.float32)

def embed(x):
    m = librosa.feature.mfcc(y=x, sr=SR, n_mfcc=20, n_fft=512,
                             hop_length=160, n_mels=40)
    m = m[:, 5:-5]                       # drop edge frames
    return np.concatenate([m.mean(axis=1), m.std(axis=1)])

def cos(a, b):
    return float(a @ b / (np.linalg.norm(a) * np.linalg.norm(b)))

A = [embed(voice(120, (500, 1500), s)) for s in (1, 2)]
B = [embed(voice(205, (350, 2100), s)) for s in (3, 4)]

print('embedding dimension:', A[0].size, '(20 MFCCs x mean and std)')
print('same speaker  A1-A2: %+.3f' % cos(A[0], A[1]))
print('same speaker  B1-B2: %+.3f' % cos(B[0], B[1]))
print('different     A1-B1: %+.3f' % cos(A[0], B[0]))
print('different     A1-B2: %+.3f' % cos(A[0], B[1]))
gap = min(cos(A[0], A[1]), cos(B[0], B[1])) - max(cos(A[0], B[0]), cos(A[0], B[1]))
print('margin between the closest same pair and the closest different pair: %+.3f' % gap)

# The energy coefficient c0 dominates an un-normalized cosine. Drop it and compare.
strip = lambda v: np.concatenate([v[1:20], v[21:]])
print('same margin with c0 removed: %+.3f' % (
    min(cos(strip(A[0]), strip(A[1])), cos(strip(B[0]), strip(B[1])))
    - max(cos(strip(A[0]), strip(B[0])), cos(strip(A[0]), strip(B[1])))))
  • pip install numpy librosa soundfile. librosa pulls in scipy, which it needs for the MFCC filterbank.
  • Expected output: both same-speaker pairs score higher than both different-speaker pairs, with a positive margin. The absolute values depend on the synthetic signal, so read the margin, not the individual numbers.
  • The last two lines are the useful part. c0 (log energy) is nearly identical across these signals and inflates every similarity toward 1.0, which makes the margin look smaller than it is. Removing it is the standard first fix for a cosine comparison that will not separate.
  • For a real system, replace embed() with a trained x-vector or ECAPA-TDNN model. The scoring code below it does not change.

Where this goes wrong

Treating the dimension count as a quality signal

A 512-dimensional x-vector and a 192-dimensional ECAPA-TDNN are not "more" and "less" of the same thing. ECAPA-TDNN is smaller and better on every published benchmark, because the training objective and the pooling are what carry the information. Vendors quoting a dimension count as evidence of quality are telling you nothing; ask for EER on a named evaluation set with the trial protocol.

Cosine similarity on raw MFCC statistics without normalization

The energy coefficient (c0) and the overall level of the recording dominate the dot product, so two loud recordings of different people score higher than two quiet recordings of the same person. L2-normalize the vector, and consider dropping or mean-normalizing c0. Teams that skip this see similarity scores clustered between 0.95 and 1.00 that separate nothing.

Enrolling on one channel and testing on another

A single recording session, microphone, or language contributes a large, consistent offset to the embedding. If every enrollment utterance is a headset recording and the test utterances are phone calls, the channel difference alone can exceed the speaker difference. Match the enrollment channel to the deployment channel, or use enough enrollment audio from enough sessions that the average cancels.

Averaging embeddings from very different conditions

Concatenating or averaging enrollment embeddings from mismatched conditions produces a centroid that sits between two clusters and matches neither. Averaging works when the enrollment utterances are homogeneous; when they are not, score against each enrollment utterance separately and take the maximum, or apply score normalization before averaging.

Using one threshold across different embedding models

The score scale is a property of the model and the training data. A threshold of 0.65 that works for one x-vector checkpoint is meaningless for the next one, and switching from cosine to PLDA changes the scale entirely. Store the threshold with the model version and re-derive it on a matched development set every time either changes.

When to buy the data instead of building the pipeline

Buy when you need speaker diversity you cannot assemble yourself: hundreds of speakers, multiple sessions per speaker, and consistent channel metadata so enrollment and test can be split by session. Buy when the language or accent coverage has to match a deployment population that is not the one you can record locally — an embedding trained and evaluated only on North American English speakers has an unknown error rate everywhere else. Building it yourself is the right call when you have a specific speaker set you control and you only need the embedding for that set; a weekend of recording gives you cleaner control than any purchase. The purchase becomes necessary when the evaluation itself has to be defensible.

How buying training data works →

More technical reference

All technical reference →

Need data for Speaker Embedding?

Tell us the language, the hours, and what the data needs to look like. You will get a real number and a real timeline — not a range. If we cannot source it well, we will tell you that instead.

  • Pilot batch before the full run, so problems surface early.
  • Consent documentation delivered with the data.
  • No medical or clinical data. No recorded telephone calls.

We reply within two business days. Your details are used only to answer this request. See our privacy policy.

Contact

Talk to a human

Send a specification and we will come back with a real number and timeline.

Submit a sourcing request

Or email hello@linguacorpus.com