Speaker verification with embeddings: setting a threshold you can defend

Verification answers one question: are these two recordings the same person. The model produces one vector per recording and a similarity between the two vectors, and the whole engineering problem is choosing the number at which you say yes. Everything below is about making that number mean something outside the dataset it was chosen on.

Get embeddings from a model you intend to keep

With pyannote: model = Model.from_pretrained("pyannote/embedding", use_auth_token=TOKEN), then inference = Inference(model, window="whole"), and emb = inference("a.wav"). The whole window averages over the file, which is fine for a short utterance and wrong for a ten-minute recording that contains several conditions.

For long audio, use a sliding window and score the best-matching window rather than the average, because an average over a long file describes the recording setup as much as the speaker. The embedding model is part of the system, not a preprocessing step: a threshold is only valid for the model that produced the vectors.

Build the trial list before tuning anything

A trial is a pair of recordings with a target or non-target label, and the list decides the result more than the model does.

Four rules keep it honest. Enroll and test on different sessions, never the same recording. Include same-session impostors if the deployment will see them, and if you exclude them, write down that you did. Report the number of trials and the number of distinct speakers on each side, because five hundred trials drawn from five speakers are not five hundred independent trials. Tune on a split and report on the rest.

Then state the prior, because EER is computed on a balanced list by construction and deployment is never balanced. If one attempt in a hundred is a genuine match, a system at 5 percent EER will still be wrong most of the time it says yes.

EER and minDCF, and which one to quote

EER is the threshold where false accepts equal false rejects. It needs no cost assumption, which is why it is the standard way to compare systems, and it is also why it is a poor description of how you will run one.

The detection cost function encodes the assumption: Cdet = Cmiss * Pmiss * Ptarget + Cfa * Pfa * (1 - Ptarget), normalized by the cost of the trivial always-accept or always-reject decision. With the usual equal costs and a target prior of 0.01, the operating point sits at a much lower false-accept rate than the EER point does.

Quote EER to compare models. Quote minDCF to describe the operating point you would actually run. A system with a better EER can have a worse minDCF at your prior, and that is not a paradox, it is the point of having two metrics.

One arithmetic caution on the sweep itself: with a hundred non-target trials, the smallest measurable false-accept rate is one percent, so an EER below that is not measurable and should be reported as a bound with the trial counts attached.

Normalize the scores or the threshold will not travel

Raw cosine scores shift with recording condition, so a single threshold over-accepts on the easy condition and under-accepts on the hard one. Cohort normalization fixes most of it: for each test embedding, compute similarities against a few hundred non-target embeddings from a cohort set, then subtract the cohort mean and divide by its standard deviation. The adaptive variant uses only the top scoring cohort members, with around two hundred as a typical count.

Verify that it worked by computing the mean and standard deviation of target scores per condition before and after. If the spread between conditions shrinks, the normalization is doing its job and one threshold becomes defensible across the corpus.

Thresholds, costs, and one limit worth stating

Decide the cost ratio first, then read the threshold off the trial scores at that ratio rather than at the EER point. Report the operating point as a false-accept and false-reject pair with the trial counts, the conditions, and the embedding model named alongside it.

Re-check the threshold on a later session. A threshold that holds in week one and drifts by week four is usually a channel change rather than a model change, and that is a finding about the deployment, not about the model.

The limit: an embedding model produces a similarity, not a probability. If the product needs a probability, calibrate — fit a logistic regression on the trial scores with the true prior — and remember that the calibration only holds for the population it was fitted on.