Scoring code-switched audio without penalizing the model twice

A standard scoring pipeline assumes one language and one script per utterance. Code-switched audio breaks both assumptions at once, and it breaks them in a way that shows up as a worse error rate rather than as an obvious failure.

The result is a number that punishes a model for something it got right, and a team that responds by collecting more data for a problem it does not have. The fix is to score the same audio four ways and read the four numbers together.

The double penalty

Suppose the reference writes an embedded English technical term in the matrix language script, and the model outputs the same word in Latin letters, correctly. The aligner sees two different strings and records a substitution. The model heard the word and is scored as having missed it.

It gets worse around the boundary. A substitution changes the alignment path, so a single script difference can push the surrounding tokens into an insert-delete pair, turning one formatting mismatch into three errors. On a set with a high switch rate, this can add several points to the headline without any acoustic failure at all.

The test is cheap: take the utterances with the largest error counts, print the alignments from jiwer.process_words, and read the chunks. If the substitutions are script variants of the same word, the score is measuring orthography rather than recognition.

Score it four ways and read them together

Once the reference carries a language tag per token and the index of each switch, each of these is a filter over the token list rather than a new pipeline.

• Matrix-language WER, computed only over tokens tagged with the base language. This is the number most comparable to a monolingual baseline.
• Embedded-language WER, computed only over the switched tokens. A large gap here is a coverage problem, and more embedded-language training audio is the fix.
• Switch-point WER, computed over tokens within two positions of a boundary, on both sides. If this is much worse than the other two, the model handles both languages and fails at the transition, which is a context or data-mix problem.
• Script-tolerant WER, where an embedded token counts as correct if it matches any accepted written form. This is the floor, and the gap between it and the strict number is the size of the orthography penalty.

How to build the script-tolerant comparison

The cheap version transliterates the embedded tokens on both sides into one script with a deterministic function before scoring, then scores that copy and reports it beside the strict score. unidecode is enough when one side is Latin and the other is not, since it maps non-ASCII characters to an ASCII approximation.

The limitation is real and worth stating in the report. Transliteration is not reversible and it collapses distinctions, so a tolerant score is a lower bound rather than a replacement. It will also accept errors that a human would call wrong, which is why the strict number stays in the table.

The more precise version stores a list of acceptable forms per token position and takes the minimum edit cost across the options. That requires the annotation guideline to define the acceptable forms up front, which is the same work as defining the script policy, done once and written down.

The boundary labels have to exist before scoring

None of the four numbers can be derived reliably from the reference text after the fact. Guessing which words are switched by looking at the script fails on loanwords, proper nouns and numerals, which are exactly the tokens where annotators disagree.

So the annotation specification needs two things: a language tag per token rather than per utterance, and the index of each switch. With those two fields, the switch-point window is a slice over a token list, and the window size is a parameter you can report. Two positions on each side is a reasonable default and a defensible one to state.

Report the switch rate next to the score

Two sets both described as code-switched can differ enormously. In one, five percent of utterances contain a switch and the set behaves almost like a monolingual one. In another, the average utterance contains three switches and the boundary failure dominates everything.

Three numbers make the set comparable to another set: the share of utterances containing at least one switch, the mean number of switches per utterance, and the share of tokens in the embedded language. Without them, a score on this audio is not comparable to any other score on any other set, and the comparison will be made anyway.

One construction to check for, since it is common in assembled test sets: clips from two monolingual recordings spliced into one file. The splice preserves clean acoustic context on both sides and puts every switch at a fixed offset, which makes the boundary artificially easy. If the model errors cluster at the same offsets as the splices, the set is measuring splicing.

What the four numbers let you say

The value is not a flattering score. It is being able to state which failure you have, because the three candidates have different remedies and only one of them is fixed by buying more audio.

Coverage failures need embedded-language data. Boundary failures need data with more switches and better context handling. Script failures need an annotation convention, and they are the only one of the three that costs nothing to fix.