Linguacorpus

Code-Switching Speech Recognition

Short answer. Code-switching ASR transcribes speech that alternates between two languages mid-sentence, and it is scored against language-tagged tokens so the reference records which language each word was in as well as what was said. The failure mode unique to this task is normalization: if the reference writes a borrowed word in Devanagari and the system outputs the Latin spelling, an unnormalized WER counts every character as an error and the score approaches 1.0 on a transcript that is substantially correct. Fix the script policy, the tag convention, and the loanword spelling before you train or evaluate anything.

How to do it

Tag the language per token, not per utterance

A code-switched utterance has more than one language label, so an utterance-level language field cannot describe it. The reference needs a language ID on every token, or explicit switch-point markers, or both. The usual serialization is inline tags — `<hi>मुझे <en>computer <hi>चाहिए` — which survives a copy and paste into a spreadsheet and is readable by an annotator without tooling.

This is not bookkeeping. If the product routes the transcript to a downstream language-specific model, or if you want to measure how well the system handles the switch itself, the tags are the deliverable. A pipeline that strips them before scoring has measured transcription and called it code-switching performance.

Pick one script policy and write it into the guideline

The same audio can be transcribed correctly in at least three ways, and they are not interchangeable strings.

All three are defensible. What is not defensible is letting each annotator choose, because then the corpus contains all three conventions and the model learns that the same sound has three spellings.

  • Matrix-language script throughout: the English word is written in Devanagari as कंप्यूटर, matching how the sentence reads as Hindi. Natural for a Hindi-first reader, and it hides the language boundary from any text-only model.
  • Each language in its own script: मुझे computer चाहिए. The boundary is visible in the text, which makes switch detection learnable from the transcript alone.
  • Romanized throughout: mujhe computer chahiye. Cheapest to type and to keyboard, and it destroys the orthographic distinction between the two languages entirely.

Normalize both sides with the same function, or you are measuring spelling

Before computing WER, run the reference and the hypothesis through the identical normalizer: strip the language tags, casefold, collapse whitespace, and apply a fixed mapping for numbers and for a listed set of loanwords. The mapping is the important part. If your reference spells the borrowed word in Devanagari and your system outputs Latin, a normalizer that maps both to one canonical form is the difference between a 0.25 WER and a 0.0 WER on the same correct transcript.

Write the normalizer down and publish it with the number. Two teams reporting code-switching WER without stating the normalization are reporting numbers that cannot be compared, and the gap between the strictest and loosest reasonable policy on the same output is routinely 15 to 30 points.

Report switch-point performance separately from the overall number

Overall WER on a code-switched test set is dominated by the matrix language, because the matrix language is most of the words. A system can report 12% overall WER and still fail on every single switch, and the overall number will not show it. Score the tokens within one or two positions of a switch point as their own set, and report the switch rate of the test set — switches per 100 words — alongside the WER.

The other number worth having is the language identification accuracy on the switched tokens themselves, because a system that transcribes the word correctly but tags it with the wrong language will break every downstream consumer that trusts the tag.

Code

Score one code-switched utterance under four script policies. The audio is identical and every word was heard correctly; only the reference convention changes.

import jiwer

def parse(tagged):
    """'<hi>mujhe <en>computer' -> [('hi', 'mujhe'), ('en', 'computer')]"""
    out, lang = [], 'hi'
    for tok in tagged.split():
        if tok.startswith('<') and tok.endswith('>'):
            lang = tok[1:-1]
        else:
            out.append((lang, tok))
    return out

def wer(ref, hyp):
    return jiwer.wer(' '.join(t for _, t in ref), ' '.join(t for _, t in hyp))

def switches(tokens):
    return sum(1 for a, b in zip(tokens, tokens[1:]) if a[0] != b[0])

REF_DEV = '<hi>मुझे <en>computer <hi>चाहिए <en>please'
REF_ROMAN = '<hi>mujhe <en>computer <hi>chahiye <en>please'
HYP_DEV = 'मुझे computer चाहिए please'
HYP_ROMAN = 'mujhe computer chahiye please'
HYP_DEV_LOAN = 'मुझे कंप्यूटर चाहिए please'      # loanword written in Devanagari

ref = parse(REF_DEV)
print('reference: %d tokens, %d switch points' % (len(ref), switches(ref)))

tests = [
    ('tags stripped on both sides', ref, parse(HYP_DEV)),
    ('romanized ref vs romanized output', parse(REF_ROMAN), parse(HYP_ROMAN)),
    ('romanized ref vs Devanagari output', parse(REF_ROMAN), parse(HYP_DEV)),
    ('Devanagari ref vs Devanagari output, Latin loanword', ref, parse(HYP_DEV_LOAN)),
]

for name, r, h in tests:
    print('%-52s WER %.3f' % (name, wer(r, h)))

print('tokens still correct in the last row: 3 of 4')
print('the only difference is which script the borrowed word was written in')
  • pip install jiwer. Everything else is the standard library.
  • Expected output: rows 1 and 2 score 0.000 — the transcripts match once the tags are gone. Row 3 scores 0.500 because the Devanagari output and the romanized reference share no characters for two of the four tokens. Row 4 scores 0.250 because one borrowed word was written in the other script.
  • Nothing changed in the audio between rows 1 and 4. The 0.0-to-0.5 spread is entirely the reference convention, which is why the convention has to be specified in the data contract rather than left to the annotator.
  • The file is UTF-8 by default in Python 3, so the Devanagari strings need no encoding declaration. If your terminal cannot render them, the WER numbers are unaffected.

Where this goes wrong

Evaluating on artificially concatenated code-switching

The cheap way to build a code-switching test set is to splice two monolingual sentences at a sentence boundary. Real switches do not happen there: they happen inside the noun phrase, on the borrowed term, mid-intonation-unit, with no pause. Systems evaluated on concatenated data score 10 to 25 points better than on natural switches, and the gap is largest exactly where the product will be used. If the test set was built by splicing, say so.

Balancing the switch rate and calling it natural

Natural switch density varies enormously by community and register — from under 1 per 100 words to over 15. A test set built to 50% switched tokens measures a task nobody speaks. Report switches per 100 words for the test set and for the deployment population; if they differ by more than a factor of two, reweight before you quote the number.

Normalizing the hypothesis but not the reference

This happens when the normalizer lives in the model output pipeline rather than in the scorer. The reference keeps its tags and its own script convention, the hypothesis arrives cleaned, and the score is dominated by the mismatch rather than the errors. The normalizer must run on both sides, in the same order, inside the same function that computes the metric.

Treating a loanword as one decision instead of three

A borrowed English word spoken inside a Hindi sentence can be written in Latin (computer), transliterated into Devanagari (कंप्यूटर), or in a community spelling that matches neither (kompyuter). Annotators will each pick a different one unless the guideline names the categories explicitly — technical terms, established loanwords, proper nouns, numerals — and gives the rule for each. This one paragraph of guideline usually moves inter-annotator agreement more than any amount of training.

Scoring the whole utterance and stopping there

Overall WER hides the switch. A model with a strong matrix-language language model can produce a fluent transcript that silently translates or drops the switched segment, which reads as a small number of errors in the total but as a total failure on the tokens that carry the meaning. Always break the score out by distance to the nearest switch point.

When to buy the data instead of building the pipeline

Buy when you need word-level switch boundaries and both-script renderings of the same audio, because those are the two things a generic corpus will not have and that annotators cannot add later without hearing the audio again. Buy when the language pair includes a script split, since the transliteration decisions have to be made consistently across thousands of utterances and that is a guideline problem more than an annotation problem. Buy when you need natural switch density at a stated rate, because synthetic splicing is the alternative and it produces a test set that flatters every system. If you have a bilingual in-house team and a single product surface, annotating a few hours yourself is cheaper and gives you control over the guideline — the purchase starts to make sense past a few hundred hours or a second language pair.

How buying training data works →

More technical reference

All technical reference →

Need data for Code-Switching Speech Recognition?

Tell us the language, the hours, and what the data needs to look like. You will get a real number and a real timeline — not a range. If we cannot source it well, we will tell you that instead.

  • Pilot batch before the full run, so problems surface early.
  • Consent documentation delivered with the data.
  • No medical or clinical data. No recorded telephone calls.

We reply within two business days. Your details are used only to answer this request. See our privacy policy.

Contact

Talk to a human

Send a specification and we will come back with a real number and timeline.

Submit a sourcing request

Or email hello@linguacorpus.com