Managing a multi-language data program

Planning ends where management starts

A multi-language program is a set of workstreams sharing one calendar, one budget, and one release. The planning decisions — which languages, in what order, at what volume — are made once. Everything after that is a series of small deviations, and the job is to absorb them without renegotiating the whole plan.

Two things break programs. The first is quality that cannot be compared across languages, so nobody can tell whether a workstream is healthy. The second is a single language that slips and takes the release down with it. Both have concrete fixes, and both depend on artifacts that have to exist before the first batch.

Make quality comparable without comparing it

The trap is comparing outcome metrics across languages. Agreement between annotators runs lower in a language with a contested orthography, and error rates are not comparable between languages with different morphology. A program that ranks languages by those numbers will spend its attention on the languages that are hardest to measure rather than the ones that are worst.

Compare process metrics instead, computed identically everywhere.

• Acceptance rate at first review, which reflects whether the guideline is being followed rather than whether the language is difficult.
• Rework hours as a fraction of production hours, which stays comparable because it is a ratio measured inside each language.
• Agreement on a double-annotated subset, tracked against that language's own baseline and its own trend rather than against other languages.
• Defect mix by reason code, using one shared reason-code list across every language. Without a shared list, a language that looks worse may only be labelled differently.
• Forecast movement: how far the projected completion date moved this week. This predicts trouble before any quality number moves.

The anchor set is what keeps reviewers aligned

Each language needs a small adjudicated reference: twenty to thirty minutes of material with agreed ground truth, produced once by the central team and the language reviewer together, with every disputed decision resolved and written down.

The anchor set does two jobs. It qualifies a new annotator or reviewer, who must reproduce it within a stated tolerance before touching production material. And it settles disagreements, because it converts an argument about whether a batch is acceptable into a comparison against a fixed reference.

Keep the anchor set out of training. It is an instrument, and an instrument that has been trained on stops measuring.

Stagger the start dates on purpose

Languages can run in parallel; the review function usually cannot, and that is where the shared constraint lives. If three languages reach their first-batch review in the same week, the review becomes a queue and all three slip together.

The lever is the start date. A practical rule: do not open two new languages inside the same two-week window unless the review capacity is genuinely separate. When the calendar forces an overlap, offset them by stage instead — one language in production while the other is still recruiting — so their peaks land in different weeks.

The same logic applies to pilots. A pilot is review-heavy relative to the hours it produces, so a program that pilots four languages at once carries four times the review load for the same amount of data.

When one language slips

The first question is not how to recover the language. It is whether the language is on the shared path — whether anything else in the release is waiting on it. A delayed language that feeds nothing else is a scheduling problem for one workstream. A delayed language that supplies a shared evaluation set or a joint release is a program problem.

If it is off the shared path, decouple it: ship the release without it and add it later. That only works if the release was built to accept additions, which means the manifest and the splits have to be language-addressable from the beginning, and speaker identifiers have to be stable so that later material appends rather than replaces.

If it is on the shared path, three options in order of cost. Reduce the scope of the delayed language to hit the date, which is the buyer's decision and should be recorded in writing. Split the delivery into a partial release now and the remainder later, with the partial coverage declared in the manifest so anyone training on it can account for the gap. Or move the date, which is sometimes the only option that preserves both scope and quality, and is cheaper than discovering the shortfall at delivery.

• Never ship fewer hours than specified without saying so. An undeclared shortfall becomes an unexplained evaluation result months later, and by then the cause is unrecoverable.
• Never let a slipping language quietly set the quality bar for the others. A workstream under time pressure tends to relax its review, and the relaxation is invisible in the aggregate numbers.

One log, one row per language per week

The artifact that makes all of this manageable is a single table with a row for each language for each week: current stage, units accepted, units in review, open blockers, and a forecast completion date.

The forecast column is the important one. A single slip is noise. A forecast that moves three weeks running is a structural problem, and it is visible weeks earlier than the quality numbers would show it. Set the escalation threshold in advance — a forecast that has moved beyond a stated margin, or a process metric below its baseline for two consecutive reviews — so that escalation is a rule rather than an argument.

Close each language with a handover rather than a delivery. The anchor set, the reason-code list, the guideline addendum, and the agreed conventions for that language are what let the next batch, or another team, work in it without repeating the calibration.