How to evaluate an annotation provider with a test you control

Why the usual evidence does not discriminate

Every annotation provider will say they follow client guidelines, run quality checks, and employ trained annotators. As written, those statements cannot be tested, which is why evaluations built on them tend to produce a ranking that matches the order the proposals arrived in.

The one piece of evidence that does discriminate is a controlled sample you score yourself, against answers you already hold. It takes an afternoon to build, and it separates providers faster than any number of reference calls.

Two things need designing: the task you give them, and the rubric you score it with. The rubric comes first, because a rubric written after the results arrive will describe the results.

Design the qualification task

The task is a fixed set of items, identical for every candidate, with a reference you control. Six decisions shape it:

• Draw the items from your own material. A public corpus can be found and pre-annotated; your files cannot.
• Cover every stratum and every hard case you already know about, such as overlapping speech, an unintelligible stretch, a mid-sentence language switch, a numeral, or a proper noun with two plausible spellings. The hard stratum decides the ranking, not the easy items.
• Size it so that one lucky guess does not move the score. A few dozen items is the order of magnitude that makes a difference of a few items meaningful.
• Plant defects on purpose: remove a label that should be present, shift a segment boundary, normalize a numeral the wrong way. You are testing whether they catch and flag problems, not only whether they reproduce your conventions.
• Ask for two outputs per item: the annotation, and a short note wherever the guideline does not settle the case. The notes reveal whether the document was read or skimmed.
• Do not let the task be renegotiated. A provider who asks to change the item set before starting has already told you how they will treat the specification later.

Score against your reference, and split the score

A single number hides the thing you are looking for. Compute these separately and read them together:

• Agreement with your reference on the unambiguous items. This is a floor rather than a differentiator, and most providers clear it.
• Agreement on the hard stratum, reported on its own. Averaging hard and easy items into one figure flatters a provider who is reliable only on the easy half.
• Whether the planted defects were caught. A provider who flags every planted problem has a working review step. One who flags none is either not reading carefully or not permitted to escalate, and both are disqualifying at production scale.
• The quality of their questions. Good questions point at real gaps in the guideline. Questions that ask you to restate what the document already says indicate it was not read.
• Turnaround, measured from handover to the returned result including their internal review. A provider who cannot hold a small deadline will not hold a large one.

The phrases that should slow you down

Certain answers recur across providers, and each one has a reading less flattering than it sounds:

• "All our annotators are native speakers." Nativeness answers a question about language, not about convention. Two native speakers will disagree about how to write a filler word unless a guideline picks one answer.
• "We guarantee accuracy above a fixed threshold." Against what reference, on what sample, measured by whom? A guarantee with no reference set is a sentence, not a measurement.
• "We will follow your guideline exactly, no questions." Every usable guideline has gaps. No questions means either the gaps went unnoticed or the answers will be invented silently.
• "We use a multi-layer quality process." Layer count is not a metric. Ask what is measured at each layer, on what fraction of the work, and what happens when a layer fails.
• "We can start immediately." An immediate start usually means no calibration step, so the first production batch becomes the calibration and you pay for it.
• "We have worked on similar projects." Ask which stage this particular team ran, and who on the current team was there.

Run it as a comparison, not a conversation

The test produces a ranking only if it is applied uniformly: the same item set, the same rubric, the same reviewer, and the same deadline for every candidate, with the rubric written down before the first response arrives.

If two people are scoring, score independently and then reconcile the disagreements in writing. Where the two scores differ, the reason for the difference usually exposes an unstated assumption about what quality means, which is worth resolving before the contract rather than after.

Keep the returned annotations. Your reference plus their corrections is a better reference set than the one you started with, and it becomes the material for acceptance sampling once production begins.

What the test cannot tell you

The qualification task measures whether a team can annotate to a guideline. It does not measure whether they can sustain that quality across a large volume, whether they can recruit the speakers you need, or whether the best annotator on their bench is the person who did your test.

So the test is a filter rather than a decision. It removes candidates before they cost you a pilot, and it produces the reference material that pilot will need. A paid pilot at production rates and production sampling is what shows whether the quality holds at scale, and running it after a qualification test is far cheaper than running it to discover the same thing.