What is Inter-Annotator Agreement?
Having different people annotate the same batch and measuring how far apart they land. It is the test of whether a guideline is genuinely unambiguous.
High agreement means the guideline is clear and the task is executable. Low agreement is usually not an annotator problem; it means the guideline itself is ambiguous.
You can ask a supplier for this number. The ones who can produce it usually run a formal annotation process.
Related terms
-
Annotation
Attaching machine-readable labels to raw data — a transcript, an intent, a speaker identity.
-
Ground Truth
Human-verified labels good enough to serve as the training target.
Keep reading
-
Full glossary
Every term we explain, in one list.
-
How to buy training data
Where these terms actually show up, and which ones change a quote.
-
Help center
How a project runs, from specification to delivery.
Submit a sourcing request
Tell us the language, the hours, and what the data needs to look like. You will get a real number and a real timeline — not a range. If we cannot source it well, we will tell you that instead.
- Pilot batch before the full run, so problems surface early.
- Consent documentation delivered with the data.
- No medical or clinical data. No recorded telephone calls.