What is RLHF, and why preference data is a different purchase entirely

Reinforcement learning from human feedback is the stage where a model is tuned toward the responses people actually prefer, rather than toward responses that merely match a reference. The commercial consequence is easy to miss: the data it consumes is not a better version of instruction data. It is a different shape.

Supervised fine-tuning asks a person to write the answer you want. RLHF asks a person to judge between two answers that already exist. Everything difficult about the second task follows from that single change: you cannot write it from a style guide, you cannot check it against a reference, and the person doing it has to hold one standard steady across thousands of judgments.

One row is a triple, not a pair

The minimum unit is a prompt, two candidate responses, and a judgment about which is better. A useful set adds a few more fields: how much better, which dimension the judgment rests on, and an optional written reason.

Ties need their own bucket. A set that forces a choice on every row converts genuine ties into coin flips, and a reward model trained on those learns noise as signal. Recording "about the same" as a valid outcome is far cheaper than adjudicating every tie later.

The prompt distribution is part of the specification too, and it is the part most often left to the supplier. Prompts sampled from a model's own outputs cluster around what the model already handles well. Prompts drawn from real user traffic are harder and more informative. Ask which one you are getting, because the two produce preference data of different value.

Where the candidates come from is a data decision

The two responses in a row are usually generated rather than written. They come from sampling a model at different checkpoints, temperatures, or with different prompts, and the comparisons that teach the most are the close ones.

This makes a preference set partly a sampling design. If one response is obviously better, the judgment was not work and the row teaches almost nothing. Producing a set with a high share of informative rows means generating more candidates than you need and screening out the lopsided ones, and that screening is where a large part of the budget goes.

It also creates a dependency worth writing into a contract: the candidates came from some model, and if that model is not the one you are training, part of what the labels encode is the other model's habits. The InstructGPT paper describes the pipeline that popularized this stage, and the same structure appears across published work. Preference data is model-relative in a way that transcription is not.

Disagreement is the signal, not the mess

Two qualified annotators will disagree on a meaningful fraction of rows. The instinct is to treat that as a defect. It is more useful to treat it as information: it marks the prompts where "better" is genuinely contested, which is usually where the product decision has not been made yet.

The protocol has three parts. Run a shared subset through two or three annotators and report the agreement rate as a number. Then set a rule for the disagreements: adjudicate with a third person, drop the row, or keep it with a split label. Then read a sample of the disagreements yourself, because they usually show that the guideline is silent on a dimension the annotators care about.

Writing a preference guideline is harder than writing a transcription guideline. Transcription has a reference. Preference has dimensions — accuracy, helpfulness, tone, safety, length — and the guideline has to state which one wins when two of them conflict. A set without that ranking encodes each annotator's personal priority order.

Why the set stays small

Compare the two shapes honestly. A supervised pair is one response, reviewed once. A preference row is two or more responses generated, both read in full by a judge, with a qualification step before that judge is allowed to work and an adjudication step on a fraction of the rows. Reading time alone makes the cost per row several times higher, and the qualification requirement raises it again.

The planning consequence: for the same budget, a preference set is usually an order of magnitude smaller than a supervised set. That is not a reason to pad it. A small set with measured agreement beats a large one with noisy labels, because a reward model trained on noisy labels will find the noise and optimize against it.

Questions worth asking a supplier

One more check belongs on your side rather than the supplier's. Hold back a set of prompts, run the same comparisons with a different group of judges, and compare the rankings. If the two groups produce different winners on the same prompts, the preference you are about to optimize against is narrower than the data suggests.

• Which model produced the candidate responses, at which settings, and can that be reproduced exactly?
• How were lopsided pairs screened out, and what share of generated pairs survived screening?
• What is the agreement rate on a shared subset, and how was it measured?
• Are ties recorded as ties, or forced into a choice?
• What is the annotator qualification, and is the pool large enough that one person's taste does not become the label?
• Does each row carry a written rationale, and is it included in the delivery?