Building a Preference Dataset for RLHF

Why judging beats writing

Here is something you already know in your bones: it is far easier to tell whether a meal is good than to cook it yourself. You can spot a clumsy sentence you could never have written, pick the sharper of two designs, catch a wrong answer at a glance. Judging is easier than creating. Reinforcement learning from human feedback - RLHF - is built on that one asymmetry.

Remember the problem SFT leaves behind. To write a demonstration, you have to produce the ideal answer yourself. That is fine for a chemistry problem, but what is the single ideal answer to "write me a poem about autumn" or "explain recursion simply"? There isn't one. Ask ten experts and you get ten good answers and a lot of arguing.

RLHF sidesteps the whole mess. Instead of writing the perfect answer, you look at two answers and say which one is better. Do that thousands of times and the model picks up your taste - what counts as clearer, more correct, more helpful - without anyone ever writing the perfect response from scratch. You are not the author anymore. You are the judge.

What a comparison captures

One preference example is a small package: a prompt, two responses to it, and a label saying which response a human preferred. That is all. The preferred one usually gets called chosen and the other rejected.

A quick example. Prompt: "Is a tomato a fruit or a vegetable?" Response A: "A fruit." Response B: "Botanically it is a fruit - it grows from a flower and carries seeds - but in the kitchen it is treated as a vegetable." Both are technically right. Most people pick B, because it is more complete and actually useful. That one choice is a data point, and it carries more signal than it looks: prefer the answer that is correct and helpful, not just the one that is not wrong.

Notice what you did not do - you did not score each answer out of ten. That is deliberate. People are far steadier at "which of these two is better" than at "rate this a 6 or a 7," where everyone draws the lines differently and the numbers drift toward a mushy middle. A straight comparison is the most reliable judgment a human can make, which is why pairs, not ratings, are the workhorse of preference data. You can rank three or more when you need finer signal, but the humble pair does most of the work.

How choices become taste

On its own, a pile of "B beat A" judgments cannot train anything. The trick is to turn those judgments into a number. A second model - the reward model - learns to score any response so that the ones people preferred score higher than the ones they rejected.

If that sounds familiar, it should - it is the same idea behind chess rankings. Nobody measures a grandmaster's skill directly. You infer a rating from who beats whom. The math behind it - a model called Bradley-Terry - turns a long list of wins and losses into one number per player. The reward model does exactly that for responses: from thousands of pairwise wins and losses, it learns a score that stands in for human taste.

Once that score exists, the language model gets fine-tuned to chase it, producing answers the reward model rates highly. That final step is the reinforcement-learning part. For our purposes the important thing sits upstream: the reward model is only ever as good as the comparisons it learned from. Garbage judgments, garbage taste.

Judging on the right thing

So what should you compare on? Honestly, whatever the task calls for - usually some mix of correct, helpful, honest, and appropriately safe. The skill is holding those in view and not getting seduced by surface polish.

Here is where it gets subtle. Prompt: "What is the boiling point of water?" Response A, three confident sentences: "Water boils at 100 degrees Celsius. This is a fundamental physical constant used throughout science and cooking. It is one of the most important reference points in all of chemistry." Response B, one line: "At sea level, 100 degrees Celsius (212 Fahrenheit) - and it drops as you go higher up." A is longer and sounds authoritative. B is shorter and more correct, because it names the thing that actually matters: pressure. The better pick is B, and a good judge has to resist the tug of the longer, glossier answer. That tug is the single biggest problem in this whole field - which is exactly where we head next.

Where it goes wrong, and how to hold the line

Preference data fails in sneakier ways than SFT data, because the failures hide inside judgments that look perfectly reasonable.

• Length bias. The big one. People tend to prefer longer, more elaborate answers, even when a short one is better. The reward models trained on them pick up the same taste. Left alone, the model learns that more words means better, and pads everything.
• Format bias. A close cousin. Bullet points, bold headers, and a confident tone read as "high quality" even when the substance is thin.
• Sycophancy. We tend to reward answers that agree with us or flatter the question. Train on that and you get a model that tells people what they want to hear instead of what is true.
• Disagreement and drift. Two annotators often split on the same pair, and the same annotator will contradict themselves on a tired afternoon. Some of that is real; a lot of it is a vague guideline in disguise.
• Thin coverage. If every prompt comes from the easy middle of the distribution, the model never learns the weird, adversarial, or high-stakes cases - usually the ones that matter most.

The defense is the same handful of habits. Write down what "better" means before you start. A one-line rubric - correct first, then helpful, then concise - turns taste into a repeatable call, and kills half your disagreement. Watch length as you go. Did you prefer the better answer, or just the longer one? Balance the set so the longer response wins about as often as the shorter. Seed in the hard cases on purpose; do not just sample the comfortable middle. Put more than one annotator on a shared slice and track how often they agree. Low agreement usually means a fuzzy rubric, not bad annotators - our companion guide, How Your Work Is Checked, digs into this. Keep sessions short; judgment drifts when you are tired. And once the reward model is trained, test it on held-out pairs where you know the answer. If it is quietly rewarding length or formatting, catch it there, not in production.

What careful comparisons buy

RLHF works because judging is easier than creating - and that is also its weak spot, because a lazy judgment is easy to make too. Every comparison you label is a small vote for what "good" means, and the model tallies those votes into a sense of taste it will apply to millions of answers. So judge on what actually matters, stay honest with yourself about length and polish, and write the rules down before the disagreement starts. Get that right and a few tens of thousands of careful comparisons will teach a model taste that no pile of hand-written answers ever could.

Additional resources

• Illustrating Reinforcement Learning from Human Feedback (RLHF)
  The classic illustrated explainer of how preference data trains a reward model and then tunes the model.
• RLHF: Reinforcement Learning from Human Feedback
  A clear, intuition-first walkthrough of the three phases behind models like ChatGPT.
• RLHF Learning Resources
  A curated reading list from a leading RLHF researcher, for going deeper at any level.