Supervised Fine-Tuning Data: A Field Guide

What supervised fine-tuning teaches

Picture a brilliant new analyst on her first day. She has read more papers than you have and knows the theory cold. What she has not seen is how your team actually answers a client, lays out a report, or handles an awkward question. So you do the obvious thing: you hand her a folder of strong past examples and say, "here, do it like these." That is supervised fine-tuning, almost to the letter.

A base model shows up in much the same state. It has read a huge slice of the internet, so it already knows a staggering amount. What it cannot do yet is hold a conversation. Ask a raw pretrained model a question and it will happily continue your sentence, tack on a second question, or wander somewhere unrelated - because continuing text is the only job it has ever had. Supervised fine-tuning, or SFT, is the step that teaches it to answer instead. You show it demonstrations - a prompt paired with the reply you want back - over and over, until answering your way becomes the default. You will also hear it called instruction tuning or behavior cloning.

It helps to know where SFT fits. Pretraining builds the raw ability. SFT aims that ability at something useful. Later, a preference stage like RLHF or DPO, or a verifiable-reward stage like RLVR, decides which of several good answers is best. SFT is the middle step, where a pile of worked examples - the ones you write - does the teaching.

Mechanically, nothing exotic is going on. For each demonstration, the model is nudged to make your response a little more likely, one token at a time - the same training signal as pretraining, just aimed at examples you picked by hand instead of whatever the web happened to contain. No clever loss, no secret sauce. The examples are the whole game, which is exactly why writing them is expert work, not the kind of thing a click-through labeling queue can produce.

What a demonstration is

Strip away the jargon and a demonstration is just one pair: a prompt someone might type, and the exact reply you wish the model would give.

Two things to hold onto. First, the response is the finished answer - not a hint, not a score out of ten, but the actual thing you want to come out the other end. Second, and this is the part people underestimate: the model copies all of it. Your structure, your tone, how deep you go, and any mistake you happen to leave in. Write a hedgy answer and you get a hedgy model. Skip a step and it learns to skip it. Get a fact wrong and it will repeat that fact to a thousand users without blinking.

A quick one. Prompt: "Convert 2.5 kilometers to meters." Good response: "There are 1,000 meters in a kilometer, so 2.5 km = 2,500 m." It is short, it is right, and it shows the one thing that matters - the conversion factor - instead of just blurting "2,500." Do that a few thousand times across different questions and the model picks up the habit: name the rule, then apply it.

Why quality beats quantity

Here is the fact that trips up newcomers: good SFT needs astonishingly little data. When OpenAI built InstructGPT, it fine-tuned on about 13,000 human-written demonstrations. A 1.3-billion-parameter model trained on them beat the raw 175-billion-parameter GPT-3 in blind human ratings - a model more than a hundred times its size. Those demonstrations taught GPT-3 nothing new. They switched on what it already knew.

A famous follow-up, LIMA, pushed the idea to its limit. A thousand hand-picked pairs - that was the whole training set - made a 65-billion-parameter base model hold its own against much larger models that had been through heavy human feedback. The researchers named the idea behind it the Superficial Alignment Hypothesis: most of what a model knows is baked in during pretraining, and fine-tuning mostly teaches it which voice and format to reach for.

Take that as a helpful lens, not scripture. It explains why a small, clean set beats a big messy one, and why a handful of contradictory or careless examples do so much damage. But it undersells the hard cases. Teaching a model to show its working, hold a strict output format, or handle a niche domain is more than teaching style. There, the shape of your set - how correct, how varied, how difficult - matters enormously. Either way the rule of thumb survives: a few thousand excellent demonstrations beat ten times as many mediocre ones, and every wrong example is a lesson pointed in the wrong direction. For a cautionary contrast, Stanford's Alpaca leaned on 52,000 machine-generated demonstrations - cheap to produce, weaker per example, and a tidy reminder that volume from a shaky source does not buy you quality.

What a strong demonstration looks like

Line up a good dataset and the strong demonstrations keep sharing the same handful of traits:

• Correct. Right on the facts and right on the steps. This one is non-negotiable, because a wrong demonstration simply teaches the model to be wrong.
• The ideal answer, not a passable one. Write the best version you can. Models settle toward the average of what they see, so the floor you set becomes their ceiling.
• One consistent style. Same structure, tone, and formatting across the set, so the model learns a single voice instead of a committee of them.
• The right length. Long enough to answer, short enough to respect the reader. Padding and throat-clearing get copied too.
• The working, where it counts. For math, code, or anything that needs reasoning, show the steps. A model can only learn a method it can actually see.

Seeing it beats describing it. Prompt: "Is 91 a prime number? Explain." A weak answer: "No, 91 is not prime." A strong one: "No. A prime has no divisors except 1 and itself. Try a few small primes: 91 ÷ 7 = 13 exactly, so 91 = 7 × 13. Since it has a factor other than 1 and 91, it is composite." Both land on the right verdict. Only the second shows the move - go hunting for a factor - that the model will need when you hand it 143 or 187 tomorrow. The first just teaches it to announce answers it cannot back up.

Where a set goes wrong, and how to keep it clean

When an SFT set goes bad, it usually goes bad in one of a few familiar ways:

• Wrong answers - the expensive mistake, because the model copies it faithfully. One bad answer in a batch does more harm than a gap where an answer should be.
• Mixed formats - half your answers are bulleted, half are prose; some cite sources, some do not. The model learns the muddle and picks at random.
• Too little variety - the set leans on one kind of task or one turn of phrase, the model overfits it, and it flounders on everything else.
• Padding and hedging - answers that waffle, over-qualify, or open with filler teach the model to do exactly that.
• Teaching to the test - if your demonstrations overlap the eval set, the score climbs while the model stands still.
• Leftover junk - stray "As an AI language model..." openers, tool-output formatting, and copy-paste debris, all learned word for word.

Keeping a set clean is mostly discipline. Verify every response before it goes in - if it is checkable, check it, do not just skim. Write the ideal answer rather than a good-enough one, since the model copies your floor. Fix a house style early and hold every demonstration to it, and keep the prompts varied so the model generalizes instead of memorizing a template. Show the reasoning where the method matters, trim the filler where it does not, and decontaminate against your eval sets so a rising score means real learning and not leakage.

None of that is trustworthy on your word alone, which is why a serious set is reviewed, not just written. A second qualified expert checks each response for correctness; disagreements are settled against a written guideline, and how often reviewers disagree is tracked as a read on how clear that guideline is - the same inter-annotator agreement machinery used across evaluation work. A held-out slice, never trained on, tells you whether the model learned the behavior or only memorized the items. Done properly, the work leaves a trail: who wrote each demonstration, who checked it, and against which standard.

The takeaway

SFT is the bluntest, most direct way to shape how a model behaves, because it copies your demonstrations almost to the letter. That works for you and against you in equal measure. Careful, consistent examples make a careful, consistent model; sloppy ones do the reverse, just as reliably. So treat the job as teaching, because that is what it is - be right, be clear, show your working, and cover enough ground that the lesson holds when the questions change. A few thousand demonstrations written with that kind of care will beat ten times as many written without it.

Additional resources

• Aligning Language Models to Follow Instructions
  OpenAI's accessible write-up of InstructGPT: how labeler-written demonstrations, then rankings, produced a far smaller model people preferred.
• Instruction Finetuning and LoRA Experiments
  A practitioner's notes on instruction finetuning and the data decisions behind it.
• RLHF: Reinforcement Learning from Human Feedback
  A readable walkthrough of the training pipeline; its supervised-finetuning section is the clearest short intro to what SFT does.
• Illustrating Reinforcement Learning from Human Feedback (RLHF)
  An illustrated tour of the fine-tuning pipeline, starting from the supervised step SFT builds on.