Fine-Tuning and RLHF: Teaching AI to Follow Instructions
Understand the training stages that turn a text predictor into a helpful assistant.
Pre-training produces a base model that knows an enormous amount and has no manners. Ask it “How do I boil an egg?” and it might carry on with “How do I poach an egg? How do I fry an egg?”, because that’s what a web page of egg questions looks like. Turning it into a helpful assistant takes two more, much smaller, stages.
Supervised fine-tuning comes first. People write thousands of example conversations: a question, followed by the kind of answer a great assistant would give. The model does what it always does, next-token prediction, but now only on these examples. It learns the format: when someone asks, answer; follow instructions; stop when you’re done. It also learns the chat template and the end-of-turn token from the generation lesson.
Preference tuning polishes the behaviour. Writing perfect answers is hard; judging which of two answers is better is easy. So people compare pairs of the model’s own answers and pick the better one. A reward model learns to predict those choices, and the LLM is then trained with reinforcement learning to produce answers the reward model scores highly. That’s RLHF, reinforcement learning from human feedback, the step that made ChatGPT feel so different from earlier models in 2022.
The proportions are striking. Pre-training eats almost all the compute and supplies almost all the knowledge. The later stages are tiny by comparison, yet they shape nearly everything about how the model behaves: its tone, helpfulness and willingness to refuse harmful requests. They can add quirks too. A model that learned people like confident, agreeable answers may become a little too confident, and a little too agreeable.
Key takeaways
- Pre-training on huge amounts of text gives knowledge, but produces a model that just continues text.
- Supervised fine-tuning on example conversations teaches it to act as an assistant.
- Preference tuning (RLHF) uses human comparisons to make answers more helpful, honest and safe.
- Pre-training costs by far the most; the later stages are small but decide how the model behaves.
Check your understanding
Go deeper (more technical)
Scale: Llama 3 was pre-trained on about 15 trillion tokens. Supervised fine-tuning (SFT) datasets are typically thousands to millions of examples. InstructGPT (2022) popularised the SFT → reward model → reinforcement learning (PPO) recipe behind ChatGPT.
Newer methods such as DPO (2023) skip the separate reward model and learn directly from preference pairs. Reasoning models are additionally trained with reinforcement learning on problems with checkable answers, a few lessons from now.
The over-agreeable habit has a name, sycophancy: telling people what they seem to want to hear. Research has linked it to preference training, and model makers now measure and train against it.