RLHF: The Human Feedback Loop That Unlocked ChatGPT
RLHF, teaching language models to follow instructions by learning from human preferences, made ChatGPT possible.
GPT-3 was a powerful language model that would complete whatever text you gave it. Ask it a question, and it would continue the sentence, sometimes answering, sometimes continuing as if writing a question paper, sometimes generating a fictional character's response. It was a text predictor, not an assistant. The gap between "predicts next tokens well" and "follows instructions helpfully and safely" was the problem that Reinforcement Learning from Human Feedback (RLHF) solved.
The key insight, developed at OpenAI in work culminating in InstructGPT (2022), was to separate two objectives: language model capability (learned by predicting text at scale) and alignment with human intent (learned from human preference data). First, fine-tune the model on demonstration data, humans writing ideal responses to prompts. Then train a reward model: show human raters pairs of model outputs and ask which is better; the reward model learns to score outputs. Finally, use reinforcement learning (Proximal Policy Optimisation) to fine-tune the language model to maximise the reward model's score.
InstructGPT, the result, was preferred by human evaluators over the raw GPT-3 in 85% of comparisons, despite GPT-3 having 100× more parameters. The improvement came not from scale but from alignment: the model had learned what humans actually wanted from a response. ChatGPT, launched in November 2022, was a version of this system made accessible to the public. The 100-million-user adoption in 60 days was driven by the RLHF-enabled conversational quality, not by raw capability.
RLHF has become the standard recipe for aligning language models, used by OpenAI, Anthropic, Google, and Meta in their frontier models. It has also introduced new problems: reward model overoptimisation (gaming the reward signal in unexpected ways), difficulty capturing nuanced human preferences, and the question of whose preferences the reward model should learn. Anthropic's Constitutional AI and others have developed variants that reduce dependence on human annotation. But the core insight, that learning from human preference signals transforms a text predictor into an assistant, has not been superseded.
What it actually meant
RLHF proved that human preference is a training signal, not just an evaluation metric. What humans prefer can be learned and optimised, not only measured. This is powerful and carries a real risk: optimising for measured human preference and optimising for what is actually correct are not the same thing.
What enterprises get wrong today
Enterprises deploy RLHF-trained models and assume preferred by users equals correct. RLHF optimises for human preference, which correlates with confidence, fluency, and agreeableness rather than accuracy. A model that answers confidently and wrongly is preferred over one that hedges and is right. Alignment with human preference and alignment with factual correctness are different properties.
Related capabilities
More from Algorithms
Ask about this story
RLHF is the technique that turned GPT-3, a powerful but unwieldy text predictor, into something that could follow instructions and have a conversation. InstructGPT preferred over GPT-3 by humans 85% of the time, with 100x fewer parameters. What would you like to explore?