Get the 2026 ML Training Cookbook | 52 recipes — GRPO, Flow Matching, World Models, and everything in between Download Now →
RLHF (Reinforcement Learning from Human Feedback)
A technique that aligns language models with human preferences by training a reward model on human feedback and optimizing the policy via reinforcement learning.
Reinforcement Learning from Human Feedback (RLHF) is a technique for aligning language model outputs with human preferences. The process operates in three stages: first, a model is supervised fine-tuned on demonstrations to produce an initial policy. Second, human annotators rank or rate pairs of model outputs, and this preference data is used to train a separate reward model that predicts which outputs humans would prefer. Third, the language model (policy) is optimized against the reward model using a reinforcement learning algorithm such as PPO.
The reward model learns to assign scalar scores to model outputs based on human preference patterns. During RL optimization, the policy generates outputs, the reward model scores them, and the policy is updated to maximize the expected reward. A KL divergence penalty is typically applied to prevent the policy from drifting too far from the supervised fine-tuned model, which maintains output quality and prevents reward hacking — where the model exploits imperfections in the reward model to achieve high scores without genuinely satisfying human preferences.
RLHF was instrumental in training models like InstructGPT and ChatGPT, transforming base language models into helpful, harmless, and honest assistants. While effective, RLHF is complex, expensive, and sensitive to the quality of human annotation. Alternative approaches such as Direct Preference Optimization (DPO) and Group Relative Policy Optimization (GRPO) have been developed to simplify or improve upon the RLHF pipeline.
- ■Three stages: SFT, reward model training from human preferences, and RL optimization
- ■Reward model predicts human preference between pairs of model outputs
- ■KL penalty prevents the policy from drifting too far from the SFT model
- ■Used to train InstructGPT and ChatGPT; aligns models with human values
- ■Sensitive to annotation quality; vulnerable to reward hacking
Training a chatbot to produce helpful, harmless responses by collecting human ratings on model outputs and optimizing the policy to maximize predicted human preference.