Get the 2026 ML Training Cookbook | 52 recipes — GRPO, Flow Matching, World Models, and everything in between Download Now →
DPO (Direct Preference Optimization)
A preference optimization method that directly optimizes the policy from preference data without training a separate reward model.
Direct Preference Optimization (DPO) is a preference optimization method that simplifies RLHF by eliminating the reward model and the reinforcement learning loop entirely. Instead of training a separate reward model and then using RL to optimize the policy against it, DPO reparameterizes the reward function in terms of the policy itself, deriving a closed-form loss function that directly optimizes the policy from preference data.
Given pairs of outputs where a human (or a stronger model) has indicated a preference, DPO trains the policy to increase the probability of the preferred output while decreasing the probability of the dispreferred output, relative to a reference model. The loss function includes a KL divergence term that keeps the policy close to the reference model, similar to the KL penalty in RLHF. This formulation is mathematically equivalent to optimizing a reward model and then doing RL, but in a single training stage.
DPO is simpler, more stable, and cheaper than RLHF because it avoids the instabilities of RL training (such as reward hacking and value function estimation) and requires no separate reward model. However, it can be less effective than RLHF in scenarios that require online exploration or iterative improvement, since it learns only from a fixed preference dataset. DPO has become a popular alternative to RLHF for aligning open-source language models.
- ■Eliminates the reward model and RL loop; optimizes directly from preference data
- ■Derives a closed-form loss using the policy itself as the reward function
- ■Includes a KL term keeping the policy close to the reference model
- ■Simpler, more stable, and cheaper than RLHF
- ■Less effective for online exploration; learns from fixed preference datasets
Aligning an open-source language model using a dataset of preferred vs. dispreferred response pairs, without the complexity of training a reward model and running PPO.