Get the 2026 ML Training Cookbook | 52 recipes — GRPO, Flow Matching, World Models, and everything in between Download Now →
PPO (Proximal Policy Optimization)
A policy gradient RL algorithm that uses clipped surrogate objectives to ensure stable, conservative policy updates.
Proximal Policy Optimization (PPO) is a policy gradient reinforcement learning algorithm designed to provide stable and reliable policy updates. PPO improves upon earlier policy gradient methods by constraining the size of each update, preventing the policy from changing too drastically in a single step — a common failure mode that leads to training instability. It has been the workhorse algorithm for RLHF in large language models.
PPO achieves this stability through a clipped surrogate objective. For each update, PPO computes the ratio of the new policy's probability to the old policy's probability for each action. This ratio is clipped to a range [1−ε, 1+ε] (where ε is a hyperparameter, typically 0.1–0.2), ensuring that the policy does not move too far from its previous version in a single update. A separate value (critic) network estimates the value of each state to compute advantages, which reduce the variance of policy gradient estimates.
In the context of RLHF for language models, PPO is used to optimize the language model policy against a learned reward model. The policy generates text, the reward model scores it, and PPO updates the policy to maximize reward. PPO requires maintaining four models in memory simultaneously: the policy, the reference model (for KL penalty), the reward model, and the value/critic model — making it memory-intensive. This limitation motivated the development of more efficient alternatives like GRPO.
- ■Uses a clipped surrogate objective to limit policy update size
- ■Clip ratio constrained to [1−ε, 1+ε], with ε typically 0.1–0.2
- ■Requires a separate value (critic) network for advantage estimation
- ■In RLHF, maintains four models in memory: policy, reference, reward, critic
- ■Memory-intensive; motivated development of GRPO and DPO
Using PPO to optimize a language model's responses against a reward model that scores helpfulness, with the clip parameter preventing destructive large updates.