Get the 2026 ML Training Cookbook | 52 recipes — GRPO, Flow Matching, World Models, and everything in between Download Now →
Group Relative Policy Optimization (GRPO)
A reinforcement learning algorithm that simplifies PPO by eliminating the critic network, computing advantages relative to a group of sampled outputs per prompt.
Group Relative Policy Optimization (GRPO) is a reinforcement learning algorithm introduced by DeepSeek that simplifies Proximal Policy Optimization (PPO) by eliminating the value (critic) network entirely. In standard PPO, a separate critic model estimates the value of each state to compute advantages, which requires maintaining and training an additional neural network alongside the policy. GRPO removes this requirement by using group statistics instead.
For each prompt, GRPO samples a group of G outputs from the current policy. A reward model (or verifier) scores each output. The algorithm then normalizes the rewards using group statistics: the advantage for each output is computed as (reward − group_mean) / group_std. This group-relative baseline replaces the learned value function. The policy is then updated via a clipped policy gradient objective, with a KL divergence penalty keeping the policy close to a reference model.
By removing the critic, GRPO reduces memory and compute by approximately 50% compared to PPO, since only one model (the policy) needs to be trained and stored in memory rather than two. GRPO has been used to train models such as DeepSeek-Math and DeepSeek-R1, demonstrating strong reasoning capabilities. The trade-off is that GRPO requires larger per-prompt batch sizes (typically G = 8–64 samples) to obtain stable group statistics.
- ■Eliminates the critic/value network used in PPO, reducing memory and compute by ~50%
- ■Advantage = (reward − group_mean) / group_std, computed over G sampled outputs per prompt
- ■Uses a KL divergence penalty to keep the policy near the reference model
- ■Requires large per-prompt group sizes (8–64) for stable advantage estimates
- ■Used to train DeepSeek-Math and DeepSeek-R1
Training a reasoning model to solve math problems, where each correct answer receives a positive reward and GRPO identifies which of the sampled solutions outperform the group average.