Group Relative Policy Optimization

Improve reasoning capabilities through group-based advantage estimation, eliminating the need for a separate value/critic network. Instead of training a separate value network to estimate advantages, GRPO samples a group of responses from the policy, scores each response with a reward model, and computes advantages relative to the group mean. The policy is then updated to increase the probability of responses that scored above average. A KL penalty keeps the policy from drifting too far from the reference model.