GRPO: grade a model's answers against its own samples, skip the critic

Group Relative Policy Optimization (GRPO) is, in the Policy gradient method article's own understated words, "a minor variant of PPO that omits the value function estimator". It was first proposed by DeepSeek researchers in the context of training reasoning language models, and the Reasoning model article records DeepSeek R1's January 2025 release as demonstrating its effectiveness. (Summarised from the sources at the bottom; edited, not verified.)

The mechanics

Standard PPO needs a learned value function V to estimate how good a state is, and uses the gap between realised and expected return as the advantage signal. GRPO deletes that machinery:

1. For each state s (an LLM prompt), sample G actions (complete responses) from the current policy.
2. Score each with the reward r(s, a). For verifiable reasoning tasks — the motivating case — the outcome reward can be binary: 1 if the final answer checks out, 0 otherwise, from a "verifier" rather than a learned reward model.
3. Compute the group-relative advantage as the standard score of the reward within the group: A = (r − μ)/σ, μ and σ being the mean and standard deviation of the G rewards.
4. Maximise the usual PPO clipped objective, averaged over the group. The KL penalty against a reference policy can be applied on top, as in RLHF generally.

The article's own intuition sentence is the whole idea: each update makes the policy more likely to answer a prompt with an action that performed relatively better than the other things it tried on that same prompt, and less likely to repeat the relatively worse ones. The baseline PPO needed a trained critic to supply is replaced by the group's own average.

Why skip the critic (my inference, labelled)

The articles state the omission, not its payoff. The obvious reading: in PPO-for-LLM the critic is roughly a second model-sized network to train and store, so a variant that needs only the reward (or a verifier) drops a large fraction of the memory and tuning burden. The article's framing of verifiable binary rewards fits neatly: if correctness is checked, estimating value with an in-group average is all the credit assignment the problem needs.

Where it can fail (inference — say so, because the articles don't)

• The group is the baseline, so the group must vary. If all G samples get the same reward — all wrong, or all right — σ is zero and the standardised advantage collapses: no gradient signal from a prompt the model uniformly fails or uniformly aces. Verifier-based training therefore spends a lot of compute sampling prompts near the model's competence frontier, and hard prompts contribute nothing until the model can sometimes solve them. The sources do not discuss this; it follows from the formula they do give.
• "Minor variant" is doing quiet work. Everything PPO can be blamed for — reward hacking, drift the KL term only slows — transfers wholesale, because GRPO keeps the PPO objective intact. The RLHF page's warning that models learn to game what is rated rather than what is good applies unchanged where the reward is a learned model; a verifier shrinks that surface but reintroduces the shortcut problem (any error in the checker is now a gradient direction).
• Normalisation hides absolute progress. Advantages are relative within a group at a moment in training; the objective never sees whether the model's typical reward is rising. Monitor raw reward separately or the run can look healthy while the policy stalls.

The sources themselves leave the RLHF-vs-GRPO-and-DPO argument open, in the same unsettled spirit as RLHF and its alternatives describes the preference-tuning debate.

Sources: Wikipedia, "Policy gradient method" (section Group Relative Policy Optimization) and "Reasoning model" (sections Reinforcement learning, Outcome reward model, timeline 2025), read 2026-09-08. Summary plus labelled inference — edited, not verified. Related: RLHF and its alternatives (what the reward model is for), Benchmark contamination (why verifier-backed training and verifier-backed evaluation fail differently).