GRPO and Verifiable Rewards

Module FT14 · Course 3 — LLM Fine-Tuning Masterclass

90 minutes · Pillar 4 — the flagship 2025–2026 module. Reasoning models (o1 / R1 / Qwen3) are trained via RL on verifiable rewards.

Pillar 4 — Reasoning Models

The decision: DPO vs GRPO

The most important judgment in this module. It comes down to one question.

Why DPO is wrong for reasoning

GRPO: drop the critic

GRPO (Group Relative Policy Optimization, DeepSeek — DeepSeekMath, arXiv:2402.03300). The key move: eliminate the learned value function from PPO.

The clarification: GRPO ≈ RLOO

RLOO (Kool et al. 2019): baseline for response i = mean of the other N−1. GRPO: baseline = mean of all N (including i). A small bias/variance tradeoff, not a different family.

The GRPO++ landscape

GRPO works. Production recipes patch specific failure modes. (Cameron Wolfe's survey is the map.)

The DeepSeek-R1 pipeline (canonical)

R1-Zero proved reasoning emerges from RL alone. R1 fixes the readability with a 4-stage pipeline. (arXiv:2501.12948 + Nature s41586-025-09422-z.)

The Qwen3 pipeline (the other canonical reference)

Qwen3 technical report (arXiv:2505.09388, May 2025). Industrializes the R1 recipe.

The reward-verification loop

The heart of the method. The verifier is the load-bearing component — its quality bounds the whole run.

1. Generate N responses per prompt from the current policy
2. Verify each: parse answer → execute/check (run the code, compare the math answer, validate the tool output) → scalar reward
3. Advantage: A_i = (r_i − mean(r)) / std(r) — group-relative, no critic
4. Update policy toward high-advantage responses, with a KL anchor to the reference

Tooling: TRL, verl, OpenRLHF

The verifiable reward function (math)

import re

def math_reward(prompts, completions, answer, **kwargs):
    """Verifiable: parse the model's \\boxed{} answer, compare exactly."""
    rewards = []
    for comp, gold in zip(completions, answer):
        # parse the last \\boxed{...} from the completion
        m = re.findall(r"\\\\boxed\{([^}]*)\}", comp)
        pred = m[-1].strip() if m else None
        rewards.append(1.0 if pred is not None and pred == gold.strip() else 0.0)
    return rewards

Anti-patterns

• DPO for reasoning — forfeits on-policy exploration on a verifiable task. Use GRPO + a verifier.
• No reward verification (reward hacking) — a hackable reward (substring, leniency) → model games it, scores high, outputs wrong. Use a verifier, not a judge.
• Skipping the cold-start SFT — the R1-Zero path can work but produces readability problems. Give the model a CoT scaffold first (one SFT stage).
• Ignoring the readability problem of pure-RL — RL optimizes correctness, not human-facing format. Plan the readability layer (cold-start SFT, rejection-sampling SFT).
• Treating GRPO variants as interchangeable — "use GRPO" is underspecified. Mid-training stalls often need a specific patch (clip-higher, length-debias, sequence-level weighting). Know which failure each patch addresses.

The lab — "GRPO on a Math Task"

Run TRL GRPOTrainer on a small base with a verifiable math reward (GSM8K-style correctness checked by a Python reward function). Watch the reward curve climb.

Next: FT15 — CoT Distillation & Rejection-Sampling FT

What you can now do

1. Explain why RL, not DPO, for reasoning — verifiable rewards enable on-policy exploration offline DPO cannot do.
2. State the GRPO mechanism — N samples, group-relative advantage, no critic → ~half the memory — and that GRPO ≈ RLOO.
3. Map the GRPO++ landscape (DAPO, Dr. GRPO, GSPO) to the failure each patches; name OLMo 3 as a production recipe.
4. Recite the DeepSeek-R1 pipeline (R1-Zero → 4-stage R1 → distillation) and the Qwen3 4-stage pipeline + thinking budget.
5. Choose between TRL GRPOTrainer (moderate) and verl (production), and state the OpenRLHF caveat.
6. Write a verifiable reward function and explain why verifier quality bounds the whole run.

DPO taught you preference. Now you have exploration — the tool that builds reasoning models.