# Module FT14 — GRPO and Verifiable Rewards

**Course**: Course 3 — LLM Fine-Tuning Masterclass
**Module**: FT14 — GRPO and Verifiable Rewards
**Duration**: 90 minutes
**Level**: Senior Engineer and above
**Prerequisites**: FT13 — The DPO Family and Preferences

---

## Learning Objectives

After completing this module, you will be able to:

1. Explain **why RL — not DPO — is the correct tool for reasoning**: verifiable rewards (math correctness, code execution, tool-use success) enable on-policy exploration that offline preference datasets structurally cannot provide. DPO learns from fixed pairs; GRPO generates, verifies, and learns from its own attempts.
2. State the GRPO (Group Relative Policy Optimization, DeepSeek) mechanism — sample N responses per prompt, compute advantages relative to the *group mean* rather than a learned value function — and explain how eliminating the critic from PPO roughly halves the memory footprint.
3. Articulate the clarification that GRPO is essentially RLOO (REINFORCE Leave-One-Out) with a slightly different advantage computation, and why this matters for how you read the literature.
4. Map the GRPO++ landscape (DAPO, Dr. GRPO, GSPO) to the specific failure modes each variant fixes, and name OLMo 3 as a concrete production recipe.
5. Recite the DeepSeek-R1 pipeline — R1-Zero (pure RL, no cold start) → R1 (cold-start SFT → reasoning RL → rejection-sampling SFT → final RL) → distillation — and explain why each stage exists.
6. Recite the Qwen3 4-stage post-training pipeline and the thinking-budget mechanism.
7. Choose between TRL `GRPOTrainer` (moderate scale) and verl (production-grade distributed RL on Ray) based on your scale, and name OpenRLHF's caveat.

---

# 14.1 — Why RL, Not DPO, for Reasoning

*The single most important judgment in this module. Get it wrong and you reach for an offline method on a problem that demands exploration.*

## The structural difference

FT13 covered the DPO family. DPO takes a *preference dataset* — pairs of (chosen, rejected) responses — and trains the model to assign higher likelihood to the chosen one. The dataset is fixed before training begins. The model never generates a new response during the DPO loss; it only re-scores the ones a human or a judge already wrote.

That is exactly right for preference alignment ("this answer is more helpful than that one"), because preferences are subjective and a fixed dataset of judgments is the signal you have.

It is exactly wrong for reasoning, for one reason: **reasoning rewards are verifiable.** A math answer is correct or it is not — you can execute the check. A program compiles and passes tests or it does not — you can run it. A tool call retrieved the right document or it did not. You do not need a human to judge these; you need a *verifier*. And the moment you have a verifier, the right move is to let the model *generate its own attempts, verify them, and learn from the distribution of its own successes and failures*.

That is on-policy exploration. DPO cannot do it. DPO learns from attempts a human (or a stronger model) already made. GRPO generates N fresh attempts per prompt, scores them with the verifier, and shapes the policy toward the ones that passed. The model is learning from *its own* current behavior, not from a frozen snapshot of someone else's.

## The intuition that makes it click

Imagine teaching a student long division two ways:

- **DPO way:** show them 1,000 worked examples where someone else divided correctly and incorrectly, labeled "good" and "bad." They learn to imitate the good ones. But you never see *their* mistakes — only the mistakes in your fixed set.
- **RL way:** give them a problem, let them attempt it N times, check each attempt against the answer key, and tell them which attempts were right. They learn from the *distribution of their own errors* — including errors your fixed dataset never contained.

For subjective taste (writing style), the DPO way is fine. For verifiable skill (math, code, tool use), the RL way is strictly more informative, because the verifier lets you cheaply generate an infinite curriculum of labeled attempts tailored to *this model's* current weak spots. The frontier of reasoning quality in 2025–2026 is built on this fact.

> **The decision rule:** if your reward is verifiable (you can write a function that returns correct/incorrect), use RL on that reward. If your reward is only expressible as a preference (a human or a judge must compare two outputs), use DPO. Reasoning is the canonical verifiable domain; hence RL.

## The "emergence" claim, stated precisely

DeepSeek-R1-Zero showed that reasoning behavior can *emerge* from RL alone on a base model — no SFT cold start. This is the headline finding that made the field pivot to RL in early 2025. But read it precisely: what emerged was not new *knowledge*. The base model (DeepSeek-V3-Base) already had the capability from pretraining. What emerged was reliable *use* of that capability — self-verification, backtracking, longer chains of thought — because RL with a correctness reward *selects for* the behaviors that produce correct answers. This is the FT00 thesis ("steering, not teaching") in its purest form: RL is steering the model to use reasoning pathways it already has.

---

# 14.2 — GRPO: Group Relative Policy Optimization

*The algorithm that became the default for reasoning RL. Introduced in DeepSeekMath, used in DeepSeek-R1.*

## The mechanism

GRPO (Shao et al., DeepSeekMath, arXiv:2402.03300) is a simplification of PPO for the LLM setting. The key move: **drop the learned value function (the critic).**

Standard PPO has two networks: the policy (the model you are training) and a value function (the critic) that estimates the expected reward from each state, used to compute the advantage (how much better an action was than expected). The critic is typically the same size as the policy. So PPO for an LLM roughly *doubles* the memory: one copy for the policy, one for the critic, plus a reference policy for the KL penalty. Three models in memory.

GRPO eliminates the critic by computing the advantage **relative to a group of sampled responses**, not relative to a learned expectation. The recipe, per prompt:

1. Sample **N** responses from the current policy (e.g., N=8 or 16).
2. Score each with the reward function (verifier).
3. Compute the group mean reward.
4. The advantage of each response is its reward minus the group mean: `A_i = (r_i − mean(r)) / std(r)` (normalized).
5. Update the policy via a clipped PPO-style objective, with a KL penalty to the reference policy to stay anchored.

No critic. The group *is* the baseline. Memory drops from three models (policy + critic + reference) to two (policy + reference), often further reducible because the reference can be the frozen pre-RL weights and the KL can be computed cheaply.

## The clarification: GRPO ≈ RLOO

Here is the clarification that saves you from reading the literature wrong: **GRPO is essentially RLOO (REINFORCE Leave-One-Out) with a slightly different advantage computation.** Both sample N responses per prompt and use a leave-one-out / group-mean baseline. They are not fundamentally different algorithms. The REINFORCE-Leave-One-Out estimator (Kool et al., 2019) computes the baseline as the mean of the *other* N−1 responses for each response i; GRPO uses the mean of all N (including i). The difference is a small bias/variance tradeoff, not a different family. When you see "GRPO" and "RLOO" discussed as alternatives, understand they are siblings, not strangers — both are variance-reduced REINFORCE with a group baseline and no critic. This is why TRL ships `GRPOTrainer` and `RLOOTrainer` side by side and they behave nearly identically in practice.

> **Mental model:** PPO = policy + learned critic (estimates baseline). GRPO/RLOO = policy + group-mean baseline (computed from the N samples, no learning). The critic is replaced by Monte Carlo estimation over the group.

## Why the group baseline works

The advantage's job is to tell the policy "this response was better or worse than expected." A learned critic estimates "expected" from state features. But when you can cheaply sample N responses, the *empirical mean of those N* is a lower-variance, bias-free estimate of the expected reward under the current policy — and it requires no extra parameters, no extra training, no extra memory. The tradeoff: you pay N forward passes per prompt instead of one. For reasoning, where the reward is cheap to compute (run the verifier) and the sample diversity is the whole point, this trade is overwhelmingly favorable.

---

# 14.3 — The GRPO++ Landscape

*GRPO works, but production recipes patch specific failure modes. The variant names matter because the 2025–2026 literature uses them constantly. Cameron Wolfe's survey is the canonical map.*

## The failure modes and their fixes

**DAPO (ByteDance, arXiv:2503.14476)** addresses two issues. *Clip-higher* decouples the PPO clip range for positive vs. negative advantages — standard symmetric clipping suppresses exploration once the policy is doing well, because good responses can only be upweighted up to the clip. Clip-higher lets positive advantages push further. *Dynamic sampling* filters out prompts where all N samples get the same reward (all-correct or all-wrong) — those contribute zero gradient signal (the advantage is zero for every sample) and waste compute. Skip them, spend the budget on prompts with reward variance.

**Dr. GRPO** removes a length bias. The standard GRPO objective, when averaged per-token, introduces a bias toward shorter responses (shorter sequences have fewer tokens to dilute the per-token advantage). Dr. GRPO re-weights to remove this, preventing the model from learning "shorter = higher reward" as a spurious shortcut.

**GSPO (Group Sequence Policy Optimization)** does sequence-level importance weighting rather than token-level. The argument: token-level importance ratios in GRPO/PPO are noisy and can mis-credit long sequences. GSPO computes one importance weight per full sequence, which stabilizes training on long chain-of-thought outputs where the reasoning spans hundreds of tokens.

## OLMo 3 as a concrete recipe

The Allen Institute's OLMo 3 (2025) is a useful production anchor: it publishes a recipe that combines GRPO-family RL with a verifiable-reward curriculum, demonstrating the pattern at open-data scale. When someone asks "what does a real, reproducible reasoning-RL pipeline look like end to end?", OLMo 3 is a better citation than the closed frontier labs. Read it as the open counterpoint to DeepSeek-R1 and Qwen3.

> **How to read the landscape:** the base algorithm is GRPO/RLOO. The variants are *patches*. DAPO patches exploration (clip-higher) and sample efficiency (dynamic sampling). Dr. GRPO patches length bias. GSPO patches credit assignment. In practice, production recipes mix and match — "GRPO with clip-higher and dynamic sampling" is a common description, and that means "GRPO + the DAPO patches." The name GRPO++ is the umbrella for this patched family.

---

# 14.4 — The DeepSeek-R1 Pipeline

*The canonical recipe. Published as R1 (arXiv:2501.12948) and in Nature (s41586-025-09422-z). Every reasoning-RL paper since cites it.*

## R1-Zero: the proof that RL alone surfaces reasoning

R1-Zero is the experiment that changed the field's mind. DeepSeek took DeepSeek-V3-Base — a base model, no instruction tuning, no SFT — and applied RL directly with verifiable rewards (math, code, plus a formatting reward to encourage a readable chain-of-thought structure). No cold-start SFT.

The result: reasoning behavior *emerged*. The model began to self-verify, backtrack, and produce longer, more careful chains of thought — not because it was taught to, but because those behaviors were *selected for* by the correctness reward. Behaviors that lead to correct answers get upweighted; the policy drifts toward them. This is pure steering of capability the base already had (FT00), but the steering is strong enough that the *behavioral phenotype* looks like new capability.

R1-Zero's limitation: readability and format. The emergent chains were often messy, mixed languages, and hard to parse. Pure RL optimizes for correctness, not for the human-facing format. This is why R1 (the full pipeline) adds the SFT stages.

## R1: the multi-stage recipe

The published DeepSeek-R1 pipeline is four stages:

1. **Cold-start SFT.** A small amount of high-quality reasoning data (long CoT examples, curated) is used to SFT the base. This gives the model a readable chain-of-thought *format* to build on — solving R1-Zero's readability problem before RL starts. You are not teaching reasoning here; you are giving the model a clean scaffold.

2. **Reasoning RL.** GRPO on verifiable reasoning rewards. This is where the reasoning capability is sharpened. The model explores, the verifier rewards correct answers, the policy moves toward behaviors that produce them.

3. **Rejection-sampling SFT.** Take the RL-trained model, generate many candidate responses across a diverse prompt set, *filter by the reward* (keep the correct ones), and SFT a fresh model on this curated set. This is the critical bridge: RL produced high-quality reasoning traces; rejection sampling distills them into a clean SFT dataset; SFT on that set bakes the quality in *without* the instability of continued RL. (This is the technique FT15 covers in depth — CoT distillation and rejection-sampling FT.)

4. **Final RL for alignment.** A second RL stage, this time with a broader reward mix (reasoning correctness + safety/helpfulness) to align the model for general use. The reasoning capability from stage 2/3 is preserved; alignment is layered on top.

The published Nature paper frames this as the recipe that produced a frontier-class reasoning model from an open-weights base.

## Distillation: the surprise

The R1 paper's most practically influential finding: **distillation works better than expected.** DeepSeek took the R1 model and distilled it into 6 smaller dense models (1.5B to 70B, Qwen and Llama bases) via **SFT only** — no RL — on roughly 800K curated reasoning samples generated by R1. The resulting DeepSeek-R1-Distill-Qwen-32B *beats OpenAI o1-mini* on reasoning benchmarks.

This single result made chain-of-thought distillation a standard technique. The implication: once you have one strong reasoning teacher (trained via the expensive RL pipeline), you can stamp out cheaper reasoning students via SFT alone. That is the entire subject of FT15. The reason it works, again, is the FT00 thesis: the student bases already had the capability; the distilled CoT data steers them to use it. No new knowledge is transferred — only the behavioral pattern of careful, verified reasoning.

---

# 14.5 — The Qwen3 Pipeline

*The other canonical 2025 reference. Qwen3 technical report (arXiv:2505.09388, May 2025). Reads as a refinement and industrialization of the R1 recipe.*

## The 4-stage post-training

Qwen3's post-training is four stages:

1. **Long-CoT cold start.** SFT on long chain-of-thought data (curated, high-quality). Like R1's cold-start SFT, this establishes a readable reasoning scaffold. Qwen3 emphasizes the *long* aspect — the cold-start data features extended reasoning traces, setting the model's prior toward thorough CoT.

2. **Reasoning RL.** GRPO-style RL on verifiable rewards. This sharpens reasoning, exactly as in R1 stage 2. Scale here is large — Qwen3 reports substantial RL compute.

3. **Thinking mode fusion.** A Qwen3-specific step: merge the "thinking" mode (long CoT) with a "non-thinking" mode (fast, direct answers) into a single model. The model learns to do both, controlled by a mode flag. This is the practical answer to "sometimes you want the model to reason for 30 seconds, sometimes you want a one-line answer" — you should not need two models.

4. **General RL.** A broad RL stage for capability and alignment across the full task distribution, preserving the reasoning from stages 2–3 while improving general helpfulness.

## Pretraining scale and the thinking budget

Qwen3 reports 36T+ pretraining tokens — among the largest openly-disclosed pretraining budgets. The scale matters because reasoning RL *amplifies* what the base can do; a richer base gives RL more to work with. The thinking-budget mechanism is the deployment-side complement: the model can be instructed (via a token budget) to spend a bounded amount of "thinking" compute before answering, trading latency for accuracy adaptively. Easy questions get short budgets; hard questions get long budgets. This is the runtime analog of the training-time exploration: spend compute where it pays off.

> **R1 vs Qwen3, in one line:** R1 established that RL-on-verifiable-rewards surfaces reasoning and that distillation propagates it; Qwen3 industrialized the recipe (thinking/non-thinking fusion, thinking budgets, massive pretraining) into a single controllable model. Both are canonical; cite both.

---

# 14.6 — Tooling: TRL, verl, OpenRLHF

*Which trainer do you actually reach for? The answer is determined by scale.*

## TRL GRPOTrainer — moderate scale, the default starting point

HuggingFace TRL ships `GRPOTrainer` (and the sibling `RLOOTrainer`). It integrates with the standard transformers/peft/accelerate stack you already use for SFT and DPO. You supply a reward function (plain Python: takes a list of generated strings, returns a list of floats), a dataset of prompts, and a config. The trainer handles the group sampling, advantage computation, clipped objective, and KL penalty.

This is the right starting point for: research prototypes, small models (1B–7B on a single GPU or small cluster), and the lab in this module. It is what you use to *learn* GRPO and to validate a reward function before scaling. Its limitation: it is not built for the multi-node, high-throughput generation that frontier-scale RL requires. When your generation cost dominates and you need thousands of parallel rollouts, you outgrow TRL.

## verl (HybridFlow, ByteDance) — production-grade distributed RL

verl is the standard for large-scale GRPO/PPO. Built on Ray for distributed orchestration, it separates the *generation* (rollout) phase from the *training* (gradient) phase — the architecture that makes RL at scale tractable, because generation and training have very different compute profiles (generation is memory-bandwidth-bound and embarrassingly parallel; training is compute-bound and needs careful orchestration). verl handles the placement, the actor/rollout/reference decomposition, and the large-batch generation that reasoning RL needs. If you are training a 30B+ model on a cluster, this is the tool. ByteDance and others use it for production reasoning RL.

## OpenRLHF — the caveat

OpenRLHF is faster than verl on some micro-benchmarks (single-node, specific configs). But a `masked_mean` bug was found in its advantage/loss computation that silently miscalculated the per-token mean under certain padding conditions — leading to subtly wrong gradients on long sequences with heavy padding (exactly the reasoning-CoT setting). The bug has been addressed, but the community consensus crystallized as: **"TRL or verl and you're fine."** Use TRL for prototyping, verl for scale. Treat OpenRLHF as opt-in only if you have verified your specific config against the patched version.

> **The decision tree:** single GPU, learning, small model → TRL `GRPOTrainer`. Cluster, large model, production → verl. OpenRLHF only with a specific reason and a check that the masked_mean issue does not affect your padding pattern.

---

# 14.7 — The Reward-Verification Loop

*The heart of the method. Get this right and everything else is plumbing.*

The loop, per training step:

1. **Generate** N responses per prompt from the current policy.
2. **Verify** each response with the reward function: parse the answer, execute the check (run the code, compare the math answer, validate the tool output), return a scalar reward.
3. **Compute advantages** relative to the group: `A_i = (r_i − mean) / std`.
4. **Update** the policy toward high-advantage responses, away from low-advantage ones, with a KL anchor to the reference.

The verifier is the load-bearing component. A weak verifier (e.g., substring matching that is fooled by formatting) produces a reward signal the model will *game* — reward hacking. A strong verifier (execute the code and check the output; compare the parsed math answer exactly) produces a signal that genuinely tracks correctness. **The quality of your reasoning RL is bounded by the quality of your verifier.** This is the analog of FT00's "your data matters more than your algorithm": here, your *reward* matters more than your optimizer. A great GRPO config on a hackable reward steers you into a wall.

This is why the lab uses a *code-execution* reward function for math — it parses the model's answer and checks it against the ground truth, which is far more robust than substring or LLM-judge matching for correctness. Learn to write verifiers as carefully as you write training data.

---

## Anti-Patterns

### Using DPO for reasoning

DPO on a fixed preference dataset for a verifiable task. You forfeit on-policy exploration — the model never learns from *its own* mistakes, only from the mistakes in your fixed set. For math/code/tool-use, where the reward is checkable, this is leaving the main signal on the table. Use GRPO with a verifier.

### No reward verification (reward hacking)

A reward function that can be gamed (substring match, leniency, a judge that rewards confidence over correctness). The model will find the shortcut and optimize for it — producing outputs that score high on the reward but are wrong. The fix is a *verifier*, not a *judge*: execute the code, compare the parsed answer exactly, validate the tool output structurally. If you must use a judge (for non-verifiable rewards), you are back in DPO territory, not GRPO.

### Skipping the cold-start SFT

Going straight to RL on a raw base (the R1-Zero path) *can* work — it is the proof-of-emergence experiment — but it produces the R1-Zero problems: poor readability, mixed-language chains, hard-to-parse formats. The R1 full pipeline adds cold-start SFT *for a reason*. Unless you are specifically studying emergence, give the model a readable CoT scaffold before RL. It is one SFT stage and it saves you the readability cleanup later.

### Ignoring the readability problem of pure-RL

The complement of the above: even if pure-RL reasoning emerges, the output is often not in a deployable format. Production reasoning models need readable, parseable chains. Plan for the readability layer (cold-start SFT, and/or rejection-sampling SFT to clean up RL outputs) — do not assume RL alone yields a shippable artifact.

### Treating GRPO variants as interchangeable without reading the patches

"Use GRPO" is underspecified. DAPO's clip-higher changes exploration; Dr. GRPO's length-debias changes the length distribution of outputs; GSPO changes credit assignment. If your model stops improving mid-training, the fix is often a *specific* GRPO++ patch, not "more compute." Know which failure mode each patch addresses.

---

## Key Terms

| Term | Definition |
| --- | --- |
| **GRPO** | Group Relative Policy Optimization (DeepSeek). Samples N responses per prompt, computes advantages relative to the group mean. Eliminates the PPO critic → ~halves memory. |
| **RLOO** | REINFORCE Leave-One-Out. The sibling algorithm GRPO is essentially equivalent to — both use a group/leave-one-out baseline, no learned critic. |
| **Verifiable reward** | A reward computable by a deterministic checker (math correctness, code execution, tool-use success) — enables on-policy RL exploration. Contrast with preference rewards (DPO). |
| **On-policy exploration** | The model generates its own training attempts (rollouts) and learns from their verified outcomes, rather than from a fixed offline dataset. |
| **Advantage (group-relative)** | `A_i = (r_i − mean(r)) / std(r)` — how much better response i was than the group average. Replaces the PPO learned-critic advantage. |
| **R1-Zero** | DeepSeek's pure-RL-on-base experiment — proved reasoning can emerge from RL alone, no SFT cold start. Has readability/format issues. |
| **R1 pipeline** | Cold-start SFT → reasoning RL → rejection-sampling SFT → final alignment RL. The canonical multi-stage recipe. |
| **Rejection-sampling SFT** | Generate candidates with the RL model, filter by reward (keep correct), SFT a fresh model on the curated set. The bridge from RL to a stable SFT model (FT15). |
| **Distillation (CoT)** | SFT a smaller student on a strong reasoning teacher's CoT outputs. R1-Distill-Qwen-32B (SFT-only) beats o1-mini. (FT15.) |
| **DAPO** | ByteDance GRPO variant: clip-higher (decoupled clip for exploration) + dynamic sampling (skip zero-variance groups). |
| **Dr. GRPO** | Variant that removes the length bias in per-token advantage averaging. |
| **GSPO** | Group Sequence Policy Optimization — sequence-level (not token-level) importance weighting. |
| **Thinking budget** | Qwen3's runtime mechanism: bound the "thinking" compute per query, trading latency for accuracy adaptively. |
| **verl** | ByteDance's HybridFlow — production-grade distributed RL on Ray. The standard for large-scale GRPO. |
| **Reward hacking** | The model exploits a weak verifier's shortcut, scoring high on the reward while being wrong. Bounded by verifier quality. |

---

## Lab Exercise

See `07-lab-spec.md`. The "GRPO on a Math Task" lab: run TRL `GRPOTrainer` on a small base with a verifiable math reward (GSM8K correctness checked by a Python reward function that parses the answer and verifies it). Watch the reward curve climb. Heavy lab — single consumer GPU for the small model, cloud GPU stretch goal. Full runnable Python including the reward function, the GRPO config, training, and eval.

---

## References

1. **Shao et al. (2024)** — *DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models*. arXiv:2402.03300. Introduces GRPO (group-relative advantage, no critic).
2. **DeepSeek-AI (2025)** — *DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning*. arXiv:2501.12948. The R1-Zero and R1 pipeline.
3. **DeepSeek-AI (2025)** — *DeepSeek-R1* (Nature). s41586-025-09422-z. The peer-reviewed R1 pipeline.
4. **Qwen Team (2025)** — *Qwen3 Technical Report*. arXiv:2505.09388. The 4-stage post-training, thinking-budget, 36T pretraining.
5. **Kool et al. (2019)** — *Buy 4 REINFORCE Samples, Get a Baseline for Free*. The RLOO estimator — the sibling GRPO is essentially equivalent to.
6. **Cameron Wolfe** — *GRPO++ survey* (Distill, Substack). The canonical map of DAPO, Dr. GRPO, GSPO and the failure modes they patch.
7. **Yu et al. (2025)** — *DAPO: An Open-Source LLM Reinforcement Learning System*. arXiv:2503.14476. Clip-higher + dynamic sampling.
8. **Allen AI (2025)** — *OLMo 3*. Open-data production reasoning-RL recipe.
9. **HuggingFace TRL** — *GRPOTrainer documentation*. The moderate-scale entry point.
10. **HybridFlow / verl** — *volcengine/verl* (GitHub). ByteDance's production distributed RL framework on Ray.
11. **Interconnects (Nathan Lambert)** — Coverage of the RL-for-reasoning landscape, GRPO variants, and the OpenRLHF masked_mean discussion.
12. **FT13 — The DPO Family and Preferences** — The prerequisite. The offline-preference baseline this module contrasts against.
13. **FT15 — CoT Distillation and Rejection-Sampling FT** — The follow-on. Covers the distillation and RFT stages that R1/Qwen3 use after RL.