Module: FT14 — GRPO and Verifiable Rewards Diagram count: 6 Tool: Mermaid (primary). Each diagram validated in Mermaid Live Editor.
Type: Architecture comparison Purpose: The single structural insight that defines GRPO. PPO needs a learned value function (critic) the same size as the policy to estimate the advantage. GRPO computes the advantage relative to the group mean of N sampled responses — no critic, ~half the memory. Reading the diagram: Top row = PPO (policy + critic + reference = three models). Bottom row = GRPO (policy + reference = two models; the group IS the baseline).
flowchart TD
subgraph PPO["PPO — three models in memory"]
direction LR
PP["POLICY\n(trainable)\nthe LLM"]
PC["CRITIC / VALUE\n(trainable)\n~same size as policy\nestimates expected reward"]
PR["REFERENCE\n(frozen)\nfor KL anchor"]
PP -.-> PC
end
subgraph GRPO["GRPO — critic dropped"]
direction LR
GP["POLICY\n(trainable)\nthe LLM"]
GR["REFERENCE\n(frozen)\nfor KL anchor"]
GG["GROUP of N samples\n(the baseline)\nno params, no training"]
GG -.-> GP
end
PPO --> MemPPO["~3× model memory\n(policy + critic + reference)\ncritic must be trained\nextra instability"]
GRPO --> MemGRPO["~2× model memory\n(policy + reference)\nno critic to train\nbaseline = mean(r_1..r_N)"]
MemPPO -.->|"drop the critic"| MemGRPO
style PP fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#5eead4
style PC fill:#14141f,stroke:#f0a868,stroke-width:1.5px,color:#f0a868
style PR fill:#14141f,stroke:rgba(148,148,160,0.5),color:#9494a0
style GP fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#5eead4
style GR fill:#14141f,stroke:rgba(148,148,160,0.5),color:#9494a0
style GG fill:#14141f,stroke:rgba(94,234,212,0.6),color:#e4e4e8
style MemPPO fill:#08080c,stroke:rgba(240,128,128,0.5),color:#f08080
style MemGRPO fill:#08080c,stroke:rgba(130,224,170,0.5),color:#82e0aa
style PPO fill:#08080c,stroke:rgba(255,255,255,0.06),color:#9494a0
style GRPO fill:#08080c,stroke:rgba(255,255,255,0.06),color:#9494a0
Type: Sequential pipeline (the canonical recipe) Purpose: The full R1 story in one diagram. R1-Zero proves emergence (pure RL, no SFT). R1 fixes the readability with a 4-stage pipeline. Distillation propagates the capability to cheaper students via SFT only. Reading the diagram: Three tracks. Left = R1-Zero (one step, has problems). Middle = R1 (four stages). Right = distillation (SFT-only, 6 students, beats o1-mini).
flowchart LR
Base["DeepSeek-V3-Base\n(base model, no SFT)"]
Base --> R1Z["R1-ZERO\npure RL on verifiable rewards\nNO cold-start SFT"]
R1Z --> Emergence["reasoning EMERGES\nself-verify, backtrack, long CoT\n(capability was in the base —\nRL selects for using it)"]
Emergence --> R1ZProb["PROBLEM:\npoor readability\nmixed languages\nhard to parse"]
Base --> R1["R1 — full pipeline\n4 stages"]
R1 --> S1["1. COLD-START SFT\n(reasoning data)\ngives a readable CoT scaffold"]
S1 --> S2["2. REASONING RL\nGRPO on verifiable rewards\nsharpens the capability"]
S2 --> S3["3. REJECTION-SAMPLING SFT\ngenerate, filter by reward,\nSFT a fresh model on the\nsurvivors (→ FT15)"]
S3 --> S4["4. FINAL RL (alignment)\nreasoning + safety/helpfulness\npreserves reasoning, aligns"]
S4 --> R1Out["DeepSeek-R1\nfrontier-class reasoner"]
R1Out --> Distill["DISTILLATION\nSFT ONLY (no RL)\n~800K curated CoT samples\nfrom R1"]
Distill --> Students["6 dense students\n1.5B–70B (Qwen/Llama bases)\nR1-Distill-Qwen-32B\nbeats OpenAI o1-mini"]
style Base fill:#14141f,stroke:rgba(255,255,255,0.12),color:#e4e4e8
style R1Z fill:#14141f,stroke:#f0a868,stroke-width:1.5px,color:#f0a868
style Emergence fill:#14141f,stroke:rgba(94,234,212,0.6),color:#e4e4e8
style R1ZProb fill:#08080c,stroke:rgba(240,128,128,0.5),color:#f08080
style R1 fill:#14141f,stroke:#5eead4,stroke-width:2px,color:#5eead4
style S1 fill:#14141f,stroke:#5eead4,color:#e4e4e8
style S2 fill:#14141f,stroke:#5eead4,color:#e4e4e8
style S3 fill:#14141f,stroke:#5eead4,color:#e4e4e8
style S4 fill:#14141f,stroke:#5eead4,color:#e4e4e8
style R1Out fill:#14141f,stroke:rgba(130,224,170,0.6),color:#82e0aa
style Distill fill:#14141f,stroke:rgba(130,224,170,0.6),color:#82e0aa
style Students fill:#14141f,stroke:rgba(130,224,170,0.7),stroke-width:1.5px,color:#82e0aa
Type: Sequential pipeline (the industrialized recipe) Purpose: The other canonical 2025 reference. Reads as a refinement of R1 — adds long-CoT cold start, thinking-mode fusion, and a thinking budget at runtime. Reading the diagram: Pretraining (36T tokens) feeds the 4 post-training stages, ending in a single model that does both thinking and non-thinking modes with an adaptive compute budget.
flowchart LR
Pre["PRETRAINING\n36T+ tokens\n(scale gives RL more to amplify)"]
Pre --> Q1["1. LONG-CoT COLD START\nSFT on long chain-of-thought\nsets a prior toward thorough reasoning"]
Q1 --> Q2["2. REASONING RL\nGRPO-style on verifiable rewards\n(large-scale)"]
Q2 --> Q3["3. THINKING MODE FUSION\nmerge thinking (long CoT)\n+ non-thinking (fast, direct)\ninto ONE model, mode-flagged"]
Q3 --> Q4["4. GENERAL RL\nbroad capability + alignment\npreserves reasoning from 2-3"]
Q4 --> Out["Qwen3\nsingle model, two modes"]
Out --> Budget["THINKING BUDGET (runtime)\nbound the 'thinking' compute\nper query\nshort budget → fast answer\nlong budget → hard problems\nadaptive latency/accuracy trade"]
style Pre fill:#14141f,stroke:rgba(255,255,255,0.12),color:#e4e4e8
style Q1 fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#5eead4
style Q2 fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#5eead4
style Q3 fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#5eead4
style Q4 fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#5eead4
style Out fill:#14141f,stroke:rgba(130,224,170,0.6),color:#82e0aa
style Budget fill:#08080c,stroke:rgba(130,224,170,0.5),color:#82e0aa
Type: Decision / comparison Purpose: The most important judgment in the module. If the reward is verifiable, use RL (GRPO) — on-policy exploration. If it is only a preference, use DPO — offline. Reading the diagram: The decision diamond splits on "can you write a verifier?" Yes → GRPO (generate, verify, learn from your own attempts). No → DPO (learn from a fixed preference dataset).
flowchart TD
Task["You want to steer the model's behavior"]
Task --> Q{"Can you write a VERIFIER?\n(deterministic correct/incorrect)"}
Q -->|"YES — math, code, tool use"| GRPO["GRPO / RL on verifiable rewards"]
Q -->|"NO — style, helpfulness, tone"| DPO["DPO family (FT13)\noffline preference pairs"]
GRPO --> Explore["ON-POLICY EXPLORATION\nmodel generates N attempts per prompt\nverifier scores each\nlearn from the distribution\nof YOUR OWN errors"]
DPO --> Offline["OFFLINE PREFERENCE\nfixed (chosen, rejected) pairs\nmodel re-scores, never generates\nlearn from SOMEONE ELSE's\nattempts"]
Explore --> Out1["correct for reasoning\n(verifiable reward enables it)"]
Offline --> Out2["correct for preference\n(subjective, no verifier)"]
style Task fill:#14141f,stroke:rgba(255,255,255,0.12),color:#e4e4e8
style Q fill:#08080c,stroke:rgba(94,234,212,0.5),color:#e4e4e8
style GRPO fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#5eead4
style DPO fill:#14141f,stroke:#f0a868,stroke-width:1.5px,color:#f0a868
style Explore fill:#14141f,stroke:#5eead4,color:#e4e4e8
style Offline fill:#14141f,stroke:#f0a868,color:#e4e4e8
style Out1 fill:#08080c,stroke:rgba(130,224,170,0.5),color:#82e0aa
style Out2 fill:#08080c,stroke:rgba(130,224,170,0.5),color:#82e0aa
Type: Failure-mode → fix map Purpose: GRPO works, but production recipes patch specific failure modes. Each variant name corresponds to a specific problem. Per Cameron Wolfe's survey. Reading the diagram: The base is GRPO/RLOO. Three patches, each addressing a distinct issue. In practice, recipes mix and match — "GRPO with clip-higher and dynamic sampling" means GRPO + DAPO patches.
flowchart TD
Base["GRPO / RLOO (base)\ngroup-relative advantage, no critic\nthe family GRPO++ builds on"]
Base --> F1["FAILURE: exploration stalls\nonce policy is doing well\n(symmetric clip caps upweighting)"]
Base --> F2["FAILURE: length bias\nper-token averaging favors\nshorter sequences"]
Base --> F3["FAILURE: noisy credit assignment\non long CoT (token-level\nimportance ratios)"]
F1 --> DAPO["DAPO (ByteDance)\n• CLIP-HIGHER: decouple clip range\n for + vs - advantages\n• DYNAMIC SAMPLING: skip groups\n with zero reward variance\n (all-correct or all-wrong)"]
F2 --> DrGRPO["Dr. GRPO\nremoves the length bias in\nper-token advantage averaging\n(no 'shorter = higher reward' shortcut)"]
F3 --> GSPO["GSPO\nSEQUENCE-LEVEL importance weighting\n(one weight per full sequence,\nnot per token) → stable long-CoT"]
DAPO --> Verdict["In production: mix & match\n'GRPO + clip-higher + dynamic sampling'\n= GRPO with DAPO patches"]
DrGRPO --> Verdict
GSPO --> Verdict
Verdict --> OLMo["OLMo 3 (2025)\nopen-data production recipe\nthe reproducible reference"]
style Base fill:#14141f,stroke:#5eead4,stroke-width:2px,color:#5eead4
style F1 fill:#08080c,stroke:rgba(240,128,128,0.5),color:#f08080
style F2 fill:#08080c,stroke:rgba(240,128,128,0.5),color:#f08080
style F3 fill:#08080c,stroke:rgba(240,128,128,0.5),color:#f08080
style DAPO fill:#14141f,stroke:rgba(94,234,212,0.7),color:#e4e4e8
style DrGRPO fill:#14141f,stroke:rgba(94,234,212,0.7),color:#e4e4e8
style GSPO fill:#14141f,stroke:rgba(94,234,212,0.7),color:#e4e4e8
style Verdict fill:#08080c,stroke:rgba(94,234,212,0.4),color:#e4e4e8
style OLMo fill:#14141f,stroke:rgba(130,224,170,0.6),color:#82e0aa
Type: The training loop, made explicit Purpose: The heart of the method. Per step: generate N → verify each → compute group-relative advantage → update policy. The verifier is the load-bearing component — its quality bounds the whole run. Reading the diagram: A cyclic loop. The verifier sits between generation and advantage — if it is hackable, the model games it (reward hacking).
flowchart LR
Prompt["prompt"]
Prompt --> Gen["1. GENERATE\nN responses from current policy\n(N=8 or 16)"]
Gen --> Samples["response_1 ... response_N"]
Samples --> Verify["2. VERIFY (reward function)\nparse answer → execute/check\n• math: compare parsed answer\n• code: run, check output\n• tools: validate output structure"]
Verify --> Rewards["r_1 ... r_N\n(scalar rewards)"]
Rewards --> Adv["3. ADVANTAGE (group-relative)\nA_i = (r_i − mean(r)) / std(r)\nNO learned critic"]
Adv --> Update["4. UPDATE POLICY\nclipped PPO-style objective\n+ KL anchor to reference\nmove toward high-A_i responses"]
Update --> Gen
Verify -.->|"WEAK verifier\n(substring, leniency)"| Hack["REWARD HACKING\nmodel finds the shortcut\nscores high, outputs wrong"]
Verify -.->|"STRONG verifier\n(execute, exact compare)"| Good["signal tracks true correctness\nthe whole run is bounded by this"]
style Prompt fill:#14141f,stroke:rgba(255,255,255,0.12),color:#e4e4e8
style Gen fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#5eead4
style Samples fill:#08080c,stroke:rgba(94,234,212,0.4),color:#e4e4e8
style Verify fill:#14141f,stroke:#5eead4,stroke-width:2px,color:#5eead4
style Rewards fill:#08080c,stroke:rgba(94,234,212,0.4),color:#e4e4e8
style Adv fill:#14141f,stroke:#5eead4,color:#e4e4e8
style Update fill:#14141f,stroke:#5eead4,color:#e4e4e8
style Hack fill:#08080c,stroke:rgba(240,128,128,0.5),color:#f08080
style Good fill:#08080c,stroke:rgba(130,224,170,0.5),color:#82e0aa
#14141f panel fill, #5eead4 (--accent) for the primary/learned path, rgba(94,234,212,0.4-0.6) for decision diamonds and group nodes, #f08080 (--danger) for failure/anti-pattern/loss nodes, #f0a868 (--warn) for the weaker/DPO/R1-Zero-problem path, #82e0aa (--ok) for the recommended/success path, #e4e4e8/#9494a0 for text.flowchart LR/TD, subgraph, decision diamonds {}, dashed -.label.-> annotation links) supported in current Mermaid (v10.4+).# Diagrams — Module FT14: GRPO and Verifiable Rewards
**Module**: FT14 — GRPO and Verifiable Rewards
**Diagram count**: 6
**Tool**: Mermaid (primary). Each diagram validated in [Mermaid Live Editor](https://mermaid.live).
---
## Diagram 1 — GRPO vs PPO: Dropping the Critic
**Type**: Architecture comparison
**Purpose**: The single structural insight that defines GRPO. PPO needs a learned value function (critic) the same size as the policy to estimate the advantage. GRPO computes the advantage relative to the group mean of N sampled responses — no critic, ~half the memory.
**Reading the diagram**: Top row = PPO (policy + critic + reference = three models). Bottom row = GRPO (policy + reference = two models; the group IS the baseline).
```mermaid
flowchart TD
subgraph PPO["PPO — three models in memory"]
direction LR
PP["POLICY\n(trainable)\nthe LLM"]
PC["CRITIC / VALUE\n(trainable)\n~same size as policy\nestimates expected reward"]
PR["REFERENCE\n(frozen)\nfor KL anchor"]
PP -.-> PC
end
subgraph GRPO["GRPO — critic dropped"]
direction LR
GP["POLICY\n(trainable)\nthe LLM"]
GR["REFERENCE\n(frozen)\nfor KL anchor"]
GG["GROUP of N samples\n(the baseline)\nno params, no training"]
GG -.-> GP
end
PPO --> MemPPO["~3× model memory\n(policy + critic + reference)\ncritic must be trained\nextra instability"]
GRPO --> MemGRPO["~2× model memory\n(policy + reference)\nno critic to train\nbaseline = mean(r_1..r_N)"]
MemPPO -.->|"drop the critic"| MemGRPO
style PP fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#5eead4
style PC fill:#14141f,stroke:#f0a868,stroke-width:1.5px,color:#f0a868
style PR fill:#14141f,stroke:rgba(148,148,160,0.5),color:#9494a0
style GP fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#5eead4
style GR fill:#14141f,stroke:rgba(148,148,160,0.5),color:#9494a0
style GG fill:#14141f,stroke:rgba(94,234,212,0.6),color:#e4e4e8
style MemPPO fill:#08080c,stroke:rgba(240,128,128,0.5),color:#f08080
style MemGRPO fill:#08080c,stroke:rgba(130,224,170,0.5),color:#82e0aa
style PPO fill:#08080c,stroke:rgba(255,255,255,0.06),color:#9494a0
style GRPO fill:#08080c,stroke:rgba(255,255,255,0.06),color:#9494a0
```
---
## Diagram 2 — The DeepSeek-R1 Pipeline: R1-Zero → R1 → Distillation
**Type**: Sequential pipeline (the canonical recipe)
**Purpose**: The full R1 story in one diagram. R1-Zero proves emergence (pure RL, no SFT). R1 fixes the readability with a 4-stage pipeline. Distillation propagates the capability to cheaper students via SFT only.
**Reading the diagram**: Three tracks. Left = R1-Zero (one step, has problems). Middle = R1 (four stages). Right = distillation (SFT-only, 6 students, beats o1-mini).
```mermaid
flowchart LR
Base["DeepSeek-V3-Base\n(base model, no SFT)"]
Base --> R1Z["R1-ZERO\npure RL on verifiable rewards\nNO cold-start SFT"]
R1Z --> Emergence["reasoning EMERGES\nself-verify, backtrack, long CoT\n(capability was in the base —\nRL selects for using it)"]
Emergence --> R1ZProb["PROBLEM:\npoor readability\nmixed languages\nhard to parse"]
Base --> R1["R1 — full pipeline\n4 stages"]
R1 --> S1["1. COLD-START SFT\n(reasoning data)\ngives a readable CoT scaffold"]
S1 --> S2["2. REASONING RL\nGRPO on verifiable rewards\nsharpens the capability"]
S2 --> S3["3. REJECTION-SAMPLING SFT\ngenerate, filter by reward,\nSFT a fresh model on the\nsurvivors (→ FT15)"]
S3 --> S4["4. FINAL RL (alignment)\nreasoning + safety/helpfulness\npreserves reasoning, aligns"]
S4 --> R1Out["DeepSeek-R1\nfrontier-class reasoner"]
R1Out --> Distill["DISTILLATION\nSFT ONLY (no RL)\n~800K curated CoT samples\nfrom R1"]
Distill --> Students["6 dense students\n1.5B–70B (Qwen/Llama bases)\nR1-Distill-Qwen-32B\nbeats OpenAI o1-mini"]
style Base fill:#14141f,stroke:rgba(255,255,255,0.12),color:#e4e4e8
style R1Z fill:#14141f,stroke:#f0a868,stroke-width:1.5px,color:#f0a868
style Emergence fill:#14141f,stroke:rgba(94,234,212,0.6),color:#e4e4e8
style R1ZProb fill:#08080c,stroke:rgba(240,128,128,0.5),color:#f08080
style R1 fill:#14141f,stroke:#5eead4,stroke-width:2px,color:#5eead4
style S1 fill:#14141f,stroke:#5eead4,color:#e4e4e8
style S2 fill:#14141f,stroke:#5eead4,color:#e4e4e8
style S3 fill:#14141f,stroke:#5eead4,color:#e4e4e8
style S4 fill:#14141f,stroke:#5eead4,color:#e4e4e8
style R1Out fill:#14141f,stroke:rgba(130,224,170,0.6),color:#82e0aa
style Distill fill:#14141f,stroke:rgba(130,224,170,0.6),color:#82e0aa
style Students fill:#14141f,stroke:rgba(130,224,170,0.7),stroke-width:1.5px,color:#82e0aa
```
---
## Diagram 3 — The Qwen3 4-Stage Post-Training Pipeline
**Type**: Sequential pipeline (the industrialized recipe)
**Purpose**: The other canonical 2025 reference. Reads as a refinement of R1 — adds long-CoT cold start, thinking-mode fusion, and a thinking budget at runtime.
**Reading the diagram**: Pretraining (36T tokens) feeds the 4 post-training stages, ending in a single model that does both thinking and non-thinking modes with an adaptive compute budget.
```mermaid
flowchart LR
Pre["PRETRAINING\n36T+ tokens\n(scale gives RL more to amplify)"]
Pre --> Q1["1. LONG-CoT COLD START\nSFT on long chain-of-thought\nsets a prior toward thorough reasoning"]
Q1 --> Q2["2. REASONING RL\nGRPO-style on verifiable rewards\n(large-scale)"]
Q2 --> Q3["3. THINKING MODE FUSION\nmerge thinking (long CoT)\n+ non-thinking (fast, direct)\ninto ONE model, mode-flagged"]
Q3 --> Q4["4. GENERAL RL\nbroad capability + alignment\npreserves reasoning from 2-3"]
Q4 --> Out["Qwen3\nsingle model, two modes"]
Out --> Budget["THINKING BUDGET (runtime)\nbound the 'thinking' compute\nper query\nshort budget → fast answer\nlong budget → hard problems\nadaptive latency/accuracy trade"]
style Pre fill:#14141f,stroke:rgba(255,255,255,0.12),color:#e4e4e8
style Q1 fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#5eead4
style Q2 fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#5eead4
style Q3 fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#5eead4
style Q4 fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#5eead4
style Out fill:#14141f,stroke:rgba(130,224,170,0.6),color:#82e0aa
style Budget fill:#08080c,stroke:rgba(130,224,170,0.5),color:#82e0aa
```
---
## Diagram 4 — DPO vs GRPO: Offline Preference vs On-Policy Verifiable
**Type**: Decision / comparison
**Purpose**: The most important judgment in the module. If the reward is verifiable, use RL (GRPO) — on-policy exploration. If it is only a preference, use DPO — offline.
**Reading the diagram**: The decision diamond splits on "can you write a verifier?" Yes → GRPO (generate, verify, learn from your own attempts). No → DPO (learn from a fixed preference dataset).
```mermaid
flowchart TD
Task["You want to steer the model's behavior"]
Task --> Q{"Can you write a VERIFIER?\n(deterministic correct/incorrect)"}
Q -->|"YES — math, code, tool use"| GRPO["GRPO / RL on verifiable rewards"]
Q -->|"NO — style, helpfulness, tone"| DPO["DPO family (FT13)\noffline preference pairs"]
GRPO --> Explore["ON-POLICY EXPLORATION\nmodel generates N attempts per prompt\nverifier scores each\nlearn from the distribution\nof YOUR OWN errors"]
DPO --> Offline["OFFLINE PREFERENCE\nfixed (chosen, rejected) pairs\nmodel re-scores, never generates\nlearn from SOMEONE ELSE's\nattempts"]
Explore --> Out1["correct for reasoning\n(verifiable reward enables it)"]
Offline --> Out2["correct for preference\n(subjective, no verifier)"]
style Task fill:#14141f,stroke:rgba(255,255,255,0.12),color:#e4e4e8
style Q fill:#08080c,stroke:rgba(94,234,212,0.5),color:#e4e4e8
style GRPO fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#5eead4
style DPO fill:#14141f,stroke:#f0a868,stroke-width:1.5px,color:#f0a868
style Explore fill:#14141f,stroke:#5eead4,color:#e4e4e8
style Offline fill:#14141f,stroke:#f0a868,color:#e4e4e8
style Out1 fill:#08080c,stroke:rgba(130,224,170,0.5),color:#82e0aa
style Out2 fill:#08080c,stroke:rgba(130,224,170,0.5),color:#82e0aa
```
---
## Diagram 5 — The GRPO++ Variant Landscape
**Type**: Failure-mode → fix map
**Purpose**: GRPO works, but production recipes patch specific failure modes. Each variant name corresponds to a specific problem. Per Cameron Wolfe's survey.
**Reading the diagram**: The base is GRPO/RLOO. Three patches, each addressing a distinct issue. In practice, recipes mix and match — "GRPO with clip-higher and dynamic sampling" means GRPO + DAPO patches.
```mermaid
flowchart TD
Base["GRPO / RLOO (base)\ngroup-relative advantage, no critic\nthe family GRPO++ builds on"]
Base --> F1["FAILURE: exploration stalls\nonce policy is doing well\n(symmetric clip caps upweighting)"]
Base --> F2["FAILURE: length bias\nper-token averaging favors\nshorter sequences"]
Base --> F3["FAILURE: noisy credit assignment\non long CoT (token-level\nimportance ratios)"]
F1 --> DAPO["DAPO (ByteDance)\n• CLIP-HIGHER: decouple clip range\n for + vs - advantages\n• DYNAMIC SAMPLING: skip groups\n with zero reward variance\n (all-correct or all-wrong)"]
F2 --> DrGRPO["Dr. GRPO\nremoves the length bias in\nper-token advantage averaging\n(no 'shorter = higher reward' shortcut)"]
F3 --> GSPO["GSPO\nSEQUENCE-LEVEL importance weighting\n(one weight per full sequence,\nnot per token) → stable long-CoT"]
DAPO --> Verdict["In production: mix & match\n'GRPO + clip-higher + dynamic sampling'\n= GRPO with DAPO patches"]
DrGRPO --> Verdict
GSPO --> Verdict
Verdict --> OLMo["OLMo 3 (2025)\nopen-data production recipe\nthe reproducible reference"]
style Base fill:#14141f,stroke:#5eead4,stroke-width:2px,color:#5eead4
style F1 fill:#08080c,stroke:rgba(240,128,128,0.5),color:#f08080
style F2 fill:#08080c,stroke:rgba(240,128,128,0.5),color:#f08080
style F3 fill:#08080c,stroke:rgba(240,128,128,0.5),color:#f08080
style DAPO fill:#14141f,stroke:rgba(94,234,212,0.7),color:#e4e4e8
style DrGRPO fill:#14141f,stroke:rgba(94,234,212,0.7),color:#e4e4e8
style GSPO fill:#14141f,stroke:rgba(94,234,212,0.7),color:#e4e4e8
style Verdict fill:#08080c,stroke:rgba(94,234,212,0.4),color:#e4e4e8
style OLMo fill:#14141f,stroke:rgba(130,224,170,0.6),color:#82e0aa
```
---
## Diagram 6 — The Reward-Verification Loop
**Type**: The training loop, made explicit
**Purpose**: The heart of the method. Per step: generate N → verify each → compute group-relative advantage → update policy. The verifier is the load-bearing component — its quality bounds the whole run.
**Reading the diagram**: A cyclic loop. The verifier sits between generation and advantage — if it is hackable, the model games it (reward hacking).
```mermaid
flowchart LR
Prompt["prompt"]
Prompt --> Gen["1. GENERATE\nN responses from current policy\n(N=8 or 16)"]
Gen --> Samples["response_1 ... response_N"]
Samples --> Verify["2. VERIFY (reward function)\nparse answer → execute/check\n• math: compare parsed answer\n• code: run, check output\n• tools: validate output structure"]
Verify --> Rewards["r_1 ... r_N\n(scalar rewards)"]
Rewards --> Adv["3. ADVANTAGE (group-relative)\nA_i = (r_i − mean(r)) / std(r)\nNO learned critic"]
Adv --> Update["4. UPDATE POLICY\nclipped PPO-style objective\n+ KL anchor to reference\nmove toward high-A_i responses"]
Update --> Gen
Verify -.->|"WEAK verifier\n(substring, leniency)"| Hack["REWARD HACKING\nmodel finds the shortcut\nscores high, outputs wrong"]
Verify -.->|"STRONG verifier\n(execute, exact compare)"| Good["signal tracks true correctness\nthe whole run is bounded by this"]
style Prompt fill:#14141f,stroke:rgba(255,255,255,0.12),color:#e4e4e8
style Gen fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#5eead4
style Samples fill:#08080c,stroke:rgba(94,234,212,0.4),color:#e4e4e8
style Verify fill:#14141f,stroke:#5eead4,stroke-width:2px,color:#5eead4
style Rewards fill:#08080c,stroke:rgba(94,234,212,0.4),color:#e4e4e8
style Adv fill:#14141f,stroke:#5eead4,color:#e4e4e8
style Update fill:#14141f,stroke:#5eead4,color:#e4e4e8
style Hack fill:#08080c,stroke:rgba(240,128,128,0.5),color:#f08080
style Good fill:#08080c,stroke:rgba(130,224,170,0.5),color:#82e0aa
```
---
## Validation notes
- All six diagrams use the course design system colors: `#14141f` panel fill, `#5eead4` (`--accent`) for the primary/learned path, `rgba(94,234,212,0.4-0.6)` for decision diamonds and group nodes, `#f08080` (`--danger`) for failure/anti-pattern/loss nodes, `#f0a868` (`--warn`) for the weaker/DPO/R1-Zero-problem path, `#82e0aa` (`--ok`) for the recommended/success path, `#e4e4e8`/`#9494a0` for text.
- Paste each into [Mermaid Live Editor](https://mermaid.live) to render. All use stable Mermaid syntax (`flowchart LR/TD`, `subgraph`, decision diamonds `{}`, dashed `-.label.->` annotation links) supported in current Mermaid (v10.4+).
- For the slide deck (artifact 03), these are rendered as static captures from Mermaid Live, inlined into reveal.js.