Lab Specification — Module FT14: GRPO on a Math Task

Course: Course 3 — LLM Fine-Tuning Masterclass Module: FT14 — GRPO and Verifiable Rewards Duration: 45–70 minutes of GPU time (plus ~10 min setup) on a consumer GPU; cloud GPU for the stretch goals Environment: Python 3.10+, a CUDA GPU with ≥24 GB VRAM (RTX 4090 / 3090 / A10G / A100 / Colab Pro A100). For the smaller base, a 16 GB card (Colab T4 / RTX 4060 Ti) works with reduced num_generations.

This is the flagship lab. By the end you will have run GRPO (Group Relative Policy Optimization) via TRL's GRPOTrainer on a small base model with a verifiable math reward — a Python reward function that parses the model's \boxed{} answer and checks it against ground truth exactly. You will watch the reward curve climb step-over-step, which is on-policy exploration working in front of you. This is the loop that builds reasoning models (o1 / R1 / Qwen3-style), at a scale you can run on one GPU.

Setup (one time)

python3 -m venv ft14-env && source ft14-env/bin/activate
pip install -q -U "torch>=2.3" "transformers>=4.46" "trl>=0.14" \
    "peft>=0.13" "accelerate>=0.34" "datasets>=3.0"
# bitsandbytes only if you use the 4-bit loading option below
pip install -q -U "bitsandbytes>=0.43"

On Colab: Runtime → Change runtime type → T4 GPU (free) or A100 (Pro). Then run the same pip install (drop the venv lines).

You do not need a Hugging Face token for the default base (Qwen/Qwen2.5-1.5B-Instruct is open).

TRL version note. GRPO support in TRL landed in trl>=0.14 and has been refined through the 0.15–0.16 line. If from trl import GRPOConfig, GRPOTrainer fails, upgrade: pip install -U "trl>=0.14". The API below (GRPOConfig, GRPOTrainer, reward function signature (prompts, completions, **kwargs)) is stable across those versions.

Learning objectives

By the end of this lab you will have:

  1. Written a verifiable reward function — a Python function that parses the model's \boxed{} answer and compares it exactly to ground truth. This is the load-bearing component of any reasoning-RL run; its quality bounds the whole pipeline.
  2. Run GRPO end to end — sample N responses per prompt, compute group-relative advantages (no critic), update the policy with a clipped PPO-style objective and a KL anchor — and seen the reward climb.
  3. Felt the on-policy exploration — watched the model learn from the distribution of its own verified attempts, the thing DPO structurally cannot do (FT13).
  4. Evaluated on held-out math — confirmed the reward gain reflects real correctness, not reward hacking, by checking accuracy on unseen problems.
  5. Internalized the decision rule — verifiable reward → GRPO; preference-only → DPO. You will have run the verifiable side with your own hands.

Phase 0 — Pick your base (2 min)

The default is Qwen/Qwen2.5-1.5B-Instruct — open (no gating), ChatML, and small enough that GRPO with num_generations=8 fits on a 24 GB card. It is an instruct model (already SFT'd), which gives the model a usable starting format — a reasonable analog to having done the cold-start SFT (R1 stage 1) without us doing it here.

Base Why Notes
Qwen/Qwen2.5-1.5B-Instruct Default. Open, ChatML, ~3 GB. Fits 24 GB with N=8; fits 16 GB with N=4.
Qwen/Qwen2.5-0.5B-Instruct Smaller — for 16 GB cards / Colab T4. Faster iterations; weaker starting math.
Qwen/Qwen2.5-Math-1.5B-Instruct Math-tuned base — stronger start. Good for seeing faster reward climb. 
MODEL_ID = "Qwen/Qwen2.5-1.5B-Instruct"   # default; swap freely

Phase 1 — Build the math dataset (5 min)

We use a small, hand-built set of grade-school arithmetic word problems with known integer answers. Each example has a prompt (the question, with an instruction to box the final answer) and an answer (the ground-truth string). The reward function will parse \boxed{} from the model's completion and compare to answer exactly.

The point is a verifiable reward — we know the correct answer, so we can check deterministically. This is what makes GRPO the right tool (vs DPO).

Create make_data.py:

# make_data.py — build a small verifiable math dataset
import json, random
random.seed(42)

# Each: (question, ground-truth answer as a string)
PROBLEMS = [
    ("A baker has 12 trays of muffins. Each tray holds 8 muffins. "
     "She sells 47 muffins. How many muffins are left?", "49"),
    ("A train travels 60 miles per hour for 3 hours, then 45 miles per hour "
     "for 2 hours. What is the total distance traveled in miles?", "270"),
    ("A store sells pencils for $0.25 each. If you buy 14 pencils, "
     "how much do you spend in dollars?", "3.5"),
    ("A rectangle has a length of 9 and a width of 5. What is its area?", "45"),
    ("There are 144 eggs. They are packed into cartons of 12. "
     "How many cartons are there?", "12"),
    ("A book has 360 pages. You read 45 pages a day. "
     "How many days to finish the book?", "8"),
    ("A car uses 8 liters of fuel per 100 km. How many liters for 350 km?", "28"),
    ("A class has 28 students. 3/4 of them play a sport. "
     "How many students play a sport?", "21"),
    ("A pizza is cut into 8 slices. 3 people eat 2 slices each. "
     "How many slices are left?", "2"),
    ("A farmer has 15 chickens and 9 ducks. Each chicken lays 5 eggs a week. "
     "How many chicken eggs per week?", "75"),
    ("A tank holds 500 liters. It is filled at 25 liters per minute. "
     "How many minutes to fill it?", "20"),
    ("A shirt costs $24. It is on sale for 25% off. What is the sale price in dollars?", "18"),
    ("A runner jogs 4 km in 20 minutes. What is the speed in km per hour?", "12"),
    ("A box has 7 red, 5 blue, and 3 green balls. How many balls total?", "15"),
    ("You have $50. You buy 3 books at $9 each. How much is left in dollars?", "23"),
    ("A garden has 6 rows of 11 plants. How many plants total?", "66"),
    ("A clock loses 3 minutes every hour. How many minutes does it lose in 9 hours?", "27"),
    ("A recipe needs 2 cups of flour for 12 cookies. "
     "How many cups for 60 cookies?", "10"),
    ("A bus has 40 seats. 5/8 are full. How many passengers are there?", "25"),
    ("A worker earns $18 per hour and works 35 hours. "
     "What is the total pay in dollars?", "630"),
]

PROMPT_TEMPLATE = (
    "Solve the following math problem step by step. "
    "Put your final numerical answer inside \\boxed{}.\n\n"
    "Problem: {q}\n\nAnswer:"
)

def make_prompt(q):
    return [{"role": "user", "content": PROMPT_TEMPLATE.format(q=q)}]

# 30 training rows (each problem used once; GRPO samples N per prompt)
train_rows = [{"prompt": make_prompt(q), "answer": a} for q, a in PROBLEMS[:18]]
# hold out 2 for eval
eval_rows = [{"prompt": make_prompt(q), "answer": a} for q, a in PROBLEMS[18:]]

with open("math_train.jsonl", "w") as f:
    for r in train_rows:
        f.write(json.dumps(r) + "\n")
with open("math_eval.jsonl", "w") as f:
    for r in eval_rows:
        f.write(json.dumps(r) + "\n")
print(f"Wrote {len(train_rows)} train, {len(eval_rows)} eval")

Run it:

python make_data.py
head -1 math_train.jsonl | python -m json.tool

Why so few training rows? GRPO does not need a large dataset — it needs prompts with verifiable answers. Each prompt generates N rollouts, so 18 prompts × N=8 = 144 generated-and-scored attempts per epoch. The curriculum is generated on-policy, not pre-collected. This is the structural difference from SFT/DPO (which need large fixed datasets). Add more problems if you want a longer run.

Phase 2 — Write the verifiable reward function (5 min)

This is the heart of the lab. The reward function is what makes this GRPO (on-policy RL on a verifiable reward) rather than DPO (offline preference). It is plain Python: TRL calls it with the lists of prompts and completions (plus any extra dataset columns as kwargs), and it returns a list of float rewards.

Create reward.py:

# reward.py — the verifiable reward function (load-bearing)
import re

def extract_boxed(text: str):
    """Pull the content of the LAST \\boxed{...} in the text.
    Handles one level of nesting via a simple brace-matching scan."""
    # find all \boxed{ start positions
    starts = [m.end() for m in re.finditer(r"\\boxed\{", text)]
    if not starts:
        return None
    # scan from the last occurrence
    start = starts[-1]
    depth = 1
    i = start
    while i < len(text) and depth > 0:
        if text[i] == "{":
            depth += 1
        elif text[i] == "}":
            depth -= 1
        i += 1
    if depth != 0:
        return None  # unbalanced
    return text[start:i - 1].strip()

def normalize_answer(s: str) -> str:
    """Normalize a numeric answer string for exact comparison."""
    s = s.strip()
    # strip trailing zeros after decimal: 3.50 -> 3.5, 18.0 -> 18
    if "." in s:
        s = s.rstrip("0").rstrip(".")
    return s

def math_reward(prompts, completions, answer, **kwargs):
    """Verifiable reward: parse \\boxed{}, compare exactly to ground truth.

    Returns 1.0 for an exact (normalized) match, 0.0 otherwise.
    This is a VERIFIER (deterministic), not a judge — which is what makes
    GRPO the right tool here (vs DPO, which needs a preference dataset).
    """
    rewards = []
    for comp, gold in zip(completions, answer):
        pred = extract_boxed(comp)
        if pred is None:
            rewards.append(0.0)
            continue
        rewards.append(1.0 if normalize_answer(pred) == normalize_answer(gold) else 0.0)
    return rewards

# Quick self-test (run: python -m reward or python reward.py)
if __name__ == "__main__":
    tests = [
        ("The answer is \\boxed{49}.", "49", 1.0),
        ("\\boxed{270}", "270", 1.0),
        ("\\boxed{3.50}", "3.5", 1.0),   # normalization
        ("\\boxed{18.0}", "18", 1.0),    # normalization
        ("I think \\boxed{45} muffins.", "45", 1.0),
        ("\\boxed{50}", "49", 0.0),      # wrong answer
        ("the answer is 49", "49", 0.0), # no \boxed{} -> 0.0
        ("\\boxed{}", "49", 0.0),        # empty box
    ]
    for comp, gold, expected in tests:
        got = math_reward(["x"], [comp], [gold])[0]
        status = "OK" if got == expected else "FAIL"
        print(f"[{status}] comp={comp!r:40} gold={gold!r:6} -> {got} (expected {expected})")

Run the self-test to confirm the verifier behaves correctly:

python reward.py

Expected: all 8 tests print OK. If any print FAIL, fix the function before training — a broken verifier produces a broken reward signal, and GRPO will dutifully optimize whatever (wrong) signal it gets. This is the "your reward matters more than your optimizer" principle in action.

Why this is a verifier, not a judge. It parses a structured field (\boxed{}) and compares the parsed value exactly to ground truth. There is no model judging "is this reasonable?" — there is a deterministic check. A judge-based reward (an LLM scoring the response) would be hackable (the model could learn to sound confident); this is not. For verifiable tasks, always prefer a verifier. This is what makes GRPO correct where DPO is wrong.

Phase 3 — Configure and run GRPO (10–30 min GPU)

Create train.py. We load the base, configure GRPOConfig, and launch GRPOTrainer. The trainer handles the loop: sample N per prompt → score with math_reward → group-relative advantage → clipped update + KL anchor.

# train.py — GRPO on a verifiable math reward
import torch
from datasets import load_dataset
from transformers import AutoTokenizer
from trl import GRPOConfig, GRPOTrainer
from reward import math_reward

MODEL_ID = "Qwen/Qwen2.5-1.5B-Instruct"

tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
if tokenizer.pad_token_id is None:
    tokenizer.pad_token = tokenizer.eos_token

# Load the model inside the trainer (GRPOTrainer takes a model id or model obj)
dataset = load_dataset("json", data_files="math_train.jsonl", split="train")

# --- GRPO config ---
# num_generations = N (the group size). Per prompt, the trainer samples N
# responses and computes group-relative advantages (no critic).
config = GRPOConfig(
    output_dir="./grpo-math",
    num_train_epochs=3,
    per_device_train_batch_size=4,       # prompts per step (each -> N generations)
    num_generations=8,                   # N: the group size for advantage
    max_prompt_length=256,
    max_completion_length=256,
    learning_rate=5e-6,                  # RL LRs are much lower than SFT (~10x)
    lr_scheduler_type="cosine",
    warmup_ratio=0.05,
    logging_steps=1,                     # log every step (few steps total)
    save_strategy="epoch",
    bf16=True,                           # use fp16=True on older cards (V100)
    gradient_checkpointing=True,
    report_to="none",
    # generation kwargs for the rollout
    generation_kwargs={
        "do_sample": True,
        "temperature": 0.7,
        "top_p": 0.9,
    },
    # KL anchor to the reference (keeps the policy from drifting too far)
    beta=0.04,                           # KL coefficient
)

trainer = GRPOTrainer(
    model=MODEL_ID,
    reward_funcs=math_reward,            # our verifiable reward function
    args=config,
    train_dataset=dataset,
    processing_class=tokenizer,
)

trainer.train()
trainer.save_model("./grpo-math/final")
print("Done. Model saved to ./grpo-math/final")

Run it:

python train.py

What you will see in the logs. TRL logs reward (mean reward per step), reward_std (the group reward variance — your gradient signal), and the KL. Watch the reward climb over steps. That climb is on-policy exploration working: the model generates N attempts, the verifier scores them, and the policy moves toward the behaviors (step-by-step arithmetic, boxing the answer) that produce correct results. If reward_std is near zero for many steps, those prompts have no gradient signal (all-correct or all-wrong) — that is the failure DAPO's dynamic sampling patches.

Watch the VRAM with nvidia-smi -l 2 in a second terminal. With N=8 and a 1.5B model, peak lands around 18–22 GB (8 concurrent generations are memory-heavy). If you OOM: reduce num_generations to 4 (and per_device_train_batch_size to 2), or use the 0.5B base, or enable 4-bit loading (see Stretch Goal 3).

Sanity checkpoints:

Phase 4 — Inspect the rollouts (3 min)

Add this snippet at the end of train.py (or run interactively) to see what the model actually generated — this is where you catch reward hacking early:

# Peek at one batch of rollouts BEFORE you trust the reward curve
batch = next(iter(trainer.get_train_dataloader()))
prompts = batch["prompt"]
# the trainer will generate; to inspect, generate manually once:
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained(MODEL_ID, torch_dtype=torch.bfloat16, device_map="auto")
for p in prompts[:2]:
    text = tokenizer.apply_chat_template(p, tokenize=False, add_generation_prompt=True)
    inp = tokenizer(text, return_tensors="pt").to(model.device)
    out = model.generate(**inp, max_new_tokens=200, do_sample=True, temperature=0.7, top_p=0.9)
    print("---")
    print(tokenizer.decode(out[0][inp.input_ids.shape[-1]:], skip_special_tokens=True))

Read the outputs. You should see step-by-step arithmetic ending in \boxed{<number>}. If you see the model emitting \boxed{} with a random number and no work, but the reward is climbing — that is reward hacking (the verifier is somehow being gamed; re-check extract_boxed). If the model ignores the \boxed{} instruction entirely, the prompt template or the base's chat format is misconfigured.

Phase 5 — Evaluate on held-out math (5 min)

The reward curve climbing is necessary but not sufficient — you must confirm the gain reflects real correctness on unseen problems, not overfitting to the 18 training prompts.

Create eval.py:

# eval.py — held-out accuracy check
import torch, json, re
from transformers import AutoModelForCausalLM, AutoTokenizer
from reward import extract_boxed, normalize_answer

MODEL_PATH = "./grpo-math/final"
MODEL_ID = "Qwen/Qwen2.5-1.5B-Instruct"   # for comparison (un-trained baseline)

def evaluate(model_path, eval_file="math_eval.jsonl"):
    tok = AutoTokenizer.from_pretrained(model_path)
    model = AutoModelForCausalLM.from_pretrained(model_path, torch_dtype=torch.bfloat16, device_map="auto")
    rows = [json.loads(l) for l in open(eval_file)]
    correct = 0
    for r in rows:
        prompt = r["prompt"]
        gold = r["answer"]
        text = tok.apply_chat_template(prompt, tokenize=False, add_generation_prompt=True)
        inp = tok(text, return_tensors="pt").to(model.device)
        with torch.no_grad():
            out = model.generate(**inp, max_new_tokens=256, do_sample=False)  # greedy for eval
        comp = tok.decode(out[0][inp.input_ids.shape[-1]:], skip_special_tokens=True)
        pred = extract_boxed(comp)
        ok = pred is not None and normalize_answer(pred) == normalize_answer(gold)
        correct += int(ok)
        print(f"{'OK ' if ok else 'X  '} gold={gold:6} pred={pred!r:10} | {comp[:80].strip()}...")
    print(f"\nAccuracy: {correct}/{len(rows)} = {correct/len(rows)*100:.0f}%")
    return correct / len(rows)

print("=== GRPO-trained model ===")
evaluate(MODEL_PATH)
print("\n=== Baseline (un-trained instruct) ===")
evaluate(MODEL_ID)

Run:

python eval.py

Expected: the GRPO-trained model scores higher than the un-trained baseline on the held-out problems. The gap is your RL run — on-policy exploration on a verifiable reward, producing real correctness gains. If both score the same (or the trained model is worse), the RL did not generalize: check for overfitting (too many epochs on 18 prompts) or a reward function that does not track true correctness.

Deliverables

Submit ft14-lab-report.md containing:

Solution key

These are defensible answers, not the only wording.

Reward function self-test

All 8 tests should print OK. The key behaviors: \boxed{49} → matches 49; 3.50 normalizes to 3.5; 18.0 normalizes to 18; missing \boxed{} → 0.0; empty box → 0.0; wrong number → 0.0. If extract_boxed fails on nested braces, the simple scanner handles one level; for deeply nested LaTeX you would reach for a proper parser, but for numeric answers the simple version suffices.

Reward trajectory

Expect: start reward ~0.1–0.3 (the 1.5B instruct base gets some grade-school arithmetic right by luck), trending upward over the steps to ~0.5–0.8 by the final epoch. reward_std should be non-zero on most steps (mix of correct/incorrect) — that is your gradient signal. If reward_std is frequently 0, many prompts are all-correct or all-wrong (zero gradient) — that is the failure DAPO's dynamic sampling patches. With only 18 prompts × 3 epochs / batch 4, you get ~13 steps; increase num_train_epochs to 5–8 for a clearer upward curve.

VRAM

Peak ~18–22 GB for Qwen2.5-1.5B with N=8, max_completion_length=256, bf16, gradient checkpointing. The 8 concurrent generations (the group) are the memory driver — each holds KV cache for 256 tokens. If OOM: N=4 + batch 2 (~12–14 GB), or the 0.5B base (~8–10 GB at N=8), or 4-bit loading (Stretch 3).

Eval

GRPO model: expect ~60–90% on the 2 held-out problems (small sample, so variance is high — add more eval problems for a stable number). Baseline: ~30–60%. The gap (often +1–2 problems) is the RL gain. With only 2 eval problems, run the eval a few times with do_sample=True (temperature 0.7) and average, or add more held-out problems, to reduce variance. The point is to confirm the gain is real correctness, not training-set memorization.

Reflection (model answer)

GRPO was the right tool because the reward is verifiable: we have the ground-truth answer, so we can write a deterministic verifier (parse \boxed{}, compare exactly). That verifier enables on-policy exploration — the model generates N attempts per prompt, we score each, and the policy learns from the distribution of its own successes and failures, including error patterns no fixed dataset contained. With DPO, we would have needed a pre-collected preference dataset of (chosen, rejected) pairs, and the model would only re-score those — never generating, never exploring its own current error distribution. For a verifiable skill like arithmetic, the verifier generates an infinite curriculum tailored to the model's live weak spots; DPO forfeits that. The reward curve climbing is on-policy exploration made visible.

Stretch goals

  1. verl for scale. Port the reward function and dataset to verl (HybridFlow, ByteDance) on a multi-GPU cluster. verl separates the generation (rollout) phase from the training (gradient) phase on Ray — the architecture that makes 30B+ reasoning RL tractable. The reward function is the same; the orchestration is what scales. This is the production path (vs TRL for prototyping).

  2. Code-execution reward. Replace the math task with a coding task: the prompt asks for a Python function, the reward function execs the generated code (in a sandbox) and checks the output against test cases. The loop is identical — generate, verify (execute), learn. This generalizes "verifiable reward" beyond math to any task with a deterministic checker.

  3. 4-bit base loading. Load the base in 4-bit (NF4 + double quant, the QLoRA innovations from FT08) before GRPO to halve the base memory, letting you run a 3B–7B model on a 24 GB card. Combine with a LoRA adapter (PEFT) so only the adapter trains — GRPO on a QLoRA base. This is the memory-efficiency play for running reasoning RL on consumer hardware.

  4. DAPO patches. Add clip-higher (decouple the ± PPO clip range) and dynamic sampling (skip prompts where all N samples get the same reward) to your config. Compare the reward curve and exploration diversity to the baseline GRPO run. Observe: with clip-higher, exploration stays alive longer (the reward keeps climbing past where vanilla GRPO plateaus); with dynamic sampling, compute is spent on informative prompts.

  5. Cold-start SFT first. Before GRPO, run a quick SFT stage on a few dozen step-by-step CoT examples ending in \boxed{} (the R1 stage-1 cold-start, at tiny scale). Then run GRPO. Compare the reward climb and the readability of the outputs to the no-cold-start run. This demonstrates the R1 pipeline's rationale: cold-start SFT gives a readable scaffold that pure RL (R1-Zero) lacks.

# Lab Specification — Module FT14: GRPO on a Math Task

**Course**: Course 3 — LLM Fine-Tuning Masterclass
**Module**: FT14 — GRPO and Verifiable Rewards
**Duration**: 45–70 minutes of GPU time (plus ~10 min setup) on a consumer GPU; cloud GPU for the stretch goals
**Environment**: Python 3.10+, a CUDA GPU with ≥24 GB VRAM (RTX 4090 / 3090 / A10G / A100 / Colab Pro A100). For the smaller base, a 16 GB card (Colab T4 / RTX 4060 Ti) works with reduced `num_generations`.

> **This is the flagship lab.** By the end you will have run GRPO (Group Relative Policy Optimization) via TRL's `GRPOTrainer` on a small base model with a *verifiable* math reward — a Python reward function that parses the model's `\boxed{}` answer and checks it against ground truth exactly. You will watch the reward curve climb step-over-step, which is on-policy exploration working in front of you. This is the loop that builds reasoning models (o1 / R1 / Qwen3-style), at a scale you can run on one GPU.

---

## Setup (one time)

```bash
python3 -m venv ft14-env && source ft14-env/bin/activate
pip install -q -U "torch>=2.3" "transformers>=4.46" "trl>=0.14" \
    "peft>=0.13" "accelerate>=0.34" "datasets>=3.0"
# bitsandbytes only if you use the 4-bit loading option below
pip install -q -U "bitsandbytes>=0.43"
```

On Colab: Runtime → Change runtime type → T4 GPU (free) or A100 (Pro). Then run the same `pip install` (drop the venv lines).

You do **not** need a Hugging Face token for the default base (`Qwen/Qwen2.5-1.5B-Instruct` is open).

> **TRL version note.** GRPO support in TRL landed in `trl>=0.14` and has been refined through the 0.15–0.16 line. If `from trl import GRPOConfig, GRPOTrainer` fails, upgrade: `pip install -U "trl>=0.14"`. The API below (`GRPOConfig`, `GRPOTrainer`, reward function signature `(prompts, completions, **kwargs)`) is stable across those versions.

---

## Learning objectives

By the end of this lab you will have:

1. **Written a verifiable reward function** — a Python function that parses the model's `\boxed{}` answer and compares it exactly to ground truth. This is the load-bearing component of any reasoning-RL run; its quality bounds the whole pipeline.
2. **Run GRPO end to end** — sample N responses per prompt, compute group-relative advantages (no critic), update the policy with a clipped PPO-style objective and a KL anchor — and seen the reward climb.
3. **Felt the on-policy exploration** — watched the model learn from the distribution of *its own* verified attempts, the thing DPO structurally cannot do (FT13).
4. **Evaluated on held-out math** — confirmed the reward gain reflects real correctness, not reward hacking, by checking accuracy on unseen problems.
5. **Internalized the decision rule** — verifiable reward → GRPO; preference-only → DPO. You will have run the verifiable side with your own hands.

---

## Phase 0 — Pick your base (2 min)

The default is **`Qwen/Qwen2.5-1.5B-Instruct`** — open (no gating), ChatML, and small enough that GRPO with `num_generations=8` fits on a 24 GB card. It is an *instruct* model (already SFT'd), which gives the model a usable starting format — a reasonable analog to having done the cold-start SFT (R1 stage 1) without us doing it here.

| Base | Why | Notes |
| --- | --- | --- |
| `Qwen/Qwen2.5-1.5B-Instruct` | **Default.** Open, ChatML, ~3 GB. | Fits 24 GB with N=8; fits 16 GB with N=4. |
| `Qwen/Qwen2.5-0.5B-Instruct` | Smaller — for 16 GB cards / Colab T4. | Faster iterations; weaker starting math. |
| `Qwen/Qwen2.5-Math-1.5B-Instruct` | Math-tuned base — stronger start. | Good for seeing faster reward climb. |

```python
MODEL_ID = "Qwen/Qwen2.5-1.5B-Instruct"   # default; swap freely
```

---

## Phase 1 — Build the math dataset (5 min)

We use a small, hand-built set of grade-school arithmetic word problems with known integer answers. Each example has a `prompt` (the question, with an instruction to box the final answer) and an `answer` (the ground-truth string). The reward function will parse `\boxed{}` from the model's completion and compare to `answer` exactly.

The point is a *verifiable* reward — we know the correct answer, so we can check deterministically. This is what makes GRPO the right tool (vs DPO).

Create `make_data.py`:

```python
# make_data.py — build a small verifiable math dataset
import json, random
random.seed(42)

# Each: (question, ground-truth answer as a string)
PROBLEMS = [
    ("A baker has 12 trays of muffins. Each tray holds 8 muffins. "
     "She sells 47 muffins. How many muffins are left?", "49"),
    ("A train travels 60 miles per hour for 3 hours, then 45 miles per hour "
     "for 2 hours. What is the total distance traveled in miles?", "270"),
    ("A store sells pencils for $0.25 each. If you buy 14 pencils, "
     "how much do you spend in dollars?", "3.5"),
    ("A rectangle has a length of 9 and a width of 5. What is its area?", "45"),
    ("There are 144 eggs. They are packed into cartons of 12. "
     "How many cartons are there?", "12"),
    ("A book has 360 pages. You read 45 pages a day. "
     "How many days to finish the book?", "8"),
    ("A car uses 8 liters of fuel per 100 km. How many liters for 350 km?", "28"),
    ("A class has 28 students. 3/4 of them play a sport. "
     "How many students play a sport?", "21"),
    ("A pizza is cut into 8 slices. 3 people eat 2 slices each. "
     "How many slices are left?", "2"),
    ("A farmer has 15 chickens and 9 ducks. Each chicken lays 5 eggs a week. "
     "How many chicken eggs per week?", "75"),
    ("A tank holds 500 liters. It is filled at 25 liters per minute. "
     "How many minutes to fill it?", "20"),
    ("A shirt costs $24. It is on sale for 25% off. What is the sale price in dollars?", "18"),
    ("A runner jogs 4 km in 20 minutes. What is the speed in km per hour?", "12"),
    ("A box has 7 red, 5 blue, and 3 green balls. How many balls total?", "15"),
    ("You have $50. You buy 3 books at $9 each. How much is left in dollars?", "23"),
    ("A garden has 6 rows of 11 plants. How many plants total?", "66"),
    ("A clock loses 3 minutes every hour. How many minutes does it lose in 9 hours?", "27"),
    ("A recipe needs 2 cups of flour for 12 cookies. "
     "How many cups for 60 cookies?", "10"),
    ("A bus has 40 seats. 5/8 are full. How many passengers are there?", "25"),
    ("A worker earns $18 per hour and works 35 hours. "
     "What is the total pay in dollars?", "630"),
]

PROMPT_TEMPLATE = (
    "Solve the following math problem step by step. "
    "Put your final numerical answer inside \\boxed{}.\n\n"
    "Problem: {q}\n\nAnswer:"
)

def make_prompt(q):
    return [{"role": "user", "content": PROMPT_TEMPLATE.format(q=q)}]

# 30 training rows (each problem used once; GRPO samples N per prompt)
train_rows = [{"prompt": make_prompt(q), "answer": a} for q, a in PROBLEMS[:18]]
# hold out 2 for eval
eval_rows = [{"prompt": make_prompt(q), "answer": a} for q, a in PROBLEMS[18:]]

with open("math_train.jsonl", "w") as f:
    for r in train_rows:
        f.write(json.dumps(r) + "\n")
with open("math_eval.jsonl", "w") as f:
    for r in eval_rows:
        f.write(json.dumps(r) + "\n")
print(f"Wrote {len(train_rows)} train, {len(eval_rows)} eval")
```

Run it:

```bash
python make_data.py
head -1 math_train.jsonl | python -m json.tool
```

> **Why so few training rows?** GRPO does not need a large dataset — it needs *prompts with verifiable answers*. Each prompt generates N rollouts, so 18 prompts × N=8 = 144 generated-and-scored attempts per epoch. The curriculum is generated on-policy, not pre-collected. This is the structural difference from SFT/DPO (which need large fixed datasets). Add more problems if you want a longer run.

---

## Phase 2 — Write the verifiable reward function (5 min)

This is the heart of the lab. The reward function is what makes this GRPO (on-policy RL on a verifiable reward) rather than DPO (offline preference). It is plain Python: TRL calls it with the lists of prompts and completions (plus any extra dataset columns as kwargs), and it returns a list of float rewards.

Create `reward.py`:

```python
# reward.py — the verifiable reward function (load-bearing)
import re

def extract_boxed(text: str):
    """Pull the content of the LAST \\boxed{...} in the text.
    Handles one level of nesting via a simple brace-matching scan."""
    # find all \boxed{ start positions
    starts = [m.end() for m in re.finditer(r"\\boxed\{", text)]
    if not starts:
        return None
    # scan from the last occurrence
    start = starts[-1]
    depth = 1
    i = start
    while i < len(text) and depth > 0:
        if text[i] == "{":
            depth += 1
        elif text[i] == "}":
            depth -= 1
        i += 1
    if depth != 0:
        return None  # unbalanced
    return text[start:i - 1].strip()

def normalize_answer(s: str) -> str:
    """Normalize a numeric answer string for exact comparison."""
    s = s.strip()
    # strip trailing zeros after decimal: 3.50 -> 3.5, 18.0 -> 18
    if "." in s:
        s = s.rstrip("0").rstrip(".")
    return s

def math_reward(prompts, completions, answer, **kwargs):
    """Verifiable reward: parse \\boxed{}, compare exactly to ground truth.

    Returns 1.0 for an exact (normalized) match, 0.0 otherwise.
    This is a VERIFIER (deterministic), not a judge — which is what makes
    GRPO the right tool here (vs DPO, which needs a preference dataset).
    """
    rewards = []
    for comp, gold in zip(completions, answer):
        pred = extract_boxed(comp)
        if pred is None:
            rewards.append(0.0)
            continue
        rewards.append(1.0 if normalize_answer(pred) == normalize_answer(gold) else 0.0)
    return rewards

# Quick self-test (run: python -m reward or python reward.py)
if __name__ == "__main__":
    tests = [
        ("The answer is \\boxed{49}.", "49", 1.0),
        ("\\boxed{270}", "270", 1.0),
        ("\\boxed{3.50}", "3.5", 1.0),   # normalization
        ("\\boxed{18.0}", "18", 1.0),    # normalization
        ("I think \\boxed{45} muffins.", "45", 1.0),
        ("\\boxed{50}", "49", 0.0),      # wrong answer
        ("the answer is 49", "49", 0.0), # no \boxed{} -> 0.0
        ("\\boxed{}", "49", 0.0),        # empty box
    ]
    for comp, gold, expected in tests:
        got = math_reward(["x"], [comp], [gold])[0]
        status = "OK" if got == expected else "FAIL"
        print(f"[{status}] comp={comp!r:40} gold={gold!r:6} -> {got} (expected {expected})")
```

Run the self-test to confirm the verifier behaves correctly:

```bash
python reward.py
```

**Expected:** all 8 tests print `OK`. If any print `FAIL`, fix the function before training — a broken verifier produces a broken reward signal, and GRPO will dutifully optimize whatever (wrong) signal it gets. This is the "your reward matters more than your optimizer" principle in action.

> **Why this is a verifier, not a judge.** It parses a structured field (`\boxed{}`) and compares the parsed value *exactly* to ground truth. There is no model judging "is this reasonable?" — there is a deterministic check. A judge-based reward (an LLM scoring the response) would be hackable (the model could learn to sound confident); this is not. For verifiable tasks, always prefer a verifier. This is what makes GRPO correct where DPO is wrong.

---

## Phase 3 — Configure and run GRPO (10–30 min GPU)

Create `train.py`. We load the base, configure `GRPOConfig`, and launch `GRPOTrainer`. The trainer handles the loop: sample N per prompt → score with `math_reward` → group-relative advantage → clipped update + KL anchor.

```python
# train.py — GRPO on a verifiable math reward
import torch
from datasets import load_dataset
from transformers import AutoTokenizer
from trl import GRPOConfig, GRPOTrainer
from reward import math_reward

MODEL_ID = "Qwen/Qwen2.5-1.5B-Instruct"

tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
if tokenizer.pad_token_id is None:
    tokenizer.pad_token = tokenizer.eos_token

# Load the model inside the trainer (GRPOTrainer takes a model id or model obj)
dataset = load_dataset("json", data_files="math_train.jsonl", split="train")

# --- GRPO config ---
# num_generations = N (the group size). Per prompt, the trainer samples N
# responses and computes group-relative advantages (no critic).
config = GRPOConfig(
    output_dir="./grpo-math",
    num_train_epochs=3,
    per_device_train_batch_size=4,       # prompts per step (each -> N generations)
    num_generations=8,                   # N: the group size for advantage
    max_prompt_length=256,
    max_completion_length=256,
    learning_rate=5e-6,                  # RL LRs are much lower than SFT (~10x)
    lr_scheduler_type="cosine",
    warmup_ratio=0.05,
    logging_steps=1,                     # log every step (few steps total)
    save_strategy="epoch",
    bf16=True,                           # use fp16=True on older cards (V100)
    gradient_checkpointing=True,
    report_to="none",
    # generation kwargs for the rollout
    generation_kwargs={
        "do_sample": True,
        "temperature": 0.7,
        "top_p": 0.9,
    },
    # KL anchor to the reference (keeps the policy from drifting too far)
    beta=0.04,                           # KL coefficient
)

trainer = GRPOTrainer(
    model=MODEL_ID,
    reward_funcs=math_reward,            # our verifiable reward function
    args=config,
    train_dataset=dataset,
    processing_class=tokenizer,
)

trainer.train()
trainer.save_model("./grpo-math/final")
print("Done. Model saved to ./grpo-math/final")
```

Run it:

```bash
python train.py
```

**What you will see in the logs.** TRL logs `reward` (mean reward per step), `reward_std` (the group reward variance — your gradient signal), and the KL. Watch the **reward climb** over steps. That climb is on-policy exploration working: the model generates N attempts, the verifier scores them, and the policy moves toward the behaviors (step-by-step arithmetic, boxing the answer) that produce correct results. If `reward_std` is near zero for many steps, those prompts have no gradient signal (all-correct or all-wrong) — that is the failure DAPO's dynamic sampling patches.

**Watch the VRAM** with `nvidia-smi -l 2` in a second terminal. With N=8 and a 1.5B model, peak lands around **18–22 GB** (8 concurrent generations are memory-heavy). If you OOM: reduce `num_generations` to 4 (and `per_device_train_batch_size` to 2), or use the 0.5B base, or enable 4-bit loading (see Stretch Goal 3).

**Sanity checkpoints:**
- Reward starts somewhere in 0.0–0.3 (the base gets some right by luck).
- Over ~30–60 steps (18 prompts × 3 epochs / batch 4 ≈ 13 steps; increase epochs for a clearer curve), reward trends upward.
- If reward is stuck at exactly 0.0 or 1.0 from step 1 with zero variance, your reward function or prompt format is broken — re-run the `reward.py` self-test and inspect a raw completion.

---

## Phase 4 — Inspect the rollouts (3 min)

Add this snippet at the end of `train.py` (or run interactively) to see what the model actually generated — this is where you catch reward hacking early:

```python
# Peek at one batch of rollouts BEFORE you trust the reward curve
batch = next(iter(trainer.get_train_dataloader()))
prompts = batch["prompt"]
# the trainer will generate; to inspect, generate manually once:
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained(MODEL_ID, torch_dtype=torch.bfloat16, device_map="auto")
for p in prompts[:2]:
    text = tokenizer.apply_chat_template(p, tokenize=False, add_generation_prompt=True)
    inp = tokenizer(text, return_tensors="pt").to(model.device)
    out = model.generate(**inp, max_new_tokens=200, do_sample=True, temperature=0.7, top_p=0.9)
    print("---")
    print(tokenizer.decode(out[0][inp.input_ids.shape[-1]:], skip_special_tokens=True))
```

**Read the outputs.** You should see step-by-step arithmetic ending in `\boxed{<number>}`. If you see the model emitting `\boxed{}` with a random number and no work, but the reward is climbing — that is *reward hacking* (the verifier is somehow being gamed; re-check `extract_boxed`). If the model ignores the `\boxed{}` instruction entirely, the prompt template or the base's chat format is misconfigured.

---

## Phase 5 — Evaluate on held-out math (5 min)

The reward curve climbing is necessary but not sufficient — you must confirm the gain reflects *real* correctness on *unseen* problems, not overfitting to the 18 training prompts.

Create `eval.py`:

```python
# eval.py — held-out accuracy check
import torch, json, re
from transformers import AutoModelForCausalLM, AutoTokenizer
from reward import extract_boxed, normalize_answer

MODEL_PATH = "./grpo-math/final"
MODEL_ID = "Qwen/Qwen2.5-1.5B-Instruct"   # for comparison (un-trained baseline)

def evaluate(model_path, eval_file="math_eval.jsonl"):
    tok = AutoTokenizer.from_pretrained(model_path)
    model = AutoModelForCausalLM.from_pretrained(model_path, torch_dtype=torch.bfloat16, device_map="auto")
    rows = [json.loads(l) for l in open(eval_file)]
    correct = 0
    for r in rows:
        prompt = r["prompt"]
        gold = r["answer"]
        text = tok.apply_chat_template(prompt, tokenize=False, add_generation_prompt=True)
        inp = tok(text, return_tensors="pt").to(model.device)
        with torch.no_grad():
            out = model.generate(**inp, max_new_tokens=256, do_sample=False)  # greedy for eval
        comp = tok.decode(out[0][inp.input_ids.shape[-1]:], skip_special_tokens=True)
        pred = extract_boxed(comp)
        ok = pred is not None and normalize_answer(pred) == normalize_answer(gold)
        correct += int(ok)
        print(f"{'OK ' if ok else 'X  '} gold={gold:6} pred={pred!r:10} | {comp[:80].strip()}...")
    print(f"\nAccuracy: {correct}/{len(rows)} = {correct/len(rows)*100:.0f}%")
    return correct / len(rows)

print("=== GRPO-trained model ===")
evaluate(MODEL_PATH)
print("\n=== Baseline (un-trained instruct) ===")
evaluate(MODEL_ID)
```

Run:

```bash
python eval.py
```

**Expected:** the GRPO-trained model scores higher than the un-trained baseline on the held-out problems. The gap is your RL run — on-policy exploration on a verifiable reward, producing real correctness gains. If both score the same (or the trained model is worse), the RL did not generalize: check for overfitting (too many epochs on 18 prompts) or a reward function that does not track true correctness.

---

## Deliverables

Submit `ft14-lab-report.md` containing:

- [ ] **Your `make_data.py`** and 3 sample training rows (question + ground-truth answer).
- [ ] **The reward function self-test output** (`python reward.py`) — all 8 tests must print `OK`. This is your proof the verifier works before training.
- [ ] **The GRPO training log** — the step-by-step `reward` and `reward_std` values (TRL logs these). Plot or list them. Report start reward, end reward, and the trend.
- [ ] **Peak VRAM observed** (`nvidia-smi`). State your `num_generations` and base size.
- [ ] **Two sampled rollouts** from Phase 4 — the actual text the model generated, showing step-by-step work ending in `\boxed{}`. Confirm there is no reward hacking (real reasoning, not a gamed shortcut).
- [ ] **Held-out eval accuracy** (Phase 5) for both the GRPO model and the un-trained baseline. The gap is your evidence the RL generalized.
- [ ] A 4–6 sentence reflection: why was GRPO (not DPO) the right tool here? Tie it to the verifiability of the reward and on-policy exploration. What would have been lost with an offline preference dataset?

---

## Solution key

These are defensible answers, not the only wording.

### Reward function self-test
All 8 tests should print `OK`. The key behaviors: `\boxed{49}` → matches `49`; `3.50` normalizes to `3.5`; `18.0` normalizes to `18`; missing `\boxed{}` → `0.0`; empty box → `0.0`; wrong number → `0.0`. If `extract_boxed` fails on nested braces, the simple scanner handles one level; for deeply nested LaTeX you would reach for a proper parser, but for numeric answers the simple version suffices.

### Reward trajectory
Expect: start reward ~0.1–0.3 (the 1.5B instruct base gets some grade-school arithmetic right by luck), trending upward over the steps to ~0.5–0.8 by the final epoch. `reward_std` should be non-zero on most steps (mix of correct/incorrect) — that is your gradient signal. If `reward_std` is frequently 0, many prompts are all-correct or all-wrong (zero gradient) — that is the failure DAPO's dynamic sampling patches. With only 18 prompts × 3 epochs / batch 4, you get ~13 steps; increase `num_train_epochs` to 5–8 for a clearer upward curve.

### VRAM
Peak ~18–22 GB for Qwen2.5-1.5B with N=8, max_completion_length=256, bf16, gradient checkpointing. The 8 concurrent generations (the group) are the memory driver — each holds KV cache for 256 tokens. If OOM: N=4 + batch 2 (~12–14 GB), or the 0.5B base (~8–10 GB at N=8), or 4-bit loading (Stretch 3).

### Eval
GRPO model: expect ~60–90% on the 2 held-out problems (small sample, so variance is high — add more eval problems for a stable number). Baseline: ~30–60%. The gap (often +1–2 problems) is the RL gain. With only 2 eval problems, run the eval a few times with `do_sample=True` (temperature 0.7) and average, or add more held-out problems, to reduce variance. The point is to confirm the gain is real correctness, not training-set memorization.

### Reflection (model answer)
GRPO was the right tool because the reward is *verifiable*: we have the ground-truth answer, so we can write a deterministic verifier (parse `\boxed{}`, compare exactly). That verifier enables on-policy exploration — the model generates N attempts per prompt, we score each, and the policy learns from the distribution of its own successes and failures, including error patterns no fixed dataset contained. With DPO, we would have needed a pre-collected preference dataset of (chosen, rejected) pairs, and the model would only re-score those — never generating, never exploring its own current error distribution. For a verifiable skill like arithmetic, the verifier generates an infinite curriculum tailored to the model's live weak spots; DPO forfeits that. The reward curve climbing is on-policy exploration made visible.

---

## Stretch goals

1. **verl for scale.** Port the reward function and dataset to [verl](https://github.com/volcengine/verl) (HybridFlow, ByteDance) on a multi-GPU cluster. verl separates the generation (rollout) phase from the training (gradient) phase on Ray — the architecture that makes 30B+ reasoning RL tractable. The reward function is the same; the orchestration is what scales. This is the production path (vs TRL for prototyping).

2. **Code-execution reward.** Replace the math task with a coding task: the prompt asks for a Python function, the reward function `exec`s the generated code (in a sandbox) and checks the output against test cases. The loop is identical — generate, verify (execute), learn. This generalizes "verifiable reward" beyond math to any task with a deterministic checker.

3. **4-bit base loading.** Load the base in 4-bit (NF4 + double quant, the QLoRA innovations from FT08) before GRPO to halve the base memory, letting you run a 3B–7B model on a 24 GB card. Combine with a LoRA adapter (PEFT) so only the adapter trains — GRPO on a QLoRA base. This is the memory-efficiency play for running reasoning RL on consumer hardware.

4. **DAPO patches.** Add clip-higher (decouple the ± PPO clip range) and dynamic sampling (skip prompts where all N samples get the same reward) to your config. Compare the reward curve and exploration diversity to the baseline GRPO run. Observe: with clip-higher, exploration stays alive longer (the reward keeps climbing past where vanilla GRPO plateaus); with dynamic sampling, compute is spent on informative prompts.

5. **Cold-start SFT first.** Before GRPO, run a quick SFT stage on a few dozen step-by-step CoT examples ending in `\boxed{}` (the R1 stage-1 cold-start, at tiny scale). Then run GRPO. Compare the reward climb and the readability of the outputs to the no-cold-start run. This demonstrates the R1 pipeline's rationale: cold-start SFT gives a readable scaffold that pure RL (R1-Zero) lacks.