"Question"	"Answer"	"Tag"
"State the decision rule for choosing DPO vs GRPO."	"If your reward is VERIFIABLE (you can write a deterministic function returning correct/incorrect — math correctness, code execution, tool-use success), use GRPO/RL for on-policy exploration. If your reward is only expressible as a PREFERENCE (a human or judge must compare two outputs — style, helpfulness, tone), use DPO. Reasoning is the canonical verifiable domain; hence RL."	c3::ft14::recall
"Why is DPO structurally wrong for reasoning tasks?"	"DPO learns from a FIXED dataset of (chosen, rejected) pairs — the model re-scores them, never generates new responses during the loss. It learns from someone else's attempts, frozen before training. For verifiable rewards (math, code), this forfeits on-policy exploration: the model never learns from the distribution of ITS OWN current errors, only from errors in the fixed set. GRPO generates, verifies, and learns from its own attempts."	c3::ft14::recall
"What does 'on-policy exploration' mean in the RL-for-reasoning context, and why does it matter?"	"The model generates its own training attempts (rollouts) from the current policy and learns from their verified outcomes, rather than from a fixed offline dataset. It matters because the verifier lets you cheaply generate an infinite curriculum of labeled attempts tailored to THIS model's current weak spots — including errors no fixed dataset contained. This is strictly more informative than offline learning for verifiable skill."	c3::ft14::recall
"State the GRPO mechanism and how it eliminates the PPO critic."	"GRPO (Group Relative Policy Optimization, DeepSeek, DeepSeekMath arXiv:2402.03300): per prompt, sample N responses from the current policy, score each with the verifier, compute the advantage relative to the GROUP MEAN: A_i = (r_i − mean(r)) / std(r). No learned value function (critic) is needed — the group IS the baseline. Memory drops from ~3 models (policy + critic + reference) to ~2 (policy + reference) → roughly halves memory vs PPO."	c3::ft14::recall
"What is the GRPO advantage formula, and what replaces the PPO learned critic?"	"A_i = (r_i − mean(r)) / std(r), where r_1..r_N are the rewards of the N sampled responses for a prompt. The GROUP MEAN (and std) replaces the PPO learned value function as the baseline. It is a Monte Carlo estimate of expected reward under the current policy — bias-free, lower-variance than single-sample REINFORCE, and requires no extra parameters, training, or memory. Cost: N forward passes per prompt instead of one."	c3::ft14::recall
"Explain the clarification that GRPO is essentially RLOO."	"GRPO is essentially RLOO (REINFORCE Leave-One-Out, Kool et al. 2019) with a slightly different advantage computation. BOTH sample N responses per prompt and use a group/leave-one-out baseline with NO learned critic. RLOO's baseline for response i = mean of the OTHER N−1 responses; GRPO's = mean of all N (including i). A small bias/variance tradeoff, NOT a different family. This is why TRL ships GRPOTrainer and RLOOTrainer side by side and they behave nearly identically."	c3::ft14::recall
"Contrast PPO, GRPO, and RLOO on baseline and critic."	"PPO: baseline = learned value function (critic, ~same size as policy, must be trained); ~3 models in memory. GRPO: baseline = mean of all N samples (including i); no critic; ~2 models. RLOO: baseline = mean of the other N−1 samples (leave-one-out); no critic; ~2 models. GRPO and RLOO are siblings (variance-reduced REINFORCE + group baseline); PPO is the distinct family that learns the baseline."	c3::ft14::recall
"Name the three GRPO++ variants and the specific failure mode each patches."	"DAPO (ByteDance): clip-higher (decouple ± clip range for exploration) + dynamic sampling (skip zero-variance groups — all-correct/all-wrong — that waste compute). Dr. GRPO: removes the length bias in per-token advantage averaging (prevents 'shorter = higher reward' shortcut). GSPO: sequence-level importance weighting (one weight per full sequence, not per token) → stable credit assignment on long CoT. All are PATCHES on GRPO/RLOO, not different algorithms."	c3::ft14::recall
"What does DAPO's clip-higher fix, and how?"	"Standard symmetric PPO clipping suppresses exploration once the policy is doing well — good responses can only be upweighted up to the clip, so positive advantages are capped. Clip-higher DECOUPLES the clip range for positive vs negative advantages, letting positive advantages push further. This keeps exploration alive late in training when the policy is already mostly correct."	c3::ft14::recall
"What does DAPO's dynamic sampling do, and why?"	"It filters out prompts where all N sampled responses get the SAME reward (all-correct or all-wrong). Those groups contribute ZERO gradient signal — the advantage A_i = (r_i − mean)/std is zero for every sample (std=0 or all-equal). They waste compute. Dynamic sampling skips them and spends the budget on prompts with reward variance, where the gradient signal is non-zero."	c3::ft14::recall
"What length bias does Dr. GRPO remove, and why does it matter?"	"Standard GRPO averages the advantage per-token, which introduces a bias toward SHORTER responses — shorter sequences have fewer tokens to dilute the per-token advantage, so the model can learn 'shorter = effectively higher reward' as a spurious shortcut. Dr. GRPO re-weights to remove this bias, preventing degeneration toward terse outputs when the task rewards correctness regardless of length."	c3::ft14::recall
"What is GSPO's change vs GRPO, and what failure does it address?"	"GSPO (Group Sequence Policy Optimization) computes importance weighting at the SEQUENCE level (one weight per full response) rather than TOKEN level. Token-level importance ratios in GRPO/PPO are noisy and can mis-credit long chain-of-thought sequences. Sequence-level weighting stabilizes training on long CoT outputs where reasoning spans hundreds of tokens."	c3::ft14::recall
"What is the DeepSeek-R1-Zero experiment and what did it prove?"	"R1-Zero: DeepSeek applied PURE RL (GRPO on verifiable rewards — math, code, + a formatting reward) directly to DeepSeek-V3-Base, with NO SFT cold start. It proved reasoning behavior can EMERGE from RL alone — self-verification, backtracking, long CoT appeared because those behaviors were SELECTED FOR by the correctness reward. This is the FT00 thesis in purest form: the base already had the capability; RL steers the model to USE it. Limitation: poor readability, mixed languages, hard-to-parse formats."	c3::ft14::recall
"State the 4 stages of the DeepSeek-R1 pipeline and why each exists."	"(1) Cold-start SFT — high-quality reasoning data gives a readable CoT scaffold (fixes R1-Zero's readability). (2) Reasoning RL — GRPO on verifiable rewards sharpens the capability. (3) Rejection-sampling SFT — generate candidates with the RL model, filter by reward (keep correct), SFT a fresh model on survivors (bakes quality in without RL instability; → FT15). (4) Final RL for alignment — broader reward mix (reasoning + safety/helpfulness), preserves reasoning, aligns for general use."	c3::ft14::recall
"What was the R1 distillation finding, and why is it practically influential?"	"DeepSeek distilled R1 into 6 smaller dense models (1.5B–70B, Qwen/Llama bases) via SFT ONLY (no RL) on ~800K curated reasoning samples from R1. R1-Distill-Qwen-32B BEATS OpenAI o1-mini. This made CoT distillation a standard technique: once you have one strong reasoning teacher (from expensive RL), you stamp out cheaper students via SFT alone. It works (FT00) because students already had the capability — the CoT data steers them to use it. Subject of FT15."	c3::ft14::recall
"State the Qwen3 4-stage post-training pipeline."	"(1) Long-CoT cold start — SFT on long chain-of-thought, sets a prior toward thorough reasoning. (2) Reasoning RL — GRPO-style on verifiable rewards (large-scale). (3) Thinking mode fusion — merge thinking (long CoT) + non-thinking (fast, direct) into ONE model, mode-flagged. (4) General RL — broad capability + alignment, preserves reasoning from 2–3. Pretraining: 36T+ tokens. Plus a runtime thinking-budget mechanism for adaptive compute."	c3::ft14::recall
"What is Qwen3's thinking-budget mechanism and what problem does it solve?"	"A runtime control: instruct the model (via a token budget) to spend a BOUNDED amount of 'thinking' compute before answering. Short budget → fast answer (easy questions); long budget → thorough reasoning (hard questions). Solves 'sometimes you want 30s of reasoning, sometimes a one-line answer' without needing two models. It is the runtime analog of training-time exploration: spend compute where it pays off — adaptive latency/accuracy trade."	c3::ft14::recall
"Name the three RL training tools and their use cases."	"TRL GRPOTrainer — moderate scale (research, small models 1B–7B, learning); integrates with transformers/peft/accelerate; supply a Python reward function; the default starting point. verl (HybridFlow, ByteDance) — production-grade distributed RL on Ray; separates generation (rollout) from training; the standard for 30B+ on a cluster. OpenRLHF — faster on some micro-benchmarks BUT a masked_mean bug was found (silently wrong on padded long CoT); consensus: 'TRL or verl and you're fine.'"	c3::ft14::recall
"What is the OpenRLHF caveat, and what is the community consensus?"	"A masked_mean bug was found in OpenRLHF's advantage/loss computation that silently miscalculated the per-token mean under certain padding conditions → subtly wrong gradients on long sequences with heavy padding (exactly the reasoning-CoT setting). It has been addressed, but the consensus crystallized as: 'TRL or verl and you're fine.' Use OpenRLHF only with a specific reason AND a check that the masked_mean issue does not affect your padding pattern."	c3::ft14::recall
"State the reward-verification loop (the 4 steps per training step)."	"(1) GENERATE N responses per prompt from the current policy. (2) VERIFY each with the reward function: parse answer, execute/check (run code, compare math answer, validate tool output) → scalar reward. (3) Compute ADVANTAGE group-relative: A_i = (r_i − mean(r)) / std(r) — no learned critic. (4) UPDATE policy toward high-advantage responses, away from low-advantage, with a KL anchor to the reference. The verifier is the load-bearing component."	c3::ft14::recall
"What is reward hacking, and how do you prevent it?"	"A weak verifier (substring match, leniency, a judge rewarding confidence over correctness) produces a reward signal the model will GAME — finding the shortcut, scoring high, outputting wrong answers. Prevent it by using a VERIFIER, not a JUDGE: execute the code and check output; compare the parsed math answer EXACTLY; validate tool output structurally. Reasoning RL quality is bounded by verifier quality — the analog of FT00's 'your data matters more than your algorithm.'"	c3::ft14::recall
"Why does the verifier quality bound the entire GRPO run?"	"GRPO optimizes the policy to maximize the reward signal. If the reward signal comes from a hackable verifier, the policy will find and exploit the shortcut — producing outputs that score high on the reward but are actually wrong. The optimizer is doing exactly its job; the problem is the reward does not track true correctness. So the ceiling on achievable true quality is the verifier's fidelity to correctness. A great GRPO config on a hackable reward steers you into a wall. This is why verifiers must be execution-based, not judge-based, for verifiable tasks."	c3::ft14::analysis
"A team uses DPO on a math-reasoning dataset of (correct, incorrect) pairs. Why will they underperform a GRPO approach, even with equally good data?"	"DPO is offline: the model re-scores fixed pairs, never generates. It learns to prefer correct over incorrect in the fixed set, but never explores its OWN current error distribution. GRPO generates N attempts per prompt from the current policy, verifies them, and learns from the distribution of its own successes/failures — including error patterns no fixed dataset contained. For verifiable tasks, the verifier generates an infinite curriculum tailored to the model's live weak spots. DPO forfeits this. The team should use GRPO with a verifier."	c3::ft14::analysis
"DeepSeek-R1-Zero showed reasoning 'emerges' from pure RL. Reconcile this with the FT00 thesis ('fine-tuning steers behavior; it does not teach knowledge')."	"No contradiction — it is the thesis in purest form. What emerged was NOT new knowledge: DeepSeek-V3-Base already had the reasoning capability from pretraining. What emerged was reliable USE of that capability — self-verification, backtracking, long CoT — because RL with a correctness reward SELECTS FOR behaviors that produce correct answers. Those behaviors get upweighted; the policy drifts toward them. RL steered the model to use reasoning pathways it already had. The behavioral phenotype looks like new capability, but it is steering of existing capability — exactly FT00."	c3::ft14::analysis
"Why does dropping the PPO critic roughly HALVE memory, and what is the tradeoff GRPO accepts to do so?"	"PPO needs policy + critic (~same size as policy, must be trained) + reference = ~3 models. GRPO needs policy + reference = ~2 models (the group mean replaces the critic, no params). So memory drops ~from 3 to 2 model-copies, roughly a third to a half reduction depending on whether the reference is held in full or approximated. The tradeoff: GRPO pays N forward passes per prompt (the group) instead of one, plus it forgoes the critic's learned state-value estimate (which can be lower-variance per-sample). For reasoning — cheap verifiable rewards, sample diversity is the point — the trade is overwhelmingly favorable."	c3::ft14::analysis
"A GRPO run's reward plateaus mid-training even though the verifier is strong. Which GRPO++ patch addresses each symptom: (a) the model stopped exploring; (b) outputs are suspiciously short; (c) long CoT credit seems noisy?"	"(a) Model stopped exploring → DAPO's clip-higher (decouple ± clip so positive advantages can push further once the policy is mostly correct). (b) Outputs suspiciously short → Dr. GRPO (remove the per-token-averaging length bias that rewards brevity). (c) Long CoT credit noisy → GSPO (sequence-level importance weighting, one weight per full sequence instead of per-token). 'Use GRPO' is underspecified; mid-training stalls often need a SPECIFIC patch, not more compute."	c3::ft14::application
"Write a verifiable math reward function for TRL GRPOTrainer (parse \\boxed{}, compare to ground truth)."	"def math_reward(prompts, completions, answer, **kwargs):\n    import re\n    rewards = []\n    for comp, gold in zip(completions, answer):\n        m = re.findall(r'\\\\boxed\\{([^}]*)\\}', comp)\n        pred = m[-1].strip() if m else None\n        rewards.append(1.0 if pred is not None and pred == gold.strip() else 0.0)\n    return rewards\n# Verifiable: parses the last \\boxed{}, compares EXACTLY to gold. Not a substring match, not a judge. The ceiling on the RL run."	c3::ft14::application
"You are choosing between TRL GRPOTrainer and verl for a project. State the decision criteria."	"TRL GRPOTrainer: single GPU or small cluster, models ~1B–7B, research/prototyping/learning, integrates with transformers/peft/accelerate, supply a Python reward function. Use this to validate a reward function and learn GRPO. verl (HybridFlow, ByteDance): production scale, 30B+ models on a cluster, built on Ray, separates generation (rollout) from training (gradient) — the architecture that makes frontier-scale RL tractable. Use this when generation cost dominates and you need thousands of parallel rollouts. Outgrow TRL → verl."	c3::ft14::application
"A team skips the cold-start SFT and goes straight to GRPO RL on a base model (the R1-Zero path). What two problems will they likely hit, and what is the fix?"	"(1) Poor READABILITY — emergent CoT is messy, mixed-language, hard to parse (the R1-Zero problem). RL optimizes correctness, not human-facing format. (2) Hard-to-parse formats — production reasoning models need readable, parseable chains. Fix: add the cold-start SFT stage (one SFT stage on high-quality reasoning data) to give a readable CoT scaffold BEFORE RL — this is R1 stage 1, and it exists for exactly this reason. R1-Zero is the proof-of-emergence experiment, not the production recipe."	c3::ft14::application
"Map the R1 pipeline stages to their FT00 interpretation: which stages are steering and which (if any) teach?"	"ALL four stages are STEERING, none teach. (1) Cold-start SFT: steers format/readability of CoT (base already reasons; we give it a clean scaffold). (2) Reasoning RL: steers the model to USE reasoning pathways it already has (FT00 purest form — RL selects for behaviors producing correct answers). (3) Rejection-sampling SFT: steers a fresh model toward the quality pattern the RL model produced (distillation of behavior, not knowledge). (4) Final RL: steers alignment. No new knowledge enters at any stage — the base had the capability; every stage redirects probability mass. Had any stage required knowledge injection, it would be continued pretraining, not these stages."	c3::ft14::analysis
"Why did R1-Distill-Qwen-32B (SFT-only, no RL) beat OpenAI o1-mini, and what does this imply for the economics of reasoning models?"	"Because the student (Qwen-32B base) already had the reasoning capability from pretraining; the ~800K curated CoT samples from R1 STEERED it to use that capability reliably — no new knowledge transferred, only the behavioral pattern of careful verified reasoning (FT00). Implication: the expensive RL pipeline (R1) needs to be run ONCE to produce a strong teacher; then cheaper students are stamped out via SFT alone. You pay the RL cost once, then distill repeatedly. This made CoT distillation a standard technique and is why FT15 is a whole module."	c3::ft14::analysis
"Contrast R1 and Qwen3 as canonical references — what did each establish?"	"R1 (arXiv:2501.12948 + Nature): established that (a) reasoning EMERGES from RL alone on a base (R1-Zero), (b) the 4-stage pipeline (cold-start SFT → reasoning RL → rejection-sampling SFT → final RL) fixes R1-Zero's readability, (c) distillation (SFT-only) propagates the capability to cheaper students that beat o1-mini. Qwen3 (arXiv:2505.09388): INDUSTRIALIZED the recipe — long-CoT cold start, thinking/non-thinking mode fusion (one model, two modes), thinking-budget runtime mechanism (adaptive compute), 36T+ pretraining tokens. R1 = the science; Qwen3 = the engineering. Both canonical; cite both."	c3::ft14::analysis
"Given a choice between a substring-match reward and an execution-based reward for a coding task, which do you choose for GRPO and why?"	"Execution-based, always. A substring-match reward is HACKABLE — the model can learn to emit the expected output string as a substring without the code being correct, gaming the reward. An execution-based reward runs the generated code and checks the actual output against test cases — it tracks true correctness. GRPO will optimize whatever signal the verifier provides; if the signal is hackable, the policy exploits the shortcut (reward hacking). The verifier's fidelity to correctness is the ceiling on the run. For verifiable tasks, use a verifier (execute), not a judge or a matcher."	c3::ft14::application
"Why is the 'group mean as baseline' in GRPO bias-free and lower-variance than single-sample REINFORCE, and what does it cost?"	"Bias-free: the empirical mean of N samples from the current policy is an unbiased Monte Carlo estimate of E[r] under that policy — using it as the baseline does not bias the gradient (a standard REINFORCE-with-baseline result). Lower-variance: subtracting the group mean centers the rewards, reducing the variance of the advantage estimates vs. raw rewards (single-sample REINFORCE uses reward directly, high variance). Cost: N forward passes per prompt instead of 1, plus you forgo a learned critic's per-state estimate (which can be even lower variance but requires training and memory). For reasoning (cheap rewards, diversity wanted), the trade is favorable."	c3::ft14::analysis