A fictional inventory exercise describes incoming and outgoing batches. The model must return a small JSON object containing the final whole-number stock. An independent verifier computes the answer from structured data without executing the model's text.
See four responses to the same problem, their rewards, relative advantages, and an update. Compare warm-start SFT, further SFT, and GRPO on new problems, including the cost of generation.
Download the complete project, revision r001
The answer must match the calculation
StockArithmetic-v1 contains 1,400 problems: 1,000 for training, 200 for validation, and 200 for testing. Each problem gives an initial stock, two deliveries, and two withdrawals; every intermediate stock remains non-negative. Tuples are deduplicated after deliveries and withdrawals are sorted separately, so a permutation of the same movements cannot cross partitions. Language patterns and arithmetic operations are shared: this small corpus does not represent real inventories.
Training and validation operands range from 0 to 50. The test set contains 100 problems from that range and 100 with at least one operand between 51 and 100, labelled OOD100. Results for those two populations are kept separate. The first 128 training targets are used for warm-starting; the other 872 are used for further SFT or RL prompt draws. Targets are calculated, not labelled by a person or produced by an LLM.
The model sees only the problem statement and a formatting instruction. The verifier receives the problem structure separately. It requires exactly one complete JSON object, limited to 256 characters, containing exactly final_stock with an integer from 0 to 300. Booleans, floats, extra fields, duplicates, Markdown, multiple objects, and non-finite numbers are rejected. A generated ID absent from the tokenizer also invalidates the response; it cannot silently disappear during decoding. No generated text is executed. A response stopped at the token limit receives zero, even if it starts with correct JSON.
The reward is one for a complete correct response and zero in every other case. There is no style bonus, partial reward, or constrained JSON decoding. The deterministic solver already knows how to calculate every target: expensive learning may be unnecessary for this task.
Three different policies
Qwen2.5-0.5B-Instruct is obtained at revision 7ae557604adf67be50417f59c2c2f167def9a775. Its 494032768 unique parameters are all optimized in FP32, with eager attention and TF32 disabled. The architecture and tied input/output embedding weights are checked. Logits predict the next token; only response tokens are supervised, including EOS 151645. The prompt and padding are masked by position. The prompt has at most 256 tokens, the response 64, and the total 320; no oversized supervised example is silently truncated.
Warm-start SFT learns from 128 examples for three epochs, or 48 updates of eight examples. Further SFT starts from that state and learns from 872 examples for two epochs, or 218 updates. Both use AdamW at 2e−5. Cross-entropy is normalized over all active tokens in an accumulation, rather than as an average of response averages.
GRPO starts from the same warm start. The fixed reference is a strictly reloaded copy of that state, frozen and without gradients. The old policy is the state that produced an iteration's responses; its log probabilities are captured during generation and remain detached. The current policy receives gradients. There is no learned value critic. These three roles are not interchangeable.
From the group to the advantage
An iteration selects four training prompts. For each one, it samples G=4 sequential responses at temperature 1, without top-k, top-p truncation, or token removal from the vocabulary. Behaviour probabilities come from the actual logits when sampling occurs. Tokens, scores, policy state, stop reasons, and times are retained. Eval mode disables dropout during generation AND updates; eval does not prohibit autograd.
Within each group, A=(reward−mean)/(standard deviation+1e−8), using the population standard deviation with correction=0. Rewards and advantages are detached. For [1,0,1,0], advantages are approximately [1,−1,1,−1]; epsilon explains why they are not exactly those integers. If all four rewards are identical, every advantage is exactly zero. We measure how often these groups occur without inventing a winner for them.
For every response token, rho=exp(logp_current−logp_old). Before the first update, scoring the entire sequence must recover the scores captured at prefixes within 2e−4; kernel shapes can create a small numerical difference. The old score is never replaced by detached new logits after learning.
The term to maximize is min(rho A, clip(rho,0.8,1.2) A). The sign of A is essential: tests check both sides and the gradients of clipped branches. The loss is the opposite of this term, averaged first over each response's tokens, then over responses, and finally over prompts. A long response does not automatically receive more total weight; this normalization still has length and difficulty biases that a small experiment does not remove.
Exact KL on visited prefixes
The variant is called “educational GRPO, exact KL on prefixes.” For every prefix actually visited, it computes sum_v p_current(v) × [log p_current(v)−log p_reference(v)] over the whole vocabulary. The vocabulary is processed in blocks of 4096, using full logsumexp normalizers. Normalizing every block separately would change the divergence.
This categorical sum is exact up to rounding for each prefix. Prefixes remain those from the rollouts: it does not enumerate every possible sequence. The coefficient is 0.02. Its reduction follows the same token/response/prompt average as the surrogate. Values and gradients on small tensors are compared with PyTorch's kl_div in FP64 and FP32; a few real full distributions serve as independent probes.
The paper uses a different estimate of the KL penalty. TRL offers several reductions, including DAPO; we do not claim either its defaults or its trainer's identity. The practical exercise explicitly implements its own variant and does not install TRL. Groups with zero advantage have no reward gradient, but KL regularization and optimizer rules can still act.
Budget, costs, and resume
The minimum GRPO run has 100 iterations, four prompts per iteration, four responses per prompt, and two updates on every set of rollouts: 1,600 actual responses and 200 updates. AdamW uses lr=1e−6, weight_decay=0.01, gradient clipping at 1, a 10% warm-up, then linear decay to 10% of the maximum rate. Warm-starting, SFT, and RL do not consume the same examples, tokens, forwards, or GPU-seconds; no equality of cost is assumed.
Generation without a KV cache favours understandable scoring. Times for generation, old-score verification, optimization with the reference, the complete loop, and commands are distinguished. State fingerprints also have a cost. The costs of the continuous/resumed comparison must remain visible in addition to those of reference continuous training.
GRPO resumes after a fully optimized iteration, meaning after both updates. The complete model, AdamW, scheduler, prompt permutation/cursor, generation RNG, system RNG, and list of consumed rollouts are saved. Interrupting at 40, continuing to 100, and resuming the remaining 60 must end in exactly identical states. A rollout cannot be consumed twice. Actual durations lie outside comparable numerical state, but their evidence is retained. SFT also compares interruption after one epoch, continuous execution, and resume.
The fifteen complete states of the five branches remain local, losslessly compressed with gzip level 1; five final models are also reloaded from Safetensors. Sizes are measured and no compression ratio is promised. Loading Torch objects is permitted here only because they were produced locally; do not replace them with a third-party file.
Evaluate a policy, not its best response
Warm-start, SFT, and GRPO policies each generate one greedy response on the same 200 test problems: that is greedy pass@1. They also produce four independent stochastic draws per problem with fixed seeds. The per-draw success rate uses 800 responses; it is not the success of the best of four. The ID50 and OOD100 populations, and their total, remain distinct.
JSON validity, correctness, absence of EOS, lengths, and times are calculated across every output. SFT validation measures NLL without selecting a checkpoint; final states are fixed before trials. Test responses do not enter updates. A lower loss does not prove improved correctness, and a result unfavourable to GRPO can be published.
Exercise: two group sizes
Constant groups are first studied using constructed rewards, clearly separated from model responses. Then G=2 and G=8 are actually trained for 20 iterations from the same warm start, with the same 80 prompt presentations and 40 updates. They produce 160 and 640 rollouts respectively: the same number of updates does not mean the same cost. Both policies are evaluated on the same 200 problems and four draws.
The main G=4 run has 100 iterations. Comparing it directly with the 20-iteration branches would not be an ablation isolating group size alone; this limitation is stated. The exercise protects SHA-256 hashes of the main weights, results, and configuration. The verifier remains unchanged, even if every group is wrong.
Write the problem and its independent verifier
A reward that does not depend on the model
Error to diagnose: an output contains two JSON objects, one of them correct; require one complete response.
Invariant: 1,000/200/200 distinct problems; exact solver on fixtures and edge cases.
File src/grpo_recompense_verifiable/verifier.py
import json, random
def compute_final_stock(problem):
if not isinstance(problem, dict) or set(problem) != {
"initial",
"deliveries",
"withdrawals",
}:
raise ValueError("Expected initial, deliveries and withdrawals only")
if type(problem["initial"]) is not int or not 0 <= problem["initial"] <= 100:
raise ValueError("Initial stock must be an integer in 0..100")
stock = problem["initial"]
for key, sign in [("deliveries", 1), ("withdrawals", -1)]:
values = problem[key]
if (
not isinstance(values, list)
or len(values) != 2
or any(type(v) is not int or not 0 <= v <= 100 for v in values)
):
raise ValueError("Exactly two bounded integer movements required")
for value in values:
stock += sign * value
if stock < 0:
raise ValueError("An intermediate stock cannot be negative")
return stock
def normalized_problem(problem):
compute_final_stock(problem)
return (
problem["initial"],
tuple(sorted(problem["deliveries"])),
tuple(sorted(problem["withdrawals"])),
)
def generate_stock_problem(rng, maximum=50):
if maximum not in [50, 100]:
raise ValueError("Named ID50 or OOD100 range required")
initial = rng.randint(0, maximum)
deliveries = [rng.randint(0, maximum) for _ in range(2)]
available = initial + sum(deliveries)
first = rng.randint(0, min(maximum, available))
second = rng.randint(0, min(maximum, available - first))
return {
"initial": initial,
"deliveries": deliveries,
"withdrawals": [first, second],
}
def unique_object(pairs):
result = {}
for key, value in pairs:
if key in result:
raise ValueError("Duplicate JSON field")
result[key] = value
return result
def reject_constant(value):
raise ValueError("Non-finite JSON number forbidden")
def parse_generated_json(text, finish_reason):
if finish_reason != "eos":
return {"valid": False, "value": None, "reason": "completion_without_eos"}
if not isinstance(text, str) or len(text) > 256:
return {"valid": False, "value": None, "reason": "oversized_or_nontext"}
try:
obj = json.loads(
text, object_pairs_hook=unique_object, parse_constant=reject_constant
)
if type(obj) is not dict or set(obj) != {"final_stock"}:
raise ValueError("One exact field required")
value = obj["final_stock"]
if type(value) is not int or not 0 <= value <= 300:
raise ValueError("A bounded integer excluding bool/float is required")
return {"valid": True, "value": value, "reason": None}
except (ValueError, TypeError, RecursionError):
return {"valid": False, "value": None, "reason": "invalid_json_schema"}
def score_generated_json(text, finish_reason, problem, tokenization_valid=True):
expected = compute_final_stock(problem)
parsed = parse_generated_json(text, finish_reason)
if type(tokenization_valid) is not bool:
raise ValueError("Explicit tokenization validity required")
if not tokenization_valid:
parsed = {"valid": False, "value": None, "reason": "undecodable_token"}
return {
**parsed,
"reward": int(parsed["valid"] and parsed["value"] == expected),
"expected": expected,
}
Form a group and inspect its scores
Rollouts and behaviour probabilities
Error to diagnose: active dropout makes log probabilities inconsistent; set the policy to evaluation mode while allowing autograd during updates.
Invariant: every group has a unique prompt_id and four responses with compatible scoring.
File src/grpo_recompense_verifiable/rollout.py
import time
import torch
from .scores import format_prompt, rollout_completion_row, model_scores
from .loss import group_advantages
from .verifier import score_generated_json
def capture_behavior_logprobs(logits, token):
# Probability captured at the actual sampling prefix, not reconstructed from
# the current policy after an optimizer step. Temperature is fixed to one.
return float(torch.log_softmax(logits.float(), dim=-1)[token].detach())
@torch.no_grad()
def sample_response(
model,
tokenizer,
page,
config,
generator,
mode="sample",
policy_id=None,
draw=0,
capture_probe=False,
):
if model.training:
raise ValueError("Sampling requires eval mode; dropout must be off")
if config["temperature"] != 1.0 or config["top_p"] != 1.0 or config["top_k"] != 0:
raise ValueError(
"The public protocol requires full support and temperature one"
)
if mode not in ["sample", "greedy"]:
raise ValueError("Named generation mode required")
messages, prompt = format_prompt(
tokenizer, page["statement"], config["template_version"]
)
if not prompt or len(prompt) > config["max_prompt_tokens"]:
raise ValueError("Prompt exceeds the declared budget; no truncation")
ids = torch.tensor([prompt], device="cuda")
response = []
behavior = []
first_distribution = None
torch.cuda.synchronize()
start = time.perf_counter()
for position in range(config["max_new_tokens"]):
# No KV cache: sampling and teacher-forced scoring both use eager FP32
# causal attention. Prefix/full-sequence kernels may differ numerically.
logits = (
model(
input_ids=ids,
attention_mask=torch.ones_like(ids),
use_cache=False,
return_dict=True,
)
.logits[0, -1]
.float()
)
probabilities = torch.softmax(logits, dim=-1)
if not torch.isfinite(probabilities).all():
raise ValueError("Non-finite sampling distribution")
token = (
int(torch.argmax(logits))
if mode == "greedy"
else int(torch.multinomial(probabilities, 1, generator=generator))
)
behavior.append(capture_behavior_logprobs(logits, token))
response.append(token)
if position == 0 and capture_probe:
first_distribution = {
"prefix_ids": prompt,
"logits": logits.cpu().tolist(),
"target": token,
"selected_logprob": behavior[-1],
}
ids = torch.cat([ids, torch.tensor([[token]], device="cuda")], dim=1)
if token == tokenizer.eos_token_id:
break
torch.cuda.synchronize()
seconds = time.perf_counter() - start
finish = "eos" if response[-1] == tokenizer.eos_token_id else "length"
# Keep non-EOS special tokens in decoded text: silently dropping generated
# controls could turn an invalid response into apparently valid JSON.
decoded_ids = response[:-1] if finish == "eos" else response
text = tokenizer.decode(
decoded_ids, skip_special_tokens=False, clean_up_tokenization_spaces=False
)
tokenization_valid = all(
tokenizer.convert_ids_to_tokens(token) is not None for token in response
)
return {
"document_id": page["document_id"],
"messages": messages,
"prompt_ids": prompt,
"response_ids": response,
"old_token_logprobs": behavior,
"response_tokens": len(response),
"finish_reason": finish,
"text": text,
"tokenization_valid": tokenization_valid,
"generation_seconds": seconds,
"mode": mode,
"draw": draw,
"policy_id": policy_id,
"temperature": 1.0,
"top_p": 1.0,
"top_k": 0,
"dropout_active": False,
"first_prefix_probe": first_distribution,
}
@torch.no_grad()
def sample_group(
model,
tokenizer,
page,
problem,
config,
generator,
policy_id,
group_id,
group_size=None,
capture_probe=False,
):
size = config["group_size"] if group_size is None else group_size
if type(size) is not int or size < 2:
raise ValueError("At least two actual stochastic responses required")
responses = []
for member in range(size):
row = sample_response(
model,
tokenizer,
page,
config,
generator,
policy_id=policy_id,
draw=member,
capture_probe=capture_probe and member < 2,
)
row["rollout_id"] = f"{group_id}/response-{member:02d}"
row["verifier"] = score_generated_json(
row["text"], row["finish_reason"], problem, row["tokenization_valid"]
)
responses.append(row)
rewards = torch.tensor(
[[r["verifier"]["reward"] for r in responses]], dtype=torch.float32
)
advantages, mean, std = group_advantages(rewards, config["advantage_epsilon"])
return {
"group_id": group_id,
"document_id": page["document_id"],
"behavior_policy_id": policy_id,
"responses": responses,
"rewards": rewards[0].tolist(),
"advantages": advantages[0].tolist(),
"reward_mean": float(mean[0, 0]),
"population_std": float(std[0, 0]),
"zero_variance": bool(std[0, 0] == 0),
"std_correction": 0,
"epsilon": config["advantage_epsilon"],
}
@torch.no_grad()
def verify_behavior_before_update(model, group, tolerance=2e-4):
if model.training:
raise ValueError("Ratio verification requires eval mode")
maximum = 0.0
for response in group["responses"]:
_, _, actual, _, _ = model_scores(model, [rollout_completion_row(response)])
recorded = torch.tensor(
response["old_token_logprobs"], device=actual.device, dtype=actual.dtype
)
error = float((actual - recorded).abs().max())
maximum = max(maximum, error)
if error > tolerance:
raise ValueError(
"Behavior/full-prefix scoring disagreement before update: " + str(error)
)
return {
"maximum_absolute_logprob_difference": maximum,
"tolerance": tolerance,
"old_logprobs_source": "captured at actual generation, unchanged",
"before_any_update": True,
}
Warm-start and compare direct SFT
A learning reference without RL
Error to diagnose: downloading a checkpoint from an earlier practical exercise replaces the minimum; redo the warm start in this project.
Invariant: no test example in the warm start; costs and initial hash captured.
File src/grpo_recompense_verifiable/warmstart.py
import math, time
from pathlib import Path
import torch
from .config import read_json, save_json, sha, seed_all, rng_state, restore_rng
from .model import (
load_base,
load_tokenizer,
reload_learned,
save_learned,
parameter_manifest,
manifest_digest,
)
from .scores import completion_row, model_scores
from .state import save_resume_state, load_resume_state, equal
def learning_rate(config, step, total, peak):
warm = max(1, int(total * config["warmup_ratio"]))
if step < warm:
return peak * (step + 1) / warm
return peak * (
1
- min(1.0, (step - warm) / max(1, total - warm - 1))
* (1 - config["minimum_lr_ratio"])
)
def assert_learning_scope():
for name in [
"data/generated/input",
"data/generated/labels/gold.json",
"outputs/inference",
]:
if Path(name).exists():
raise ValueError(
"Learning must not mount reserved inputs, gold or inference outputs: "
+ name
)
def encoded_split(config, tokenizer, split):
pages = read_json(f"data/generated/{split}/pages.json")
labels = {
r["document_id"]: r for r in read_json(f"data/generated/labels/{split}.json")
}
return [
completion_row(
tokenizer,
p["statement"],
labels[p["document_id"]]["response"],
config,
p["document_id"],
)
for p in pages
]
@torch.no_grad()
def validation_nll(model, rows):
model.eval()
result = []
for row in rows:
sums, counts, values, _, targets = model_scores(model, [row])
result.append(
{
"document_id": row["document_id"],
"sum_logprob": float(sums[0]),
"tokens": int(counts[0]),
"token_logprobs": values.cpu().tolist(),
"targets": targets.cpu().tolist(),
}
)
return {
"examples": len(rows),
"tokens": sum(r["tokens"] for r in result),
"nll": -sum(r["sum_logprob"] for r in result)
/ sum(r["tokens"] for r in result),
"rows": result,
"used_for_selection": False,
}
def freeze_initial_reference():
model, metadata = reload_learned("assets/checkpoints/warmstart-resume.safetensors")
for parameter in model.parameters():
parameter.requires_grad_(False)
model = model.cuda().eval()
if any(p.requires_grad or p.grad is not None for p in model.parameters()):
raise ValueError("Reference must be frozen without gradients")
return model, metadata
def train_warmstart(config, branch="warmstart", mode="continuous"):
if branch not in ["warmstart", "sft"] or mode not in [
"interrupt",
"continuous",
"resume",
]:
raise ValueError("Unknown SFT phase")
assert_learning_scope()
seed_all(config["seed"])
tokenizer = load_tokenizer()
split = "warmstart" if branch == "warmstart" else "train"
rows = encoded_split(config, tokenizer, split)
validation = encoded_split(config, tokenizer, "validation")
if len(rows) != (128 if branch == "warmstart" else 872) or len(validation) != 200:
raise ValueError("Frozen SFT population changed")
model = (
load_base()
if branch == "warmstart"
else reload_learned("assets/checkpoints/warmstart-resume.safetensors")[0]
)
model = model.cuda().eval()
parameters = list(model.parameters())
initial = parameter_manifest(model)
digest = manifest_digest(initial)
if sum(p.numel() for p in parameters) != 494032768 or any(
not p.requires_grad or p.dtype != torch.float32 for p in parameters
):
raise ValueError("Expected full FP32 policy")
epochs = (
config["warmstart_epochs"] if branch == "warmstart" else config["sft_epochs"]
)
peak = (
config["warmstart_learning_rate"]
if branch == "warmstart"
else config["sft_learning_rate"]
)
total = math.ceil(len(rows) / config["accumulation_examples"]) * epochs
identity = {
"branch": branch,
"config": config,
"initial_model_digest": digest,
"model_sha256": sha("assets/model/model.safetensors"),
"tokenizer_sha256": sha("assets/model/tokenizer.json"),
"data_sha256": {
name: sha("data/generated/" + name)
for name in [
f"{split}/pages.json",
f"labels/{split}.json",
"validation/pages.json",
"labels/validation.json",
]
},
"code_sha256": {
name: sha("src/grpo_recompense_verifiable/" + name)
for name in ["model.py", "scores.py", "warmstart.py", "state.py"]
},
}
optimizer = torch.optim.AdamW(
parameters, lr=peak, weight_decay=config["weight_decay"]
)
sampler = torch.Generator().manual_seed(config["seed"] + 100)
root = Path("assets/checkpoints")
destination = root / f"{branch}-{mode}.pt.gz"
if destination.exists():
raise ValueError("Preserve existing full checkpoint")
updates = start = 0
history = []
validation_history = []
if mode == "resume":
state = load_resume_state(root / f"{branch}-interrupt.pt.gz")
if state["identity"] != identity:
raise ValueError("SFT resume identity changed")
model.load_state_dict(state["model"], strict=True)
optimizer.load_state_dict(state["optimizer"])
sampler.set_state(state["sampler"])
restore_rng(state["rng"])
updates = state["updates"]
start = state["epoch"]
history = state["history"]
validation_history = state["validation_history"]
if state["scheduler"] != {
"next_update": updates,
"next_lr": learning_rate(config, updates, total, peak),
}:
raise ValueError("SFT scheduler changed")
del state
stop = 1 if mode == "interrupt" else epochs
torch.cuda.reset_peak_memory_stats()
torch.cuda.synchronize()
begun = time.perf_counter()
optimization_seconds = 0.0
for epoch in range(start, stop):
order = torch.randperm(len(rows), generator=sampler).tolist()
torch.cuda.synchronize()
start_epoch = time.perf_counter()
for offset in range(0, len(rows), config["accumulation_examples"]):
selected = order[offset : offset + config["accumulation_examples"]]
denominator = sum(rows[i]["response_tokens"] for i in selected)
lr = learning_rate(config, updates, total, peak)
for group in optimizer.param_groups:
group["lr"] = lr
optimizer.zero_grad(set_to_none=True)
details = []
for index in selected:
sums, counts, *_ = model_scores(model, [rows[index]])
loss = -sums.sum() / denominator
if not torch.isfinite(loss):
raise ValueError("Nonfinite SFT objective")
loss.backward()
details.append(
{
"document_id": rows[index]["document_id"],
"sum_logprob": float(sums.detach()[0]),
"response_tokens": int(counts[0]),
}
)
norm = torch.nn.utils.clip_grad_norm_(
parameters, config["gradient_clip"], error_if_nonfinite=True
)
if any(p.grad is None or p.grad.dtype != torch.float32 for p in parameters):
raise ValueError("All full-policy gradients required")
optimizer.step()
updates += 1
history.append(
{
"update": updates,
"epoch": epoch + 1,
"document_ids": [rows[i]["document_id"] for i in selected],
"response_token_denominator": denominator,
"nll": -sum(r["sum_logprob"] for r in details) / denominator,
"learning_rate": lr,
"gradient_norm_before_clip": float(norm),
"rows": details,
}
)
torch.cuda.synchronize()
optimization_seconds += time.perf_counter() - start_epoch
observed = validation_nll(model, validation)
validation_history.append({"epoch": epoch + 1, **observed})
print(
{
"branch": branch,
"epoch": epoch + 1,
"updates": updates,
"validation_nll": observed["nll"],
},
flush=True,
)
torch.cuda.synchronize()
elapsed = time.perf_counter() - begun
peak_memory = torch.cuda.max_memory_allocated()
final = parameter_manifest(model)
final_digest = manifest_digest(final)
if final_digest == digest:
raise ValueError("No actual SFT weight update")
state = {
"identity": identity,
"epoch": stop,
"updates": updates,
"model": model.state_dict(),
"optimizer": optimizer.state_dict(),
"sampler": sampler.get_state(),
"scheduler": {
"next_update": updates,
"next_lr": learning_rate(config, updates, total, peak),
},
"rng": rng_state(),
"history": history,
"validation_history": validation_history,
}
compression = save_resume_state(destination, state)
if mode == "resume":
save_learned(model, root / f"{branch}-resume.safetensors", identity)
save_json(
f"outputs/learning/train-{branch}-{mode}.json",
{
"branch": branch,
"mode": mode,
"identity": identity,
"epochs": stop,
"updates": updates,
"history": history,
"validation_history": validation_history,
"tokenization": rows,
"initial_parameters": initial,
"final_parameters": final,
"initial_model_digest": digest,
"final_model_digest": final_digest,
"trainable_parameters": 494032768,
"checkpoint_sha256": sha(destination),
"checkpoint_bytes": destination.stat().st_size,
"checkpoint_compression": compression,
"optimization_seconds_this_process": optimization_seconds,
"training_and_validation_seconds_this_process": elapsed,
"max_torch_allocated_bytes": peak_memory,
"test_visible": False,
"dropout_active": False,
},
)
def verify_resume(branch):
root = Path("assets/checkpoints")
a = load_resume_state(root / f"{branch}-continuous.pt.gz")
b = load_resume_state(root / f"{branch}-resume.pt.gz")
checks = {k: equal(a[k], b[k]) for k in a}
if a.keys() != b.keys() or not all(checks.values()):
raise ValueError("Full resume differs: " + str(checks))
save_json(
f"outputs/learning/resume-{branch}.json",
{
"status": "PASS",
"all_state_exact": True,
"checks": checks,
"boundary": "Full epoch for SFT; completed rollout iteration with both optimizer updates for GRPO. Timings live outside resumable numerical state.",
},
)
Code advantages, clipping, and KL
Online group update
Error to diagnose: using detached current log probabilities as old at every microbatch erases history; capture them during generation.
Invariant: clipping-sign, constant-group, and gradient tests pass; the policy is actually updated.
File src/grpo_recompense_verifiable/train.py
import time
from pathlib import Path
import torch
from .config import read_json, save_json, sha, seed_all, rng_state, restore_rng
from .model import (
load_tokenizer,
reload_learned,
save_learned,
parameter_manifest,
manifest_digest,
)
from .scores import rollout_completion_row, model_scores
from .rollout import sample_group, verify_behavior_before_update
from .loss import group_advantages, clipped_group_surrogate, exact_prefix_kl
from .warmstart import assert_learning_scope, freeze_initial_reference, learning_rate
from .state import save_resume_state, load_resume_state
def next_prompt_indices(sampler, population, order, cursor, count):
selected = []
while len(selected) < count:
if cursor == len(order):
order = torch.randperm(population, generator=sampler).tolist()
cursor = 0
selected.append(order[cursor])
cursor += 1
return selected, order, cursor
def train_grpo(config, branch="grpo", mode="continuous"):
if branch not in ["grpo", "grpo-g2", "grpo-g8"] or mode not in [
"interrupt",
"continuous",
"resume",
]:
raise ValueError("Unknown GRPO branch or process")
group_size = (
config["group_size"] if branch == "grpo" else int(branch.rsplit("g", 1)[1])
)
iterations = (
config["iterations"] if branch == "grpo" else config["exercise_iterations"]
)
boundary = config["interrupt_iteration"] if branch == "grpo" else 8
if config["std_correction"] != 0 or config["updates_per_rollout"] != 2:
raise ValueError(
"Frozen variant requires population std and two updates per rollout"
)
assert_learning_scope()
seed_all(config["seed"])
tokenizer = load_tokenizer()
pages = read_json("data/generated/train/pages.json")
gold = {r["document_id"]: r for r in read_json("data/generated/labels/train.json")}
if (
len(pages) != 872
or any(p["split"] != "train" for p in pages)
or any(r["warmstart"] for r in gold.values())
):
raise ValueError("RL must use only the 872 remaining train problems")
model, metadata = reload_learned("assets/checkpoints/warmstart-resume.safetensors")
model = model.cuda().eval()
reference, ref_metadata = freeze_initial_reference()
parameters = list(model.parameters())
initial = parameter_manifest(model)
initial_digest = manifest_digest(initial)
if (
initial_digest != ref_metadata["model_digest"]
or sum(p.numel() for p in parameters) != 494032768
or any(not p.requires_grad or p.dtype != torch.float32 for p in parameters)
):
raise ValueError(
"Policy and fixed reference must start from the same full FP32 local SFT"
)
identity = {
"branch": branch,
"config": config,
"group_size": group_size,
"iterations": iterations,
"initial_model_digest": initial_digest,
"reference_sha256": sha("assets/checkpoints/warmstart-resume.safetensors"),
"reference_model_digest": ref_metadata["model_digest"],
"tokenizer_sha256": sha("assets/model/tokenizer.json"),
"variant": "GRPO pédagogique, KL exacte sur préfixes",
"data_sha256": {
n: sha("data/generated/" + n)
for n in ["train/pages.json", "labels/train.json"]
},
"code_sha256": {
n: sha("src/grpo_recompense_verifiable/" + n)
for n in [
"model.py",
"scores.py",
"loss.py",
"rollout.py",
"verifier.py",
"train.py",
"state.py",
]
},
}
optimizer = torch.optim.AdamW(
parameters, lr=config["learning_rate"], weight_decay=config["weight_decay"]
)
sampler = torch.Generator().manual_seed(config["seed"] + 100)
generation_rng = torch.Generator(device="cuda").manual_seed(config["seed"] + 200)
root = Path("assets/checkpoints")
destination = root / f"{branch}-{mode}.pt.gz"
if destination.exists():
raise ValueError("Preserve existing local GRPO state")
start = updates = cursor = 0
order = []
history = []
consumed = []
total_updates = iterations * config["updates_per_rollout"]
if mode == "resume":
state = load_resume_state(root / f"{branch}-interrupt.pt.gz")
if state["identity"] != identity:
raise ValueError("GRPO resume identity differs")
model.load_state_dict(state["model"], strict=True)
optimizer.load_state_dict(state["optimizer"])
sampler.set_state(state["sampler"])
generation_rng.set_state(state["generation_rng"])
restore_rng(state["rng"])
start = state["iteration"]
updates = state["updates"]
cursor = state["cursor"]
order = state["order"]
history = state["history"]
consumed = state["consumed_rollout_ids"]
if (
updates != start * config["updates_per_rollout"]
or state["pending_rollouts"] != []
):
raise ValueError("Resume only at a completed rollout boundary")
if state["scheduler"] != {
"next_update": updates,
"next_lr": learning_rate(
config, updates, total_updates, config["learning_rate"]
),
}:
raise ValueError("GRPO scheduler identity differs")
del state
stop = boundary if mode == "interrupt" else iterations
torch.cuda.reset_peak_memory_stats()
torch.cuda.synchronize()
begun = time.perf_counter()
generation_seconds = scoring_seconds = optimization_seconds = 0.0
for iteration in range(start, stop):
selected, order, cursor = next_prompt_indices(
sampler, len(pages), order, cursor, config["prompts_per_iteration"]
)
behavior_digest = manifest_digest(parameter_manifest(model))
groups = []
for prompt_position, index in enumerate(selected):
group_id = (
f"{branch}/iteration-{iteration:03d}/prompt-{prompt_position:02d}"
)
group = sample_group(
model,
tokenizer,
pages[index],
gold[pages[index]["document_id"]]["problem"],
config,
generation_rng,
behavior_digest,
group_id,
group_size=group_size,
capture_probe=iteration == 0 and prompt_position == 0,
)
generation_seconds += sum(
r["generation_seconds"] for r in group["responses"]
)
torch.cuda.synchronize()
score_start = time.perf_counter()
group["before_update"] = verify_behavior_before_update(model, group)
torch.cuda.synchronize()
scoring_seconds += time.perf_counter() - score_start
groups.append(group)
rollout_path = Path(
f"outputs/learning/rollouts/{branch}/{mode}/iteration-{iteration:03d}.json"
)
if rollout_path.exists():
raise ValueError("Existing actual rollout evidence must be preserved")
save_json(
rollout_path,
{
"branch": branch,
"mode": mode,
"iteration": iteration,
"behavior_model_digest": behavior_digest,
"groups": groups,
"reference_model_digest": identity["reference_model_digest"],
},
)
current_ids = [r["rollout_id"] for g in groups for r in g["responses"]]
if len(set(current_ids)) != len(current_ids) or set(current_ids) & set(
consumed
):
raise ValueError("Duplicate rollout would be optimized twice")
update_rows = []
torch.cuda.synchronize()
update_start = time.perf_counter()
for inner_update in range(config["updates_per_rollout"]):
lr = learning_rate(config, updates, total_updates, config["learning_rate"])
for parameter_group in optimizer.param_groups:
parameter_group["lr"] = lr
optimizer.zero_grad(set_to_none=True)
details = []
for group in groups:
for member, response in enumerate(group["responses"]):
row = rollout_completion_row(response)
with torch.no_grad():
_, _, reference_values, reference_logits, _ = model_scores(
reference, [row]
)
_, _, current_values, current_logits, targets = model_scores(
model, [row]
)
old = torch.tensor(
response["old_token_logprobs"],
device="cuda",
dtype=current_values.dtype,
).unsqueeze(0)
advantage = torch.tensor(
[group["advantages"][member]],
device="cuda",
dtype=current_values.dtype,
)
mask = torch.ones_like(old, dtype=torch.bool)
surrogate, audit = clipped_group_surrogate(
current_values.unsqueeze(0),
old,
advantage,
mask,
config["clip_epsilon"],
)
kl = exact_prefix_kl(
current_logits, reference_logits, config["kl_vocabulary_block"]
)
contribution = (
surrogate.mean() + config["kl_beta"] * kl.mean()
) / (len(groups) * group_size)
if not torch.isfinite(contribution) or not torch.isfinite(kl).all():
raise ValueError("Nonfinite actual GRPO objective")
detail = {
"rollout_id": response["rollout_id"],
"tokens": len(response["response_ids"]),
"old_token_logprobs": response["old_token_logprobs"],
"current_token_logprobs": current_values.detach()
.cpu()
.tolist(),
"reference_token_logprobs": reference_values.cpu().tolist(),
"advantages": float(advantage[0]),
"token_ratios": audit["ratio"][0].cpu().tolist(),
"surrogate": float(surrogate.detach()[0]),
"exact_prefix_kl": kl.detach().cpu().tolist(),
"active_clip_fraction": float(audit["active_clip_fraction"][0]),
"outside_interval_fraction": float(
audit["outside_interval_fraction"][0]
),
"scaled_contribution": float(contribution.detach()),
}
if iteration == 0 and inner_update == 0 and len(details) < 2:
detail["kl_probe"] = {
"current_logits": current_logits[0].detach().cpu().tolist(),
"reference_logits": reference_logits[0].cpu().tolist(),
"target": int(targets[0]),
"exact_kl": float(kl.detach()[0]),
}
contribution.backward()
details.append(detail)
del (
current_logits,
reference_logits,
current_values,
reference_values,
kl,
contribution,
surrogate,
)
norm = torch.nn.utils.clip_grad_norm_(
parameters, config["gradient_clip"], error_if_nonfinite=True
)
if any(p.grad is None or p.grad.dtype != torch.float32 for p in parameters):
raise ValueError("Full policy gradients missing")
if (
any(
p.requires_grad or p.grad is not None
for p in reference.parameters()
)
or reference.training
or model.training
):
raise ValueError("Reference/dropout invariant broken")
optimizer.step()
updates += 1
update_rows.append(
{
"update": updates,
"inner_update": inner_update,
"learning_rate": lr,
"gradient_norm_before_clip": float(norm),
"prompt_denominator": len(groups),
"group_denominator": group_size,
"surrogate": sum(r["surrogate"] for r in details) / len(details),
"mean_exact_kl": sum(
sum(r["exact_prefix_kl"]) / r["tokens"] for r in details
)
/ len(details),
"active_clip_fraction": sum(
r["active_clip_fraction"] for r in details
)
/ len(details),
"total_loss": sum(r["scaled_contribution"] for r in details),
"responses": details,
}
)
torch.cuda.synchronize()
optimization_seconds += time.perf_counter() - update_start
consumed.extend(current_ids)
# Keep timing out of resumable state: wall-clock durations cannot be equal
# across processes. Actual raw rollouts and timings are retained above.
summary = {
"iteration": iteration,
"behavior_model_digest": behavior_digest,
"document_ids": [pages[i]["document_id"] for i in selected],
"group_size": group_size,
"groups": [
{
"group_id": g["group_id"],
"rewards": g["rewards"],
"advantages": g["advantages"],
"population_std": g["population_std"],
"zero_variance": g["zero_variance"],
"before_update": g["before_update"],
"response_ids": [r["response_ids"] for r in g["responses"]],
}
for g in groups
],
"updates": update_rows,
}
history.append(summary)
print(
{
"branch": branch,
"iteration": iteration + 1,
"updates": updates,
"reward_mean": sum(g["reward_mean"] for g in groups) / len(groups),
"zero_variance_fraction": sum(g["zero_variance"] for g in groups)
/ len(groups),
"kl": update_rows[-1]["mean_exact_kl"],
},
flush=True,
)
torch.cuda.synchronize()
elapsed = time.perf_counter() - begun
peak_memory = torch.cuda.max_memory_allocated()
final = parameter_manifest(model)
final_digest = manifest_digest(final)
ref_final = manifest_digest(parameter_manifest(reference))
if (
ref_final != identity["reference_model_digest"]
or sha("assets/checkpoints/warmstart-resume.safetensors")
!= identity["reference_sha256"]
):
raise ValueError("Frozen reference changed")
# Zero reward variation can leave a policy unchanged; this is a valid measured
# limitation. Never require a fabricated improvement or weight change here.
state = {
"identity": identity,
"iteration": stop,
"updates": updates,
"model": model.state_dict(),
"optimizer": optimizer.state_dict(),
"sampler": sampler.get_state(),
"generation_rng": generation_rng.get_state(),
"order": order,
"cursor": cursor,
"scheduler": {
"next_update": updates,
"next_lr": learning_rate(
config, updates, total_updates, config["learning_rate"]
),
},
"rng": rng_state(),
"history": history,
"consumed_rollout_ids": consumed,
"pending_rollouts": [],
}
compression = save_resume_state(destination, state)
if mode == "resume":
save_learned(model, root / f"{branch}-resume.safetensors", identity)
save_json(
f"outputs/learning/train-{branch}-{mode}.json",
{
"branch": branch,
"mode": mode,
"identity": identity,
"iterations": stop,
"updates": updates,
"history": history,
"consumed_rollout_ids": consumed,
"initial_parameters": initial,
"final_parameters": final,
"initial_model_digest": initial_digest,
"final_model_digest": final_digest,
"reference_model_digest_after": ref_final,
"weights_changed": final_digest != initial_digest,
"trainable_parameters": 494032768,
"reference_trainable_parameters": 0,
"reference_gradients": 0,
"checkpoint_sha256": sha(destination),
"checkpoint_bytes": destination.stat().st_size,
"checkpoint_compression": compression,
"generation_seconds_this_process": generation_seconds,
"behavior_scoring_seconds_this_process": scoring_seconds,
"optimization_and_reference_scoring_seconds_this_process": optimization_seconds,
"total_loop_seconds_this_process": elapsed,
"max_torch_allocated_bytes": peak_memory,
"test_visible": False,
"dropout_active": False,
},
)
Evaluate quality and total cost
Pass@1 and no signal
Error to diagnose: publish the best draw as pass@1; name the exact sampling budget.
Invariant: 200 IDs per branch, invalid outputs retained, and generation cost not omitted.
File src/grpo_recompense_verifiable/evaluate.py
import time
from pathlib import Path
import torch
from .config import read_json, save_json, sha, seed_all
from .model import load_tokenizer, reload_learned, manifest_digest, parameter_manifest
from .rollout import sample_response
from .verifier import score_generated_json
MAIN = ["warmstart", "sft", "grpo"]
EXERCISE = ["grpo-g2", "grpo-g8"]
@torch.no_grad()
def generate_policy(config, branch):
if branch not in MAIN + EXERCISE:
raise ValueError("Unknown evaluation policy")
if Path("data/generated/labels").exists() or Path("outputs/learning").exists():
raise ValueError("Inference must not mount labels or learning outputs")
seed_all(config["seed"])
tokenizer = load_tokenizer()
model, metadata = reload_learned(f"assets/checkpoints/{branch}-resume.safetensors")
model = model.cuda().eval()
pages = [
p for p in read_json("data/generated/input/pages.json") if p["split"] == "test"
]
if len(pages) != 200:
raise ValueError("Frozen test population changed")
root = Path("outputs/inference") / branch
if root.exists():
raise ValueError("Preserve existing actual predictions")
torch.cuda.reset_peak_memory_stats()
torch.cuda.synchronize()
begun = time.perf_counter()
files = {}
for mode in ["greedy", "sample"]:
results = []
for index, page in enumerate(pages):
for draw in range(1 if mode == "greedy" else config["evaluation_draws"]):
# Fixed independent draws, common seeds across policies; no
# selection of the best response and no label-dependent retry.
seed = 480000 + index * config["evaluation_draws"] + draw
generator = torch.Generator(device="cuda").manual_seed(seed)
row = sample_response(
model,
tokenizer,
page,
config,
generator,
mode=mode,
policy_id=metadata["model_digest"],
draw=draw,
capture_probe=index < 2 and draw == 0,
)
row["sampling_seed"] = seed if mode == "sample" else None
row["population"] = page["population"]
results.append(row)
if (index + 1) % 50 == 0:
print(
{"policy": branch, "mode": mode, "problems": index + 1}, flush=True
)
path = root / f"{mode}.json"
save_json(
path,
{
"branch": branch,
"mode": mode,
"problems": 200,
"draws_per_problem": (
1 if mode == "greedy" else config["evaluation_draws"]
),
"model_digest": metadata["model_digest"],
"rows": results,
"labels_visible": False,
},
)
files[mode] = {"sha256": sha(path), "responses": len(results)}
torch.cuda.synchronize()
elapsed = time.perf_counter() - begun
if manifest_digest(parameter_manifest(model)) != metadata["model_digest"]:
raise ValueError("Evaluation changed the policy")
save_json(
root / "audit.json",
{
"branch": branch,
"model_digest": metadata["model_digest"],
"seconds": elapsed,
"max_torch_allocated_bytes": torch.cuda.max_memory_allocated(),
"files": files,
"labels_visible": False,
"generation_support": "full vocabulary; temperature1; no top-k/top-p truncation; no JSON constraint",
},
)
def metrics(rows, labels):
result = {}
for population in ["ALL", "ID50", "OOD100"]:
selected = [
r for r in rows if population == "ALL" or r["population"] == population
]
scored = []
for row in selected:
observed = score_generated_json(
row["text"],
row["finish_reason"],
labels[row["document_id"]]["problem"],
row["tokenization_valid"],
)
scored.append(
{
"document_id": row["document_id"],
"draw": row["draw"],
"response_tokens": row["response_tokens"],
"finish_reason": row["finish_reason"],
**observed,
}
)
n = len(scored)
if n == 0:
raise ValueError("A declared evaluation population is empty")
result[population] = {
"responses": n,
"problems": len({r["document_id"] for r in scored}),
"correct": sum(r["reward"] for r in scored),
"success_rate": sum(r["reward"] for r in scored) / n,
"valid_json": sum(r["valid"] for r in scored),
"json_validity": sum(r["valid"] for r in scored) / n,
"without_eos": sum(r["finish_reason"] != "eos" for r in scored),
"mean_response_tokens": sum(r["response_tokens"] for r in scored) / n,
"rows": scored,
}
return result
def evaluate(config, exercise=False):
branches = EXERCISE if exercise else MAIN
labels = {r["document_id"]: r for r in read_json("data/generated/labels/gold.json")}
results = {}
for branch in branches:
root = Path("outputs/inference") / branch
greedy = read_json(root / "greedy.json")
sample = read_json(root / "sample.json")
if (
len(greedy["rows"]) != 200
or len(sample["rows"]) != 200 * config["evaluation_draws"]
):
raise ValueError("Incomplete actual evaluation")
for source, draws in [(greedy, 1), (sample, config["evaluation_draws"])]:
keys = [(r["document_id"], r["draw"]) for r in source["rows"]]
if len(set(keys)) != len(keys) or any(
sum(r["document_id"] == identifier for r in source["rows"]) != draws
for identifier, _ in keys
):
raise ValueError("Missing/duplicated test draw")
results[branch] = {
"greedy_pass_at_1": metrics(greedy["rows"], labels),
"empirical_per_draw_success": metrics(sample["rows"], labels),
"draws_per_problem": config["evaluation_draws"],
"best_of_draws_reported_as_pass_at_1": False,
"generation": read_json(root / "audit.json"),
}
save_json(
"outputs/exercise/results.json" if exercise else "outputs/results.json",
{
"policies": results,
"deterministic_solver": "Exact on all generated problems; a production stock calculation does not need a language model",
"shared_templates": "Synthetic controlled arithmetic, not real operational evidence",
"quality_gain_required": False,
},
)
return results
Interpret groups without a winner
Zero signal without manufacturing a reward
Error to diagnose: adding an arbitrary bonus to force a downward curve is prohibited; record the null result.
Invariant: zero advantages for constant groups; no modification to the verifier.
File src/grpo_recompense_verifiable/exercise.py
from pathlib import Path
import torch
from .config import read_json, save_json, sha
from .loss import group_advantages, clipped_group_surrogate
def main_artifacts():
names = ["configs/minimum.yaml", "outputs/results.json"]
for branch in ["warmstart", "sft", "grpo"]:
names += [
f"assets/checkpoints/{branch}-resume.safetensors",
f"outputs/learning/resume-{branch}.json",
f"outputs/inference/{branch}/greedy.json",
f"outputs/inference/{branch}/sample.json",
]
return names
def preserve_main():
path = Path("outputs/exercise/protected.json")
if path.exists():
raise ValueError("Preserve existing exercise snapshot")
save_json(path, {name: sha(name) for name in main_artifacts()})
def run_group_size_ablation(config):
protected = read_json("outputs/exercise/protected.json")
if protected != {name: sha(name) for name in main_artifacts()}:
raise ValueError("Exercise changed the main artifacts")
analytic = []
for rewards in [[0.0, 0.0, 0.0, 0.0], [1.0, 1.0, 1.0, 1.0], [1.0, 0.0, 1.0, 0.0]]:
advantages, mean, std = group_advantages(
torch.tensor([rewards], dtype=torch.float64), config["advantage_epsilon"]
)
current = torch.full((4, 2), -1.0, dtype=torch.float64, requires_grad=True)
old = current.detach().clone()
loss, _ = clipped_group_surrogate(
current, old, advantages[0], torch.ones_like(current, dtype=torch.bool)
)
loss.mean().backward()
analytic.append(
{
"rewards": rewards,
"advantages": advantages[0].tolist(),
"population_std": float(std[0, 0]),
"reward_surrogate": float(loss.detach().mean()),
"token_gradients": current.grad.tolist(),
"source": "Constructed illustrative rewards, not actual model outputs",
}
)
runs = {
b: read_json(f"outputs/learning/train-{b}-continuous.json")
for b in ["grpo-g2", "grpo-g8"]
}
if any(r["updates"] != 40 or r["iterations"] != 20 for r in runs.values()):
raise ValueError("Frozen group-size ablation budget changed")
if len({r["initial_model_digest"] for r in runs.values()}) != 1:
raise ValueError("Ablation initialization differs")
if [[h["document_ids"] for h in r["history"]] for r in runs.values()][0] != [
[h["document_ids"] for h in r["history"]] for r in runs.values()
][1]:
raise ValueError("Ablation prompt sequence differs")
budgets = {
b: {
"updates": r["updates"],
"prompts": sum(len(h["document_ids"]) for h in r["history"]),
"rollouts": len(r["consumed_rollout_ids"]),
"seconds": r["total_loop_seconds_this_process"],
}
for b, r in runs.items()
}
result = read_json("outputs/exercise/results.json")
if set(result["policies"]) != {"grpo-g2", "grpo-g8"}:
raise ValueError("Missing actual heldout ablation generation")
save_json(
"outputs/exercise/group-size.json",
{
"analytic_constant_groups": analytic,
"actual_fresh_training": True,
"budgets": budgets,
"same_initialization_and_prompt_order": True,
"equal_compute_claimed": False,
"main_100_iteration_g4_is_not_a_controlled_group_size_only_comparison": True,
"main_artifacts_unchanged": protected
== {name: sha(name) for name in main_artifacts()},
"protected_sha256": protected,
"verifier_changed": False,
"quality_gain_required": False,
},
)
Sources
DeepSeekMath v3 describes the group-relative objective. TRL v0.23.1 documents different reductions and variants. These sources clarify the choices; no published result or complete trainer is reproduced. Frozen Qwen base.
Development observations
Values recalculated from this run's raw outputs. They are not performance thresholds; the path from the ZIP recalculates them on the same populations.
{
"status": "PASS",
"metrics": {
"warmstart": {
"greedy_pass_at_1": {
"responses": 200,
"problems": 200,
"correct": 3,
"success_rate": 0.015,
"valid_json": 200,
"json_validity": 1.0,
"without_eos": 0,
"mean_response_tokens": 7.85
},
"empirical_per_draw_success": {
"responses": 800,
"problems": 200,
"correct": 15,
"success_rate": 0.01875,
"valid_json": 799,
"json_validity": 0.99875,
"without_eos": 0,
"mean_response_tokens": 7.78125
}
},
"sft": {
"greedy_pass_at_1": {
"responses": 200,
"problems": 200,
"correct": 16,
"success_rate": 0.08,
"valid_json": 200,
"json_validity": 1.0,
"without_eos": 0,
"mean_response_tokens": 7.865
},
"empirical_per_draw_success": {
"responses": 800,
"problems": 200,
"correct": 30,
"success_rate": 0.0375,
"valid_json": 799,
"json_validity": 0.99875,
"without_eos": 0,
"mean_response_tokens": 7.81125
}
},
"grpo": {
"greedy_pass_at_1": {
"responses": 200,
"problems": 200,
"correct": 3,
"success_rate": 0.015,
"valid_json": 200,
"json_validity": 1.0,
"without_eos": 0,
"mean_response_tokens": 7.825
},
"empirical_per_draw_success": {
"responses": 800,
"problems": 200,
"correct": 14,
"success_rate": 0.0175,
"valid_json": 799,
"json_validity": 0.99875,
"without_eos": 0,
"mean_response_tokens": 7.74375
}
},
"grpo-g2": {
"greedy_pass_at_1": {
"responses": 200,
"problems": 200,
"correct": 4,
"success_rate": 0.02,
"valid_json": 200,
"json_validity": 1.0,
"without_eos": 0,
"mean_response_tokens": 7.9
},
"empirical_per_draw_success": {
"responses": 800,
"problems": 200,
"correct": 11,
"success_rate": 0.01375,
"valid_json": 800,
"json_validity": 1.0,
"without_eos": 0,
"mean_response_tokens": 7.805
}
},
"grpo-g8": {
"greedy_pass_at_1": {
"responses": 200,
"problems": 200,
"correct": 3,
"success_rate": 0.015,
"valid_json": 200,
"json_validity": 1.0,
"without_eos": 0,
"mean_response_tokens": 7.96
},
"empirical_per_draw_success": {
"responses": 800,
"problems": 200,
"correct": 13,
"success_rate": 0.01625,
"valid_json": 799,
"json_validity": 0.99875,
"without_eos": 0,
"mean_response_tokens": 7.79875
}
}
},
"training_branches": 5,
"full_state_resume_checks": 5,
"actual_test_generations": 5000,
"scope": "Scalar reward/ratio/categorical-probe/metrics plus captured full-state comparisons; no scalar full Transformer or CUDA multinomial replay claimed",
"quality_gain_required": false
}
