A fictional tool turns incident facts into a brief report that separates observation from hypothesis. Pairs illustrate a faithful summary and a summary that adds an unsubstantiated cause. The task is neither medical nor equipment control.
Compare chosen-only SFT and ORPO from the same instruct weights. Display chosen NLL, log-odds, and summary quality, including an example where a numerically naive formula fails.
Download the complete project, revision r003
First the fact, then the report
IncidentNotesPrefs-v1 contains 300 families of four pairs, or 1,200 examples. Families 0–209 provide 840 training pairs; 210–254 provide 180 validation pairs; 255–299 provide 180 test pairs. Each case names a fictional unit, its stopped/running state, a known or unmeasured pressure, and a confirmed or unknown cause. An old note gives a contradictory state and a suggested cause; the current reading explicitly replaces it. No action on real equipment is proposed.
The chosen response states the current facts. The rejected response reverses one item only: cause in two formulations, pressure in one, and state in one. Some causes are confirmed: always writing "unknown" is then wrong. A polite response can nevertheless invent a cause. Courtesy phrases alternate to produce a shorter chosen response, one of equal word length, or a longer one. The words and these regimes are audited; final strata use the actual tokens.
All variants of a family remain in one partition. Identifiers are distinct, but elementary facts and language forms are shared: this split does not prove generalization to real incidents or domains. Annotations are simulated, with no human feedback claimed. A word-level audit does not demonstrate the absence of every semantic shortcut.
A distribution over tokens
The Qwen2.5-0.5B-Instruct base is acquired at revision 7ae557604adf67be50417f59c2c2f167def9a775. Its 494,032,768 unique parameters are all trainable in FP32. The architecture is checked: 24 blocks, width 896, 14 Q heads and two K/V heads, MLP 4864, vocabulary 151936, and tied input/output embeddings. Eager attention and TF32 are disabled; one card is explicitly selected. There is no LoRA, quantization, or reference copy.
The tokenizer template assembles the system, incident, and assistant boundary. The response is encoded separately, with one final EOS=151645. This contract does not claim equivalence to retokenizing the entire string. Limits are 256 prompt tokens, 128 response tokens including EOS, and at most 512 total. Any overflow raises an identified error, never a truncation of training data.
Logits at position t−1 predict token t. After shifting, only response labels remain active; prompt and padding are −100. PAD=151643 and EOS are distinct, and real positions determine masks. The prompt is not directly supervised, but it conditions every response score.
For a response of n active tokens, a sum s of log-probabilities is computed, then ell=s/n. q=exp(ell) is the geometric mean of token probabilities, not the joint probability of the response. EOS is included in n. An empty response is rejected. Unlike DPO, the selected ORPO objective uses ell, not s.
From probabilities to odds
The odds of q are q/(1−q). Their logarithm is ell−log(1−exp(ell)). For chosen and rejected responses, the two log-odds are subtracted; a positive difference favors the chosen response. The per-pair penalty is −logsigmoid(difference). The total loss adds supervision: chosen NLL + lambda × mean pair penalties.
Two distinct reductions matter. NLL is the sum of all chosen cross-entropies divided by all their active tokens. Preference is a mean per pair. In an accumulation of eight pairs with different lengths, each chosen cross-entropy shares the total active-token denominator of those eight pairs, while each penalty is divided by eight. Averaging cross-entropy averages would change the objective.
Lambda=0 must recover SFT loss and gradients with the same masks and reductions. The actual SFT branch does not compute rejected responses; the lambda=0 test may compute them to verify their zero gradient. Identical responses yield a relative penalty of log(2), with no useful relative signal; chosen NLL remains present. Identical pairs are rejected in the corpus.
Avoid an inaccurate subtraction
Computing log(1−exp(ell)) directly can yield log(0) near zero: exp(ell) rounds to 1. We use log1p(−exp(ell)) if ell < −log(2), and log(−expm1(ell)) in the other branch. Only the useful branch is evaluated for each value; evaluating both and then selecting with torch.where could leave a NaN gradient from the inactive branch.
The two sides are stacked before this shared operation. This symmetry lets opposite gradients from one input cancel exactly where they meet, without changing the formula or relaxing the test.
The domain is explicit: FP32 or FP64, strictly negative finite ell, with a magnitude at least equal to finfo.tiny. Zero and subnormal magnitudes are rejected because the log-odds or its derivative is not representable under the selected contract. There is no silent clamp that creates a plateau. Tests include −1000, the −log(2) boundary, −1e−300 in FP64, and −1e−30 in FP32. Failures of the naive formula are retained with a status and null for non-finite numbers; never a false JSON numeric result.
The value and gradients of the preference term are compared with odds_ratio_loss from TRL v0.23.1, commit 4529a1c8b1813480a85b02c9c8e7f75a29085d65, in the ordinary numerical domain: absolute tolerance 1e−12 in FP64 and 2e−6 in FP32. This Apache-2.0 excerpt is provided with provenance and license; the trainer is not imported. Masked NLL is compared separately with PyTorch CrossEntropyLoss.
This comparison does not reproduce the whole TRL trainer: in the inspected version, it rebuilds chosen labels from IDs and attention, including the prompt. The tutorial explicitly uses response-only labels. The authors' original repository was also inspected; its masks and final BF16 conversion are not reused. The preference formula and stated reductions are compared, without claiming identity with an entire training infrastructure.
Three training runs and a shared base
The minimum comparison covers the acquired base, chosen-only SFT, and ORPO lambda=0.1. The two trained policies independently start again from the same instruct file, with no imported SFT or warm start from another tutorial. Each policy sees the same 840 prompts for two epochs, in the same deterministic order: 1,680 presentations and 210 updates, with a microbatch of one pair and accumulation of eight.
AdamW uses lr=0.000002, weight_decay=0.01, clipping at 1, a warm-up of 21 updates, then linear decay to 10% of the maximum rate. The seed is 47. Full FP32 parameters and gradients are checked. Validation is measured after each epoch, without selecting a checkpoint or lambda on the test set. ORPO also processes rejected responses; the same prompt exposure does not mean the same budget of tokens, forwards, or time.
Each phase includes an interruption after one epoch, continuous training for two epochs, and resumption of the second. Final states for the model, AdamW, sampler, scheduler, Python/NumPy/PyTorch CPU/CUDA RNG, and histories must be exactly equal. Complete checkpoints are serialized locally with lossless gzip level 1, with actual sizes recorded, then published by atomic rename. Their compression does not reduce VRAM and does not guarantee a compression ratio. They are trusted because they are produced locally; do not load a downloaded Torch object in their place. Final models are also kept in Safetensors and reloaded strictly.
The exercise adds a third trained policy, called sum-ablation, with the same initial weights, prompt order, optimizer, and updates. Its odds term receives SUMS rather than means: it is a named ablation, not the reference ORPO. The nine complete states of the three phases remain local. Optimization, validation, full-command durations, and measured peaks are distinguished.
Evaluate candidates and free text
Each base/SFT/ORPO/sum-ablation state scores the same 180 test pairs. The report publishes chosen NLL per token, log-odds margin, penalty, and accuracy with half credit for ties. Results are separated by whether the chosen response is shorter, equal, or longer in tokens. To compare policies, this evaluation always uses odds of means, including for sum-ablation weights; the model name does not secretly alter the metric.
Each state also generates 180 free reports, with only the incident in the prompt: greedy decoding, one beam, at most 128 new tokens, EOS=151645. IDs, raw text, stop reason, length, and durations remain available. A response stopped at the limit fails even if its beginning appears complete. No output is sent to or applied to a real system.
The section is a grammar independent of generation. It recognizes controlled clauses for unit/state, pressure, and cause. Any unrecognized sentence, repeated field, tag, or stop at the limit makes the note incomplete. A correct natural paraphrase outside the grammar can therefore fail. The score is neither a general judge of faithfulness nor a measure of satisfaction.
Correctness requires all four fields to be correct across ALL 180 cases. Known and unknown causes have separate denominators. A recognized cause different from the reading is counted as unsupported. A recognized abstention for a known cause is counted separately; a missing field in unrecognized prose is not automatically called abstention. A recognized pressure when no measurement exists is reported. Unrecognized sentences remain uncertified, even if these partial counters are zero. Failures are never removed from denominators.
Length exercise
The first lines of the exercise are analytical: constant log-probabilities of −1 for chosen tokens and −1.2 for rejected ones, with lengths 4, 16, and 64. Means, sums, odds, and gradients are compared for the nine length pairs. With means, constant average quality does not depend on length; with sums, a long chosen response can lose to a short rejected response. This is not a measurement on business texts.
Then, the real scores of the main ORPO policy's 180 pairs are fed to both formulas without changing its weights. Finally, the sum-ablation policy is actually trained and evaluated in free generation. Its results are separated from the analytical experiment and simple formula recalculation. SHA-256 values for models, main results, and the minimum configuration are protected; the main version keeps means.
A numerically stable loss can learn poor preferences. A positive margin on two candidates does not guarantee good new text. One seed and bounded synthetic templates strongly limit conclusions; no ORPO improvement over SFT is required.
Define preferences without a revealing word
Observation and causality
Common mistake: every chosen response says 'unconfirmed'; add confirmed causes and paraphrases without changing the test after the run.
Invariant: 840/180/180 pairs, no incident leakage, diverse patterns.
File src/orpo_sans_reference/data.py
import hashlib, re
from pathlib import Path
from .config import save_json, sha
CAUSES = ["filter blockage", "power loss", "sensor fault"]
COURTESY = [
"This is the recorded incident note.",
"Further details may be documented separately.",
]
def split_incident_families(family):
return "train" if family < 210 else "validation" if family < 255 else "test"
def facts_for(family):
return {
"unit": f"P{family+1000}",
"state": "stopped" if family % 3 else "running",
"pressure": None if family % 4 < 2 else 10 + (family // 4) % 7,
"cause": None if family % 2 == 0 else CAUSES[(family // 2) % 3],
}
def incident_text(family, j, facts):
previous = "running" if facts["state"] == "stopped" else "stopped"
old_cause = CAUSES[(family // 2 + 1) % 3]
pressure = (
"not measured" if facts["pressure"] is None else str(facts["pressure"]) + " kPa"
)
cause = "not confirmed" if facts["cause"] is None else "confirmed " + facts["cause"]
return f"Incident N{family:03d}-{j}. Superseded note: state {previous}; suggested cause {old_cause}. Current verified record: unit {facts['unit']}; state {facts['state']}; pressure {pressure}; cause {cause}. Use only the current record."
def reply(facts, j, courtesy=False):
first = [
f"{facts['unit']} is {facts['state']}.",
f"Unit {facts['unit']} is {facts['state']}.",
f"The state of {facts['unit']} is {facts['state']}.",
f"The current state of {facts['unit']} is {facts['state']}.",
][j]
pressure = (
f"The pressure is {facts['pressure']} kPa."
if facts["pressure"] is not None
else ("Pressure was not measured." if j % 2 else "The pressure is unknown.")
)
cause = (
f"The confirmed cause is {facts['cause']}."
if facts["cause"] is not None
else (
"The cause remains unconfirmed."
if j % 2
else "The confirmed cause is not known."
)
)
clauses = [first, pressure, cause]
if courtesy:
clauses = COURTESY[:1] + clauses + COURTESY[1:]
return " ".join(clauses)
def audit_causal_labels(labels):
n = len(labels)
lengths = {"chosen_shorter": 0, "same_words": 0, "chosen_longer": 0}
known = sum(r["expected"]["cause"] is not None for r in labels)
features = []
words = sorted(
{
w
for r in labels
for side in ["chosen", "rejected"]
for w in re.findall("[a-z]+", r[side].lower())
}
)
for row in labels:
a, b = len(row["chosen"].split()), len(row["rejected"].split())
lengths[
"chosen_shorter" if a < b else "chosen_longer" if a > b else "same_words"
] += 1
for word in words:
a, b = [
sum(word in set(re.findall("[a-z]+", r[side].lower())) for r in labels)
for side in ["chosen", "rejected"]
]
accuracy = (a + n - b) / (2 * n)
features.append(
{
"word": word,
"chosen_presence": a,
"rejected_presence": b,
"best_binary_label_accuracy": max(accuracy, 1 - accuracy),
}
)
return {
"pairs": n,
"known_cause": known,
"unknown_cause": n - known,
"unknown_pressure": sum(r["expected"]["pressure"] is None for r in labels),
"word_length_regimes": lengths,
"best_length_preference_accuracy_half_ties": max(
(lengths["chosen_longer"] + 0.5 * lengths["same_words"]) / n,
(lengths["chosen_shorter"] + 0.5 * lengths["same_words"]) / n,
),
"word_features": features,
"maximum_word_accuracy": max(r["best_binary_label_accuracy"] for r in features),
"scope": "Original words, not every possible semantic/BPE shortcut",
}
def generate_incident_pairs():
pages = []
labels = []
for family in range(300):
facts = facts_for(family)
for j in range(4):
identifier = f"N{family:03d}-{j}"
text = incident_text(family, j, facts)
wrong = dict(facts)
reason = ["cause", "pressure", "state", "cause"][j]
if reason == "cause":
wrong["cause"] = (
CAUSES[(family // 2) % 3] if facts["cause"] is None else None
)
elif reason == "pressure":
wrong["pressure"] = (
10 + (family // 4) % 7 if facts["pressure"] is None else None
)
else:
wrong["state"] = "running" if facts["state"] == "stopped" else "stopped"
chosen = reply(facts, j, courtesy=j == 1)
rejected = reply(wrong, j, courtesy=j == 2)
if chosen == rejected:
raise ValueError("Preference pair is identical")
pages.append(
{
"document_id": identifier,
"family": family,
"formulation": j,
"split": split_incident_families(family),
"incident": text,
"record": facts,
"content_sha256": hashlib.sha256(text.encode()).hexdigest(),
}
)
labels.append(
{
"document_id": identifier,
"chosen": chosen,
"rejected": rejected,
"expected": facts,
"preference_reason": reason,
"simulated_preference": True,
}
)
if len({p["incident"] for p in pages}) != 1200:
raise ValueError("Duplicate incident")
return pages, labels
def prepare(config):
root = Path("data/generated")
if (root / "manifest.json").exists():
raise ValueError("Preserve existing corpus; use a new extraction")
pages, labels = generate_incident_pairs()
gold = {r["document_id"]: r for r in labels}
save_json(root / "input/pages.json", pages)
save_json(root / "labels/gold.json", labels)
counts = {}
audits = {}
for split in ["train", "validation", "test"]:
selected = [p for p in pages if p["split"] == split]
chosen = [gold[p["document_id"]] for p in selected]
counts[split] = {
"pairs": len(selected),
"families": len({p["family"] for p in selected}),
}
audits[split] = audit_causal_labels(chosen)
if split != "test":
save_json(root / f"{split}/pages.json", selected)
save_json(root / f"labels/{split}.json", chosen)
save_json(
"outputs/data-audit.json",
{
"dataset": "IncidentNotesPrefs-v1",
"rights": "CC0-1.0",
"counts": counts,
"shortcuts": audits,
"human_preferences": False,
"shared_semantic_and_language_primitives": True,
"current_record_supersedes_previous_note": True,
"reference_model_or_cache": False,
},
)
save_json(
root / "manifest.json",
{
"dataset": "IncidentNotesPrefs-v1",
"seed": config["seed"],
"counts": counts,
"files": {
str(f.relative_to(root)): sha(f) for f in root.rglob("*") if f.is_file()
},
},
)
return counts
Compute token means
Masks, length, and score q
Common mistake: a batch mean wrongly weights every sentence; name token mean and pair mean separately.
Invariant: PAD invariants, non-empty responses, controlled reduction.
File src/orpo_sans_reference/scores.py
import torch
from .config import save_json
SYSTEM = "Write an incident note using only the current verified record. State the unit and its current state, the pressure if measured, and the confirmed cause if known. Distinguish observations from causal conclusions. A superseded suggestion is not a current fact. Do not add facts, actions or explanations."
def format_prompt(tokenizer, incident, version):
if version != "incident-v1":
raise ValueError("Unknown incident template")
messages = [
{"role": "system", "content": SYSTEM},
{"role": "user", "content": incident},
]
return messages, tokenizer.apply_chat_template(
messages, tokenize=True, add_generation_prompt=True
)
def completion_row(tokenizer, incident, response, config, identifier, version=None):
version = version or config["template_version"]
messages, prompt = format_prompt(tokenizer, incident, version)
target = tokenizer.encode(response, add_special_tokens=False) + [151645]
if (
not prompt
or len(prompt) > config["max_prompt_tokens"]
or not 1 < len(target) <= config["max_response_tokens_including_eos"]
or len(prompt) + len(target) > config["context"]
):
raise ValueError(
identifier + ": declared token budget exceeded; do not truncate"
)
return {
"document_id": identifier,
"messages": messages,
"input_ids": prompt + target,
"labels": [-100] * len(prompt) + target,
"prompt_tokens": len(prompt),
"response_tokens": len(target),
"template_version": version,
}
def paired_completion_batch(tokenizer, page, label, config, version=None):
chosen = completion_row(
tokenizer,
page["incident"],
label["chosen"],
config,
page["document_id"],
version,
)
rejected = completion_row(
tokenizer,
page["incident"],
label["rejected"],
config,
page["document_id"],
version,
)
if (
chosen["input_ids"] == rejected["input_ids"]
and chosen["labels"] == rejected["labels"]
):
raise ValueError("Identical completions are not an informative preference")
return {
"document_id": page["document_id"],
"chosen": chosen,
"rejected": rejected,
"response_length_delta": chosen["response_tokens"]
- rejected["response_tokens"],
}
def collate(rows, device="cuda", pad_to=None):
if not rows:
raise ValueError("Empty completion batch")
width = max(len(r["input_ids"]) for r in rows) if pad_to is None else pad_to
ids = torch.full((len(rows), width), 151643, dtype=torch.long, device=device)
mask = torch.zeros_like(ids, dtype=torch.bool)
labels = torch.full_like(ids, -100)
for i, row in enumerate(rows):
n = len(row["input_ids"])
if n > width:
raise ValueError("Padding width would truncate a response")
ids[i, :n] = torch.tensor(row["input_ids"], device=device)
mask[i, :n] = True
labels[i, :n] = torch.tensor(row["labels"], device=device)
return ids, mask, labels
def sequence_logprob(logits, labels):
if logits.shape[:2] != labels.shape:
raise ValueError("Logits and labels shape mismatch")
active = labels[:, 1:] != -100
counts = active.sum(dim=1)
if (counts == 0).any():
raise ValueError("Every sequence needs an active response target")
selected = logits[:, :-1][active]
targets = labels[:, 1:][active]
values = (
torch.log_softmax(selected, dim=-1)
.gather(-1, targets.unsqueeze(-1))
.squeeze(-1)
)
sums = torch.stack([v.sum() for v in values.split(counts.tolist())])
return sums, counts, values, selected, targets
def model_scores(model, rows):
ids, mask, labels = collate(rows)
logits = model(
input_ids=ids, attention_mask=mask, use_cache=False, return_dict=True
).logits
return sequence_logprob(logits.float(), labels)
def mean_completion_logprob(logits, labels):
sums, counts, values, selected, targets = sequence_logprob(logits, labels)
return sums / counts, counts, values, selected, targets
def chosen_token_nll(logits, labels):
sums, counts, _, _, _ = sequence_logprob(logits, labels)
return -sums.sum() / counts.sum()
def inspect_pair_lengths(config):
from .model import load_tokenizer
from .data import generate_incident_pairs
tokenizer = load_tokenizer()
pages, labels = generate_incident_pairs()
pairs = [
paired_completion_batch(tokenizer, pages[i], labels[i], config)
for i in range(4)
]
save_json(
"outputs/tokenization-mechanism.json",
{
"pairs": pairs,
"preference_reduction": "mean_response_logprob_including_EOS",
"nll_reduction": "all_chosen_active_tokens",
"prompt_and_padding_excluded": True,
},
)
Implement odds stably
log1mexp and sigmoid
Common mistake: a silent clamp creates a gradient plateau; make it explicit and test its active range.
Invariant: FP64/FP32 tests and reference agreement at the predefined tolerances.
File src/orpo_sans_reference/loss.py
import math
from types import SimpleNamespace
import torch
from .config import save_json
def log1mexp(a):
if a.dtype not in [torch.float32, torch.float64] or not torch.isfinite(a).all():
raise ValueError("Use finite FP32/FP64 log probabilities")
if (a >= 0).any() or (a.abs() < torch.finfo(a.dtype).tiny).any():
raise ValueError(
"Log probability must be strictly negative with a representable finite derivative; no silent clamp"
)
# Compute only the selected branch: torch.where on two evaluated formulas
# could propagate NaN gradients from its inactive near-zero branch.
far = a < -math.log(2)
result = torch.empty_like(a)
result[far] = torch.log1p(-torch.exp(a[far]))
result[~far] = torch.log(-torch.expm1(a[~far]))
return result
def orpo_preference_loss(chosen_logps, rejected_logps):
if chosen_logps.shape != rejected_logps.shape or not chosen_logps.numel():
raise ValueError("Matching nonempty paired score vectors required")
paired = torch.stack([chosen_logps, rejected_logps])
odds = paired - log1mexp(paired)
margin = odds[0] - odds[1]
return -torch.nn.functional.logsigmoid(margin), margin
def orpo_total_loss(
chosen_sums,
rejected_sums,
chosen_counts,
rejected_counts,
coefficient=0.1,
normalization="mean",
):
if (
coefficient < 0
or not math.isfinite(coefficient)
or normalization not in ["mean", "sum"]
):
raise ValueError("Explicit nonnegative lambda and named normalization required")
if (chosen_counts <= 0).any() or (rejected_counts <= 0).any():
raise ValueError("Responses must have active tokens")
chosen = chosen_sums / chosen_counts if normalization == "mean" else chosen_sums
rejected = (
rejected_sums / rejected_counts if normalization == "mean" else rejected_sums
)
nll = -chosen_sums.sum() / chosen_counts.sum()
preference, margins = orpo_preference_loss(chosen, rejected)
return nll + coefficient * preference.mean(), nll, preference, margins
def numeric_mechanism(config):
from .reference_loss import odds_ratio_loss
reference_rows = []
for dtype, tolerance in [(torch.float64, 1e-12), (torch.float32, 2e-6)]:
chosen = torch.tensor([-1.0, -3.0, -0.1], dtype=dtype, requires_grad=True)
rejected = torch.tensor([-2.0, -1.0, -0.5], dtype=dtype, requires_grad=True)
loss, margin = orpo_preference_loss(chosen, rejected)
control = SimpleNamespace(
beta=config["lambda_orpo"], accelerator=SimpleNamespace(device="cpu")
)
reference = -odds_ratio_loss(control, chosen, rejected)[0]
a = torch.autograd.grad(
config["lambda_orpo"] * loss.mean(), (chosen, rejected), retain_graph=True
)
b = torch.autograd.grad(reference.mean(), (chosen, rejected))
checks = [
torch.allclose(
config["lambda_orpo"] * loss, reference, atol=tolerance, rtol=0
),
*(torch.allclose(x, y, atol=tolerance, rtol=0) for x, y in zip(a, b)),
]
if not all(checks):
raise ValueError("Pinned ORPO reference value/gradient comparison failed")
reference_rows.append(
{
"dtype": str(dtype),
"chosen_mean_logps": chosen.tolist(),
"rejected_mean_logps": rejected.tolist(),
"preference_losses": loss.tolist(),
"log_odds_margins": margin.tolist(),
"weighted_reference_losses": reference.tolist(),
"chosen_gradient": a[0].tolist(),
"rejected_gradient": a[1].tolist(),
"reference_chosen_gradient": b[0].tolist(),
"reference_rejected_gradient": b[1].tolist(),
"lambda": config["lambda_orpo"],
"atol": tolerance,
}
)
stability = []
for dtype, values in [
(
torch.float64,
[-1000.0, -100.0, -1.0, -math.log(2), -0.1, -1e-10, -1e-100, -1e-300],
),
(
torch.float32,
[-1000.0, -100.0, -1.0, -math.log(2), -0.1, -1e-5, -1e-10, -1e-30],
),
]:
x = torch.tensor(values, dtype=dtype, requires_grad=True)
stable = log1mexp(x)
gradient = torch.autograd.grad(stable.sum(), x)[0]
naive = torch.log(1 - torch.exp(x))
naive_grad = torch.autograd.grad(naive.sum(), x)[0]
if not torch.isfinite(stable).all() or not torch.isfinite(gradient).all():
raise ValueError("Stable numerical branch is not finite")
for a, s, g, n, ng in zip(
x.tolist(),
stable.tolist(),
gradient.tolist(),
naive.tolist(),
naive_grad.tolist(),
):
expected = (
math.log1p(-math.exp(a))
if a < -math.log(2)
else math.log(-math.expm1(a))
)
stability.append(
{
"dtype": str(dtype),
"log_probability": a,
"stable": s,
"stable_gradient": g,
"analytic_float64_value": expected,
"absolute_error": abs(s - expected),
"naive": n if math.isfinite(n) else None,
"naive_gradient": ng if math.isfinite(ng) else None,
"naive_finite": math.isfinite(n) and math.isfinite(ng),
}
)
save_json(
"outputs/numeric-mechanism.json",
{
"reference": reference_rows,
"stability": stability,
"data_source": "Controlled analytical log probabilities, not incident quality measurements",
"zero_or_subnormal_magnitude_policy": "Explicit rejection, no clamp",
"reference_model_loaded": False,
},
)
Train SFT and ORPO from the same base
Two comparable objectives
Common mistake: copying a DPO configuration imports an unused reference; audit objects and allocations.
Invariant: actual updates, checkpoints, and resumption verified; no reference model created.
File src/orpo_sans_reference/train.py
import math, time
from pathlib import Path
import torch
from .config import read_json, save_json, sha, seed_all, rng_state, restore_rng
from .model import (
load_base,
load_tokenizer,
parameter_manifest,
manifest_digest,
save_learned,
)
from .scores import paired_completion_batch, model_scores
from .loss import orpo_preference_loss
from .state import save_resume_state, load_resume_state, equal
BRANCHES = ["sft", "orpo", "sum-ablation"]
def learning_rate(config, step, total=210):
peak = config["learning_rate"]
warmup = max(1, int(total * config["warmup_ratio"]))
if step < warmup:
return peak * (step + 1) / warmup
return peak * (
1
- min(1.0, (step - warmup) / max(1, total - warmup - 1))
* (1 - config["minimum_lr_ratio"])
)
def encoded_split(config, tokenizer, split):
pages = read_json(f"data/generated/{split}/pages.json")
gold = {
r["document_id"]: r for r in read_json(f"data/generated/labels/{split}.json")
}
if any(p["split"] != split for p in pages):
raise ValueError("Family partition differs")
return [
paired_completion_batch(tokenizer, p, gold[p["document_id"]], config)
for p in pages
]
@torch.no_grad()
def validate_policy(model, pairs):
model.eval()
rows = []
total_nll = 0.0
total_tokens = 0
wins = 0.0
for pair in pairs:
chosen, nc, *_ = model_scores(model, [pair["chosen"]])
rejected, nr, *_ = model_scores(model, [pair["rejected"]])
a = chosen / nc
b = rejected / nr
preference, margin = orpo_preference_loss(a, b)
total_nll -= float(chosen[0])
total_tokens += int(nc[0])
wins += 1 if float(margin[0]) > 0 else 0.5 if float(margin[0]) == 0 else 0
rows.append(
{
"document_id": pair["document_id"],
"chosen_sum": float(chosen[0]),
"rejected_sum": float(rejected[0]),
"chosen_tokens": int(nc[0]),
"rejected_tokens": int(nr[0]),
"chosen_mean": float(a[0]),
"rejected_mean": float(b[0]),
"mean_log_odds_margin": float(margin[0]),
"mean_odds_preference_loss": float(preference[0]),
}
)
model.train()
return {
"pairs": len(pairs),
"chosen_tokens": total_tokens,
"chosen_nll_nats_per_token": total_nll / total_tokens,
"mean_odds_accuracy_half_ties": wins / len(pairs),
"rows": rows,
"used_for_checkpoint_selection": False,
"evaluation_normalization": "mean_even_for_sum_ablation",
}
def train_policy(config, branch="orpo", mode="continuous"):
if branch not in BRANCHES or mode not in ["interrupt", "continuous", "resume"]:
raise ValueError("Unknown training branch/mode")
normalization = "sum" if branch == "sum-ablation" else "mean"
if config.get("normalization", "mean") != normalization:
raise ValueError("Sum ablation requires its explicit exercise config")
if (
Path("data/generated/input").exists()
or Path("data/generated/labels/gold.json").exists()
or Path("assets/reference").exists()
or Path("outputs/reference-cache").exists()
or Path("outputs/inference").exists()
):
raise ValueError(
"Learning must not mount heldout outputs, labels or any reference model/cache"
)
seed_all(config["seed"])
tokenizer = load_tokenizer()
rows = encoded_split(config, tokenizer, "train")
validation = encoded_split(config, tokenizer, "validation")
if len(rows) != 840 or len(validation) != 180:
raise ValueError("Frozen population differs")
model = load_base().cuda()
parameters = list(model.parameters())
initial = parameter_manifest(model)
initial_digest = manifest_digest(initial)
if (
sum(p.numel() for p in parameters) != 494032768
or any(not p.requires_grad or p.dtype != torch.float32 for p in parameters)
or len({id(p) for p in parameters}) != len(parameters)
):
raise ValueError("All unique FP32 policy parameters must be optimized")
identity = {
"branch": branch,
"normalization": normalization,
"lambda": 0.0 if branch == "sft" else config["lambda_orpo"],
"config": config,
"initial_model_digest": initial_digest,
"model_sha256": sha("assets/model/model.safetensors"),
"tokenizer_sha256": sha("assets/model/tokenizer.json"),
"implementation_sha256": {
name: sha("src/orpo_sans_reference/" + name)
for name in ["model.py", "scores.py", "loss.py", "train.py", "state.py"]
},
"data_sha256": {
name: sha("data/generated/" + name)
for name in [
"train/pages.json",
"labels/train.json",
"validation/pages.json",
"labels/validation.json",
]
},
"reference_models": 0,
"reference_caches": 0,
}
optimizer = torch.optim.AdamW(
parameters, lr=config["learning_rate"], weight_decay=config["weight_decay"]
)
sampler = torch.Generator().manual_seed(config["seed"] + 100)
per_epoch = math.ceil(len(rows) / config["accumulation_pairs"])
total = per_epoch * config["epochs"]
root = Path("assets/checkpoints")
destination = root / f"{branch}-{mode}.pt.gz"
updates = start = 0
history = []
validation_history = []
if destination.exists():
raise ValueError("Preserve existing local checkpoint")
if mode == "resume":
state = load_resume_state(root / f"{branch}-interrupt.pt.gz")
if state["identity"] != identity:
raise ValueError("Resume identity changed, including lambda/normalization")
model.load_state_dict(state["model"], strict=True)
optimizer.load_state_dict(state["optimizer"])
sampler.set_state(state["sampler"])
restore_rng(state["rng"])
updates = state["updates"]
start = state["epoch"]
history = state["history"]
validation_history = state["validation_history"]
if state["scheduler"] != {
"next_update": updates,
"next_lr": learning_rate(config, updates, total),
}:
raise ValueError("Scheduler identity changed")
del state
stop = config["interrupt_epoch"] if mode == "interrupt" else config["epochs"]
model.train()
torch.cuda.reset_peak_memory_stats()
torch.cuda.synchronize()
begun = time.perf_counter()
optimization_seconds = 0.0
for epoch in range(start, stop):
order = torch.randperm(len(rows), generator=sampler).tolist()
torch.cuda.synchronize()
epoch_begun = time.perf_counter()
for offset in range(0, len(rows), config["accumulation_pairs"]):
selected = order[offset : offset + config["accumulation_pairs"]]
chosen_denominator = sum(
rows[i]["chosen"]["response_tokens"] for i in selected
)
lr = learning_rate(config, updates, total)
for group in optimizer.param_groups:
group["lr"] = lr
optimizer.zero_grad(set_to_none=True)
details = []
nll_numerator = preference_numerator = 0.0
policy_tokens = 0
for index in selected:
pair = rows[index]
chosen, nc, *_ = model_scores(model, [pair["chosen"]])
nll = -chosen.sum()
contribution = nll / chosen_denominator
policy_tokens += int(nc[0])
detail = {
"document_id": pair["document_id"],
"chosen_sum": float(chosen.detach()[0]),
"chosen_tokens": int(nc[0]),
"rejected_sum": None,
"rejected_tokens": 0,
"preference_loss": None,
"log_odds_margin": None,
"preference_chosen_logp": None,
"preference_rejected_logp": None,
}
if branch != "sft":
rejected, nr, *_ = model_scores(model, [pair["rejected"]])
a = chosen / nc if normalization == "mean" else chosen
b = rejected / nr if normalization == "mean" else rejected
preference, margin = orpo_preference_loss(a, b)
contribution = contribution + config[
"lambda_orpo"
] * preference.sum() / len(selected)
preference_numerator += float(preference.detach()[0])
policy_tokens += int(nr[0])
detail.update(
rejected_sum=float(rejected.detach()[0]),
rejected_tokens=int(nr[0]),
preference_loss=float(preference.detach()[0]),
log_odds_margin=float(margin.detach()[0]),
preference_chosen_logp=float(a.detach()[0]),
preference_rejected_logp=float(b.detach()[0]),
)
if not torch.isfinite(contribution):
raise ValueError("Nonfinite objective")
contribution.backward()
nll_numerator += float(nll.detach())
details.append(detail)
norm = torch.nn.utils.clip_grad_norm_(
parameters, config["gradient_clip"], error_if_nonfinite=True
)
if any(p.grad is None or p.grad.dtype != torch.float32 for p in parameters):
raise ValueError("All policy gradients must exist in FP32")
optimizer.step()
updates += 1
nll = nll_numerator / chosen_denominator
preference = (
preference_numerator / len(selected) if branch != "sft" else 0.0
)
history.append(
{
"update": updates,
"epoch": epoch + 1,
"document_ids": [rows[i]["document_id"] for i in selected],
"chosen_token_denominator": chosen_denominator,
"preference_pair_denominator": (
len(selected) if branch != "sft" else 0
),
"policy_response_tokens": policy_tokens,
"chosen_nll": nll,
"preference_loss": preference,
"total_loss": nll + identity["lambda"] * preference,
"learning_rate": lr,
"gradient_norm_before_clip": float(norm),
"pairs": details,
}
)
if updates % 21 == 0:
print(
{k: v for k, v in history[-1].items() if k != "pairs"}, flush=True
)
torch.cuda.synchronize()
optimization_seconds += time.perf_counter() - epoch_begun
observed = validate_policy(model, validation)
validation_history.append({"epoch": epoch + 1, **observed})
print(
{
"branch": branch,
"epoch": epoch + 1,
**{k: v for k, v in observed.items() if k != "rows"},
},
flush=True,
)
torch.cuda.synchronize()
elapsed = time.perf_counter() - begun
peak = torch.cuda.max_memory_allocated()
final = parameter_manifest(model)
final_digest = manifest_digest(final)
if final_digest == initial_digest:
raise ValueError("No real learned parameter change")
state = {
"identity": identity,
"epoch": stop,
"updates": updates,
"model": model.state_dict(),
"optimizer": optimizer.state_dict(),
"sampler": sampler.get_state(),
"scheduler": {
"next_update": updates,
"next_lr": learning_rate(config, updates, total),
},
"rng": rng_state(),
"history": history,
"validation_history": validation_history,
}
compression = save_resume_state(destination, state)
if mode == "resume":
save_learned(model, root / f"{branch}-resume.safetensors", identity)
save_json(
f"outputs/learning/train-{branch}-{mode}.json",
{
"branch": branch,
"mode": mode,
"identity": identity,
"epochs": stop,
"updates": updates,
"history": history,
"validation_history": validation_history,
"tokenization": rows,
"initial_parameters": initial,
"final_parameters": final,
"initial_model_digest": initial_digest,
"final_model_digest": final_digest,
"trainable_parameters": 494032768,
"checkpoint_sha256": sha(destination),
"checkpoint_bytes": destination.stat().st_size,
"checkpoint_compression": compression,
"optimization_seconds_this_process": optimization_seconds,
"training_and_validation_seconds_this_process": elapsed,
"max_torch_allocated_bytes": peak,
"test_visible": False,
"reference_models_loaded": 0,
"policy_models_loaded": 1,
},
)
def verify_resume(branch):
root = Path("assets/checkpoints")
a = load_resume_state(root / f"{branch}-continuous.pt.gz")
b = load_resume_state(root / f"{branch}-resume.pt.gz")
checks = {k: equal(a[k], b[k]) for k in a}
if a.keys() != b.keys() or not all(checks.values()):
raise ValueError("Continuous/resumed full states differ: " + str(checks))
save_json(
f"outputs/learning/resume-{branch}.json",
{
"status": "PASS",
"all_state_exact": True,
"checks": checks,
"boundary": "Full epoch; model/AdamW/scheduler/sampler/RNG/history compared exactly after lossless decompression",
},
)
Observe invented causes and abstentions
Free-generation quality
Common mistake: evaluating only unknown causes rewards systematic abstention; retain both strata.
Invariant: 180 outputs per branch, known/unknown causes presented separately.
File src/orpo_sans_reference/evaluate.py
import math, re, time
from pathlib import Path
import torch
from transformers import GenerationConfig
from .config import read_json, save_json, sha, seed_all
from .model import (
load_base,
load_tokenizer,
reload_learned,
parameter_manifest,
manifest_digest,
)
from .scores import format_prompt, paired_completion_batch, model_scores
from .loss import orpo_preference_loss
from .data import COURTESY
def parse_note(text, finish):
sentences = [
s.strip() for s in re.split(r"(?<=[.!?])\s+", text.strip()) if s.strip()
]
fields = {}
unknown = []
duplicates = []
for sentence in sentences:
if sentence in COURTESY:
continue
value = sentence.rstrip(".!?").strip()
items = None
match = re.fullmatch(r"(?:Unit )?(P\d{4}) is (stopped|running)", value, re.I)
if match:
items = {"unit": match[1].upper(), "state": match[2].lower()}
else:
match = re.fullmatch(
r"The (?:current )?state of (P\d{4}) is (stopped|running)", value, re.I
)
if match:
items = {"unit": match[1].upper(), "state": match[2].lower()}
else:
match = re.fullmatch(r"The pressure is (\d{1,4}) kPa", value, re.I)
if match:
items = {"pressure": int(match[1])}
elif value.lower() in [
"pressure was not measured",
"the pressure is unknown",
]:
items = {"pressure": None}
elif value.lower() in [
"the cause remains unconfirmed",
"no cause has been confirmed",
"the confirmed cause is not known",
]:
items = {"cause": None}
else:
match = re.fullmatch(
r"The confirmed cause is (filter blockage|power loss|sensor fault)",
value,
re.I,
)
if match:
items = {"cause": match[1].lower()}
if items is None:
unknown.append(sentence)
else:
for key, item in items.items():
if key in fields:
duplicates.append(key)
else:
fields[key] = item
covered = bool(sentences) and not unknown and not duplicates
return {
"recognized": fields,
"unrecognized_sentences": unknown,
"duplicate_fields": duplicates,
"all_text_covered": covered,
"complete": finish == "eos"
and covered
and set(fields) == {"unit", "state", "pressure", "cause"},
"truncated": finish != "eos",
}
def score_causal_claims(rows, gold):
metrics = {
"documents": len(rows),
"complete_controlled_notes": 0,
"exact_facts": 0,
"truncated": 0,
"unrecognized_prose": 0,
"known_cause_cases": 0,
"unknown_cause_cases": 0,
"known_cause_correct": 0,
"unknown_cause_correct": 0,
"recognized_unsupported_cause": 0,
"recognized_known_cause_abstention": 0,
"unknown_pressure_cases": 0,
"recognized_invented_pressure": 0,
}
fields = dict.fromkeys(["unit", "state", "pressure", "cause"], 0)
details = []
for row in rows:
expected = gold[row["document_id"]]["expected"]
parsed = parse_note(row["text"], row["finish_reason"])
if row["parsed"] != parsed:
raise ValueError("Stored parser result differs")
actual = parsed["recognized"]
valid = parsed["complete"]
correct = {k: valid and actual.get(k) == v for k, v in expected.items()}
exact = all(correct.values())
metrics["complete_controlled_notes"] += valid
metrics["exact_facts"] += exact
metrics["truncated"] += parsed["truncated"]
metrics["unrecognized_prose"] += bool(parsed["unrecognized_sentences"])
for key, value in correct.items():
fields[key] += value
known = expected["cause"] is not None
metrics["known_cause_cases" if known else "unknown_cause_cases"] += 1
metrics["known_cause_correct" if known else "unknown_cause_correct"] += correct[
"cause"
]
metrics["recognized_unsupported_cause"] += (
actual.get("cause") is not None and actual["cause"] != expected["cause"]
)
metrics["recognized_known_cause_abstention"] += (
known and "cause" in actual and actual["cause"] is None
)
if expected["pressure"] is None:
metrics["unknown_pressure_cases"] += 1
metrics["recognized_invented_pressure"] += (
actual.get("pressure") is not None
)
details.append(
{
"document_id": row["document_id"],
"expected": expected,
"recognized": actual,
"complete": valid,
"fields_correct": correct,
"exact": exact,
"unrecognized_sentences": parsed["unrecognized_sentences"],
}
)
n = len(rows)
metrics.update(
exact_accuracy_over_all_cases=metrics["exact_facts"] / n,
field_correct_over_all_cases=fields,
field_accuracy_over_all_cases={k: v / n for k, v in fields.items()},
known_cause_accuracy=metrics["known_cause_correct"]
/ metrics["known_cause_cases"],
unknown_cause_accuracy=metrics["unknown_cause_correct"]
/ metrics["unknown_cause_cases"],
recognized_known_cause_abstention_rate=metrics[
"recognized_known_cause_abstention"
]
/ metrics["known_cause_cases"],
mean_generated_tokens=math.fsum(r["output_tokens"] for r in rows) / n,
)
return {
"metrics": metrics,
"rows": details,
"unrecognized_prose_is_not_certified_faithful": True,
}
def pair_metrics(rows):
result = {}
for group in ["all", "chosen_shorter", "same_tokens", "chosen_longer"]:
selected = (
rows
if group == "all"
else [
r
for r in rows
if (
"chosen_shorter"
if r["response_length_delta"] < 0
else (
"chosen_longer"
if r["response_length_delta"] > 0
else "same_tokens"
)
)
== group
]
)
n = len(selected)
count = sum(r["scores"]["chosen"]["response_tokens"] for r in selected)
result[group] = {
"pairs": n,
"chosen_response_tokens": count,
"chosen_nll_nats_per_token": (
-math.fsum(r["scores"]["chosen"]["sum_logprob"] for r in selected)
/ count
if count
else None
),
"mean_log_odds_margin": (
math.fsum(r["mean_log_odds_margin"] for r in selected) / n
if n
else None
),
"mean_odds_preference_accuracy_half_ties": (
sum(
(
1
if r["mean_log_odds_margin"] > 0
else 0.5 if r["mean_log_odds_margin"] == 0 else 0
)
for r in selected
)
/ n
if n
else None
),
"mean_odds_preference_loss": (
math.fsum(r["mean_odds_preference_loss"] for r in selected) / n
if n
else None
),
}
return result
def load_policy(branch):
if branch not in ["base", "sft", "orpo", "sum-ablation"]:
raise ValueError("Unknown evaluation branch")
path = Path(
"assets/model/model.safetensors"
if branch == "base"
else f"assets/checkpoints/{branch}-resume.safetensors"
)
model = load_base() if branch == "base" else reload_learned(path)[0]
digest = manifest_digest(parameter_manifest(model))
for parameter in model.parameters():
parameter.requires_grad_(False)
return model.cuda().eval(), {
"branch": branch,
"file_sha256": sha(path),
"model_digest": digest,
"tokenizer_sha256": sha("assets/model/tokenizer.json"),
"reference_models_loaded": 0,
}
@torch.no_grad()
def score_pairs(config, branch):
seed_all(config["seed"])
model, identity = load_policy(branch)
tokenizer = load_tokenizer()
pages = [
p for p in read_json("data/generated/input/pages.json") if p["split"] == "test"
]
gold = {g["document_id"]: g for g in read_json("data/generated/labels/gold.json")}
rows = []
probes = []
torch.cuda.reset_peak_memory_stats()
torch.cuda.synchronize()
begun = time.perf_counter()
for index, page in enumerate(pages):
pair = paired_completion_batch(
tokenizer, page, gold[page["document_id"]], config
)
scores = {}
mean_tensors = {}
for side in ["chosen", "rejected"]:
sums, counts, values, logits, targets = model_scores(model, [pair[side]])
mean_tensors[side] = sums / counts
scores[side] = {
"sum_logprob": float(sums[0]),
"mean_logprob": float(mean_tensors[side][0]),
"response_tokens": int(counts[0]),
"token_logprobs": values.cpu().tolist(),
"targets": targets.cpu().tolist(),
}
if index == 0:
probes.append(
{
"document_id": page["document_id"],
"side": side,
"response_position": 0,
"target": int(targets[0]),
"logits": logits[0].cpu().tolist(),
"logprob": float(values[0]),
}
)
loss, margin = orpo_preference_loss(
mean_tensors["chosen"], mean_tensors["rejected"]
)
rows.append(
{
"document_id": page["document_id"],
"pair": pair,
"scores": scores,
"response_length_delta": pair["response_length_delta"],
"mean_log_odds_margin": float(margin[0]),
"mean_odds_preference_loss": float(loss[0]),
}
)
torch.cuda.synchronize()
save_json(
f"outputs/evaluation/pairs-{branch}.json",
{
"branch": branch,
"identity": identity,
"rows": rows,
"probes": probes,
"metrics": pair_metrics(rows),
"seconds": time.perf_counter() - begun,
"max_torch_allocated_bytes": torch.cuda.max_memory_allocated(),
"evaluation_normalization": "mean_even_for_sum_ablation",
"test_selection": False,
},
)
@torch.no_grad()
def generate_notes(config, branch):
if (
Path("data/generated/labels").exists()
or Path("assets/reference").exists()
or Path("outputs/reference-cache").exists()
):
raise ValueError("Blind generation must not mount labels or a reference")
seed_all(config["seed"])
model, identity = load_policy(branch)
tokenizer = load_tokenizer()
pages = [
p for p in read_json("data/generated/input/pages.json") if p["split"] == "test"
]
generation = GenerationConfig(
do_sample=False,
num_beams=1,
max_new_tokens=config["max_new_tokens"],
use_cache=True,
eos_token_id=151645,
pad_token_id=151643,
bos_token_id=151643,
)
rows = []
torch.cuda.reset_peak_memory_stats()
torch.cuda.synchronize()
begun = time.perf_counter()
for index, page in enumerate(pages):
messages, ids = format_prompt(
tokenizer, page["incident"], config["template_version"]
)
if not 0 < len(ids) <= config["max_prompt_tokens"]:
raise ValueError("No silent prompt truncation")
tensor = torch.tensor([ids], device="cuda")
torch.cuda.synchronize()
started = time.perf_counter()
output = (
model.generate(
input_ids=tensor,
attention_mask=torch.ones_like(tensor),
generation_config=generation,
)[0, len(ids) :]
.cpu()
.tolist()
)
torch.cuda.synchronize()
finish = "eos" if output and output[-1] == 151645 else "length"
text = tokenizer.decode(
output[:-1] if finish == "eos" else output,
skip_special_tokens=False,
clean_up_tokenization_spaces=False,
)
rows.append(
{
"document_id": page["document_id"],
"messages": messages,
"input_ids": ids,
"output_ids": output,
"text": text,
"finish_reason": finish,
"parsed": parse_note(text, finish),
"input_tokens": len(ids),
"output_tokens": len(output),
"seconds": time.perf_counter() - started,
}
)
if (index + 1) % 30 == 0:
print({"branch": branch, "generated_notes": index + 1}, flush=True)
torch.cuda.synchronize()
save_json(
f"outputs/inference/predictions-{branch}.json",
{
"branch": branch,
"identity": identity,
"generation": generation.to_dict(),
"input_sha256": sha("data/generated/input/pages.json"),
"rows": rows,
"seconds": time.perf_counter() - begun,
"max_torch_allocated_bytes": torch.cuda.max_memory_allocated(),
"gold_visible": False,
"reference_models_loaded": 0,
"training": False,
},
)
def evaluate_notes(config, exercise=False):
gold = {g["document_id"]: g for g in read_json("data/generated/labels/gold.json")}
test_ids = [
p["document_id"]
for p in read_json("data/generated/input/pages.json")
if p["split"] == "test"
]
results = {}
for branch in ["sum-ablation"] if exercise else ["base", "sft", "orpo"]:
predictions = read_json(f"outputs/inference/predictions-{branch}.json")
pairs = read_json(f"outputs/evaluation/pairs-{branch}.json")
if (
[r["document_id"] for r in predictions["rows"]] != test_ids
or [r["document_id"] for r in pairs["rows"]] != test_ids
or len(test_ids) != 180
or predictions["identity"] != pairs["identity"]
):
raise ValueError("Same180testcases and model weights required")
results[branch] = {
"free_notes": score_causal_claims(predictions["rows"], gold),
"pairs": pair_metrics(pairs["rows"]),
}
print(
{
"branch": branch,
"free_notes": results[branch]["free_notes"]["metrics"],
"pairs": results[branch]["pairs"],
},
flush=True,
)
save_json(
(
"outputs/evaluation/exercise-comparison.json"
if exercise
else "outputs/evaluation/comparison.json"
),
{
"results": results,
"lambda_selected_before_training": config["lambda_orpo"],
"human_preferences": False,
"quality_gain_required": False,
"checkpoint_selection": False,
"reference_models_loaded": 0,
"same_prompt_presentations_not_same_compute": True,
"exercise_sum_is_not_reference_ORPO": exercise,
"no_real_incident_action": True,
},
)
Isolate the effect of length
Normalization ablation
Common mistake: mixing analytical data and trained results in one unlabeled chart; separate sources.
Invariant: both variants are named and the final ORPO keeps the mean.
File src/orpo_sans_reference/exercise.py
from pathlib import Path
import torch
from .config import read_json, save_json, sha
from .loss import orpo_preference_loss
def main_artifacts():
names = [
"configs/minimum.yaml",
"outputs/evaluation/comparison.json",
"outputs/numeric-mechanism.json",
]
for branch in ["base", "sft", "orpo"]:
names += [
f"outputs/evaluation/pairs-{branch}.json",
f"outputs/inference/predictions-{branch}.json",
]
if branch != "base":
names += [
f"assets/checkpoints/{branch}-resume.safetensors",
f"outputs/learning/resume-{branch}.json",
]
return names
def preserve_main():
path = Path("outputs/exercise/protected.json")
if path.exists():
raise ValueError("Preserve existing exercise snapshot")
save_json(path, {name: sha(name) for name in main_artifacts()})
def compare_sum_mean_odds():
rows = []
for chosen_length in [4, 16, 64]:
for rejected_length in [4, 16, 64]:
for normalization in ["mean", "sum"]:
a = torch.full(
(chosen_length,), -1.0, dtype=torch.float64, requires_grad=True
)
b = torch.full(
(rejected_length,), -1.2, dtype=torch.float64, requires_grad=True
)
pa = a.mean() if normalization == "mean" else a.sum()
pb = b.mean() if normalization == "mean" else b.sum()
loss, margin = orpo_preference_loss(pa.reshape(1), pb.reshape(1))
ga, gb = torch.autograd.grad(loss.sum(), (a, b))
rows.append(
{
"normalization": normalization,
"chosen_length": chosen_length,
"rejected_length": rejected_length,
"chosen_token_logprob": -1.0,
"rejected_token_logprob": -1.2,
"chosen_score": float(pa.detach()),
"rejected_score": float(pb.detach()),
"log_odds_margin": float(margin.detach()[0]),
"preference_loss": float(loss.detach()[0]),
"chosen_token_gradients": ga.tolist(),
"rejected_token_gradients": gb.tolist(),
}
)
return rows
def run_length_ablation(config):
if config.get("normalization") != "sum":
raise ValueError("Exercise must identify the sum ablation")
protected = read_json("outputs/exercise/protected.json")
if protected != {name: sha(name) for name in main_artifacts()}:
raise ValueError("Exercise changed main weights/results/config")
actual = read_json("outputs/evaluation/pairs-orpo.json")
rescored = []
for row in actual["rows"]:
a = torch.tensor(
[row["scores"]["chosen"]["sum_logprob"]], dtype=torch.float32, device="cuda"
)
b = torch.tensor(
[row["scores"]["rejected"]["sum_logprob"]],
dtype=torch.float32,
device="cuda",
)
na = row["scores"]["chosen"]["response_tokens"]
nb = row["scores"]["rejected"]["response_tokens"]
mean, mean_margin = orpo_preference_loss(a / na, b / nb)
summed, sum_margin = orpo_preference_loss(a, b)
if (
float(mean[0]) != row["mean_odds_preference_loss"]
or float(mean_margin[0]) != row["mean_log_odds_margin"]
):
raise ValueError("Mean scoring changed in the exercise")
rescored.append(
{
"document_id": row["document_id"],
"chosen_tokens": na,
"rejected_tokens": nb,
"mean_log_odds_margin": float(mean_margin[0]),
"sum_log_odds_margin": float(sum_margin[0]),
"mean_preference_loss": float(mean[0]),
"sum_preference_loss": float(summed[0]),
"scope": "Same already scored main ORPO policy, formula ablation only",
}
)
branches = {
b: read_json(f"outputs/learning/train-{b}-continuous.json")
for b in ["sft", "orpo", "sum-ablation"]
}
initial = {r["initial_model_digest"] for r in branches.values()}
schedules = [
[
(r["document_ids"], r["chosen_token_denominator"], r["learning_rate"])
for r in run["history"]
]
for run in branches.values()
]
if (
len(initial) != 1
or not all(schedule == schedules[0] for schedule in schedules)
or any(run["updates"] != 210 for run in branches.values())
):
raise ValueError("Ablation initial model/example budgets differ")
result = read_json("outputs/evaluation/exercise-comparison.json")
if not result["exercise_sum_is_not_reference_ORPO"]:
raise ValueError("Sum variant must not be labelled reference ORPO")
save_json(
"outputs/exercise/odds-length.json",
{
"analytic_rows": compare_sum_mean_odds(),
"analytic_source": "Constructed constant token log probabilities, not generated incident notes",
"actual_main_policy_rescoring": rescored,
"actual_fresh_sum_training": True,
"same_base_and_prompt_order": True,
"updates_per_branch": 210,
"main_normalization_restored": "mean",
"main_artifacts_unchanged": protected
== {name: sha(name) for name in main_artifacts()},
"protected_sha256": protected,
"sum_variant_is_not_reference_ORPO": True,
"quality_gain_required": False,
},
)
Sources
ORPO, original paper, the authors' pinned repository, the pinned TRL control function, Qwen2.5-0.5B-Instruct. Local data, masks, and budgets do not reproduce these authors' published results.
Measured reference
See the r2 development results (RESULTATS_DEVELOPPEMENT.md, included in the ZIP) and their explicit provenance (OBSERVATIONS.json, included in the ZIP). The scientific code is identical in r3; only a caption is corrected.
