Una herramienta ficticia transforma hechos de incidentes en un informe breve que separa observación e hipótesis. Los pares ilustran una síntesis fiel y otra que añade una causa no establecida. La tarea no es médica ni controla equipos.
Compare SFT chosen-only y ORPO desde los mismos pesos instruct. Muestre NLL elegida, log-odds y calidad de resúmenes, con un ejemplo donde falla una fórmula numéricamente ingenua.
Descargar el proyecto completo, revisión r003
Primero el hecho, después el informe
IncidentNotesPrefs-v1 contiene 300 familias de cuatro pares, 1200 ejemplos. Las familias 0–209 dan 840 pares train; 210–254, 180 validation; 255–299, 180 test. Cada expediente nombra una unidad ficticia, estado stopped/running, presión conocida o no medida y causa confirmada o desconocida. Una nota antigua ofrece estado contradictorio y causa sugerida; el registro actual la sustituye explícitamente. No se propone acción sobre un equipo real.
La respuesta elegida expresa hechos actuales. El rechazo invierte una sola información: causa en dos formulaciones, presión en una y estado en una. Algunas causas están confirmadas: escribir siempre «desconocida» sería falso. Una respuesta cortés puede inventar una causa. Las fórmulas se alternan para que la elegida sea más corta, igual o más larga en palabras. Se auditan palabras y regímenes; los estratos finales usan tokens reales.
Todas las variantes de una familia permanecen en una partición. Los ID son distintos, pero hechos elementales y formas de lenguaje se comparten: esta separación no prueba generalización a incidentes o dominios reales. Las anotaciones son simuladas, sin retorno humano. Una auditoría por palabra no demuestra la ausencia de todo atajo semántico.
Una distribución sobre tokens
La base Qwen2.5-0.5B-Instruct se adquiere en la revisión 7ae557604adf67be50417f59c2c2f167def9a775. Sus 494 032 768 parámetros únicos son entrenables en FP32. Se controla arquitectura: 24 bloques, anchura 896, 14 cabezas Q y dos K/V, MLP 4864, vocabulario 151936 y embeddings/salida vinculados. Atención eager y TF32 desactivado; una tarjeta elegida explícitamente. No hay LoRA, cuantificación ni copia de referencia.
La plantilla del tokenizer reúne sistema, incidente y frontera de asistente. La respuesta se codifica por separado con un único EOS=151645 final. Este contrato no afirma equivalencia con retokenizar la cadena entera. Límites: 256 tokens de prompt, 128 de respuesta con EOS y total máximo 512. Un exceso genera un error identificado, nunca truncamiento de datos de aprendizaje.
Los logits en posición t−1 predicen el token t. Tras desplazar, solo labels de respuesta permanecen activos; prompt y padding valen −100. PAD=151643 y EOS son distintos y las posiciones reales determinan máscaras. El prompt no se supervisa directamente, pero condiciona cada puntuación de respuesta.
Para una respuesta de n tokens activos se calcula suma s de log-probabilidades y después ell=s/n. q=exp(ell) es la media geométrica de probabilidades de tokens, no la probabilidad conjunta de la respuesta. EOS entra en n. Se rechaza respuesta vacía. A diferencia de DPO, el objetivo ORPO usa ell, no s.
De probabilidades a cuotas
La cuota de q es q/(1−q). Su logaritmo es ell−log(1−exp(ell)). Para elegida y rechazada se restan las dos log-cuotas; una diferencia positiva favorece la elegida. La penalización por par es −logsigmoid(diferencia). La pérdida total añade supervisión: NLL elegida + lambda × media de penalizaciones de par.
Dos reducciones son importantes. La NLL es suma de todas las CE elegidas dividida por todos sus tokens activos. La preferencia es una media por par. En una acumulación de ocho pares de longitudes distintas, cada CE elegida comparte el denominador total de tokens de esos ocho pares, mientras cada penalización se divide entre ocho. Una media de medias de CE cambiaría el objetivo.
Lambda=0 debe recuperar pérdida y gradientes SFT con las mismas máscaras y reducciones. La rama SFT real no calcula rechazos; la prueba lambda=0 puede calcularlos para comprobar gradiente nulo. Respuestas idénticas dan penalización relativa log(2), sin señal relativa útil; la NLL elegida sigue presente. Los pares idénticos se rechazan en el corpus.
Evitar una resta imprecisa
Calcular directamente log(1−exp(ell)) puede dar log(0) cerca de cero: exp(ell) se redondea a 1. Usamos log1p(−exp(ell)) si ell < −log(2), y log(−expm1(ell)) en la otra rama. Solo se evalúa la rama útil; evaluar ambas y seleccionar con torch.where podría dejar un gradiente NaN de la rama inactiva.
Los dos lados se apilan antes de esta operación común. Esta simetría permite que gradientes opuestos de una misma entrada se anulen exactamente al reunirse, sin cambiar la fórmula ni relajar la prueba.
El dominio es explícito: FP32 o FP64, ell finito estrictamente negativo y magnitud al menos finfo.tiny. Se rechazan cero y magnitudes subnormales porque la log-cuota o derivada no es representable. No hay clamp silencioso. Las pruebas incluyen −1000, frontera −log(2), −1e−300 FP64 y −1e−30 FP32. Los fallos ingenuos se conservan con estado y null para números no finitos; nunca un resultado JSON falso.
Valor y gradientes del término de preferencia se comparan con odds_ratio_loss de TRL v0.23.1, commit 4529a1c8b1813480a85b02c9c8e7f75a29085d65, en dominio ordinario: tolerancia absoluta 1e−12 FP64 y 2e−6 FP32. Se aporta procedencia y licencia Apache-2.0; no se importa el trainer. La NLL enmascarada se compara por separado con CrossEntropyLoss de PyTorch.
Esta comparación no reproduce todo el trainer TRL: la versión inspeccionada reconstruye labels elegidos desde ID y atención, incluido el prompt. El TP utiliza explícitamente labels de respuesta sola. También se inspeccionó el repositorio original; sus máscaras y conversión BF16 final no se reutilizan. Se comparan fórmula y reducciones anunciadas, sin reclamar identidad de una infraestructura completa.
Tres aprendizajes y una base común
El mínimo compara base adquirida, SFT chosen-only y ORPO lambda=0,1. Las dos políticas entrenadas parten independientemente del mismo archivo instruct, sin SFT importado. Cada una ve los mismos 840 prompts durante dos épocas, en el mismo orden: 1680 presentaciones y 210 actualizaciones, microbatch de un par y acumulación de ocho.
AdamW usa lr=0,000002, weight_decay=0,01, clipping a 1, warm-up de 21 actualizaciones y descenso lineal hasta 10 %. La semilla es 47. Se controlan parámetros completos y gradientes FP32. La validación se mide tras cada época, sin seleccionar checkpoint ni lambda con test. ORPO procesa también respuestas rechazadas; igual exposición de prompts no implica igual presupuesto de tokens, forwards o tiempo.
Cada fase incluye interrupción tras una época, entrenamiento continuo de dos épocas y reanudación de la segunda. Los estados finales de modelo, AdamW, sampler, scheduler, RNG Python/NumPy/PyTorch CPU/CUDA e historiales deben ser exactamente iguales. Checkpoints completos se serializan localmente en gzip 1 sin pérdida, con tamaños registrados, y se publican por renombrado atómico. Su compresión no reduce VRAM ni garantiza tasa. Son fiables por producirse localmente; no sustituya un objeto Torch descargado. Los modelos finales también se guardan en Safetensors y se recargan estrictamente.
El ejercicio añade una tercera política, sum-ablation, con mismos pesos iniciales, orden de prompts, optimizador y actualizaciones. Su término de cuotas recibe SUMAS en vez de medias: es una ablación nombrada, no el ORPO de referencia. Los nueve estados completos de tres fases permanecen locales. Se distinguen duraciones de optimización, validación, comandos completos y picos medidos.
Evaluar candidatos y textos libres
Cada estado base/SFT/ORPO/sum-ablation puntúa los mismos 180 pares test. El informe publica NLL elegida por token, margen de log-cuotas, penalización y accuracy con medio crédito en empates. Resultados separados según longitud elegida más corta, igual o más larga en tokens. Para comparar políticas, esta evaluación siempre usa cuotas de medias, también con pesos sum-ablation; el nombre no altera la métrica en secreto.
Cada estado genera también 180 informes libres, solo con el incidente en prompt: greedy, un beam, máximo 128 tokens nuevos, EOS=151645. ID, texto bruto, motivo de parada, longitud y duración siguen disponibles. Una respuesta al límite falla aunque su inicio parezca completo. Ninguna salida se envía ni aplica a un sistema real.
La rúbrica es una gramática independiente de la generación. Reconoce cláusulas de unidad/estado, presión y causa. Toda frase no reconocida, campo repetido, etiqueta o parada al límite vuelve incompleta la nota. Una paráfrasis natural correcta fuera de gramática puede fallar. La puntuación no es juez general de fidelidad ni medida de satisfacción.
La exactitud exige cuatro campos correctos en TODOS los 180 casos. Causas conocidas y desconocidas tienen denominadores separados. Una causa reconocida distinta del registro se cuenta como no sustentada. Una abstención reconocida sobre causa conocida se cuenta aparte; campo ausente en prosa incomprendida no se llama automáticamente abstención. Una presión reconocida sin medida se señala. Frases no reconocidas siguen sin certificar, incluso si contadores parciales son cero. Los fallos nunca se retiran de denominadores.
Ejercicio sobre longitud
Las primeras líneas son analíticas: log-probabilidades constantes −1 para tokens elegidos y −1,2 para rechazados, con longitudes 4, 16 y 64. Medias, sumas, cuotas y gradientes se comparan en nueve parejas. Con medias, calidad media constante no depende de longitud; con sumas, una elegida larga puede perder ante una rechazada corta. No es una medida sobre textos de negocio.
Después, puntuaciones reales de 180 pares de política ORPO principal se someten a ambas fórmulas sin modificar pesos. Finalmente se entrena realmente sum-ablation y se evalúa en generación libre. Resultados separados de experiencia analítica y recálculo de fórmula. SHA-256 de modelos, resultados principales y configuración mínima se protegen; versión principal conserva medias.
Una pérdida estable numéricamente puede aprender preferencias malas. Un margen positivo entre dos candidatos no garantiza un texto nuevo bueno. Una sola semilla y patrones sintéticos limitan fuertemente conclusiones; no se exige mejora de ORPO sobre SFT.
Definir preferencias sin palabra reveladora
Observación y causalidad
Error frecuente: Todos los chosen dicen 'unconfirmed': añadir causas confirmadas y paráfrasis sin modificar test después de la ejecución.
Invariante: 840/180/180 pares, sin fuga de incidente y diversidad de motivos.
Archivo src/orpo_sans_reference/data.py
import hashlib, re
from pathlib import Path
from .config import save_json, sha
CAUSES = ["filter blockage", "power loss", "sensor fault"]
COURTESY = [
"This is the recorded incident note.",
"Further details may be documented separately.",
]
def split_incident_families(family):
return "train" if family < 210 else "validation" if family < 255 else "test"
def facts_for(family):
return {
"unit": f"P{family+1000}",
"state": "stopped" if family % 3 else "running",
"pressure": None if family % 4 < 2 else 10 + (family // 4) % 7,
"cause": None if family % 2 == 0 else CAUSES[(family // 2) % 3],
}
def incident_text(family, j, facts):
previous = "running" if facts["state"] == "stopped" else "stopped"
old_cause = CAUSES[(family // 2 + 1) % 3]
pressure = (
"not measured" if facts["pressure"] is None else str(facts["pressure"]) + " kPa"
)
cause = "not confirmed" if facts["cause"] is None else "confirmed " + facts["cause"]
return f"Incident N{family:03d}-{j}. Superseded note: state {previous}; suggested cause {old_cause}. Current verified record: unit {facts['unit']}; state {facts['state']}; pressure {pressure}; cause {cause}. Use only the current record."
def reply(facts, j, courtesy=False):
first = [
f"{facts['unit']} is {facts['state']}.",
f"Unit {facts['unit']} is {facts['state']}.",
f"The state of {facts['unit']} is {facts['state']}.",
f"The current state of {facts['unit']} is {facts['state']}.",
][j]
pressure = (
f"The pressure is {facts['pressure']} kPa."
if facts["pressure"] is not None
else ("Pressure was not measured." if j % 2 else "The pressure is unknown.")
)
cause = (
f"The confirmed cause is {facts['cause']}."
if facts["cause"] is not None
else (
"The cause remains unconfirmed."
if j % 2
else "The confirmed cause is not known."
)
)
clauses = [first, pressure, cause]
if courtesy:
clauses = COURTESY[:1] + clauses + COURTESY[1:]
return " ".join(clauses)
def audit_causal_labels(labels):
n = len(labels)
lengths = {"chosen_shorter": 0, "same_words": 0, "chosen_longer": 0}
known = sum(r["expected"]["cause"] is not None for r in labels)
features = []
words = sorted(
{
w
for r in labels
for side in ["chosen", "rejected"]
for w in re.findall("[a-z]+", r[side].lower())
}
)
for row in labels:
a, b = len(row["chosen"].split()), len(row["rejected"].split())
lengths[
"chosen_shorter" if a < b else "chosen_longer" if a > b else "same_words"
] += 1
for word in words:
a, b = [
sum(word in set(re.findall("[a-z]+", r[side].lower())) for r in labels)
for side in ["chosen", "rejected"]
]
accuracy = (a + n - b) / (2 * n)
features.append(
{
"word": word,
"chosen_presence": a,
"rejected_presence": b,
"best_binary_label_accuracy": max(accuracy, 1 - accuracy),
}
)
return {
"pairs": n,
"known_cause": known,
"unknown_cause": n - known,
"unknown_pressure": sum(r["expected"]["pressure"] is None for r in labels),
"word_length_regimes": lengths,
"best_length_preference_accuracy_half_ties": max(
(lengths["chosen_longer"] + 0.5 * lengths["same_words"]) / n,
(lengths["chosen_shorter"] + 0.5 * lengths["same_words"]) / n,
),
"word_features": features,
"maximum_word_accuracy": max(r["best_binary_label_accuracy"] for r in features),
"scope": "Original words, not every possible semantic/BPE shortcut",
}
def generate_incident_pairs():
pages = []
labels = []
for family in range(300):
facts = facts_for(family)
for j in range(4):
identifier = f"N{family:03d}-{j}"
text = incident_text(family, j, facts)
wrong = dict(facts)
reason = ["cause", "pressure", "state", "cause"][j]
if reason == "cause":
wrong["cause"] = (
CAUSES[(family // 2) % 3] if facts["cause"] is None else None
)
elif reason == "pressure":
wrong["pressure"] = (
10 + (family // 4) % 7 if facts["pressure"] is None else None
)
else:
wrong["state"] = "running" if facts["state"] == "stopped" else "stopped"
chosen = reply(facts, j, courtesy=j == 1)
rejected = reply(wrong, j, courtesy=j == 2)
if chosen == rejected:
raise ValueError("Preference pair is identical")
pages.append(
{
"document_id": identifier,
"family": family,
"formulation": j,
"split": split_incident_families(family),
"incident": text,
"record": facts,
"content_sha256": hashlib.sha256(text.encode()).hexdigest(),
}
)
labels.append(
{
"document_id": identifier,
"chosen": chosen,
"rejected": rejected,
"expected": facts,
"preference_reason": reason,
"simulated_preference": True,
}
)
if len({p["incident"] for p in pages}) != 1200:
raise ValueError("Duplicate incident")
return pages, labels
def prepare(config):
root = Path("data/generated")
if (root / "manifest.json").exists():
raise ValueError("Preserve existing corpus; use a new extraction")
pages, labels = generate_incident_pairs()
gold = {r["document_id"]: r for r in labels}
save_json(root / "input/pages.json", pages)
save_json(root / "labels/gold.json", labels)
counts = {}
audits = {}
for split in ["train", "validation", "test"]:
selected = [p for p in pages if p["split"] == split]
chosen = [gold[p["document_id"]] for p in selected]
counts[split] = {
"pairs": len(selected),
"families": len({p["family"] for p in selected}),
}
audits[split] = audit_causal_labels(chosen)
if split != "test":
save_json(root / f"{split}/pages.json", selected)
save_json(root / f"labels/{split}.json", chosen)
save_json(
"outputs/data-audit.json",
{
"dataset": "IncidentNotesPrefs-v1",
"rights": "CC0-1.0",
"counts": counts,
"shortcuts": audits,
"human_preferences": False,
"shared_semantic_and_language_primitives": True,
"current_record_supersedes_previous_note": True,
"reference_model_or_cache": False,
},
)
save_json(
root / "manifest.json",
{
"dataset": "IncidentNotesPrefs-v1",
"seed": config["seed"],
"counts": counts,
"files": {
str(f.relative_to(root)): sha(f) for f in root.rglob("*") if f.is_file()
},
},
)
return counts
Calcular las medias de tokens
Máscaras, longitud y puntuación q
Error frecuente: Un batch medio pondera cada frase por error: nombrar por separado media de tokens y media de pares.
Invariante: PAD invariantes, respuestas no vacías y reducción controlada.
Archivo src/orpo_sans_reference/scores.py
import torch
from .config import save_json
SYSTEM = "Write an incident note using only the current verified record. State the unit and its current state, the pressure if measured, and the confirmed cause if known. Distinguish observations from causal conclusions. A superseded suggestion is not a current fact. Do not add facts, actions or explanations."
def format_prompt(tokenizer, incident, version):
if version != "incident-v1":
raise ValueError("Unknown incident template")
messages = [
{"role": "system", "content": SYSTEM},
{"role": "user", "content": incident},
]
return messages, tokenizer.apply_chat_template(
messages, tokenize=True, add_generation_prompt=True
)
def completion_row(tokenizer, incident, response, config, identifier, version=None):
version = version or config["template_version"]
messages, prompt = format_prompt(tokenizer, incident, version)
target = tokenizer.encode(response, add_special_tokens=False) + [151645]
if (
not prompt
or len(prompt) > config["max_prompt_tokens"]
or not 1 < len(target) <= config["max_response_tokens_including_eos"]
or len(prompt) + len(target) > config["context"]
):
raise ValueError(
identifier + ": declared token budget exceeded; do not truncate"
)
return {
"document_id": identifier,
"messages": messages,
"input_ids": prompt + target,
"labels": [-100] * len(prompt) + target,
"prompt_tokens": len(prompt),
"response_tokens": len(target),
"template_version": version,
}
def paired_completion_batch(tokenizer, page, label, config, version=None):
chosen = completion_row(
tokenizer,
page["incident"],
label["chosen"],
config,
page["document_id"],
version,
)
rejected = completion_row(
tokenizer,
page["incident"],
label["rejected"],
config,
page["document_id"],
version,
)
if (
chosen["input_ids"] == rejected["input_ids"]
and chosen["labels"] == rejected["labels"]
):
raise ValueError("Identical completions are not an informative preference")
return {
"document_id": page["document_id"],
"chosen": chosen,
"rejected": rejected,
"response_length_delta": chosen["response_tokens"]
- rejected["response_tokens"],
}
def collate(rows, device="cuda", pad_to=None):
if not rows:
raise ValueError("Empty completion batch")
width = max(len(r["input_ids"]) for r in rows) if pad_to is None else pad_to
ids = torch.full((len(rows), width), 151643, dtype=torch.long, device=device)
mask = torch.zeros_like(ids, dtype=torch.bool)
labels = torch.full_like(ids, -100)
for i, row in enumerate(rows):
n = len(row["input_ids"])
if n > width:
raise ValueError("Padding width would truncate a response")
ids[i, :n] = torch.tensor(row["input_ids"], device=device)
mask[i, :n] = True
labels[i, :n] = torch.tensor(row["labels"], device=device)
return ids, mask, labels
def sequence_logprob(logits, labels):
if logits.shape[:2] != labels.shape:
raise ValueError("Logits and labels shape mismatch")
active = labels[:, 1:] != -100
counts = active.sum(dim=1)
if (counts == 0).any():
raise ValueError("Every sequence needs an active response target")
selected = logits[:, :-1][active]
targets = labels[:, 1:][active]
values = (
torch.log_softmax(selected, dim=-1)
.gather(-1, targets.unsqueeze(-1))
.squeeze(-1)
)
sums = torch.stack([v.sum() for v in values.split(counts.tolist())])
return sums, counts, values, selected, targets
def model_scores(model, rows):
ids, mask, labels = collate(rows)
logits = model(
input_ids=ids, attention_mask=mask, use_cache=False, return_dict=True
).logits
return sequence_logprob(logits.float(), labels)
def mean_completion_logprob(logits, labels):
sums, counts, values, selected, targets = sequence_logprob(logits, labels)
return sums / counts, counts, values, selected, targets
def chosen_token_nll(logits, labels):
sums, counts, _, _, _ = sequence_logprob(logits, labels)
return -sums.sum() / counts.sum()
def inspect_pair_lengths(config):
from .model import load_tokenizer
from .data import generate_incident_pairs
tokenizer = load_tokenizer()
pages, labels = generate_incident_pairs()
pairs = [
paired_completion_batch(tokenizer, pages[i], labels[i], config)
for i in range(4)
]
save_json(
"outputs/tokenization-mechanism.json",
{
"pairs": pairs,
"preference_reduction": "mean_response_logprob_including_EOS",
"nll_reduction": "all_chosen_active_tokens",
"prompt_and_padding_excluded": True,
},
)
Codificar las cuotas de forma estable
log1mexp y sigmoide
Error frecuente: Un clamp silencioso crea una meseta de gradiente: hacerlo explícito y probar su zona de acción.
Invariante: Pruebas FP64/FP32 y concordancia de referencia según tolerancias prefijadas.
Archivo src/orpo_sans_reference/loss.py
import math
from types import SimpleNamespace
import torch
from .config import save_json
def log1mexp(a):
if a.dtype not in [torch.float32, torch.float64] or not torch.isfinite(a).all():
raise ValueError("Use finite FP32/FP64 log probabilities")
if (a >= 0).any() or (a.abs() < torch.finfo(a.dtype).tiny).any():
raise ValueError(
"Log probability must be strictly negative with a representable finite derivative; no silent clamp"
)
# Compute only the selected branch: torch.where on two evaluated formulas
# could propagate NaN gradients from its inactive near-zero branch.
far = a < -math.log(2)
result = torch.empty_like(a)
result[far] = torch.log1p(-torch.exp(a[far]))
result[~far] = torch.log(-torch.expm1(a[~far]))
return result
def orpo_preference_loss(chosen_logps, rejected_logps):
if chosen_logps.shape != rejected_logps.shape or not chosen_logps.numel():
raise ValueError("Matching nonempty paired score vectors required")
paired = torch.stack([chosen_logps, rejected_logps])
odds = paired - log1mexp(paired)
margin = odds[0] - odds[1]
return -torch.nn.functional.logsigmoid(margin), margin
def orpo_total_loss(
chosen_sums,
rejected_sums,
chosen_counts,
rejected_counts,
coefficient=0.1,
normalization="mean",
):
if (
coefficient < 0
or not math.isfinite(coefficient)
or normalization not in ["mean", "sum"]
):
raise ValueError("Explicit nonnegative lambda and named normalization required")
if (chosen_counts <= 0).any() or (rejected_counts <= 0).any():
raise ValueError("Responses must have active tokens")
chosen = chosen_sums / chosen_counts if normalization == "mean" else chosen_sums
rejected = (
rejected_sums / rejected_counts if normalization == "mean" else rejected_sums
)
nll = -chosen_sums.sum() / chosen_counts.sum()
preference, margins = orpo_preference_loss(chosen, rejected)
return nll + coefficient * preference.mean(), nll, preference, margins
def numeric_mechanism(config):
from .reference_loss import odds_ratio_loss
reference_rows = []
for dtype, tolerance in [(torch.float64, 1e-12), (torch.float32, 2e-6)]:
chosen = torch.tensor([-1.0, -3.0, -0.1], dtype=dtype, requires_grad=True)
rejected = torch.tensor([-2.0, -1.0, -0.5], dtype=dtype, requires_grad=True)
loss, margin = orpo_preference_loss(chosen, rejected)
control = SimpleNamespace(
beta=config["lambda_orpo"], accelerator=SimpleNamespace(device="cpu")
)
reference = -odds_ratio_loss(control, chosen, rejected)[0]
a = torch.autograd.grad(
config["lambda_orpo"] * loss.mean(), (chosen, rejected), retain_graph=True
)
b = torch.autograd.grad(reference.mean(), (chosen, rejected))
checks = [
torch.allclose(
config["lambda_orpo"] * loss, reference, atol=tolerance, rtol=0
),
*(torch.allclose(x, y, atol=tolerance, rtol=0) for x, y in zip(a, b)),
]
if not all(checks):
raise ValueError("Pinned ORPO reference value/gradient comparison failed")
reference_rows.append(
{
"dtype": str(dtype),
"chosen_mean_logps": chosen.tolist(),
"rejected_mean_logps": rejected.tolist(),
"preference_losses": loss.tolist(),
"log_odds_margins": margin.tolist(),
"weighted_reference_losses": reference.tolist(),
"chosen_gradient": a[0].tolist(),
"rejected_gradient": a[1].tolist(),
"reference_chosen_gradient": b[0].tolist(),
"reference_rejected_gradient": b[1].tolist(),
"lambda": config["lambda_orpo"],
"atol": tolerance,
}
)
stability = []
for dtype, values in [
(
torch.float64,
[-1000.0, -100.0, -1.0, -math.log(2), -0.1, -1e-10, -1e-100, -1e-300],
),
(
torch.float32,
[-1000.0, -100.0, -1.0, -math.log(2), -0.1, -1e-5, -1e-10, -1e-30],
),
]:
x = torch.tensor(values, dtype=dtype, requires_grad=True)
stable = log1mexp(x)
gradient = torch.autograd.grad(stable.sum(), x)[0]
naive = torch.log(1 - torch.exp(x))
naive_grad = torch.autograd.grad(naive.sum(), x)[0]
if not torch.isfinite(stable).all() or not torch.isfinite(gradient).all():
raise ValueError("Stable numerical branch is not finite")
for a, s, g, n, ng in zip(
x.tolist(),
stable.tolist(),
gradient.tolist(),
naive.tolist(),
naive_grad.tolist(),
):
expected = (
math.log1p(-math.exp(a))
if a < -math.log(2)
else math.log(-math.expm1(a))
)
stability.append(
{
"dtype": str(dtype),
"log_probability": a,
"stable": s,
"stable_gradient": g,
"analytic_float64_value": expected,
"absolute_error": abs(s - expected),
"naive": n if math.isfinite(n) else None,
"naive_gradient": ng if math.isfinite(ng) else None,
"naive_finite": math.isfinite(n) and math.isfinite(ng),
}
)
save_json(
"outputs/numeric-mechanism.json",
{
"reference": reference_rows,
"stability": stability,
"data_source": "Controlled analytical log probabilities, not incident quality measurements",
"zero_or_subnormal_magnitude_policy": "Explicit rejection, no clamp",
"reference_model_loaded": False,
},
)
Entrenar SFT y ORPO desde la misma base
Dos objetivos comparables
Error frecuente: Copiar una configuración DPO importa una ref no utilizada: auditar objetos y asignaciones.
Invariante: Actualizaciones reales, checkpoints y reanudación verificados; ningún modelo ref creado.
Archivo src/orpo_sans_reference/train.py
import math, time
from pathlib import Path
import torch
from .config import read_json, save_json, sha, seed_all, rng_state, restore_rng
from .model import (
load_base,
load_tokenizer,
parameter_manifest,
manifest_digest,
save_learned,
)
from .scores import paired_completion_batch, model_scores
from .loss import orpo_preference_loss
from .state import save_resume_state, load_resume_state, equal
BRANCHES = ["sft", "orpo", "sum-ablation"]
def learning_rate(config, step, total=210):
peak = config["learning_rate"]
warmup = max(1, int(total * config["warmup_ratio"]))
if step < warmup:
return peak * (step + 1) / warmup
return peak * (
1
- min(1.0, (step - warmup) / max(1, total - warmup - 1))
* (1 - config["minimum_lr_ratio"])
)
def encoded_split(config, tokenizer, split):
pages = read_json(f"data/generated/{split}/pages.json")
gold = {
r["document_id"]: r for r in read_json(f"data/generated/labels/{split}.json")
}
if any(p["split"] != split for p in pages):
raise ValueError("Family partition differs")
return [
paired_completion_batch(tokenizer, p, gold[p["document_id"]], config)
for p in pages
]
@torch.no_grad()
def validate_policy(model, pairs):
model.eval()
rows = []
total_nll = 0.0
total_tokens = 0
wins = 0.0
for pair in pairs:
chosen, nc, *_ = model_scores(model, [pair["chosen"]])
rejected, nr, *_ = model_scores(model, [pair["rejected"]])
a = chosen / nc
b = rejected / nr
preference, margin = orpo_preference_loss(a, b)
total_nll -= float(chosen[0])
total_tokens += int(nc[0])
wins += 1 if float(margin[0]) > 0 else 0.5 if float(margin[0]) == 0 else 0
rows.append(
{
"document_id": pair["document_id"],
"chosen_sum": float(chosen[0]),
"rejected_sum": float(rejected[0]),
"chosen_tokens": int(nc[0]),
"rejected_tokens": int(nr[0]),
"chosen_mean": float(a[0]),
"rejected_mean": float(b[0]),
"mean_log_odds_margin": float(margin[0]),
"mean_odds_preference_loss": float(preference[0]),
}
)
model.train()
return {
"pairs": len(pairs),
"chosen_tokens": total_tokens,
"chosen_nll_nats_per_token": total_nll / total_tokens,
"mean_odds_accuracy_half_ties": wins / len(pairs),
"rows": rows,
"used_for_checkpoint_selection": False,
"evaluation_normalization": "mean_even_for_sum_ablation",
}
def train_policy(config, branch="orpo", mode="continuous"):
if branch not in BRANCHES or mode not in ["interrupt", "continuous", "resume"]:
raise ValueError("Unknown training branch/mode")
normalization = "sum" if branch == "sum-ablation" else "mean"
if config.get("normalization", "mean") != normalization:
raise ValueError("Sum ablation requires its explicit exercise config")
if (
Path("data/generated/input").exists()
or Path("data/generated/labels/gold.json").exists()
or Path("assets/reference").exists()
or Path("outputs/reference-cache").exists()
or Path("outputs/inference").exists()
):
raise ValueError(
"Learning must not mount heldout outputs, labels or any reference model/cache"
)
seed_all(config["seed"])
tokenizer = load_tokenizer()
rows = encoded_split(config, tokenizer, "train")
validation = encoded_split(config, tokenizer, "validation")
if len(rows) != 840 or len(validation) != 180:
raise ValueError("Frozen population differs")
model = load_base().cuda()
parameters = list(model.parameters())
initial = parameter_manifest(model)
initial_digest = manifest_digest(initial)
if (
sum(p.numel() for p in parameters) != 494032768
or any(not p.requires_grad or p.dtype != torch.float32 for p in parameters)
or len({id(p) for p in parameters}) != len(parameters)
):
raise ValueError("All unique FP32 policy parameters must be optimized")
identity = {
"branch": branch,
"normalization": normalization,
"lambda": 0.0 if branch == "sft" else config["lambda_orpo"],
"config": config,
"initial_model_digest": initial_digest,
"model_sha256": sha("assets/model/model.safetensors"),
"tokenizer_sha256": sha("assets/model/tokenizer.json"),
"implementation_sha256": {
name: sha("src/orpo_sans_reference/" + name)
for name in ["model.py", "scores.py", "loss.py", "train.py", "state.py"]
},
"data_sha256": {
name: sha("data/generated/" + name)
for name in [
"train/pages.json",
"labels/train.json",
"validation/pages.json",
"labels/validation.json",
]
},
"reference_models": 0,
"reference_caches": 0,
}
optimizer = torch.optim.AdamW(
parameters, lr=config["learning_rate"], weight_decay=config["weight_decay"]
)
sampler = torch.Generator().manual_seed(config["seed"] + 100)
per_epoch = math.ceil(len(rows) / config["accumulation_pairs"])
total = per_epoch * config["epochs"]
root = Path("assets/checkpoints")
destination = root / f"{branch}-{mode}.pt.gz"
updates = start = 0
history = []
validation_history = []
if destination.exists():
raise ValueError("Preserve existing local checkpoint")
if mode == "resume":
state = load_resume_state(root / f"{branch}-interrupt.pt.gz")
if state["identity"] != identity:
raise ValueError("Resume identity changed, including lambda/normalization")
model.load_state_dict(state["model"], strict=True)
optimizer.load_state_dict(state["optimizer"])
sampler.set_state(state["sampler"])
restore_rng(state["rng"])
updates = state["updates"]
start = state["epoch"]
history = state["history"]
validation_history = state["validation_history"]
if state["scheduler"] != {
"next_update": updates,
"next_lr": learning_rate(config, updates, total),
}:
raise ValueError("Scheduler identity changed")
del state
stop = config["interrupt_epoch"] if mode == "interrupt" else config["epochs"]
model.train()
torch.cuda.reset_peak_memory_stats()
torch.cuda.synchronize()
begun = time.perf_counter()
optimization_seconds = 0.0
for epoch in range(start, stop):
order = torch.randperm(len(rows), generator=sampler).tolist()
torch.cuda.synchronize()
epoch_begun = time.perf_counter()
for offset in range(0, len(rows), config["accumulation_pairs"]):
selected = order[offset : offset + config["accumulation_pairs"]]
chosen_denominator = sum(
rows[i]["chosen"]["response_tokens"] for i in selected
)
lr = learning_rate(config, updates, total)
for group in optimizer.param_groups:
group["lr"] = lr
optimizer.zero_grad(set_to_none=True)
details = []
nll_numerator = preference_numerator = 0.0
policy_tokens = 0
for index in selected:
pair = rows[index]
chosen, nc, *_ = model_scores(model, [pair["chosen"]])
nll = -chosen.sum()
contribution = nll / chosen_denominator
policy_tokens += int(nc[0])
detail = {
"document_id": pair["document_id"],
"chosen_sum": float(chosen.detach()[0]),
"chosen_tokens": int(nc[0]),
"rejected_sum": None,
"rejected_tokens": 0,
"preference_loss": None,
"log_odds_margin": None,
"preference_chosen_logp": None,
"preference_rejected_logp": None,
}
if branch != "sft":
rejected, nr, *_ = model_scores(model, [pair["rejected"]])
a = chosen / nc if normalization == "mean" else chosen
b = rejected / nr if normalization == "mean" else rejected
preference, margin = orpo_preference_loss(a, b)
contribution = contribution + config[
"lambda_orpo"
] * preference.sum() / len(selected)
preference_numerator += float(preference.detach()[0])
policy_tokens += int(nr[0])
detail.update(
rejected_sum=float(rejected.detach()[0]),
rejected_tokens=int(nr[0]),
preference_loss=float(preference.detach()[0]),
log_odds_margin=float(margin.detach()[0]),
preference_chosen_logp=float(a.detach()[0]),
preference_rejected_logp=float(b.detach()[0]),
)
if not torch.isfinite(contribution):
raise ValueError("Nonfinite objective")
contribution.backward()
nll_numerator += float(nll.detach())
details.append(detail)
norm = torch.nn.utils.clip_grad_norm_(
parameters, config["gradient_clip"], error_if_nonfinite=True
)
if any(p.grad is None or p.grad.dtype != torch.float32 for p in parameters):
raise ValueError("All policy gradients must exist in FP32")
optimizer.step()
updates += 1
nll = nll_numerator / chosen_denominator
preference = (
preference_numerator / len(selected) if branch != "sft" else 0.0
)
history.append(
{
"update": updates,
"epoch": epoch + 1,
"document_ids": [rows[i]["document_id"] for i in selected],
"chosen_token_denominator": chosen_denominator,
"preference_pair_denominator": (
len(selected) if branch != "sft" else 0
),
"policy_response_tokens": policy_tokens,
"chosen_nll": nll,
"preference_loss": preference,
"total_loss": nll + identity["lambda"] * preference,
"learning_rate": lr,
"gradient_norm_before_clip": float(norm),
"pairs": details,
}
)
if updates % 21 == 0:
print(
{k: v for k, v in history[-1].items() if k != "pairs"}, flush=True
)
torch.cuda.synchronize()
optimization_seconds += time.perf_counter() - epoch_begun
observed = validate_policy(model, validation)
validation_history.append({"epoch": epoch + 1, **observed})
print(
{
"branch": branch,
"epoch": epoch + 1,
**{k: v for k, v in observed.items() if k != "rows"},
},
flush=True,
)
torch.cuda.synchronize()
elapsed = time.perf_counter() - begun
peak = torch.cuda.max_memory_allocated()
final = parameter_manifest(model)
final_digest = manifest_digest(final)
if final_digest == initial_digest:
raise ValueError("No real learned parameter change")
state = {
"identity": identity,
"epoch": stop,
"updates": updates,
"model": model.state_dict(),
"optimizer": optimizer.state_dict(),
"sampler": sampler.get_state(),
"scheduler": {
"next_update": updates,
"next_lr": learning_rate(config, updates, total),
},
"rng": rng_state(),
"history": history,
"validation_history": validation_history,
}
compression = save_resume_state(destination, state)
if mode == "resume":
save_learned(model, root / f"{branch}-resume.safetensors", identity)
save_json(
f"outputs/learning/train-{branch}-{mode}.json",
{
"branch": branch,
"mode": mode,
"identity": identity,
"epochs": stop,
"updates": updates,
"history": history,
"validation_history": validation_history,
"tokenization": rows,
"initial_parameters": initial,
"final_parameters": final,
"initial_model_digest": initial_digest,
"final_model_digest": final_digest,
"trainable_parameters": 494032768,
"checkpoint_sha256": sha(destination),
"checkpoint_bytes": destination.stat().st_size,
"checkpoint_compression": compression,
"optimization_seconds_this_process": optimization_seconds,
"training_and_validation_seconds_this_process": elapsed,
"max_torch_allocated_bytes": peak,
"test_visible": False,
"reference_models_loaded": 0,
"policy_models_loaded": 1,
},
)
def verify_resume(branch):
root = Path("assets/checkpoints")
a = load_resume_state(root / f"{branch}-continuous.pt.gz")
b = load_resume_state(root / f"{branch}-resume.pt.gz")
checks = {k: equal(a[k], b[k]) for k in a}
if a.keys() != b.keys() or not all(checks.values()):
raise ValueError("Continuous/resumed full states differ: " + str(checks))
save_json(
f"outputs/learning/resume-{branch}.json",
{
"status": "PASS",
"all_state_exact": True,
"checks": checks,
"boundary": "Full epoch; model/AdamW/scheduler/sampler/RNG/history compared exactly after lossless decompression",
},
)
Observar causas inventadas y abstenciones
Calidad de generación libre
Error frecuente: Evaluar solo causas desconocidas recompensa la abstención sistemática: conservar ambos estratos.
Invariante: 180 salidas por rama, causas conocidas/desconocidas presentadas por separado.
Archivo src/orpo_sans_reference/evaluate.py
import math, re, time
from pathlib import Path
import torch
from transformers import GenerationConfig
from .config import read_json, save_json, sha, seed_all
from .model import (
load_base,
load_tokenizer,
reload_learned,
parameter_manifest,
manifest_digest,
)
from .scores import format_prompt, paired_completion_batch, model_scores
from .loss import orpo_preference_loss
from .data import COURTESY
def parse_note(text, finish):
sentences = [
s.strip() for s in re.split(r"(?<=[.!?])\s+", text.strip()) if s.strip()
]
fields = {}
unknown = []
duplicates = []
for sentence in sentences:
if sentence in COURTESY:
continue
value = sentence.rstrip(".!?").strip()
items = None
match = re.fullmatch(r"(?:Unit )?(P\d{4}) is (stopped|running)", value, re.I)
if match:
items = {"unit": match[1].upper(), "state": match[2].lower()}
else:
match = re.fullmatch(
r"The (?:current )?state of (P\d{4}) is (stopped|running)", value, re.I
)
if match:
items = {"unit": match[1].upper(), "state": match[2].lower()}
else:
match = re.fullmatch(r"The pressure is (\d{1,4}) kPa", value, re.I)
if match:
items = {"pressure": int(match[1])}
elif value.lower() in [
"pressure was not measured",
"the pressure is unknown",
]:
items = {"pressure": None}
elif value.lower() in [
"the cause remains unconfirmed",
"no cause has been confirmed",
"the confirmed cause is not known",
]:
items = {"cause": None}
else:
match = re.fullmatch(
r"The confirmed cause is (filter blockage|power loss|sensor fault)",
value,
re.I,
)
if match:
items = {"cause": match[1].lower()}
if items is None:
unknown.append(sentence)
else:
for key, item in items.items():
if key in fields:
duplicates.append(key)
else:
fields[key] = item
covered = bool(sentences) and not unknown and not duplicates
return {
"recognized": fields,
"unrecognized_sentences": unknown,
"duplicate_fields": duplicates,
"all_text_covered": covered,
"complete": finish == "eos"
and covered
and set(fields) == {"unit", "state", "pressure", "cause"},
"truncated": finish != "eos",
}
def score_causal_claims(rows, gold):
metrics = {
"documents": len(rows),
"complete_controlled_notes": 0,
"exact_facts": 0,
"truncated": 0,
"unrecognized_prose": 0,
"known_cause_cases": 0,
"unknown_cause_cases": 0,
"known_cause_correct": 0,
"unknown_cause_correct": 0,
"recognized_unsupported_cause": 0,
"recognized_known_cause_abstention": 0,
"unknown_pressure_cases": 0,
"recognized_invented_pressure": 0,
}
fields = dict.fromkeys(["unit", "state", "pressure", "cause"], 0)
details = []
for row in rows:
expected = gold[row["document_id"]]["expected"]
parsed = parse_note(row["text"], row["finish_reason"])
if row["parsed"] != parsed:
raise ValueError("Stored parser result differs")
actual = parsed["recognized"]
valid = parsed["complete"]
correct = {k: valid and actual.get(k) == v for k, v in expected.items()}
exact = all(correct.values())
metrics["complete_controlled_notes"] += valid
metrics["exact_facts"] += exact
metrics["truncated"] += parsed["truncated"]
metrics["unrecognized_prose"] += bool(parsed["unrecognized_sentences"])
for key, value in correct.items():
fields[key] += value
known = expected["cause"] is not None
metrics["known_cause_cases" if known else "unknown_cause_cases"] += 1
metrics["known_cause_correct" if known else "unknown_cause_correct"] += correct[
"cause"
]
metrics["recognized_unsupported_cause"] += (
actual.get("cause") is not None and actual["cause"] != expected["cause"]
)
metrics["recognized_known_cause_abstention"] += (
known and "cause" in actual and actual["cause"] is None
)
if expected["pressure"] is None:
metrics["unknown_pressure_cases"] += 1
metrics["recognized_invented_pressure"] += (
actual.get("pressure") is not None
)
details.append(
{
"document_id": row["document_id"],
"expected": expected,
"recognized": actual,
"complete": valid,
"fields_correct": correct,
"exact": exact,
"unrecognized_sentences": parsed["unrecognized_sentences"],
}
)
n = len(rows)
metrics.update(
exact_accuracy_over_all_cases=metrics["exact_facts"] / n,
field_correct_over_all_cases=fields,
field_accuracy_over_all_cases={k: v / n for k, v in fields.items()},
known_cause_accuracy=metrics["known_cause_correct"]
/ metrics["known_cause_cases"],
unknown_cause_accuracy=metrics["unknown_cause_correct"]
/ metrics["unknown_cause_cases"],
recognized_known_cause_abstention_rate=metrics[
"recognized_known_cause_abstention"
]
/ metrics["known_cause_cases"],
mean_generated_tokens=math.fsum(r["output_tokens"] for r in rows) / n,
)
return {
"metrics": metrics,
"rows": details,
"unrecognized_prose_is_not_certified_faithful": True,
}
def pair_metrics(rows):
result = {}
for group in ["all", "chosen_shorter", "same_tokens", "chosen_longer"]:
selected = (
rows
if group == "all"
else [
r
for r in rows
if (
"chosen_shorter"
if r["response_length_delta"] < 0
else (
"chosen_longer"
if r["response_length_delta"] > 0
else "same_tokens"
)
)
== group
]
)
n = len(selected)
count = sum(r["scores"]["chosen"]["response_tokens"] for r in selected)
result[group] = {
"pairs": n,
"chosen_response_tokens": count,
"chosen_nll_nats_per_token": (
-math.fsum(r["scores"]["chosen"]["sum_logprob"] for r in selected)
/ count
if count
else None
),
"mean_log_odds_margin": (
math.fsum(r["mean_log_odds_margin"] for r in selected) / n
if n
else None
),
"mean_odds_preference_accuracy_half_ties": (
sum(
(
1
if r["mean_log_odds_margin"] > 0
else 0.5 if r["mean_log_odds_margin"] == 0 else 0
)
for r in selected
)
/ n
if n
else None
),
"mean_odds_preference_loss": (
math.fsum(r["mean_odds_preference_loss"] for r in selected) / n
if n
else None
),
}
return result
def load_policy(branch):
if branch not in ["base", "sft", "orpo", "sum-ablation"]:
raise ValueError("Unknown evaluation branch")
path = Path(
"assets/model/model.safetensors"
if branch == "base"
else f"assets/checkpoints/{branch}-resume.safetensors"
)
model = load_base() if branch == "base" else reload_learned(path)[0]
digest = manifest_digest(parameter_manifest(model))
for parameter in model.parameters():
parameter.requires_grad_(False)
return model.cuda().eval(), {
"branch": branch,
"file_sha256": sha(path),
"model_digest": digest,
"tokenizer_sha256": sha("assets/model/tokenizer.json"),
"reference_models_loaded": 0,
}
@torch.no_grad()
def score_pairs(config, branch):
seed_all(config["seed"])
model, identity = load_policy(branch)
tokenizer = load_tokenizer()
pages = [
p for p in read_json("data/generated/input/pages.json") if p["split"] == "test"
]
gold = {g["document_id"]: g for g in read_json("data/generated/labels/gold.json")}
rows = []
probes = []
torch.cuda.reset_peak_memory_stats()
torch.cuda.synchronize()
begun = time.perf_counter()
for index, page in enumerate(pages):
pair = paired_completion_batch(
tokenizer, page, gold[page["document_id"]], config
)
scores = {}
mean_tensors = {}
for side in ["chosen", "rejected"]:
sums, counts, values, logits, targets = model_scores(model, [pair[side]])
mean_tensors[side] = sums / counts
scores[side] = {
"sum_logprob": float(sums[0]),
"mean_logprob": float(mean_tensors[side][0]),
"response_tokens": int(counts[0]),
"token_logprobs": values.cpu().tolist(),
"targets": targets.cpu().tolist(),
}
if index == 0:
probes.append(
{
"document_id": page["document_id"],
"side": side,
"response_position": 0,
"target": int(targets[0]),
"logits": logits[0].cpu().tolist(),
"logprob": float(values[0]),
}
)
loss, margin = orpo_preference_loss(
mean_tensors["chosen"], mean_tensors["rejected"]
)
rows.append(
{
"document_id": page["document_id"],
"pair": pair,
"scores": scores,
"response_length_delta": pair["response_length_delta"],
"mean_log_odds_margin": float(margin[0]),
"mean_odds_preference_loss": float(loss[0]),
}
)
torch.cuda.synchronize()
save_json(
f"outputs/evaluation/pairs-{branch}.json",
{
"branch": branch,
"identity": identity,
"rows": rows,
"probes": probes,
"metrics": pair_metrics(rows),
"seconds": time.perf_counter() - begun,
"max_torch_allocated_bytes": torch.cuda.max_memory_allocated(),
"evaluation_normalization": "mean_even_for_sum_ablation",
"test_selection": False,
},
)
@torch.no_grad()
def generate_notes(config, branch):
if (
Path("data/generated/labels").exists()
or Path("assets/reference").exists()
or Path("outputs/reference-cache").exists()
):
raise ValueError("Blind generation must not mount labels or a reference")
seed_all(config["seed"])
model, identity = load_policy(branch)
tokenizer = load_tokenizer()
pages = [
p for p in read_json("data/generated/input/pages.json") if p["split"] == "test"
]
generation = GenerationConfig(
do_sample=False,
num_beams=1,
max_new_tokens=config["max_new_tokens"],
use_cache=True,
eos_token_id=151645,
pad_token_id=151643,
bos_token_id=151643,
)
rows = []
torch.cuda.reset_peak_memory_stats()
torch.cuda.synchronize()
begun = time.perf_counter()
for index, page in enumerate(pages):
messages, ids = format_prompt(
tokenizer, page["incident"], config["template_version"]
)
if not 0 < len(ids) <= config["max_prompt_tokens"]:
raise ValueError("No silent prompt truncation")
tensor = torch.tensor([ids], device="cuda")
torch.cuda.synchronize()
started = time.perf_counter()
output = (
model.generate(
input_ids=tensor,
attention_mask=torch.ones_like(tensor),
generation_config=generation,
)[0, len(ids) :]
.cpu()
.tolist()
)
torch.cuda.synchronize()
finish = "eos" if output and output[-1] == 151645 else "length"
text = tokenizer.decode(
output[:-1] if finish == "eos" else output,
skip_special_tokens=False,
clean_up_tokenization_spaces=False,
)
rows.append(
{
"document_id": page["document_id"],
"messages": messages,
"input_ids": ids,
"output_ids": output,
"text": text,
"finish_reason": finish,
"parsed": parse_note(text, finish),
"input_tokens": len(ids),
"output_tokens": len(output),
"seconds": time.perf_counter() - started,
}
)
if (index + 1) % 30 == 0:
print({"branch": branch, "generated_notes": index + 1}, flush=True)
torch.cuda.synchronize()
save_json(
f"outputs/inference/predictions-{branch}.json",
{
"branch": branch,
"identity": identity,
"generation": generation.to_dict(),
"input_sha256": sha("data/generated/input/pages.json"),
"rows": rows,
"seconds": time.perf_counter() - begun,
"max_torch_allocated_bytes": torch.cuda.max_memory_allocated(),
"gold_visible": False,
"reference_models_loaded": 0,
"training": False,
},
)
def evaluate_notes(config, exercise=False):
gold = {g["document_id"]: g for g in read_json("data/generated/labels/gold.json")}
test_ids = [
p["document_id"]
for p in read_json("data/generated/input/pages.json")
if p["split"] == "test"
]
results = {}
for branch in ["sum-ablation"] if exercise else ["base", "sft", "orpo"]:
predictions = read_json(f"outputs/inference/predictions-{branch}.json")
pairs = read_json(f"outputs/evaluation/pairs-{branch}.json")
if (
[r["document_id"] for r in predictions["rows"]] != test_ids
or [r["document_id"] for r in pairs["rows"]] != test_ids
or len(test_ids) != 180
or predictions["identity"] != pairs["identity"]
):
raise ValueError("Same180testcases and model weights required")
results[branch] = {
"free_notes": score_causal_claims(predictions["rows"], gold),
"pairs": pair_metrics(pairs["rows"]),
}
print(
{
"branch": branch,
"free_notes": results[branch]["free_notes"]["metrics"],
"pairs": results[branch]["pairs"],
},
flush=True,
)
save_json(
(
"outputs/evaluation/exercise-comparison.json"
if exercise
else "outputs/evaluation/comparison.json"
),
{
"results": results,
"lambda_selected_before_training": config["lambda_orpo"],
"human_preferences": False,
"quality_gain_required": False,
"checkpoint_selection": False,
"reference_models_loaded": 0,
"same_prompt_presentations_not_same_compute": True,
"exercise_sum_is_not_reference_ORPO": exercise,
"no_real_incident_action": True,
},
)
Aislar el efecto de la longitud
Ablación de normalización
Error frecuente: Mezclar datos analíticos y resultados entrenados en un mismo gráfico sin leyenda: separar fuentes.
Invariante: Las dos variantes están nombradas y el ORPO final conserva la media.
Archivo src/orpo_sans_reference/exercise.py
from pathlib import Path
import torch
from .config import read_json, save_json, sha
from .loss import orpo_preference_loss
def main_artifacts():
names = [
"configs/minimum.yaml",
"outputs/evaluation/comparison.json",
"outputs/numeric-mechanism.json",
]
for branch in ["base", "sft", "orpo"]:
names += [
f"outputs/evaluation/pairs-{branch}.json",
f"outputs/inference/predictions-{branch}.json",
]
if branch != "base":
names += [
f"assets/checkpoints/{branch}-resume.safetensors",
f"outputs/learning/resume-{branch}.json",
]
return names
def preserve_main():
path = Path("outputs/exercise/protected.json")
if path.exists():
raise ValueError("Preserve existing exercise snapshot")
save_json(path, {name: sha(name) for name in main_artifacts()})
def compare_sum_mean_odds():
rows = []
for chosen_length in [4, 16, 64]:
for rejected_length in [4, 16, 64]:
for normalization in ["mean", "sum"]:
a = torch.full(
(chosen_length,), -1.0, dtype=torch.float64, requires_grad=True
)
b = torch.full(
(rejected_length,), -1.2, dtype=torch.float64, requires_grad=True
)
pa = a.mean() if normalization == "mean" else a.sum()
pb = b.mean() if normalization == "mean" else b.sum()
loss, margin = orpo_preference_loss(pa.reshape(1), pb.reshape(1))
ga, gb = torch.autograd.grad(loss.sum(), (a, b))
rows.append(
{
"normalization": normalization,
"chosen_length": chosen_length,
"rejected_length": rejected_length,
"chosen_token_logprob": -1.0,
"rejected_token_logprob": -1.2,
"chosen_score": float(pa.detach()),
"rejected_score": float(pb.detach()),
"log_odds_margin": float(margin.detach()[0]),
"preference_loss": float(loss.detach()[0]),
"chosen_token_gradients": ga.tolist(),
"rejected_token_gradients": gb.tolist(),
}
)
return rows
def run_length_ablation(config):
if config.get("normalization") != "sum":
raise ValueError("Exercise must identify the sum ablation")
protected = read_json("outputs/exercise/protected.json")
if protected != {name: sha(name) for name in main_artifacts()}:
raise ValueError("Exercise changed main weights/results/config")
actual = read_json("outputs/evaluation/pairs-orpo.json")
rescored = []
for row in actual["rows"]:
a = torch.tensor(
[row["scores"]["chosen"]["sum_logprob"]], dtype=torch.float32, device="cuda"
)
b = torch.tensor(
[row["scores"]["rejected"]["sum_logprob"]],
dtype=torch.float32,
device="cuda",
)
na = row["scores"]["chosen"]["response_tokens"]
nb = row["scores"]["rejected"]["response_tokens"]
mean, mean_margin = orpo_preference_loss(a / na, b / nb)
summed, sum_margin = orpo_preference_loss(a, b)
if (
float(mean[0]) != row["mean_odds_preference_loss"]
or float(mean_margin[0]) != row["mean_log_odds_margin"]
):
raise ValueError("Mean scoring changed in the exercise")
rescored.append(
{
"document_id": row["document_id"],
"chosen_tokens": na,
"rejected_tokens": nb,
"mean_log_odds_margin": float(mean_margin[0]),
"sum_log_odds_margin": float(sum_margin[0]),
"mean_preference_loss": float(mean[0]),
"sum_preference_loss": float(summed[0]),
"scope": "Same already scored main ORPO policy, formula ablation only",
}
)
branches = {
b: read_json(f"outputs/learning/train-{b}-continuous.json")
for b in ["sft", "orpo", "sum-ablation"]
}
initial = {r["initial_model_digest"] for r in branches.values()}
schedules = [
[
(r["document_ids"], r["chosen_token_denominator"], r["learning_rate"])
for r in run["history"]
]
for run in branches.values()
]
if (
len(initial) != 1
or not all(schedule == schedules[0] for schedule in schedules)
or any(run["updates"] != 210 for run in branches.values())
):
raise ValueError("Ablation initial model/example budgets differ")
result = read_json("outputs/evaluation/exercise-comparison.json")
if not result["exercise_sum_is_not_reference_ORPO"]:
raise ValueError("Sum variant must not be labelled reference ORPO")
save_json(
"outputs/exercise/odds-length.json",
{
"analytic_rows": compare_sum_mean_odds(),
"analytic_source": "Constructed constant token log probabilities, not generated incident notes",
"actual_main_policy_rescoring": rescored,
"actual_fresh_sum_training": True,
"same_base_and_prompt_order": True,
"updates_per_branch": 210,
"main_normalization_restored": "mean",
"main_artifacts_unchanged": protected
== {name: sha(name) for name in main_artifacts()},
"protected_sha256": protected,
"sum_variant_is_not_reference_ORPO": True,
"quality_gain_required": False,
},
)
Fuentes
ORPO, artículo original, repositorio de autores fijado, función de control TRL fijada, Qwen2.5-0.5B-Instruct. Los datos, máscaras y presupuestos locales no reproducen los resultados publicados de estos autores.
Referencia medida
Consulte los resultados de desarrollo r2 (RESULTATS_DEVELOPPEMENT.md, incluido en el ZIP) y su procedencia explícita (OBSERVATIONS.json, incluido en el ZIP). El código científico es idéntico en r3; solo se corrige una leyenda.
