Un taller recibe solicitudes en texto libre. Queremos una ficha JSON que copie la información presente, elija una categoría definida y deje null la información ausente. No hay acceso a una GED ni ejecución de intervención.
Con el mismo ticket, muestre la salida con solo prompt, tras SFT completo y tras adaptación limitada a sus dos últimos bloques. Analice el JSON y compruebe los campos sin decodificación restringida para aislar el efecto del aprendizaje.
Descargar el proyecto completo, revisión r002
Una buena salida descrita por los datos
El SFT maximiza la verosimilitud de una respuesta esperada condicionada por un prompt. No se trata de hacer que el modelo relea tickets brutos: los pares ticket/ficha enseñan aquí una transformación precisa. El modelo base Qwen2.5-0.5B ya tiene pesos preentrenados; este TP no pretende aprender una lengua desde cero.
InterventionPairs contiene 400 familias de seis formulaciones, es decir, 2400 pares. Las familias 0–279 dan 1680 ejemplos de train; 280–339 dan 360 de validation; 340–399 dan 360 de test. Las formulaciones vecinas de una familia permanecen juntas. Las combinaciones de apertura/cierre e identidades de unidad diferencian las familias; el vocabulario de síntomas, lugares y formas de frase sigue compartido. Los hashes normalizados comprueban duplicados tras normalizar espacios y mayúsculas, no una ausencia universal de solapamiento semántico.
Los campos son exactamente unit, symptom, location, scheduled_date. Sus valores son cadenas o null; un síntoma no null debe pertenecer a noise, leak, heat, vibration. El esquema comprueba tipos y claves, no la verdad de una cadena ni su conformidad con un calendario. Una fecha ficticia errónea puede respetar el esquema y fallar la evaluación del campo. La igualdad de negocio se refiere a los valores exactos, no al orden de las claves JSON.
Los objetivos no recuperan información ausente del ticket: una cláusula de ausencia sustituye el hecho correspondiente. Las fechas faltan en dos formulaciones de seis; otros casos retiran un lugar, una unidad o un síntoma. Las proporciones efectivas de cada split se registran y deben superar un cuarto. Ocho demostraciones se eligen mediante ID de train fijados en el generador, antes de observar resultados de test.
Frontera de respuesta y loss
El formato local es una instrucción, Ticket: ... y después JSON:. No usamos una plantilla conversacional implícita. El tokenizer rápido procede de la base fijada. El contrato codifica por separado el prompt y la respuesta JSON, concatena sus listas de tokens y añade EOS=151643. Esta construcción no se presenta como idéntica a retokenizar una cadena concatenada: la frontera es explícita y se prueba con acentos y JSON vacío.
Los labels del prompt y del padding son −100. Cada token de respuesta y EOS tiene su objetivo. Para predecir el token siguiente se alinean logits[:, :-1] con labels[:, 1:]. El primer token de la secuencia no tiene predecesor; nunca es objetivo de este cálculo, incluida la ablación que supervisa el prompt.
PAD y EOS usan el mismo identificador. Por tanto, la máscara depende de posiciones reales y no de la igualdad con ese identificador: ignorar todos los 151643 también eliminaría la señal de parada. No se empaqueta ninguna secuencia con otra. Los prompts SFT/zero-shot se limitan a 384 tokens; las respuestas a 96, EOS incluido. Una respuesta vacía o una entrada demasiado larga provoca un error con el ID del caso; nada se trunca silenciosamente.
La CE se codifica como suma sobre posiciones activas dividida por su número exacto. Para ocho ejemplos acumulados, cada microbatch usa el denominador total de esos ocho ejemplos aunque sus respuestas tengan longitudes diferentes. Una media de medias cambiaría el peso de cada token. El decodificador y la proyección al vocabulario calculan toda la secuencia y la máscara selecciona las posiciones supervisadas para la CE. Mantener la misma forma de proyección que Transformers evita una diferencia de redondeo FP32 ligada al producto matricial sobre un subconjunto de filas. Una prueba compara logits seleccionados y loss con Transformers en las tolerancias FP32 iniciales; prompt y padding siguen excluidos de la supervisión.
¿Qué parámetros pueden cambiar?
La base es Qwen/Qwen2.5-0.5B, revisión 060db6499f32faf8b98477b0a26969ef7d8b9987. La arquitectura adquirida se verifica: anchura 896, 24 bloques, 14 cabezas de consulta, dos cabezas K/V, MLP 4864, vocabulario 151936 y contexto nativo 32768. Los embeddings de entrada y la proyección de salida comparten el mismo Parameter. El programa calcula el número exacto de parámetros únicos.
Full hace modificables todos los parámetros. Partial solo hace modificables los bloques 22 y 23; embeddings, salida vinculada, normalización final y demás bloques permanecen congelados. La selección recorre los objetos reales de la arquitectura y comprueba nombres, identidades y ausencia de duplicados en el optimizador. Los hashes antes/después deben conservar todos los parámetros congelados, mientras que parámetros autorizados deben cambiar realmente. Hacer modificable un parámetro no garantiza que cada tensor cambie numéricamente en cada paso.
Las dos ramas parten exactamente de la misma base, en FP32 con atención eager y TF32 desactivado. Usan los mismos datos, orden de extracción, dos épocas y cantidades de tokens de respuesta supervisados. AdamW, weight_decay 0,01, microbatch 1, acumulación de ocho ejemplos, clipping a 1, warm-up de 42 actualizaciones y descenso lineal hasta el 10 % de la tasa inicial. Full usa lr=0,00002 y partial lr=0,00005: esta comparación también varía la tasa de aprendizaje, no solo el número de parámetros. No se elige ninguna tasa ni checkpoint según test; el checkpoint final queda fijado tras dos épocas.
La validación de respuesta sola se mide tras cada época, sin seleccionar el mejor checkpoint. La reanudación ocurre en un límite de época, tras una época interrumpida. Restaura pesos, AdamW, estado del scheduler, RNG Python/NumPy/PyTorch CPU/CUDA, sampler e historial. El recorrido continuo de dos épocas y la reanudación deben dar estados exactamente iguales. No promete reanudar en medio de un microbatch ni de una escritura incompleta.
Los estados .pt se crean localmente, se escriben en un archivo temporal y se renombran atómicamente; no deben sustituirse por archivos no fiables. El modelo final también se guarda en Safetensors con gestión del compartir embedding/salida y se recarga en una instancia aislada. Archivos y parámetros se verifican mediante SHA-256. Los pesos aprendidos quedan fuera del ZIP público; el lector los recrea.
Baselines y generación sin restricciones
El modelo base se evalúa en zero-shot y con ocho demostraciones fijas. Estos ocho ejemplos requieren un presupuesto de contexto separado de 2048 tokens: no se fuerzan artificialmente en los 384 tokens de otros prompts. Se registra su coste real. La generación reserva 96 tokens nuevos en el contexto nativo, sin truncar el prompt.
Todas las ramas usan generación greedy, un único beam y parada EOS=151643, con un máximo de 96 tokens nuevos. No se fuerza ningún esquema durante la decodificación. ID de entrada/salida, texto bruto, motivo de parada, número de tokens y duración se conservan para cada uno de los 360 casos de validation y 360 de test. Los pesos no se ajustan en esta etapa.
El analizador distingue tres propiedades: texto JSON analizable, esquema válido y respuesta completa aceptada. Una respuesta detenida en el límite puede contener JSON bien formado; sigue rechazándose como respuesta completa. No se aplican retirada de etiquetas Markdown, añadido de campos, reparación ni enriquecimiento. Se rechazan claves duplicadas y constantes no finitas.
El exact match global y la exactitud de cada campo usan los 360 casos de test. Una salida inválida o truncada falla en todos los campos. Para información ausente, el informe distingue null correcto, valor no null inventado en una respuesta completa válida y salida inválida. La tasa de valores inventados usa todas las ocasiones donde falta el campo; la tasa de fallo al proporcionar null incluye también salidas inválidas. Así, un modelo que no produce JSON no puede parecer fiable por una tasa artificialmente nula de invención.
La NLL de respuesta sola en test se recalcula con el mismo prompt zero-shot para base, full, partial y ablación. Incluye EOS y los mismos objetivos. La generación few-shot es una comparación distinta con más contexto; se informan su calidad y sus tokens sin confundir este coste con un efecto de aprendizaje.
Ejercicio: incluir el prompt en la loss
Una tercera rama completa parte realmente de la base durante dos épocas, con el mismo orden documental y configuración full, pero también supervisa tokens del prompt. Por tanto, cuenta más tokens activos. Una bajada de su loss de entrenamiento no puede compararse directamente con la media de respuesta sola.
Se recalculan su NLL sobre las mismas respuestas de test y sus 720 generaciones de validation/test, y se comparan exact match y campos ausentes. No se impone ninguna ganancia ni retroceso. Modelos y resultados principales quedan protegidos por SHA antes del ejercicio. La máscara de respuesta sigue siendo la configuración de referencia; la ablación lleva un nombre distinto.
Límites
Estos tickets son sintéticos y regulares. Sus etiquetas reflejan un generador, no la variedad de solicitudes reales. Un JSON conforme puede contener información falsa. Una sola semilla de aprendizaje no permite estimar la variabilidad de un entrenamiento. Las reglas que producen objetivos son un oráculo pedagógico, no un descubrimiento del modelo. No se crea ni ejecuta ningún expediente documental ni acto de intervención.
Definir tarea, esquema y familias
Demostración supervisada
Error frecuente: El generador escribe una fecha objetivo ausente del ticket: probar cada campo frente a los hechos representados.
Invariante: 1 680/360/360 pares y proporciones de ausencia auditadas antes de posibles exclusiones.
Archivo src/sft_complet/data.py
import hashlib, json, re
from pathlib import Path
from .config import save_json, sha
FIELDS = ["unit", "symptom", "location", "scheduled_date"]
SYMPTOMS = ["noise", "leak", "heat", "vibration"]
LOCATIONS = [
"inlet",
"outlet",
"motor bay",
"upper housing",
"control panel",
"rear cover",
"lower bracket",
"supply line",
"return line",
"service hatch",
]
OPENINGS = [
"A technician submitted this observation.",
"The operator left a short report.",
"The shift log contains this request.",
"A maintenance note describes the following.",
"The service team received this message.",
"An internal ticket records these details.",
"The workshop inbox received a report.",
"A routine handover included this note.",
"The operator asks for a review.",
"The inspection desk received this request.",
"The shift coordinator recorded a concern.",
"A service request includes these facts.",
"The team copied this observation into a ticket.",
"The workshop clerk transcribed this report.",
"A technician added the following account.",
"The afternoon handover contains this message.",
"The maintenance inbox lists this observation.",
"The service desk forwarded this written request.",
"An operator documented the following concern.",
"A new workshop note contains these details.",
]
CLOSINGS = [
"Please retain the original observations.",
"The note contains no other confirmed details.",
"This is a request for review only.",
"No repair has been performed by this note.",
"The text is an observation, not an instruction to run equipment.",
"The report remains available for the next shift.",
"Keep unreported details unspecified.",
"The service desk can review the record.",
"This message does not confirm a diagnosis.",
"The observation is awaiting a technical review.",
"Only explicitly reported facts should be copied.",
"The ticket has not authorized any intervention.",
"Retain uncertainty when details are missing.",
"No additional findings are supplied here.",
"The report is ready for manual review.",
"This entry is a written observation only.",
"The workshop has supplied no supplementary facts.",
"The request can be filed for later inspection.",
"The maintenance team should read the original note.",
"The text does not provide a cause for the concern.",
]
ORDERS = [
["unit", "symptom", "location", "scheduled_date"],
["location", "unit", "scheduled_date", "symptom"],
["scheduled_date", "symptom", "unit", "location"],
["symptom", "location", "unit", "scheduled_date"],
["unit", "scheduled_date", "location", "symptom"],
["location", "symptom", "scheduled_date", "unit"],
]
MISSING = {
"unit": "The unit identifier was not provided.",
"symptom": "The symptom was not specified.",
"location": "The location was not stated.",
"scheduled_date": "No inspection date has been scheduled.",
}
def split_scenarios(family):
return "train" if family < 280 else "validation" if family < 340 else "test"
def normalized(text):
return " ".join(text.casefold().split())
def render_target(values):
return json.dumps(
{key: values[key] for key in FIELDS}, ensure_ascii=False, separators=(",", ":")
)
def render_fact(key, value, j):
if value is None:
return MISSING[key]
alternatives = {
"unit": [
f"The unit is {value}.",
f"Unit {value} is named in the report.",
f"The reported unit identifier is {value}.",
f"This observation concerns unit {value}.",
f"The unit was identified as {value}.",
f"The record names unit {value}.",
],
"symptom": [
f"The reported symptom is {value}.",
f"The operator reports {value}.",
f"The symptom is described as {value}.",
f"The note explicitly reports {value}.",
f"The observed symptom is {value}.",
f"The report describes {value}.",
],
"location": [
f"The location is {value}.",
f"The issue was reported near the {value}.",
f"The stated location is the {value}.",
f"The observation was made at the {value}.",
f"The report points to the {value}.",
f"The location named is the {value}.",
],
"scheduled_date": [
f"The inspection is scheduled for {value}.",
f"The scheduled inspection date is {value}.",
f"An inspection has been scheduled on {value}.",
f"The planned inspection date is {value}.",
f"The inspection appointment is set for {value}.",
f"The record gives {value} as the inspection date.",
],
}
return alternatives[key][j]
def generate_tickets():
pages = []
gold = []
templates = []
for family in range(400):
opening = OPENINGS[family // 20]
closing = CLOSINGS[family % 20]
templates.append(
{
"family": family,
"split": split_scenarios(family),
"opening": opening,
"closing": closing,
}
)
for j in range(6):
values = {
"unit": f"U{family+1000}",
"symptom": SYMPTOMS[family % 4],
"location": LOCATIONS[(family // 4) % 10],
"scheduled_date": f"2026-{family%12+1:02d}-{family//12%27+1:02d}",
}
if j in [0, 3]:
values["scheduled_date"] = None
if j == 4 and family % 3 == 0:
values["location"] = None
if j == 5 and family % 4 == 0:
values["unit"] = None
if j == 2 and family % 7 == 0:
values["symptom"] = None
clauses = [render_fact(key, values[key], j) for key in ORDERS[j]]
ticket = " ".join([opening, *clauses, closing])
identifier = f"I{family:03d}-{j}"
row = {
"document_id": identifier,
"family": family,
"formulation": j,
"split": split_scenarios(family),
"ticket": ticket,
"content_sha256": hashlib.sha256(ticket.encode()).hexdigest(),
"normalized_sha256": hashlib.sha256(
normalized(ticket).encode()
).hexdigest(),
}
pages.append(row)
gold.append(
{
"document_id": identifier,
"target": values,
"target_json": render_target(values),
"absent_fields": [key for key in FIELDS if values[key] is None],
}
)
# Every non-null label appears in a rendered clause; absent facts are
# represented only by their missingness statement, never hidden metadata.
for key in FIELDS:
if values[key] is not None:
assert str(values[key]) in clauses[ORDERS[j].index(key)]
assert len({r["normalized_sha256"] for r in pages}) == 2400
return pages, gold, templates
def prepare(config):
root = Path("data/generated")
root.mkdir(parents=True, exist_ok=True)
if (root / "manifest.json").exists():
raise ValueError("Existing frozen corpus; extract into a new directory")
pages, gold, templates = generate_tickets()
byid = {r["document_id"]: r for r in gold}
save_json(root / "input/pages.json", pages)
save_json(root / "input/templates.json", templates)
save_json(root / "labels/gold.json", gold)
audit = {}
for split in ["train", "validation", "test"]:
selected = [r for r in pages if r["split"] == split]
labels = [byid[r["document_id"]] for r in selected]
save_json(root / f"{split}/pages.json", selected)
if split != "test":
save_json(root / f"labels/{split}.json", labels)
absent = sum(bool(r["absent_fields"]) for r in labels)
assert absent / len(labels) >= 0.25
audit[split] = {
"documents": len(selected),
"families": len({r["family"] for r in selected}),
"with_absent_field": absent,
"with_absent_field_rate": absent / len(labels),
}
demo_ids = [
f"I{family:03d}-{j}"
for family, j in [
(0, 0),
(3, 1),
(8, 2),
(13, 3),
(21, 4),
(34, 5),
(55, 0),
(89, 1),
]
]
lookup = {r["document_id"]: r for r in pages}
demonstrations = [
{
"document_id": identifier,
"ticket": lookup[identifier]["ticket"],
"target_json": byid[identifier]["target_json"],
}
for identifier in demo_ids
]
assert all(lookup[i]["split"] == "train" for i in demo_ids)
save_json(root / "input/demonstrations.json", demonstrations)
save_json(
root / "manifest.json",
{
"dataset": "InterventionPairs-v1",
"rights": "CC0-1.0",
"seed": config["seed"],
"split_audit": audit,
"normalized_duplicates": 0,
"demonstration_ids": demo_ids,
"files": {str(f.relative_to(root)): sha(f) for f in root.rglob("*.json")},
},
)
save_json(
"outputs/data-audit.json",
{"splits": audit, "normalized_duplicates": 0, "demonstration_ids": demo_ids},
)
return audit
Codificar el collator de respuestas
Frontera de tokens y EOS
Error frecuente: Un truncamiento a la derecha elimina EOS y respuesta: detectarlo antes del entrenamiento.
Invariante: Todos los tokens de respuesta esperados tienen un objetivo y el prompt se ignora.
Archivo src/sft_complet/collator.py
import torch
import torch.nn.functional as F
INSTRUCTION = "Extract exactly unit, symptom, location, scheduled_date as a JSON object. Missing information must be null. A reported symptom is noise, leak, heat, or vibration. Return only the JSON object."
def prompt_text(ticket):
return INSTRUCTION + "\nTicket: " + ticket + "\nJSON:"
def encode_prompt_response(
tokenizer, ticket, response, config, identifier, supervise_prompt=False
):
# Explicit concatenation of two separately encoded token streams. We do not
# pretend it equals re-tokenizing the concatenated character string.
prompt = tokenizer.encode(prompt_text(ticket), add_special_tokens=False)
completion = tokenizer.encode(response, add_special_tokens=False) + [151643]
if not prompt or len(prompt) > config["max_prompt_tokens"]:
raise ValueError(
identifier + ": prompt exceeds the declared token budget; no truncation"
)
if (
len(completion) <= 1
or len(completion) > config["max_response_tokens_including_eos"]
):
raise ValueError(
identifier + ": missing or overlength response including EOS; no truncation"
)
ids = prompt + completion
labels = list(ids) if supervise_prompt else [-100] * len(prompt) + completion
return {
"document_id": identifier,
"ids": ids,
"labels": labels,
"prompt_tokens": len(prompt),
"response_tokens": len(completion),
"supervise_prompt": supervise_prompt,
}
def completion_labels(row):
return row["labels"]
def collate(rows, device="cuda", pad_to=None):
if not rows:
raise ValueError("Empty batch")
length = max(len(r["ids"]) for r in rows) if pad_to is None else pad_to
if length < max(len(r["ids"]) for r in rows):
raise ValueError("Silent truncation prohibited")
ids = torch.full((len(rows), length), 151643, dtype=torch.long, device=device)
labels = torch.full_like(ids, -100)
mask = torch.zeros_like(ids, dtype=torch.bool)
for i, row in enumerate(rows):
n = len(row["ids"])
ids[i, :n] = torch.tensor(row["ids"], device=device)
labels[i, :n] = torch.tensor(row["labels"], device=device)
mask[i, :n] = True
return ids, mask, labels
def completion_nll(logits, labels):
selected = labels[:, 1:] != -100
if not selected.any():
raise ValueError("No supervised next token")
values = F.cross_entropy(
logits[:, :-1][selected].float(), labels[:, 1:][selected], reduction="none"
)
return values.sum(), int(selected.sum()), values
Medir solo el prompt y elegir pesos modificables
Inicialización y espacio de adaptación
Error frecuente: Embeddings vinculados a lm_head descongelados por error: comprobar identidades de almacenamiento y lista del optimizador.
Invariante: Los dos últimos bloques están identificados y ningún otro parámetro partial está en el optimizador.
Archivo src/sft_complet/baseline.py
from .model import load_base, select_trainables, parameter_manifest, manifest_digest
from .config import save_json
def report_trainable_parameters():
model = load_base()
result = {}
for branch in ["full", "partial"]:
parameters = select_trainables(model, branch)
rows = parameter_manifest(model)
result[branch] = {
"parameters": rows,
"total_parameters": sum(r["numel"] for r in rows),
"trainable_parameters": sum(r["numel"] for r in rows if r["requires_grad"]),
"optimizer_parameter_objects": len(parameters),
"unique_optimizer_parameter_objects": len({id(p) for p in parameters}),
"model_digest": manifest_digest(rows),
"tied_embedding_output": model.lm_head.weight
is model.model.embed_tokens.weight,
}
if result["full"]["model_digest"] != result["partial"]["model_digest"]:
raise ValueError("Selecting trainable parameters changed weights")
save_json("outputs/parameter-audit.json", result)
import shutil
shutil.copyfile("assets/model/tokenizer.json", "outputs/tokenizer-reference.json")
def select_last_blocks(model):
return select_trainables(model, "partial")
def evaluate_prompt_only(config, variant):
from .evaluate import predict
if variant not in ["zeroshot", "fewshot"]:
raise ValueError("Expected frozen-base baseline")
predict(config, variant)
Realizar el SFT y su reanudación
CE condicional y estado de optimización
Error frecuente: Acumular la media de respuestas pequeñas da un gradiente incorrecto: sumar NLL y contadores.
Invariante: Pesos full y partial esperados modificados; checkpoints separados, estados del sampler y RNG restaurados.
Archivo src/sft_complet/train.py
import copy, hashlib, math, os, time
from pathlib import Path
import numpy as np
import torch
from .config import read_json, save_json, sha, seed_all, rng_state, restore_rng
from .model import (
load_base,
load_tokenizer,
select_trainables,
parameter_manifest,
manifest_digest,
save_learned,
active_response_logits,
)
from .collator import encode_prompt_response, collate
def equal(a, b):
if isinstance(a, torch.Tensor):
return isinstance(b, torch.Tensor) and torch.equal(a, b)
if isinstance(a, np.ndarray):
return isinstance(b, np.ndarray) and np.array_equal(a, b)
if isinstance(a, dict):
return (
isinstance(b, dict)
and a.keys() == b.keys()
and all(equal(a[k], b[k]) for k in a)
)
if isinstance(a, (list, tuple)):
return (
type(a) == type(b)
and len(a) == len(b)
and all(equal(x, y) for x, y in zip(a, b))
)
return a == b
def save_resume_state(path, state):
path = Path(path)
path.parent.mkdir(parents=True, exist_ok=True)
if path.exists():
raise ValueError("Existing training state is preserved")
temporary = path.with_suffix(".pending")
torch.save(state, temporary)
os.replace(temporary, path)
def encode_split(config, tokenizer, split, supervise_prompt=False):
pages = read_json(f"data/generated/{split}/pages.json")
labels = {
r["document_id"]: r for r in read_json(f"data/generated/labels/{split}.json")
}
if any(p["split"] != split for p in pages):
raise ValueError("Wrong training/validation partition")
return [
encode_prompt_response(
tokenizer,
p["ticket"],
labels[p["document_id"]]["target_json"],
config,
p["document_id"],
supervise_prompt,
)
for p in pages
]
def learning_rate(config, branch, step, total):
peak = (
config["learning_rate_partial"]
if branch == "partial"
else config["learning_rate_full"]
)
warmup = config["warmup_updates"]
if step < warmup:
return peak * (step + 1) / warmup
return peak * (
1
- min(1.0, (step - warmup) / max(1, total - warmup - 1))
* (1 - config["minimum_lr_ratio"])
)
@torch.no_grad()
def validate_response(model, rows):
model.eval()
total = 0.0
count = 0
for row in rows:
ids, mask, labels = collate([row])
_, n, values, _, _ = active_response_logits(model, ids, mask, labels)
total += float(values.double().sum())
count += n
model.train()
return {
"response_nll_nats_per_token": total / count,
"response_tokens": count,
"documents": len(rows),
}
def train_sft(config, branch="full", mode="continuous"):
if (
Path("data/generated/test").exists()
or Path("data/generated/labels/gold.json").exists()
):
raise ValueError("Learning service must not mount test inputs or gold")
if mode not in ["interrupt", "continuous", "resume"]:
raise ValueError("Unknown training mode")
seed_all(config["seed"])
tokenizer = load_tokenizer()
rows = encode_split(config, tokenizer, "train", branch == "prompt_loss")
validation = encode_split(config, tokenizer, "validation", False)
if len(rows) != 1680 or len(validation) != 360:
raise ValueError("Expected frozen1680/360 population")
model = load_base()
parameters = select_trainables(model, branch)
initial = parameter_manifest(model)
initial_digest = manifest_digest(initial)
model = model.cuda()
parameters = [p for p in model.parameters() if p.requires_grad]
optimizer = torch.optim.AdamW(
parameters, lr=config["learning_rate_full"], weight_decay=config["weight_decay"]
)
sampler = torch.Generator().manual_seed(config["seed"] + 100)
updates_per_epoch = math.ceil(len(rows) / config["accumulation_examples"])
total_updates = updates_per_epoch * config["epochs"]
start = updates = 0
history = []
validation_history = []
identity = {
"branch": branch,
"config": config,
"base_sha256": sha("assets/model/model.safetensors"),
"tokenizer_sha256": sha("assets/model/tokenizer.json"),
"model_config_sha256": sha("assets/model/config.json"),
"model_lock_sha256": sha("locks/models.lock.json"),
"implementation_sha256": {
name: sha("src/sft_complet/" + name) for name in ["model.py", "train.py"]
},
"train_input_sha256": sha("data/generated/train/pages.json"),
"train_labels_sha256": sha("data/generated/labels/train.json"),
"validation_input_sha256": sha("data/generated/validation/pages.json"),
"validation_labels_sha256": sha("data/generated/labels/validation.json"),
"collator_sha256": sha("src/sft_complet/collator.py"),
"initial_model_digest": initial_digest,
"trainable_names": [r["name"] for r in initial if r["requires_grad"]],
}
root = Path("assets/checkpoints")
destination = root / f"{branch}-{mode}.pt"
if destination.exists():
raise ValueError("Preserve existing checkpoint")
if mode == "resume":
state = torch.load(
root / f"{branch}-interrupt.pt", map_location="cpu", weights_only=False
)
if state["identity"] != identity:
raise ValueError("Resume identity changed")
model.load_state_dict(state["model"], strict=True)
optimizer.load_state_dict(state["optimizer"])
sampler.set_state(state["sampler"])
restore_rng(state["rng"])
start = state["epoch"]
updates = state["updates"]
history = state["history"]
validation_history = state["validation_history"]
if state["scheduler"] != {
"next_update": updates,
"next_lr": learning_rate(config, branch, updates, total_updates),
}:
raise ValueError("Resume scheduler changed")
del state
stop = config["interrupt_epoch"] if mode == "interrupt" else config["epochs"]
begun = time.perf_counter()
torch.cuda.reset_peak_memory_stats()
model.train()
for epoch in range(start, stop):
permutation = torch.randperm(len(rows), generator=sampler).tolist()
epoch_loss = 0.0
epoch_tokens = 0
for offset in range(0, len(rows), config["accumulation_examples"]):
selected = permutation[offset : offset + config["accumulation_examples"]]
denominator = sum(
sum(v != -100 for v in rows[i]["labels"][1:]) for i in selected
)
lr = learning_rate(config, branch, updates, total_updates)
for group in optimizer.param_groups:
group["lr"] = lr
optimizer.zero_grad(set_to_none=True)
summed = 0.0
for i in selected:
ids, mask, labels = collate([rows[i]])
loss_sum, n, values, _, _ = active_response_logits(
model, ids, mask, labels
)
if not torch.isfinite(loss_sum) or n != sum(
v != -100 for v in rows[i]["labels"][1:]
):
raise ValueError("Nonfinite or miscounted active-token loss")
(loss_sum / denominator).backward()
summed += float(values.detach().double().sum())
norm = torch.nn.utils.clip_grad_norm_(
parameters, config["gradient_clip"], error_if_nonfinite=True
)
optimizer.step()
updates += 1
epoch_loss += summed
epoch_tokens += denominator
history.append(
{
"update": updates,
"epoch": epoch + 1,
"document_ids": [rows[i]["document_id"] for i in selected],
"active_tokens": denominator,
"nll_nats_per_active_token": summed / denominator,
"learning_rate": lr,
"gradient_norm_before_clip": float(norm),
}
)
if updates % 42 == 0:
print(history[-1], flush=True)
observed = validate_response(model, validation)
validation_history.append({"epoch": epoch + 1, **observed})
print(
{
"epoch": epoch + 1,
"train_nll_nats_per_active_token": epoch_loss / epoch_tokens,
"train_active_tokens": epoch_tokens,
"validation": observed,
},
flush=True,
)
torch.cuda.synchronize()
final = parameter_manifest(model)
before = {r["name"]: r for r in initial}
changed = [r["name"] for r in final if r["sha256"] != before[r["name"]]["sha256"]]
frozen_changed = [
r["name"]
for r in final
if not before[r["name"]]["requires_grad"]
and r["sha256"] != before[r["name"]]["sha256"]
]
if not changed or frozen_changed:
raise ValueError("No learning or frozen parameters changed")
state = {
"identity": identity,
"epoch": stop,
"updates": updates,
"model": model.state_dict(),
"optimizer": optimizer.state_dict(),
"sampler": sampler.get_state(),
"scheduler": {
"next_update": updates,
"next_lr": learning_rate(config, branch, updates, total_updates),
},
"rng": rng_state(),
"history": history,
"validation_history": validation_history,
}
save_resume_state(destination, state)
if mode == "resume" or (branch == "prompt_loss" and mode == "continuous"):
save_learned(model, root / f"{branch}-{mode}.safetensors", identity)
save_json(
f"outputs/train-{branch}-{mode}.json",
{
"branch": branch,
"mode": mode,
"identity": identity,
"epochs": stop,
"updates": updates,
"history": history,
"validation_history": validation_history,
"tokenization": rows,
"initial_parameters": initial,
"final_parameters": final,
"changed_parameters": changed,
"frozen_changed": frozen_changed,
"trainable_parameters": sum(
r["numel"] for r in initial if r["requires_grad"]
),
"total_parameters": sum(r["numel"] for r in initial),
"initial_model_digest": initial_digest,
"final_model_digest": manifest_digest(final),
"checkpoint_sha256": sha(destination),
"checkpoint_bytes": destination.stat().st_size,
"max_torch_allocated_bytes": torch.cuda.max_memory_allocated(),
"seconds": time.perf_counter() - begun,
"test_visible": False,
},
)
def verify_resume(branch):
root = Path("assets/checkpoints")
a = torch.load(
root / f"{branch}-continuous.pt", map_location="cpu", weights_only=False
)
b = torch.load(root / f"{branch}-resume.pt", map_location="cpu", weights_only=False)
checks = {k: equal(a[k], b[k]) for k in a}
if not all(checks.values()):
raise ValueError("Full resume differs: " + str(checks))
save_json(
f"outputs/resume-{branch}.json",
{
"status": "PASS",
"all_state_exact": True,
"checks": checks,
"boundary": "complete epoch; not a mid-microbatch recovery claim",
},
)
Evaluar sin restricción de decodificación
Formato y semántica separados
Error frecuente: Un analizador tolerante completa campos ausentes: prohibir todo enriquecimiento silencioso.
Invariante: 360 ID por rama, sin eliminar los JSON inválidos.
Archivo src/sft_complet/evaluate.py
import json, math, time
from pathlib import Path
import torch
from transformers import GenerationConfig
from .config import read_json, save_json, sha, seed_all
from .model import (
load_base,
load_tokenizer,
reload_learned,
active_response_logits,
parameter_manifest,
manifest_digest,
)
from .collator import prompt_text, encode_prompt_response, collate
from .data import FIELDS, SYMPTOMS
def parse_generated_record(text, finish_reason):
def pairs(items):
result = {}
for key, value in items:
if key in result:
raise ValueError("Duplicate key")
result[key] = value
return result
def constant(value):
raise ValueError("Nonfinite JSON constant")
try:
record = json.loads(text, object_pairs_hook=pairs, parse_constant=constant)
except (ValueError, TypeError):
return {
"json_parsable": False,
"schema_valid": False,
"completion_valid": False,
"record": None,
"reason": "TRUNCATED" if finish_reason != "eos" else "INVALID_JSON",
}
valid = (
isinstance(record, dict)
and set(record) == set(FIELDS)
and all(v is None or type(v) == str for v in record.values())
)
if valid:
valid = record["symptom"] is None or record["symptom"] in SYMPTOMS
return {
"json_parsable": True,
"schema_valid": valid,
"completion_valid": valid and finish_reason == "eos",
"record": record if valid else None,
"reason": (
"TRUNCATED"
if finish_reason != "eos"
else "VALID_SCHEMA" if valid else "INVALID_SCHEMA"
),
}
def generation_prompt(tokenizer, ticket, demonstrations, config):
text = "\n\n".join(
prompt_text(d["ticket"]) + d["target_json"] for d in demonstrations
)
if demonstrations:
text += "\n\n"
text += prompt_text(ticket)
ids = tokenizer.encode(text, add_special_tokens=False)
limit = (
config["max_fewshot_prompt_tokens"]
if demonstrations
else config["max_prompt_tokens"]
)
if len(ids) > limit:
raise ValueError("Generation prompt exceeds declared token cap; no truncation")
return text, ids
@torch.no_grad()
def predict(config, variant):
if Path("data/generated/labels").exists():
raise ValueError("Free generation must not mount gold labels")
seed_all(config["seed"])
tokenizer = load_tokenizer()
learned = variant in ["full", "partial", "prompt_loss"]
path = Path("assets/checkpoints") / (
variant
+ (
"-continuous.safetensors"
if variant == "prompt_loss"
else "-resume.safetensors"
)
)
if learned:
model, metadata = reload_learned(path)
identity = metadata["model_digest"]
else:
model = load_base()
identity = manifest_digest(parameter_manifest(model))
model = model.cuda().eval()
pages = read_json("data/generated/input/pages.json")
selected = [r for r in pages if r["split"] in ["validation", "test"]]
demonstrations = (
read_json("data/generated/input/demonstrations.json")
if variant == "fewshot"
else []
)
if len(selected) != 720 or (variant == "fewshot" and len(demonstrations) != 8):
raise ValueError("Expected fixed720heldout cases and8train demos")
generation = GenerationConfig(
do_sample=False,
num_beams=1,
max_new_tokens=config["max_new_tokens"],
use_cache=True,
eos_token_id=151643,
pad_token_id=151643,
bos_token_id=151643,
)
rows = []
begun = time.perf_counter()
torch.cuda.reset_peak_memory_stats()
for index, page in enumerate(selected):
text, ids = generation_prompt(tokenizer, page["ticket"], demonstrations, config)
input_ids = torch.tensor([ids], device="cuda")
mask = torch.ones_like(input_ids)
started = time.perf_counter()
output = (
model.generate(
input_ids=input_ids, attention_mask=mask, generation_config=generation
)[0, len(ids) :]
.cpu()
.tolist()
)
torch.cuda.synchronize()
finish = "eos" if output and output[-1] == 151643 else "length"
content = output[:-1] if finish == "eos" else output
decoded = tokenizer.decode(
content, skip_special_tokens=False, clean_up_tokenization_spaces=False
)
parsed = parse_generated_record(decoded, finish)
rows.append(
{
"document_id": page["document_id"],
"split": page["split"],
"variant": variant,
"input_ids": ids,
"input_text": text,
"output_ids": output,
"text": decoded,
"finish_reason": finish,
"parsed": parsed,
"input_tokens": len(ids),
"output_tokens": len(output),
"seconds": time.perf_counter() - started,
}
)
if (index + 1) % 60 == 0:
print(
{"variant": variant, "generated": index + 1, "total": len(selected)},
flush=True,
)
save_json(
f"outputs/predictions-{variant}.json",
{
"variant": variant,
"model_digest": identity,
"model_file_sha256": (
sha(path) if learned else sha("assets/model/model.safetensors")
),
"tokenizer_sha256": sha("assets/model/tokenizer.json"),
"input_sha256": sha("data/generated/input/pages.json"),
"demonstration_ids": [d["document_id"] for d in demonstrations],
"generation": generation.to_dict(),
"rows": rows,
"seconds": time.perf_counter() - begun,
"max_torch_allocated_bytes": torch.cuda.max_memory_allocated(),
"gold_visible": False,
},
)
def score_missing_fields(rows, gold):
n = len(rows)
correct = {k: 0 for k in FIELDS}
absent = {k: 0 for k in FIELDS}
invented = {k: 0 for k in FIELDS}
null_correct = {k: 0 for k in FIELDS}
invalid_absent = {k: 0 for k in FIELDS}
exact = parsable = schema = truncated = 0
details = []
for row in rows:
expected = gold[row["document_id"]]["target"]
parsed = parse_generated_record(row["text"], row["finish_reason"])
assert parsed == row["parsed"]
valid = parsed["completion_valid"]
record = parsed["record"]
fields = {key: valid and record[key] == expected[key] for key in FIELDS}
row_exact = all(fields.values())
exact += row_exact
parsable += parsed["json_parsable"]
schema += parsed["schema_valid"]
truncated += row["finish_reason"] == "length"
for key in FIELDS:
correct[key] += fields[key]
if expected[key] is None:
absent[key] += 1
null_correct[key] += valid and record[key] is None
invented[key] += valid and record[key] is not None
invalid_absent[key] += not valid
details.append(
{
"document_id": row["document_id"],
"fields_correct": fields,
"exact_match": row_exact,
"absent_fields": [k for k in FIELDS if expected[k] is None],
}
)
return {
"documents": n,
"json_parsable": parsable,
"json_parsable_rate": parsable / n,
"schema_valid": schema,
"schema_valid_rate": schema / n,
"exact_match": exact,
"exact_match_rate": exact / n,
"field_correct": correct,
"field_accuracy": {k: v / n for k, v in correct.items()},
"absent_opportunities": absent,
"correct_null": null_correct,
"invented_nonnull_in_valid_record": invented,
"invalid_record_with_absent_field": invalid_absent,
"invented_rate_over_all_absent_opportunities": {
k: invented[k] / absent[k] if absent[k] else None for k in FIELDS
},
"null_failure_rate_over_all_absent_opportunities": {
k: (absent[k] - null_correct[k]) / absent[k] if absent[k] else None
for k in FIELDS
},
"truncated": truncated,
"rows": details,
}
def evaluate_branches(variants):
gold = {r["document_id"]: r for r in read_json("data/generated/labels/gold.json")}
result = {}
for variant in variants:
predictions = read_json(f"outputs/predictions-{variant}.json")
result[variant] = {}
for split in ["validation", "test"]:
rows = [r for r in predictions["rows"] if r["split"] == split]
if len(rows) != 360 or len({r["document_id"] for r in rows}) != 360:
raise ValueError("Do not omit invalid heldout outputs")
result[variant][split] = score_missing_fields(rows, gold)
name = (
"exercise-evaluation.json" if variants == ["prompt_loss"] else "evaluation.json"
)
save_json(
"outputs/" + name,
{
"results": result,
"quality_gain_required": False,
"decoding_constraints": False,
"invalid_outputs_discarded": False,
},
)
print(
{
v: {k: value for k, value in splits["test"].items() if k != "rows"}
for v, splits in result.items()
},
flush=True,
)
@torch.no_grad()
def response_nll(config, variant):
tokenizer = load_tokenizer()
learned = variant in ["full", "partial", "prompt_loss"]
path = Path("assets/checkpoints") / (
variant
+ (
"-continuous.safetensors"
if variant == "prompt_loss"
else "-resume.safetensors"
)
)
model = reload_learned(path)[0] if learned else load_base()
model = model.cuda().eval()
pages = [
r for r in read_json("data/generated/input/pages.json") if r["split"] == "test"
]
gold = {r["document_id"]: r for r in read_json("data/generated/labels/gold.json")}
records = []
probes = []
# Always the same zero-shot response-only conditioning contract, including
# the prompt-loss ablation. Few-shot cost/quality is evaluated separately.
for index, page in enumerate(pages):
row = encode_prompt_response(
tokenizer,
page["ticket"],
gold[page["document_id"]]["target_json"],
config,
page["document_id"],
False,
)
ids, mask, labels = collate([row])
_, n, losses, logits, targets = active_response_logits(model, ids, mask, labels)
records.append(
{
"document_id": page["document_id"],
"input_ids": row["ids"],
"prompt_tokens": row["prompt_tokens"],
"active_response_tokens": n,
"response_targets": targets.cpu().tolist(),
"response_nll": losses.cpu().tolist(),
}
)
if index == 0:
for pos in [0, n - 1]:
probes.append(
{
"document_id": page["document_id"],
"response_position": pos,
"target": int(targets[pos]),
"logits": logits[pos].cpu().tolist(),
"nll": float(losses[pos]),
}
)
total = math.fsum(v for row in records for v in row["response_nll"])
count = sum(r["active_response_tokens"] for r in records)
save_json(
f"outputs/response-nll-{variant}.json",
{
"variant": variant,
"conditioning": "zero-shot completion-only for every compared weight state",
"documents": len(records),
"response_tokens": count,
"response_nll_nats_per_token": total / count,
"rows": records,
"probes": probes,
},
)
Modificar la máscara para entender la señal
Ablación de supervisión
Error frecuente: Reutilizar resultados full como los de la ablación: entrenar una nueva rama y conservar su identidad.
Invariante: Tabla de tokens activos, coste y campos ausentes para cada máscara.
Archivo src/sft_complet/exercise.py
from .config import read_json, save_json, sha
def preserve_main():
paths = [
"outputs/evaluation.json",
"outputs/response-nll-full.json",
"outputs/response-nll-partial.json",
"outputs/resume-full.json",
"outputs/resume-partial.json",
"assets/checkpoints/full-resume.safetensors",
"assets/checkpoints/partial-resume.safetensors",
]
save_json("outputs/exercise-before.json", {name: sha(name) for name in paths})
def compare_response_losses():
before = read_json("outputs/exercise-before.json")
if any(sha(name) != digest for name, digest in before.items()):
raise ValueError("Exercise changed main artifacts")
main = read_json("outputs/evaluation.json")
exercise = read_json("outputs/exercise-evaluation.json")
rows = {}
for branch in ["full", "partial", "prompt_loss"]:
training = read_json(
f"outputs/train-{branch}-"
+ ("continuous" if branch == "prompt_loss" else "resume")
+ ".json"
)
nll = read_json(f"outputs/response-nll-{branch}.json")
metrics = (exercise if branch == "prompt_loss" else main)["results"][branch][
"test"
]
rows[branch] = {
"training_active_tokens": sum(
r["active_tokens"] for r in training["history"]
),
"updates": training["updates"],
"trainable_parameters": training["trainable_parameters"],
"test_response_tokens": nll["response_tokens"],
"test_response_nll_nats_per_token": nll["response_nll_nats_per_token"],
"test_metrics": {k: v for k, v in metrics.items() if k != "rows"},
"model_digest": training["final_model_digest"],
"initial_model_digest": training["initial_model_digest"],
}
if (
len({r["test_response_tokens"] for r in rows.values()}) != 1
or len({r["initial_model_digest"] for r in rows.values()}) != 1
):
raise ValueError("Ablation initial state or evaluation population changed")
if (
rows["prompt_loss"]["training_active_tokens"]
<= rows["full"]["training_active_tokens"]
):
raise ValueError("Prompt-loss branch must actually supervise additional tokens")
save_json(
"outputs/exercise/loss-masks.json",
{
"rows": rows,
"protected_sha256": before,
"main_artifacts_unchanged": True,
"fresh_prompt_loss_training": True,
"quality_gain_required": False,
"training_denominators_differ": True,
},
)
Fuentes
Modelo base Qwen2.5-0.5B, implementación Qwen2 de Transformers 4.56.2 y gestión de tensores compartidos en Safetensors 0.6.2 se consultaron para comprobar los mecanismos. El modelo es una base preentrenada, no la variante Instruct. Las versiones ejecutables y hashes figuran en locks y requirements.lock.txt.
Interpretar los resultados efectivamente obtenidos
Durante el desarrollo, las ramas full y partial alcanzan cada una 360 respuestas exactamente correctas sobre 360 tickets test. La base sin ejemplos obtiene 145; el contexto con ocho demostraciones no obtiene ninguna según el contrato de respuesta completa. Sus 360 salidas test alcanzan el límite de generación. Un ejemplo bruto comienza por un objeto JSON y continúa con una nueva instrucción y otro ticket: la base prosigue la secuencia de demostraciones. No reparamos esta salida conservando solo su primer objeto. Esta observación pertenece a este modelo base y a esta serialización; no es una regla general contra few-shot.
El corpus regular hace aquí suficientes las dos adaptaciones. La comparación full/partial incluye dos learning rates distintos; no aísla el solo efecto del número de parámetros. Los tiempos continuos de aprendizaje+validación observados son 372,91 s (full) y 152,13 s (partial), con picos PyTorch de 9 964 563 968 y 2 611 971 584 bytes. Estas medidas dependen del equipo y su carga; no incluyen adquisición, generación, repeticiones de reanudación ni serialización final. No representan toda la VRAM reservada por Docker.
El ejercicio que supervisa también el prompt alcanza igualmente 360/360. Activa 430238 tokens frente a 105052 en la loss de respuesta sola para las mismas 420 updates. Su NLLtest de respuesta es 1,06265e−5, frente a 6,89464e−7 para full. No se observa ninguna degradación de respuestas finales en este ejercicio; esto no borra la diferencia de señal y denominador. Los valores detallados y límites se dan a continuación, y el recorrido del lector los recalcula íntegramente.
Observaciones de desarrollo
Valores recalculados a partir de las salidas brutas de este recorrido. No son umbrales de ganancia; el recorrido desde el ZIP los recalcula sobre las mismas poblaciones.
{
"status": "PASS",
"documents": 2400,
"test_documents": 360,
"parameters": 494032768,
"partial_trainable_parameters": 29824768,
"metrics": {
"zeroshot": {
"documents": 360,
"json_parsable": 329,
"json_parsable_rate": 0.9138888888888889,
"schema_valid": 167,
"schema_valid_rate": 0.4638888888888889,
"exact_match": 145,
"exact_match_rate": 0.4027777777777778,
"field_correct": {
"unit": 167,
"symptom": 161,
"location": 160,
"scheduled_date": 158
},
"field_accuracy": {
"unit": 0.4638888888888889,
"symptom": 0.44722222222222224,
"location": 0.4444444444444444,
"scheduled_date": 0.4388888888888889
},
"absent_opportunities": {
"unit": 15,
"symptom": 9,
"location": 20,
"scheduled_date": 120
},
"correct_null": {
"unit": 0,
"symptom": 0,
"location": 6,
"scheduled_date": 71
},
"invented_nonnull_in_valid_record": {
"unit": 0,
"symptom": 6,
"location": 7,
"scheduled_date": 9
},
"invalid_record_with_absent_field": {
"unit": 15,
"symptom": 3,
"location": 7,
"scheduled_date": 40
},
"invented_rate_over_all_absent_opportunities": {
"unit": 0.0,
"symptom": 0.6666666666666666,
"location": 0.35,
"scheduled_date": 0.075
},
"null_failure_rate_over_all_absent_opportunities": {
"unit": 1.0,
"symptom": 1.0,
"location": 0.7,
"scheduled_date": 0.4083333333333333
},
"truncated": 31
},
"fewshot": {
"documents": 360,
"json_parsable": 0,
"json_parsable_rate": 0.0,
"schema_valid": 0,
"schema_valid_rate": 0.0,
"exact_match": 0,
"exact_match_rate": 0.0,
"field_correct": {
"unit": 0,
"symptom": 0,
"location": 0,
"scheduled_date": 0
},
"field_accuracy": {
"unit": 0.0,
"symptom": 0.0,
"location": 0.0,
"scheduled_date": 0.0
},
"absent_opportunities": {
"unit": 15,
"symptom": 9,
"location": 20,
"scheduled_date": 120
},
"correct_null": {
"unit": 0,
"symptom": 0,
"location": 0,
"scheduled_date": 0
},
"invented_nonnull_in_valid_record": {
"unit": 0,
"symptom": 0,
"location": 0,
"scheduled_date": 0
},
"invalid_record_with_absent_field": {
"unit": 15,
"symptom": 9,
"location": 20,
"scheduled_date": 120
},
"invented_rate_over_all_absent_opportunities": {
"unit": 0.0,
"symptom": 0.0,
"location": 0.0,
"scheduled_date": 0.0
},
"null_failure_rate_over_all_absent_opportunities": {
"unit": 1.0,
"symptom": 1.0,
"location": 1.0,
"scheduled_date": 1.0
},
"truncated": 360
},
"full": {
"documents": 360,
"json_parsable": 360,
"json_parsable_rate": 1.0,
"schema_valid": 360,
"schema_valid_rate": 1.0,
"exact_match": 360,
"exact_match_rate": 1.0,
"field_correct": {
"unit": 360,
"symptom": 360,
"location": 360,
"scheduled_date": 360
},
"field_accuracy": {
"unit": 1.0,
"symptom": 1.0,
"location": 1.0,
"scheduled_date": 1.0
},
"absent_opportunities": {
"unit": 15,
"symptom": 9,
"location": 20,
"scheduled_date": 120
},
"correct_null": {
"unit": 15,
"symptom": 9,
"location": 20,
"scheduled_date": 120
},
"invented_nonnull_in_valid_record": {
"unit": 0,
"symptom": 0,
"location": 0,
"scheduled_date": 0
},
"invalid_record_with_absent_field": {
"unit": 0,
"symptom": 0,
"location": 0,
"scheduled_date": 0
},
"invented_rate_over_all_absent_opportunities": {
"unit": 0.0,
"symptom": 0.0,
"location": 0.0,
"scheduled_date": 0.0
},
"null_failure_rate_over_all_absent_opportunities": {
"unit": 0.0,
"symptom": 0.0,
"location": 0.0,
"scheduled_date": 0.0
},
"truncated": 0
},
"partial": {
"documents": 360,
"json_parsable": 360,
"json_parsable_rate": 1.0,
"schema_valid": 360,
"schema_valid_rate": 1.0,
"exact_match": 360,
"exact_match_rate": 1.0,
"field_correct": {
"unit": 360,
"symptom": 360,
"location": 360,
"scheduled_date": 360
},
"field_accuracy": {
"unit": 1.0,
"symptom": 1.0,
"location": 1.0,
"scheduled_date": 1.0
},
"absent_opportunities": {
"unit": 15,
"symptom": 9,
"location": 20,
"scheduled_date": 120
},
"correct_null": {
"unit": 15,
"symptom": 9,
"location": 20,
"scheduled_date": 120
},
"invented_nonnull_in_valid_record": {
"unit": 0,
"symptom": 0,
"location": 0,
"scheduled_date": 0
},
"invalid_record_with_absent_field": {
"unit": 0,
"symptom": 0,
"location": 0,
"scheduled_date": 0
},
"invented_rate_over_all_absent_opportunities": {
"unit": 0.0,
"symptom": 0.0,
"location": 0.0,
"scheduled_date": 0.0
},
"null_failure_rate_over_all_absent_opportunities": {
"unit": 0.0,
"symptom": 0.0,
"location": 0.0,
"scheduled_date": 0.0
},
"truncated": 0
},
"prompt_loss": {
"documents": 360,
"json_parsable": 360,
"json_parsable_rate": 1.0,
"schema_valid": 360,
"schema_valid_rate": 1.0,
"exact_match": 360,
"exact_match_rate": 1.0,
"field_correct": {
"unit": 360,
"symptom": 360,
"location": 360,
"scheduled_date": 360
},
"field_accuracy": {
"unit": 1.0,
"symptom": 1.0,
"location": 1.0,
"scheduled_date": 1.0
},
"absent_opportunities": {
"unit": 15,
"symptom": 9,
"location": 20,
"scheduled_date": 120
},
"correct_null": {
"unit": 15,
"symptom": 9,
"location": 20,
"scheduled_date": 120
},
"invented_nonnull_in_valid_record": {
"unit": 0,
"symptom": 0,
"location": 0,
"scheduled_date": 0
},
"invalid_record_with_absent_field": {
"unit": 0,
"symptom": 0,
"location": 0,
"scheduled_date": 0
},
"invented_rate_over_all_absent_opportunities": {
"unit": 0.0,
"symptom": 0.0,
"location": 0.0,
"scheduled_date": 0.0
},
"null_failure_rate_over_all_absent_opportunities": {
"unit": 0.0,
"symptom": 0.0,
"location": 0.0,
"scheduled_date": 0.0
},
"truncated": 0
}
},
"response_nll": {
"zeroshot": 0.3579458403824175,
"full": 6.894639002962341e-07,
"partial": 1.5182218123592618e-05,
"prompt_loss": 1.0626541892029729e-05
},
"quality_gain_required": false,
"scope": "Data-only tokenizer decoding, label formulas, per-tensor hashes, exact active-token budgets, resume evidence, full-vocabulary CE probes and all-case metrics; not BPE re-encoding or scalar reimplementation of the 0.5B Transformer"
}
