Un catalogo fittizio contiene descrizioni eterogenee di componenti. Il compito finale trasforma una descrizione completa, talvolta abbreviata, in una riga canonica. Prima di questo compito supervisionato, una piccola rete impara a ricostruire segmenti rimossi dalle schede.
Confrontare tre normalizzatori sulle stesse schede riservate: regole, encoder-decoder casuale poi fine-tunato, ed encoder-decoder preaddestrato tramite denoising poi fine-tunato. Visualizzare anche una ricostruzione di span.
Scaricare il progetto completo, revisione r001
Due compiti e due segnali
Il denoising rimuove segmenti contigui da una scheda. La rete riceve i byte rimanenti e token sentinella; la sua destinazione contiene soltanto i segmenti rimossi, associati alle stesse sentinelle. L’auto-supervisione proviene dalla scheda iniziale.
Il corpus originale CatalogueSpan-v1 contiene 600 famiglie di dieci formulazioni, cioè 6000 schede. Le famiglie 0–419 sono train, 420–509 validation, 510–599 test: 4200/900/900 schede. Una famiglia condivide riferimento, materiale e diametro.
Le formulazioni 0 e 5 forniscono 840 coppie train e 180 coppie per ogni partizione riservata. Tutti i loro campi sono presenti. I 60 casi incompleti, separati, rimuovono un campo e testano un rifiuto tramite regole prima della generazione. Il modello non viene interrogato.
Byte, span e sentinelle
Il vocabolario è fisso: byte UTF-8 0–255, PAD=256, BOS=257, EOS=258 e 32 sentinelle 259–290. Un carattere accentato può occupare più byte e uno span può tagliarlo. La visualizzazione usa frammenti bytes espliciti; il ripristino considera i byte originali.
Il numero di byte rimossi è round(0,15 × lunghezza), limitato tra 1 e lunghezza−1. Il numero di span è vicino al numero rimosso diviso per tre, con vincoli di non-vuoto.
L’encoder riceve BOS, l’input corrotto, EOS. Il decoder riceve BOS poi target[:-1] e predice target, comprese sentinelle ed EOS. PAD non è mai una destinazione attiva.
Le maschere validation/test sono fissate per identificatore prima dell’addestramento. Le maschere train sono estratte dinamicamente con un generatore esplicito, il cui stato è salvato.
Tre percorsi di attenzione
La rete originale contiene tre blocchi encoder e tre blocchi decoder, larghezza 192, sei teste e MLP 768 con GELU. Ogni strato è allocato e inizializzato indipendentemente.
L’auto-attenzione dell’encoder è bidirezionale e maschera solo le chiavi PAD. Quella del decoder maschera il futuro e le chiavi PAD. La cross-attention consulta tutta la codifica visibile.
Protocollo di apprendimento
I semi 42 e 43 producono due catene indipendenti. Per ciascuna viene salvata un’inizializzazione. Il ramo diretto parte da questi pesi; quello trasferito riceve i pesi dopo 600 aggiornamenti di denoising sulle sole 4200 schede train.
AdamW aggiorna tutti i parametri in FP32 CUDA, con TF32 disattivato, weight_decay 0,01 e norma dei gradienti limitata a 1. Il tasso massimo è 0,0003 nel preaddestramento e 0,0001 nella normalizzazione.
Ogni fase è eseguita davvero interrotta, continua poi ripresa. Il preaddestramento si interrompe a 240 passi su 600; il SFT a 120 su 300. Modello, AdamW, scheduler, cursore, permutazione e RNG sono conservati.
Valutare liberamente
La ricostruzione è valutata prima e dopo il preaddestramento sulle stesse 900 schede test e maschere fissate, con NLL per token destinazione e dodici prime ricostruzioni libere illustrative. La normalizzazione è valutata su 180 uscite libere per ramo e seme. La decodifica greedy autorizza fino a 128 nuovi token; solo PAD e BOS sono rimossi dal supporto. Le sentinelle non vengono rimosse per forzare una riga canonica: la loro comparsa è un fallimento visibile di formato.
La validità di completamento richiede EOS e il formato completo. L’exact match grezzo confronta l’intera riga; l’esattezza per campo confronta i valori analizzati. Una riga numericamente corretta ma formattata con zeri iniziali può avere campi corretti senza exact match grezzo.
Le tabelle indicano separatamente annotazione, token trattati e tempo per fase. Le durate osservate includono validation e scrittura del checkpoint nel processo; non sono misure del tempo di kernel CUDA isolati.
Raddoppiare le annotazioni
L’esercizio prende quattro formulazioni train (0,2,5,7), cioè 1680 coppie, e riaddestra due nuovi rami dagli stessi stati iniziali, diretti o preaddestrati. I 180 casi test, due semi e 300 aggiornamenti restano fissi. Un miglioramento non è imposto; un trasferimento nullo o negativo resta nei risultati.
Creare le schede e le partizioni
Separare famiglia e formulazione
Invariante: 420/90/90 famiglie, 840/180/180 coppie a valle prima delle esclusioni documentate.
Errore frequente: un’altra formulazione dello stesso componente appare nel test; suddividere le famiglie prima della generazione.
File src/preentrainement_denoising/data.py
import hashlib, json, re
from pathlib import Path
from .config import save_json, sha
MATERIALS = ["aluminium", "acier", "cuivre", "nylon", "laiton", "bronze"]
ALIASES = ["alu", "ac", "cu", "nyl", "lait", "brz"]
FIELDS = ["reference", "materiau", "diametre_mm"]
def split_catalog(family):
return "train" if family < 420 else "validation" if family < 510 else "test"
def canonical(values):
return ";".join(key + "=" + str(values[key]) for key in FIELDS)
def render(values, formulation):
ref = values["reference"]
material = values["materiau"]
diameter = values["diametre_mm"]
alias = ALIASES[MATERIALS.index(material)] if material else "?"
ref = ref or "?"
material = material or "?"
diameter = str(diameter) if diameter is not None else "?"
patterns = [
f"Piece {ref}, {material}, diametre {diameter} mm.",
f"Reference {ref} : matiere {material} ; diametre {diameter} mm.",
f"{material} / piece {ref} / diam. {diameter}mm",
f"Diametre {diameter} mm ; reference {ref} ; materiau {material}.",
f"Fiche piece={ref} ; mat={alias} ; D={diameter} mm",
f"piece {ref} / {alias} / diam.{diameter}mm",
f"Mat. {alias}, ref. {ref}, diametre={diameter}mm.",
f"{ref} | matiere:{material} | diam:{diameter} mm",
f"D={diameter}mm pour piece {ref} en {material}.",
f"Catalogue : reference {ref}. Diametre {diameter} mm. Matiere {material}.",
]
return patterns[formulation]
def generate_catalog():
pages = []
gold = []
for family in range(600):
values = {
"reference": f"K{family:04d}",
"materiau": MATERIALS[family % 6],
"diametre_mm": 8 + 2 * ((family // 6) % 20),
}
for j in range(10):
text = render(values, j)
identifier = f"C{family:03d}-{j}"
pages.append(
{
"document_id": identifier,
"family": family,
"formulation": j,
"split": split_catalog(family),
"text": text,
"content_sha256": hashlib.sha256(text.encode()).hexdigest(),
}
)
gold.append(
{
"document_id": identifier,
"target": values.copy(),
"target_text": canonical(values),
}
)
if len({r["text"].casefold() for r in pages}) != 6000:
raise ValueError("Duplicate normalized catalogue record")
missing = []
for index in range(60):
family = 510 + index
field = FIELDS[index % 3]
values = {
"reference": f"K{family:04d}",
"materiau": MATERIALS[family % 6],
"diametre_mm": 8 + 2 * ((family // 6) % 20),
}
values[field] = None
missing.append(
{
"document_id": f"M{index:03d}",
"source_family": family,
"missing_field": field,
"text": render(values, index % 10),
}
)
return pages, gold, missing
def select_labeled_pairs(pages, gold, split, formulations):
byid = {r["document_id"]: r for r in gold}
return [
{
**p,
"target": byid[p["document_id"]]["target"],
"target_text": byid[p["document_id"]]["target_text"],
}
for p in pages
if p["split"] == split and p["formulation"] in formulations
]
def prepare(config):
root = Path("data/generated")
root.mkdir(parents=True, exist_ok=True)
if (root / "manifest.json").exists():
raise ValueError("Preserve frozen catalogue; use a new extraction")
pages, gold, missing = generate_catalog()
save_json(root / "input/pages.json", pages)
save_json(root / "labels/gold.json", gold)
save_json(root / "input/invalid.json", missing)
audit = {}
for split in ["train", "validation", "test"]:
selected = [p for p in pages if p["split"] == split]
save_json(root / f"{split}/pages.json", selected)
pairs = select_labeled_pairs(pages, gold, split, config["labeled_formulations"])
audit[split] = {
"documents": len(selected),
"families": len({p["family"] for p in selected}),
"labeled_pairs": len(pairs),
}
if split != "test":
save_json(root / f"labels/{split}.json", pairs)
exercise = select_labeled_pairs(
pages, gold, "train", config["exercise_labeled_formulations"]
)
save_json(root / "labels/train-pairs.json", exercise)
from .corruption import corrupt_with_sentinels
import random
frozen = [
{
"document_id": p["document_id"],
**corrupt_with_sentinels(
list(p["text"].encode()),
random.Random(420000 + i),
config["noise_density"],
config["mean_span_length"],
),
}
for i, p in enumerate(pages)
if p["split"] != "train"
]
save_json(root / "input/pool.json", frozen)
validation_ids = {p["document_id"] for p in pages if p["split"] == "validation"}
save_json(
root / "validation/corruptions.json",
[r for r in frozen if r["document_id"] in validation_ids],
)
manifest = {
"dataset": "CatalogueSpan-v1",
"rights": "CC0-1.0",
"seed": config["seed"],
"split_audit": audit,
"exercise_labeled_pairs": len(exercise),
"missing_cases": len(missing),
"fixed_evaluation_corruptions": len(frozen),
"normalized_duplicates": 0,
"files": {str(f.relative_to(root)): sha(f) for f in root.rglob("*.json")},
}
save_json(root / "manifest.json", manifest)
save_json("outputs/data-audit.json", manifest)
return audit
Codificare la corruzione per span
Sentinelle e destinazione auto-supervisionata
Invariante: la ricostruzione oracle ripristina tutti i byte della scheda.
Errore frequente: due buchi condividono la stessa sentinella; assegnare token unici e verificare la capacità.
File src/preentrainement_denoising/corruption.py
import random
PAD = 256
BOS = 257
EOS = 258
FIRST = 259
SENTINELS = 32
VOCAB = 291
def positive_parts(total, parts, generator):
if not 1 <= parts <= total:
raise ValueError("A span must contain at least one token")
boundaries = [0] + sorted(generator.sample(range(1, total), parts - 1)) + [total]
return [b - a for a, b in zip(boundaries, boundaries[1:])]
def sample_noise_spans(length, generator, density=0.15, mean_length=3.0):
if length < 2 or not 0 < density < 1 or mean_length <= 0:
raise ValueError("Invalid corruption parameters")
count = max(1, min(length - 1, round(length * density)))
spans = max(1, min(count, length - count, round(count / mean_length)))
if spans + 1 > SENTINELS:
raise ValueError("Not enough unique sentinels including the terminal sentinel")
noise = positive_parts(count, spans, generator)
visible = positive_parts(length - count, spans, generator)
cursor = 0
ranges = []
for a, b in zip(visible, noise):
cursor += a
ranges.append([cursor, cursor + b])
cursor += b
assert cursor == length
return ranges
def corrupt_with_sentinels(clean, generator, density=0.15, mean_length=3.0):
if not clean or any(type(v) != int or not 0 <= v < 256 for v in clean):
raise ValueError("Corrupt only raw byte tokens")
ranges = sample_noise_spans(len(clean), generator, density, mean_length)
encoder = []
target = []
cursor = 0
for index, (start, end) in enumerate(ranges):
encoder.extend(clean[cursor:start])
encoder.append(FIRST + index)
target.append(FIRST + index)
target.extend(clean[start:end])
cursor = end
encoder.extend(clean[cursor:])
target.extend([FIRST + len(ranges), EOS])
return {
"clean_ids": clean,
"encoder_ids": [BOS] + encoder + [EOS],
"target_ids": target,
"spans": ranges,
"removed_tokens": sum(b - a for a, b in ranges),
}
def restore_from_target(encoder, target):
if encoder[0] != BOS or encoder[-1] != EOS or target[-1] != EOS:
raise ValueError("Missing boundary token")
pieces = {}
current = None
for token in target[:-1]:
if FIRST <= token < FIRST + SENTINELS:
if token in pieces or token != FIRST + len(pieces):
raise ValueError("Repeated or out-of-order sentinel")
pieces[token] = []
current = token
elif 0 <= token < 256 and current is not None:
pieces[current].append(token)
else:
raise ValueError("Invalid reconstruction target")
if (
len(pieces) < 2
or pieces[max(pieces)]
or any(not v for k, v in pieces.items() if k != max(pieces))
):
raise ValueError("Nonempty spans and empty terminal sentinel required")
seen = []
result = []
for token in encoder[1:-1]:
if FIRST <= token < FIRST + SENTINELS:
if token not in pieces or token == max(pieces) or token in seen:
raise ValueError("Unmatched encoder sentinel")
seen.append(token)
result.extend(pieces[token])
elif 0 <= token < 256:
result.append(token)
else:
raise ValueError("Invalid encoder byte")
if seen != list(range(FIRST, max(pieces))):
raise ValueError("Missing or reordered span")
return result
def display(ids):
# Sentinels can split a UTF-8 character: show raw byte runs using repr,
# never treat a lossy decoder display as an exact reconstructed string.
result = []
buffer = []
for value in ids:
if value < 256:
buffer.append(value)
continue
if buffer:
result.append(repr(bytes(buffer)))
buffer = []
result.append(
{PAD: "<PAD>", BOS: "<BOS>", EOS: "<EOS>"}.get(
value, f"<extra_{value-FIRST}>"
)
)
if buffer:
result.append(repr(bytes(buffer)))
return " ".join(result)
Costruire l’encoder-decoder
Tre percorsi di attenzione distinti
Invariante: il test causale del decoder passa e la cross-attention ignora i PAD dell’input.
Errore frequente: una maschera causale è applicata all’encoder per copia; visualizzare separatamente le tre maschere.
File src/preentrainement_denoising/model.py
import math, torch
from torch import nn
from .corruption import PAD, BOS, EOS, VOCAB
class Attention(nn.Module):
def __init__(self, width, heads, dropout):
super().__init__()
if width % heads:
raise ValueError("Width must be divisible by heads")
self.width = width
self.heads = heads
self.query = nn.Linear(width, width)
self.key = nn.Linear(width, width)
self.value = nn.Linear(width, width)
self.output = nn.Linear(width, width)
self.dropout = nn.Dropout(dropout)
def forward(self, query, memory, key_mask, causal=False):
batch, steps, width = query.shape
length = memory.shape[1]
size = width // self.heads
if not key_mask.any(1).all():
raise ValueError("Attention needs at least one real key")
q = self.query(query).reshape(batch, steps, self.heads, size).transpose(1, 2)
k = self.key(memory).reshape(batch, length, self.heads, size).transpose(1, 2)
v = self.value(memory).reshape(batch, length, self.heads, size).transpose(1, 2)
allowed = key_mask[:, None, None, :].expand(batch, 1, steps, length)
if causal:
if steps != length:
raise ValueError("Self-attention causal dimensions differ")
allowed = (
allowed
& torch.ones(
steps, length, device=query.device, dtype=torch.bool
).tril()[None, None]
)
if not allowed.any(-1).all():
raise ValueError(
"Use right padding with a visible BOS, not an empty causal prefix"
)
weights = (
(q @ k.transpose(-1, -2) / math.sqrt(size))
.masked_fill(~allowed, float("-inf"))
.softmax(-1)
)
x = (
(self.dropout(weights) @ v)
.transpose(1, 2)
.contiguous()
.reshape(batch, steps, width)
)
return self.dropout(self.output(x))
def mlp(width, hidden, dropout):
return nn.Sequential(
nn.Linear(width, hidden),
nn.GELU(),
nn.Linear(hidden, width),
nn.Dropout(dropout),
)
class EncoderBlock(nn.Module):
def __init__(self, width, heads, hidden, dropout):
super().__init__()
self.norm1 = nn.LayerNorm(width)
self.attention = Attention(width, heads, dropout)
self.norm2 = nn.LayerNorm(width)
self.feedforward = mlp(width, hidden, dropout)
def forward(self, x, mask):
normalized = self.norm1(x)
x = x + self.attention(normalized, normalized, mask, False)
return x + self.feedforward(self.norm2(x))
class DecoderBlock(nn.Module):
def __init__(self, width, heads, hidden, dropout):
super().__init__()
self.norm1 = nn.LayerNorm(width)
self.self_attention = Attention(width, heads, dropout)
self.norm2 = nn.LayerNorm(width)
self.cross_attention = Attention(width, heads, dropout)
self.norm3 = nn.LayerNorm(width)
self.feedforward = mlp(width, hidden, dropout)
def forward(self, x, mask, memory, memory_mask):
normalized = self.norm1(x)
x = x + self.self_attention(normalized, normalized, mask, True)
x = x + self.cross_attention(self.norm2(x), memory, memory_mask, False)
return x + self.feedforward(self.norm3(x))
class SpanDenoiser(nn.Module):
def __init__(self, config):
super().__init__()
width = config["width"]
self.context = config["context"]
self.embedding = nn.Embedding(VOCAB, width)
self.encoder_position = nn.Embedding(self.context, width)
self.decoder_position = nn.Embedding(self.context, width)
self.dropout = nn.Dropout(config["dropout"])
args = (width, config["heads"], config["mlp_width"], config["dropout"])
self.encoders = nn.ModuleList(
[EncoderBlock(*args) for _ in range(config["encoder_blocks"])]
)
self.decoders = nn.ModuleList(
[DecoderBlock(*args) for _ in range(config["decoder_blocks"])]
)
self.encoder_norm = nn.LayerNorm(width)
self.decoder_norm = nn.LayerNorm(width)
self.apply(self.initialize)
@staticmethod
def initialize(module):
if isinstance(module, (nn.Linear, nn.Embedding)):
nn.init.normal_(module.weight, mean=0.0, std=0.02)
if isinstance(module, nn.Linear) and module.bias is not None:
nn.init.zeros_(module.bias)
def embed(self, ids, position):
if ids.ndim != 2 or ids.shape[1] > self.context:
raise ValueError("No silent sequence truncation")
return self.dropout(
self.embedding(ids)
+ position(torch.arange(ids.shape[1], device=ids.device))[None]
)
def encode(self, ids, mask):
x = self.embed(ids, self.encoder_position)
for layer in self.encoders:
x = layer(x, mask)
return self.encoder_norm(x)
def decode(self, ids, mask, memory, memory_mask):
x = self.embed(ids, self.decoder_position)
for layer in self.decoders:
x = layer(x, mask, memory, memory_mask)
return self.decoder_norm(x) @ self.embedding.weight.T
def forward(self, encoder_ids, encoder_mask, decoder_ids, decoder_mask):
return self.decode(
decoder_ids,
decoder_mask,
self.encode(encoder_ids, encoder_mask),
encoder_mask,
)
def make_batch(rows, device="cuda", pad_encoder=None, pad_target=None):
if not rows or any(not r["encoder_ids"] or not r["target_ids"] for r in rows):
raise ValueError("Empty encoder or target")
ne = max(len(r["encoder_ids"]) for r in rows)
nt = max(len(r["target_ids"]) for r in rows)
pe = ne if pad_encoder is None else pad_encoder
pt = nt if pad_target is None else pad_target
if pe < ne or pt < nt:
raise ValueError("Padding must not truncate")
encoder = torch.full((len(rows), pe), PAD, dtype=torch.long, device=device)
decoder = torch.full((len(rows), pt), PAD, dtype=torch.long, device=device)
labels = torch.full((len(rows), pt), -100, dtype=torch.long, device=device)
em = torch.zeros_like(encoder, dtype=torch.bool)
dm = torch.zeros_like(decoder, dtype=torch.bool)
for i, row in enumerate(rows):
source = row["encoder_ids"]
target = row["target_ids"]
encoder[i, : len(source)] = torch.tensor(source, device=device)
decoder[i, : len(target)] = torch.tensor([BOS] + target[:-1], device=device)
labels[i, : len(target)] = torch.tensor(target, device=device)
em[i, : len(source)] = True
dm[i, : len(target)] = True
return encoder, em, decoder, dm, labels
def target_loss(logits, labels):
active = labels != -100
if not active.any():
raise ValueError("No active target")
values = torch.nn.functional.cross_entropy(
logits[active].float(), labels[active], reduction="none"
)
return values.sum(), int(active.sum()), values
Preaddestrare poi fine-tunare i due rami
Cambio di obiettivo, stessi pesi riutilizzati
Invariante: i due rami vedono gli stessi ID supervisionati; i loro costi per fase sono registrati.
Errore frequente: caricare i pesi preaddestrati in entrambi i rami cancella la baseline; confrontare i loro hash iniziali.
File src/preentrainement_denoising/train.py
import math, random, time
from pathlib import Path
import torch
from .config import read_json, save_json, sha, seed_all, rng_state, restore_rng
from .model import SpanDenoiser, make_batch, target_loss
from .corruption import BOS, EOS, corrupt_with_sentinels
from .state import digest, equal, save_training_state, export_weights, load_weights
class ExampleSampler:
def __init__(self, size, seed):
self.size = size
self.generator = random.Random(seed)
self.order = []
self.cursor = 0
self.epoch = 0
def take(self, count):
selected = []
while len(selected) < count:
if self.cursor == len(self.order):
self.order = list(range(self.size))
self.generator.shuffle(self.order)
self.cursor = 0
self.epoch += 1
selected.append(self.order[self.cursor])
self.cursor += 1
return selected
def state_dict(self):
return {
"size": self.size,
"order": self.order,
"cursor": self.cursor,
"epoch": self.epoch,
"rng": self.generator.getstate(),
}
def load_state_dict(self, state):
if (
state["size"] != self.size
or sorted(state["order"]) != list(range(self.size))
or not 0 <= state["cursor"] <= self.size
):
raise ValueError("Invalid sampler checkpoint")
self.order = list(state["order"])
self.cursor = state["cursor"]
self.epoch = state["epoch"]
self.generator.setstate(state["rng"])
def identity_config(config):
return {k: v for k, v in config.items() if k != "exercise"}
def initialize(config, seed):
seed_all(seed)
model = SpanDenoiser(config)
root = Path("assets/checkpoints")
save_training_state(
root / f"initial-{seed}.pt",
{
"model": model.state_dict(),
"config": identity_config(config),
"seed": seed,
"tokenizer": {
"bytes": [0, 255],
"pad": 256,
"bos": 257,
"eos": 258,
"sentinels": [259, 290],
},
},
)
export_weights(f"outputs/model-initial-{seed}.json", model)
return digest(model.state_dict())
def supervised_row(page):
return {
"document_id": page["document_id"],
"encoder_ids": [BOS] + list(page["text"].encode()) + [EOS],
"target_ids": list(page["target_text"].encode()) + [EOS],
}
def phase_rows(phase, config):
if phase == "pretrain":
train = read_json("data/generated/train/pages.json")
validation_pages = read_json("data/generated/validation/pages.json")
validation_ids = {p["document_id"] for p in validation_pages}
validation = [
p
for p in read_json("data/generated/validation/corruptions.json")
if p["document_id"] in validation_ids
]
else:
path = (
"data/generated/labels/train-pairs.json"
if phase.startswith("exercise_")
else "data/generated/labels/train.json"
)
train = read_json(path)
validation = [
supervised_row(p)
for p in read_json("data/generated/labels/validation.json")
]
if any(p["split"] != "train" for p in train):
raise ValueError("Training family leakage")
expected = (
4200 if phase == "pretrain" else 1680 if phase.startswith("exercise_") else 840
)
if len(train) != expected or len(validation) != (
900 if phase == "pretrain" else 180
):
raise ValueError("Unexpected phase population")
return train, validation
def lr_at(config, phase, step, total):
peak = (
config["learning_rate_pretrain"]
if phase == "pretrain"
else config["learning_rate_finetune"]
)
warmup = max(1, round(total * config["warmup_ratio"]))
return (
peak * (step + 1) / warmup
if step < warmup
else peak
* (
1
- min(1.0, (step - warmup) / max(1, total - warmup - 1))
* (1 - config["minimum_lr_ratio"])
)
)
@torch.no_grad()
def validation_nll(model, rows, batch_size):
model.eval()
total = 0.0
count = 0
for i in range(0, len(rows), batch_size):
encoder, em, decoder, dm, labels = make_batch(rows[i : i + batch_size])
_, n, values = target_loss(model(encoder, em, decoder, dm), labels)
total += float(values.double().sum())
count += n
model.train()
return {
"documents": len(rows),
"target_tokens": count,
"target_nll_nats_per_token": total / count,
}
def train_phase(config, seed, phase, mode):
if (
Path("data/generated/test").exists()
or Path("data/generated/labels/gold.json").exists()
):
raise ValueError("Learning container must not mount test or global gold")
if phase not in [
"pretrain",
"random",
"transferred",
"exercise_random",
"exercise_transferred",
] or mode not in ["interrupt", "continuous", "resume"]:
raise ValueError("Unknown phase/mode")
seed_all(seed)
train, validation = phase_rows(phase, config)
source = Path("assets/checkpoints") / (
f"pretrain-{seed}-resume.pt"
if phase in ["transferred", "exercise_transferred"]
else f"initial-{seed}.pt"
)
initial = torch.load(source, map_location="cpu", weights_only=False)
if initial.get(
"config", initial.get("identity", {}).get("config")
) != identity_config(config):
raise ValueError("Tokenizer/config changed between phases")
model = SpanDenoiser(config)
model.load_state_dict(initial["model"], strict=True)
initial_digest = digest(model.state_dict())
del initial
model = model.cuda()
optimizer = torch.optim.AdamW(
model.parameters(),
lr=config["learning_rate_pretrain"],
weight_decay=config["weight_decay"],
)
sampler = ExampleSampler(len(train), seed + 100)
corruption_rng = random.Random(seed + 200)
total = (
config["pretrain_steps"] if phase == "pretrain" else config["finetune_steps"]
)
interrupt = (
config["pretrain_interrupt"]
if phase == "pretrain"
else config["finetune_interrupt"]
)
stop = interrupt if mode == "interrupt" else total
identity = {
"phase": phase,
"seed": seed,
"config": identity_config(config),
"source_checkpoint_sha256": sha(source),
"initial_model_sha256": initial_digest,
"training_ids": [p["document_id"] for p in train],
"training_sha256": (
sha("data/generated/train/pages.json")
if phase == "pretrain"
else sha(
"data/generated/labels/train-pairs.json"
if phase.startswith("exercise_")
else "data/generated/labels/train.json"
)
),
"implementation_sha256": {
n: sha("src/preentrainement_denoising/" + n)
for n in ["model.py", "corruption.py", "train.py", "state.py"]
},
}
start = 0
history = []
validation_history = []
root = Path("assets/checkpoints")
path = root / f"{phase}-{seed}-{mode}.pt"
if path.exists():
raise ValueError("Preserve phase checkpoint")
if mode == "resume":
state = torch.load(
root / f"{phase}-{seed}-interrupt.pt",
map_location="cpu",
weights_only=False,
)
if state["identity"] != identity or state["scheduler"] != {
"next_step": state["steps"],
"next_lr": lr_at(config, phase, state["steps"], total),
}:
raise ValueError("Resume identity/scheduler mismatch")
model.load_state_dict(state["model"], strict=True)
optimizer.load_state_dict(state["optimizer"])
sampler.load_state_dict(state["sampler"])
corruption_rng.setstate(state["corruption_rng"])
restore_rng(state["rng"])
start = state["steps"]
history = state["history"]
validation_history = state["validation_history"]
del state
model.train()
begun = time.perf_counter()
torch.cuda.reset_peak_memory_stats()
for step in range(start, stop):
selected = sampler.take(config["batch_size"])
rows = []
for index in selected:
page = train[index]
row = (
{
"document_id": page["document_id"],
**corrupt_with_sentinels(
list(page["text"].encode()),
corruption_rng,
config["noise_density"],
config["mean_span_length"],
),
}
if phase == "pretrain"
else supervised_row(page)
)
rows.append(row)
denominator = sum(len(r["target_ids"]) for r in rows)
lr = lr_at(config, phase, step, total)
for group in optimizer.param_groups:
group["lr"] = lr
optimizer.zero_grad(set_to_none=True)
loss_total = 0.0
for offset in range(0, len(rows), config["microbatch"]):
encoder, em, decoder, dm, labels = make_batch(
rows[offset : offset + config["microbatch"]]
)
summed, n, values = target_loss(model(encoder, em, decoder, dm), labels)
if not torch.isfinite(summed):
raise ValueError("Nonfinite loss")
(summed / denominator).backward()
loss_total += float(values.detach().double().sum())
norm = torch.nn.utils.clip_grad_norm_(
model.parameters(), config["gradient_clip"], error_if_nonfinite=True
)
optimizer.step()
history.append(
{
"step": step + 1,
"document_ids": [r["document_id"] for r in rows],
"spans": [r.get("spans", []) for r in rows],
"target_tokens": denominator,
"target_nll_nats_per_token": loss_total / denominator,
"learning_rate": lr,
"gradient_norm_before_clip": float(norm),
}
)
if (step + 1) % 100 == 0:
observed = validation_nll(model, validation, config["batch_size"])
validation_history.append({"step": step + 1, **observed})
print(
{
"seed": seed,
"phase": phase,
"mode": mode,
**history[-1],
"validation": observed,
},
flush=True,
)
torch.cuda.synchronize()
state = {
"identity": identity,
"steps": stop,
"model": model.state_dict(),
"optimizer": optimizer.state_dict(),
"scheduler": {"next_step": stop, "next_lr": lr_at(config, phase, stop, total)},
"sampler": sampler.state_dict(),
"corruption_rng": corruption_rng.getstate(),
"rng": rng_state(),
"history": history,
"validation_history": validation_history,
}
final = digest(model.state_dict())
if final == initial_digest:
raise ValueError("No learned parameter change")
save_training_state(path, state)
if mode == "resume":
export_weights(f"outputs/model-{phase}-{seed}.json", model)
save_json(
f"outputs/train-{phase}-{seed}-{mode}.json",
{
"seed": seed,
"phase": phase,
"mode": mode,
"identity": identity,
"steps": stop,
"history": history,
"validation_history": validation_history,
"initial_model_sha256": initial_digest,
"final_model_sha256": final,
"parameters": sum(p.numel() for p in model.parameters()),
"checkpoint_sha256": sha(path),
"checkpoint_bytes": path.stat().st_size,
"max_torch_allocated_bytes": torch.cuda.max_memory_allocated(),
"seconds": time.perf_counter() - begun,
"test_visible": False,
},
)
def verify_resume(seed, phase):
root = Path("assets/checkpoints")
a = torch.load(
root / f"{phase}-{seed}-continuous.pt", map_location="cpu", weights_only=False
)
b = torch.load(
root / f"{phase}-{seed}-resume.pt", map_location="cpu", weights_only=False
)
checks = {k: equal(a[k], b[k]) for k in a}
if not all(checks.values()):
raise ValueError("Resume differs: " + str(checks))
save_json(
f"outputs/resume-{phase}-{seed}.json",
{
"status": "PASS",
"all_state_exact": True,
"checks": checks,
"boundary": "complete optimizer update, including corruption and permutation RNG",
},
)
Confrontare la normalizzazione, non solo le loss
Metriche a valle e copertura
Invariante: 180 casi test per ramo con uno stato esplicito per ogni fallimento di formato.
Errore frequente: una differenza di formato viene contata come campo falso senza spiegarla; pubblicare exact match grezzo e campi normalizzati.
File src/preentrainement_denoising/evaluate.py
import math, time
from pathlib import Path
import torch
from .config import read_json, save_json, sha, seed_all
from .corruption import PAD, BOS, EOS, FIRST, VOCAB, restore_from_target
from .data import FIELDS
from .model import make_batch, target_loss
from .state import load_weights, digest
from .train import supervised_row
from .baseline import normalize_with_rules, parse_canonical, upstream_complete
@torch.no_grad()
def greedy(model, rows, config):
model.eval()
maximum = max(len(r["encoder_ids"]) for r in rows)
encoder = torch.full((len(rows), maximum), PAD, dtype=torch.long, device="cuda")
mask = torch.zeros_like(encoder, dtype=torch.bool)
for i, row in enumerate(rows):
encoder[i, : len(row["encoder_ids"])] = torch.tensor(
row["encoder_ids"], device="cuda"
)
mask[i, : len(row["encoder_ids"])] = True
memory = model.encode(encoder, mask)
decoder = torch.full((len(rows), 1), BOS, dtype=torch.long, device="cuda")
dm = torch.ones_like(decoder, dtype=torch.bool)
done = [False] * len(rows)
outputs = [[] for _ in rows]
probes = []
for step in range(config["max_new_tokens"]):
logits = model.decode(decoder, dm, memory, mask)[:, -1].float()
logits[:, [PAD, BOS]] = float("-inf")
chosen = logits.argmax(-1).cpu().tolist()
if step < 2:
probes.append(
{
"generation_position": step,
"logits": [
None if not math.isfinite(x) else x
for x in logits[0].cpu().tolist()
],
"chosen": chosen[0],
}
)
next_ids = []
next_mask = []
for i, token in enumerate(chosen):
if done[i]:
next_ids.append(PAD)
next_mask.append(False)
else:
outputs[i].append(token)
next_ids.append(token)
next_mask.append(True)
done[i] = token == EOS
if all(done):
break
decoder = torch.cat(
[decoder, torch.tensor(next_ids, device="cuda")[:, None]], 1
)
dm = torch.cat([dm, torch.tensor(next_mask, device="cuda")[:, None]], 1)
return outputs, probes
def decode_generated(ids):
finish = "eos" if ids and ids[-1] == EOS else "length"
content = ids[:-1] if finish == "eos" else ids
valid_bytes = all(0 <= v < 256 for v in content)
raw = bytes(content) if valid_bytes else None
try:
text = raw.decode("utf-8") if raw is not None else None
utf8 = raw is not None
except UnicodeDecodeError:
text = None
utf8 = False
return {
"finish_reason": finish,
"content_bytes_hex": raw.hex() if raw is not None else None,
"byte_only": valid_bytes,
"utf8_valid": utf8,
"text": text,
}
def selected_pages(config):
return [
p
for p in read_json("data/generated/input/pages.json")
if p["split"] == "test" and p["formulation"] in config["labeled_formulations"]
]
@torch.no_grad()
def predict(config, seed, phase):
if Path("data/generated/labels").exists():
raise ValueError("Generation service must not mount target labels")
seed_all(seed)
path = Path(f"outputs/model-{phase}-{seed}.json")
model = load_weights(path, config).cuda().eval()
pages = selected_pages(config)
if len(pages) != 180:
raise ValueError("Expected180heldout pairs")
rows = [
{
"document_id": p["document_id"],
"encoder_ids": [BOS] + list(p["text"].encode()) + [EOS],
}
for p in pages
]
result = []
probes = []
begun = time.perf_counter()
torch.cuda.reset_peak_memory_stats()
for offset in range(0, len(rows), config["generation_batch_size"]):
batch = rows[offset : offset + config["generation_batch_size"]]
generated, actual = greedy(model, batch, config)
if offset == 0:
probes = actual
for row, ids in zip(batch, generated):
decoded = decode_generated(ids)
parsed = (
parse_canonical(decoded["text"], decoded["finish_reason"])
if decoded["text"] is not None
else {
"format_valid": False,
"completion_valid": False,
"record": None,
"reason": (
"NON_BYTE_TOKEN" if not decoded["byte_only"] else "INVALID_UTF8"
),
}
)
result.append({**row, "generated_ids": ids, **decoded, "parsed": parsed})
torch.cuda.synchronize()
save_json(
f"outputs/predictions-{phase}-{seed}.json",
{
"phase": phase,
"seed": seed,
"model_sha256": digest(model.state_dict()),
"model_json_sha256": sha(path),
"test_input_sha256": sha("data/generated/input/pages.json"),
"rows": result,
"first_case_logit_probes": probes,
"seconds": time.perf_counter() - begun,
"max_torch_allocated_bytes": torch.cuda.max_memory_allocated(),
"gold_visible": False,
"decoding": "greedy291tokens; PAD/BOS suppressed; sentinels not suppressed",
},
)
@torch.no_grad()
def nll_rows(model, rows, batch_size):
results = []
probes = []
for offset in range(0, len(rows), batch_size):
batch = rows[offset : offset + batch_size]
encoder, em, decoder, dm, labels = make_batch(batch)
logits = model(encoder, em, decoder, dm)
_, _, values = target_loss(logits, labels)
cursor = 0
for row in batch:
n = len(row["target_ids"])
results.append(
{**row, "target_nll": values[cursor : cursor + n].cpu().tolist()}
)
cursor += n
if offset == 0:
for pos in [0, len(batch[0]["target_ids"]) - 1]:
probes.append(
{
"document_id": batch[0]["document_id"],
"target_position": pos,
"target": int(labels[0, pos]),
"logits": logits[0, pos].cpu().tolist(),
"target_nll": results[0]["target_nll"][pos],
}
)
values = [v for row in results for v in row["target_nll"]]
return {
"documents": len(rows),
"target_tokens": len(values),
"target_nll_nats_per_token": math.fsum(values) / len(values),
"rows": results,
"probes": probes,
}
@torch.no_grad()
def evaluate_reconstruction(config, seed, phase):
if phase not in ["initial", "pretrain"]:
raise ValueError("Reconstruction comparison is before/after pretraining")
seed_all(seed)
model = load_weights(f"outputs/model-{phase}-{seed}.json", config).cuda().eval()
test_ids = {
p["document_id"]
for p in read_json("data/generated/input/pages.json")
if p["split"] == "test"
}
rows = [
r
for r in read_json("data/generated/input/pool.json")
if r["document_id"] in test_ids
]
result = nll_rows(model, rows, config["batch_size"])
generations, probes = greedy(model, rows[:12], config)
examples = []
for row, ids in zip(rows, generations):
try:
restored = restore_from_target(row["encoder_ids"], ids)
valid = True
except (ValueError, IndexError):
restored = None
valid = False
examples.append(
{
"document_id": row["document_id"],
"encoder_ids": row["encoder_ids"],
"target_ids": row["target_ids"],
"generated_ids": ids,
"finish_reason": "eos" if ids[-1] == EOS else "length",
"valid_sentinel_structure": valid,
"restored_ids": restored,
"exact_reconstruction": valid and restored == row["clean_ids"],
}
)
save_json(
f"outputs/reconstruction-{phase}-{seed}.json",
{
"seed": seed,
"phase": phase,
"model_sha256": digest(model.state_dict()),
**result,
"fixed_first12_generations": examples,
"first_case_generation_probes": probes,
},
)
def score_fields(predictions, gold):
correct = dict.fromkeys(FIELDS, 0)
exact = valid = truncated = 0
details = []
for row in predictions:
expected = gold[row["document_id"]]
parsed = row["parsed"]
accepted = parsed["completion_valid"]
fields = {
k: accepted and parsed["record"][k] == expected["target"][k] for k in FIELDS
}
raw = accepted and row["text"] == expected["target_text"]
exact += raw
valid += accepted
truncated += row["finish_reason"] == "length"
for key in FIELDS:
correct[key] += fields[key]
details.append(
{
"document_id": row["document_id"],
"fields_correct": fields,
"raw_exact_match": raw,
}
)
n = len(predictions)
return {
"documents": n,
"valid_completions": valid,
"valid_completion_rate": valid / n,
"raw_exact_match": exact,
"raw_exact_match_rate": exact / n,
"field_correct": correct,
"field_accuracy": {k: v / n for k, v in correct.items()},
"truncated": truncated,
"rows": details,
}
def score_canonical_line(config, seed, phase):
gold = {r["document_id"]: r for r in read_json("data/generated/labels/gold.json")}
prediction = read_json(f"outputs/predictions-{phase}-{seed}.json")
metrics = score_fields(prediction["rows"], gold)
model = load_weights(f"outputs/model-{phase}-{seed}.json", config).cuda().eval()
pairs = [{**p, **gold[p["document_id"]]} for p in selected_pages(config)]
nll = nll_rows(model, [supervised_row(p) for p in pairs], config["batch_size"])
save_json(f"outputs/nll-{phase}-{seed}.json", nll)
save_json(
f"outputs/evaluation-{phase}-{seed}.json",
{
"seed": seed,
"phase": phase,
"metrics": metrics,
"response_nll_nats_per_token": nll["target_nll_nats_per_token"],
"quality_gain_required": False,
},
)
print(
{
"seed": seed,
"phase": phase,
"metrics": {k: v for k, v in metrics.items() if k != "rows"},
},
flush=True,
)
def rule_baseline(config):
gold = {r["document_id"]: r for r in read_json("data/generated/labels/gold.json")}
rows = []
for p in selected_pages(config):
observed = normalize_with_rules(p["text"])
rows.append(
{
"document_id": p["document_id"],
"text": observed["text"],
"status": observed["status"],
"finish_reason": "eos",
"parsed": {
"completion_valid": observed["status"] == "ACCEPT",
"record": observed["record"],
},
}
)
missing = read_json("data/generated/input/invalid.json")
decisions = [
{
"document_id": p["document_id"],
"missing_field": p["missing_field"],
"accepted_upstream": upstream_complete(p["text"]),
}
for p in missing
]
save_json(
"outputs/rules.json",
{
"rows": rows,
"metrics": score_fields(rows, gold),
"coverage": sum(r["status"] == "ACCEPT" for r in rows) / len(rows),
"abstentions": sum(r["status"] == "ABSTAIN" for r in rows),
"missing_cases": decisions,
"missing_rejected": sum(not r["accepted_upstream"] for r in decisions),
"scope": "Upstream rule over controlled synthetic fields; not a guarantee of factual completeness for real texts",
},
)
def compare_branches(config):
rows = {}
for seed in config["reference_seeds"]:
rows[str(seed)] = {}
for phase in ["random", "transferred"]:
report = read_json(f"outputs/evaluation-{phase}-{seed}.json")
training = read_json(f"outputs/train-{phase}-{seed}-resume.json")
pretraining = read_json(f"outputs/train-pretrain-{seed}-resume.json")
continuous = read_json(f"outputs/train-{phase}-{seed}-continuous.json")
precontinuous = read_json(f"outputs/train-pretrain-{seed}-continuous.json")
rows[str(seed)][phase] = {
"metrics": {k: v for k, v in report["metrics"].items() if k != "rows"},
"normalization_target_nll": report["response_nll_nats_per_token"],
"supervised_examples": len(training["identity"]["training_ids"]),
"supervised_presentations": sum(
len(r["document_ids"]) for r in training["history"]
),
"supervised_target_tokens": sum(
r["target_tokens"] for r in training["history"]
),
"continuous_supervised_seconds": continuous["seconds"],
"additional_pretraining_steps": 600 if phase == "transferred" else 0,
"additional_pretraining_target_tokens": (
sum(r["target_tokens"] for r in pretraining["history"])
if phase == "transferred"
else 0
),
"continuous_additional_pretraining_seconds": (
precontinuous["seconds"] if phase == "transferred" else 0
),
}
a = read_json(f"outputs/train-random-{seed}-resume.json")
b = read_json(f"outputs/train-transferred-{seed}-resume.json")
if [r["document_ids"] for r in a["history"]] != [
r["document_ids"] for r in b["history"]
]:
raise ValueError("Supervised batches differ")
save_json(
"outputs/comparison.json",
{
"seeds": rows,
"rules": {
k: v
for k, v in read_json("outputs/rules.json").items()
if k not in ["rows", "missing_cases"]
},
"equal_annotation_budget": True,
"equal_total_compute": False,
"continuous_times_include_validation_and_checkpoint_io": True,
"resume_verification_overhead_not_in_path_cost_but_in_reader_wall_time": True,
"quality_gain_required": False,
},
)
Modificare il budget di annotazione
Beneficio marginale del trasferimento
Invariante: le nuove annotazioni appartengono soltanto alle famiglie train.
Errore frequente: confrontare due test set diversi falsifica lo scarto; riutilizzare gli ID e sottoporre a hash l’elenco.
File src/preentrainement_denoising/exercise.py
from pathlib import Path
from .config import read_json, save_json, sha
def preserve_main(config):
paths = ["outputs/comparison.json", "outputs/rules.json"]
for seed in config["reference_seeds"]:
for phase in ["initial", "pretrain", "random", "transferred"]:
paths.append(f"outputs/model-{phase}-{seed}.json")
for phase in ["random", "transferred"]:
paths.extend(
[
f"outputs/evaluation-{phase}-{seed}.json",
f"outputs/predictions-{phase}-{seed}.json",
]
)
save_json("outputs/exercise-before.json", {name: sha(name) for name in paths})
def run_label_budget(config):
before = read_json("outputs/exercise-before.json")
if any(sha(name) != h for name, h in before.items()):
raise ValueError("Exercise changed main artifacts")
rows = {}
for seed in config["reference_seeds"]:
rows[str(seed)] = {}
for branch in ["random", "transferred"]:
phase = "exercise_" + branch
report = read_json(f"outputs/evaluation-{phase}-{seed}.json")
training = read_json(f"outputs/train-{phase}-{seed}-resume.json")
main = read_json(f"outputs/train-{branch}-{seed}-resume.json")
if training["initial_model_sha256"] != main["initial_model_sha256"] or len(
training["identity"]["training_ids"]
) != 2 * len(main["identity"]["training_ids"]):
raise ValueError("Initial state or doubled labels changed")
rows[str(seed)][branch] = {
"main": read_json(f"outputs/evaluation-{branch}-{seed}.json")[
"metrics"
],
"double_labels": report["metrics"],
"labeled_pairs": len(training["identity"]["training_ids"]),
"optimizer_steps": training["steps"],
"presentations": sum(
len(r["document_ids"]) for r in training["history"]
),
"initial_model_sha256": training["initial_model_sha256"],
"target_tokens": sum(r["target_tokens"] for r in training["history"]),
}
save_json(
"outputs/exercise/label-budget.json",
{
"seeds": rows,
"protected_sha256": before,
"main_artifacts_unchanged": True,
"same_pretraining_corpus_and_weights": True,
"same180test_cases": True,
"same300updates_fewer_passes_over_more_labels": True,
"quality_gain_required": False,
},
)
Fonti
L’articolo T5 e il repository degli autori sono stati consultati per il contesto dell’obiettivo. Non viene ripreso alcun codice né peso T5. Le dipendenze realmente usate sono bloccate nel progetto. Docker GPU e NVIDIA Container Toolkit documentano i prerequisiti.
Risultati osservati durante lo sviluppo GPU
Il percorso completo è riuscito con 15 test in 898,97 secondi. Due semi, 42 e 43, sono stati addestrati con 600 aggiornamenti di denoising e 300 di SFT per ramo. I confronti di ripresa ritrovano esattamente modello, ottimizzatore, sampler, RNG e storici.
Sui 180 casi test di normalizzazione, nessuna delle otto varianti neurali produce una riga interamente esatta; le regole ottengono 180/180. Il preaddestramento migliora alcuni segnali senza risolvere il compito.
Esempio conservato: per «Piece K0510, aluminium, diametre 18 mm.», il modello trasferito del seme 42 produce reference=K0024;materiau=aluminium;diametre_mm=26. Il formato e il materiale sono corretti, ma gli altri due campi sono errati.
La NLL di ricostruzione dei buchi su 900 schede e 9 150 token destinazione scende da 5,6601 a 1,0212 per il seme 42 e da 5,6971 a 1,0314 per il seme 43. Le loss condizionate e la qualità delle uscite libere rispondono a domande diverse.
L’esercizio raddoppia il numero di formulazioni supervisionate da 840 a 1 680 esempi disponibili, ma conserva 300 aggiornamenti: ogni esempio è quindi meno rivisitato. Non apporta un guadagno uniforme.
Il costo aggiuntivo del preaddestramento è incluso: 36,13 s poi 35,26 s sui percorsi continui, con circa 16–17 s per SFT. Queste durate includono validation e scrittura.
Questo corpus chiuso e le sue regole sono un supporto per apprendere il meccanismo. Il preaddestramento aggiuntivo non giustifica qui la sostituzione del parser deterministico. Due semi non bastano per stabilire una generalità aziendale.
Osservazioni sullo sviluppo
Valori ricalcolati dalle uscite grezze di questo percorso. Non sono soglie di guadagno; il percorso dallo ZIP li ricalcola sulle stesse popolazioni.
{
"status": "PASS",
"documents": 6000,
"test_normalization_cases": 180,
"seeds": [
42,
43
],
"parameters": 3269952,
"metrics": {
"random-42": {
"documents": 180,
"valid_completions": 101,
"valid_completion_rate": 0.5611111111111111,
"raw_exact_match": 0,
"raw_exact_match_rate": 0.0,
"field_correct": {
"reference": 0,
"materiau": 69,
"diametre_mm": 3
},
"field_accuracy": {
"reference": 0.0,
"materiau": 0.38333333333333336,
"diametre_mm": 0.016666666666666666
},
"truncated": 0
},
"transferred-42": {
"documents": 180,
"valid_completions": 180,
"valid_completion_rate": 1.0,
"raw_exact_match": 0,
"raw_exact_match_rate": 0.0,
"field_correct": {
"reference": 0,
"materiau": 131,
"diametre_mm": 6
},
"field_accuracy": {
"reference": 0.0,
"materiau": 0.7277777777777777,
"diametre_mm": 0.03333333333333333
},
"truncated": 0
},
"exercise_random-42": {
"documents": 180,
"valid_completions": 109,
"valid_completion_rate": 0.6055555555555555,
"raw_exact_match": 0,
"raw_exact_match_rate": 0.0,
"field_correct": {
"reference": 0,
"materiau": 59,
"diametre_mm": 8
},
"field_accuracy": {
"reference": 0.0,
"materiau": 0.3277777777777778,
"diametre_mm": 0.044444444444444446
},
"truncated": 0
},
"exercise_transferred-42": {
"documents": 180,
"valid_completions": 170,
"valid_completion_rate": 0.9444444444444444,
"raw_exact_match": 0,
"raw_exact_match_rate": 0.0,
"field_correct": {
"reference": 0,
"materiau": 109,
"diametre_mm": 7
},
"field_accuracy": {
"reference": 0.0,
"materiau": 0.6055555555555555,
"diametre_mm": 0.03888888888888889
},
"truncated": 0
},
"random-43": {
"documents": 180,
"valid_completions": 98,
"valid_completion_rate": 0.5444444444444444,
"raw_exact_match": 0,
"raw_exact_match_rate": 0.0,
"field_correct": {
"reference": 0,
"materiau": 54,
"diametre_mm": 0
},
"field_accuracy": {
"reference": 0.0,
"materiau": 0.3,
"diametre_mm": 0.0
},
"truncated": 0
},
"transferred-43": {
"documents": 180,
"valid_completions": 180,
"valid_completion_rate": 1.0,
"raw_exact_match": 0,
"raw_exact_match_rate": 0.0,
"field_correct": {
"reference": 0,
"materiau": 145,
"diametre_mm": 6
},
"field_accuracy": {
"reference": 0.0,
"materiau": 0.8055555555555556,
"diametre_mm": 0.03333333333333333
},
"truncated": 0
},
"exercise_random-43": {
"documents": 180,
"valid_completions": 178,
"valid_completion_rate": 0.9888888888888889,
"raw_exact_match": 0,
"raw_exact_match_rate": 0.0,
"field_correct": {
"reference": 0,
"materiau": 60,
"diametre_mm": 5
},
"field_accuracy": {
"reference": 0.0,
"materiau": 0.3333333333333333,
"diametre_mm": 0.027777777777777776
},
"truncated": 0
},
"exercise_transferred-43": {
"documents": 180,
"valid_completions": 180,
"valid_completion_rate": 1.0,
"raw_exact_match": 0,
"raw_exact_match_rate": 0.0,
"field_correct": {
"reference": 0,
"materiau": 146,
"diametre_mm": 6
},
"field_accuracy": {
"reference": 0.0,
"materiau": 0.8111111111111111,
"diametre_mm": 0.03333333333333333
},
"truncated": 0
}
},
"rules": {
"documents": 180,
"valid_completions": 180,
"valid_completion_rate": 1.0,
"raw_exact_match": 180,
"raw_exact_match_rate": 1.0,
"field_correct": {
"reference": 180,
"materiau": 180,
"diametre_mm": 180
},
"field_accuracy": {
"reference": 1.0,
"materiau": 1.0,
"diametre_mm": 1.0
},
"truncated": 0
},
"quality_gain_required": false,
"scope": "Data-only exact sampler/corruption replay, all target budgets, weight hashes, full-state resume evidence, CE probes, raw decoding/metrics and observed phase costs; no scalar reimplementation of the full Transformer"
}
