A fictional catalogue contains heterogeneous descriptions of parts. The final task turns a complete, sometimes abbreviated description into a canonical line. Before this supervised task, a small network learns to reconstruct segments removed from the training records.
Compare three normalizers on the same held-out records: rules, a randomly initialized encoder-decoder then fine-tuned, and an encoder-decoder pretrained by denoising then fine-tuned. Also display a span reconstruction.
Download the complete project, revision r001
Two tasks and two signals
Denoising removes contiguous segments from a record. The network receives the remaining bytes and sentinel tokens; its target contains only the removed segments, associated with the same sentinels. Self-supervision comes from the original record, without an annotator. The downstream task uses another target: a canonical line explicitly constructed for normalization. These two NLLs do not have the same target content; their values are not directly a common measure of business quality.
The original CatalogueSpan-v1 corpus contains 600 families of ten formulations, or 6,000 records. Families 0–419 are training, 420–509 validation, and 510–599 test: 4,200/900/900 records. A family shares a reference, material, and diameter. Formulations, materials, and short syntactic templates are shared across partitions; references are distinct. This tests transfer on an artificial catalogue, not the discovery of an entirely new grammar or generalization to a real catalogue.
Formulations 0 and 5 provide 840 training pairs and 180 pairs per held-out partition. All their fields are present. The 60 separate incomplete cases remove one field and test rule-based rejection before generation. The model is not queried to invent the hidden value; this experiment therefore does not demonstrate resistance to hallucinations. The rules cover the ten controlled formats and may already be perfect: if the measurements confirm it, their simplicity remains a real advantage.
Bytes, spans, and sentinels
The vocabulary is fixed: UTF-8 bytes 0–255, PAD=256, BOS=257, EOS=258, and 32 sentinels 259–290. An accented character can occupy several bytes and a span can split it. Display uses explicit byte fragments; oracle restoration reconstructs the bytes exactly before decoding, with no Unicode repair.
The number of removed bytes is round(0.15 × length), bounded between 1 and length−1. The number of spans is close to the number removed divided by three, with non-empty constraints. Two random positive compositions distribute visible and removed lengths; visible and removed segments alternate, and the final span reaches the end of the record. An average of three is a rounded target, not an exact geometric law. This edge choice is an explicit limitation. Each gap has its sentinel; the target ends with one additional sentinel and then EOS. More than 31 gaps are rejected.
The encoder receives BOS, the corrupted input, and EOS. The decoder receives BOS then target[:-1] and predicts target, including sentinels and EOS. PAD is never an active target. Cross-entropy selects active positions before summing; accumulation divides by the exact number of target tokens, without an incorrect average of microbatch averages. The collator never puts a removed segment back into the encoder input.
Validation/test masks are fixed by identifier before training. Training masks are sampled dynamically with an explicit generator whose state is saved. Held-out formulations remain physically absent from the learning service; it mounts only validation corruptions, not test corruptions.
Three attention paths
The original network contains three encoder blocks and three decoder blocks, width 192, six heads, and an MLP of 768 with GELU. Each layer is allocated and initialized independently. The input embedding and output projection reuse the same Parameter. Learned positions cover 256 tokens; an overflow is rejected, never silently truncated. Pre-LayerNorm, residuals, and dropout 0.1 are visible. This is not the T5 architecture: only the idea of an objective based on spans and sentinels inspires it.
Encoder self-attention is bidirectional and masks only PAD keys. Decoder self-attention masks the future and PAD keys. Cross-attention reads the entire visible encoding while masking encoder PAD tokens. Batches use right padding, with BOS present. Changing a future target must not change earlier logits; changing a future encoder input may change its earlier representations. Adding padding must preserve loss and gradients within the stated FP32 tolerance.
Training protocol
Seeds 42 and 43 produce two independent chains. An initialization is saved for each. The direct branch starts from these weights; the transferred branch receives the weights after 600 denoising updates on the 4,200 training records only. Both branches then perform 300 updates on the same 840 pairs, in the same supervised order. The final checkpoint is fixed; validation every 100 steps does not select a better version.
AdamW updates every parameter in FP32 CUDA, with TF32 disabled, weight_decay 0.01, and gradient norm clipped to 1. The maximum rate is 0.0003 in pretraining and 0.0001 in normalization; warm-up lasts 10% of the steps, followed by linear decay to 10% of the maximum rate. One update processes 16 examples in two microbatches of eight with an exact token denominator. The sampler traverses permutations and may cross an epoch boundary to complete a batch.
Each phase is actually run interrupted, continuously, and resumed. Pretraining stops at 240 steps out of 600; SFT at 120 out of 300. Model, AdamW, scheduler, cursor/permutation, corruption RNG, and Python/NumPy/PyTorch CPU/CUDA RNG must match the continuous result. These repetitions cost computation; the durations of continuous paths are separated from total reader time. The .pt files are created locally; JSON weights with dimensions, size, and SHA enable an audit without executing external pickle content.
Evaluate freely
Reconstruction is evaluated before and after pretraining on the same 900 test records and fixed masks, with NLL per target token and the first twelve free reconstructions as illustrations. Normalization is evaluated on 180 free outputs per branch and seed. Greedy decoding allows up to 128 new tokens; only PAD and BOS are removed from the support. Sentinels are not removed to force a canonical line: their appearance is a visible format failure. Invalid bytes, wrong formats, and truncations remain in the denominator.
Completion validity requires EOS and the complete format. Raw exact match compares the whole line; field accuracy compares parsed values, including the integer diameter. A numerically correct line formatted with leading zeros can have correct fields without a raw exact match. An output reaching 128 tokens fails even if its prefix appears complete. Downstream NLL uses teacher forcing and the same targets; it does not replace free evaluation.
The tables show annotation, processed tokens, and time per phase separately. Observed durations include validation and checkpoint writing in the process; they are not isolated measurements of CUDA kernel time. Overall resources and resume overhead are measured in the walkthrough. Transfer adds computation and is not compared at equal total budget.
Double the annotations
The exercise takes four training formulations (0, 2, 5, 7), or 1,680 pairs, and retrains two new branches from the same initial/direct or pretrained states. The 180 test cases, two seeds, and 300 updates remain fixed. The number of passes over each pair decreases: more labels do not mean more steps. The unannotated corpus and pretraining weights remain unchanged and their SHAs are checked. An improvement is not assumed; null or negative transfer remains in the results. An extension at equal total computation would require another experiment, explicitly different from this one.
Create the records and partitions
Separate family and formulation
Invariant: 420/90/90 families, 840/180/180 downstream pairs before documented exclusions.
Common mistake: another formulation of the same part appears in test; partition families before generation.
File src/preentrainement_denoising/data.py
import hashlib, json, re
from pathlib import Path
from .config import save_json, sha
MATERIALS = ["aluminium", "acier", "cuivre", "nylon", "laiton", "bronze"]
ALIASES = ["alu", "ac", "cu", "nyl", "lait", "brz"]
FIELDS = ["reference", "materiau", "diametre_mm"]
def split_catalog(family):
return "train" if family < 420 else "validation" if family < 510 else "test"
def canonical(values):
return ";".join(key + "=" + str(values[key]) for key in FIELDS)
def render(values, formulation):
ref = values["reference"]
material = values["materiau"]
diameter = values["diametre_mm"]
alias = ALIASES[MATERIALS.index(material)] if material else "?"
ref = ref or "?"
material = material or "?"
diameter = str(diameter) if diameter is not None else "?"
patterns = [
f"Piece {ref}, {material}, diametre {diameter} mm.",
f"Reference {ref} : matiere {material} ; diametre {diameter} mm.",
f"{material} / piece {ref} / diam. {diameter}mm",
f"Diametre {diameter} mm ; reference {ref} ; materiau {material}.",
f"Fiche piece={ref} ; mat={alias} ; D={diameter} mm",
f"piece {ref} / {alias} / diam.{diameter}mm",
f"Mat. {alias}, ref. {ref}, diametre={diameter}mm.",
f"{ref} | matiere:{material} | diam:{diameter} mm",
f"D={diameter}mm pour piece {ref} en {material}.",
f"Catalogue : reference {ref}. Diametre {diameter} mm. Matiere {material}.",
]
return patterns[formulation]
def generate_catalog():
pages = []
gold = []
for family in range(600):
values = {
"reference": f"K{family:04d}",
"materiau": MATERIALS[family % 6],
"diametre_mm": 8 + 2 * ((family // 6) % 20),
}
for j in range(10):
text = render(values, j)
identifier = f"C{family:03d}-{j}"
pages.append(
{
"document_id": identifier,
"family": family,
"formulation": j,
"split": split_catalog(family),
"text": text,
"content_sha256": hashlib.sha256(text.encode()).hexdigest(),
}
)
gold.append(
{
"document_id": identifier,
"target": values.copy(),
"target_text": canonical(values),
}
)
if len({r["text"].casefold() for r in pages}) != 6000:
raise ValueError("Duplicate normalized catalogue record")
missing = []
for index in range(60):
family = 510 + index
field = FIELDS[index % 3]
values = {
"reference": f"K{family:04d}",
"materiau": MATERIALS[family % 6],
"diametre_mm": 8 + 2 * ((family // 6) % 20),
}
values[field] = None
missing.append(
{
"document_id": f"M{index:03d}",
"source_family": family,
"missing_field": field,
"text": render(values, index % 10),
}
)
return pages, gold, missing
def select_labeled_pairs(pages, gold, split, formulations):
byid = {r["document_id"]: r for r in gold}
return [
{
**p,
"target": byid[p["document_id"]]["target"],
"target_text": byid[p["document_id"]]["target_text"],
}
for p in pages
if p["split"] == split and p["formulation"] in formulations
]
def prepare(config):
root = Path("data/generated")
root.mkdir(parents=True, exist_ok=True)
if (root / "manifest.json").exists():
raise ValueError("Preserve frozen catalogue; use a new extraction")
pages, gold, missing = generate_catalog()
save_json(root / "input/pages.json", pages)
save_json(root / "labels/gold.json", gold)
save_json(root / "input/invalid.json", missing)
audit = {}
for split in ["train", "validation", "test"]:
selected = [p for p in pages if p["split"] == split]
save_json(root / f"{split}/pages.json", selected)
pairs = select_labeled_pairs(pages, gold, split, config["labeled_formulations"])
audit[split] = {
"documents": len(selected),
"families": len({p["family"] for p in selected}),
"labeled_pairs": len(pairs),
}
if split != "test":
save_json(root / f"labels/{split}.json", pairs)
exercise = select_labeled_pairs(
pages, gold, "train", config["exercise_labeled_formulations"]
)
save_json(root / "labels/train-pairs.json", exercise)
from .corruption import corrupt_with_sentinels
import random
frozen = [
{
"document_id": p["document_id"],
**corrupt_with_sentinels(
list(p["text"].encode()),
random.Random(420000 + i),
config["noise_density"],
config["mean_span_length"],
),
}
for i, p in enumerate(pages)
if p["split"] != "train"
]
save_json(root / "input/pool.json", frozen)
validation_ids = {p["document_id"] for p in pages if p["split"] == "validation"}
save_json(
root / "validation/corruptions.json",
[r for r in frozen if r["document_id"] in validation_ids],
)
manifest = {
"dataset": "CatalogueSpan-v1",
"rights": "CC0-1.0",
"seed": config["seed"],
"split_audit": audit,
"exercise_labeled_pairs": len(exercise),
"missing_cases": len(missing),
"fixed_evaluation_corruptions": len(frozen),
"normalized_duplicates": 0,
"files": {str(f.relative_to(root)): sha(f) for f in root.rglob("*.json")},
}
save_json(root / "manifest.json", manifest)
save_json("outputs/data-audit.json", manifest)
return audit
Implement span corruption
Sentinels and self-supervised target
Invariant: oracle reconstruction restores every byte of the record.
Common mistake: two gaps share the same sentinel; allocate unique tokens and check capacity.
File src/preentrainement_denoising/corruption.py
import random
PAD = 256
BOS = 257
EOS = 258
FIRST = 259
SENTINELS = 32
VOCAB = 291
def positive_parts(total, parts, generator):
if not 1 <= parts <= total:
raise ValueError("A span must contain at least one token")
boundaries = [0] + sorted(generator.sample(range(1, total), parts - 1)) + [total]
return [b - a for a, b in zip(boundaries, boundaries[1:])]
def sample_noise_spans(length, generator, density=0.15, mean_length=3.0):
if length < 2 or not 0 < density < 1 or mean_length <= 0:
raise ValueError("Invalid corruption parameters")
count = max(1, min(length - 1, round(length * density)))
spans = max(1, min(count, length - count, round(count / mean_length)))
if spans + 1 > SENTINELS:
raise ValueError("Not enough unique sentinels including the terminal sentinel")
noise = positive_parts(count, spans, generator)
visible = positive_parts(length - count, spans, generator)
cursor = 0
ranges = []
for a, b in zip(visible, noise):
cursor += a
ranges.append([cursor, cursor + b])
cursor += b
assert cursor == length
return ranges
def corrupt_with_sentinels(clean, generator, density=0.15, mean_length=3.0):
if not clean or any(type(v) != int or not 0 <= v < 256 for v in clean):
raise ValueError("Corrupt only raw byte tokens")
ranges = sample_noise_spans(len(clean), generator, density, mean_length)
encoder = []
target = []
cursor = 0
for index, (start, end) in enumerate(ranges):
encoder.extend(clean[cursor:start])
encoder.append(FIRST + index)
target.append(FIRST + index)
target.extend(clean[start:end])
cursor = end
encoder.extend(clean[cursor:])
target.extend([FIRST + len(ranges), EOS])
return {
"clean_ids": clean,
"encoder_ids": [BOS] + encoder + [EOS],
"target_ids": target,
"spans": ranges,
"removed_tokens": sum(b - a for a, b in ranges),
}
def restore_from_target(encoder, target):
if encoder[0] != BOS or encoder[-1] != EOS or target[-1] != EOS:
raise ValueError("Missing boundary token")
pieces = {}
current = None
for token in target[:-1]:
if FIRST <= token < FIRST + SENTINELS:
if token in pieces or token != FIRST + len(pieces):
raise ValueError("Repeated or out-of-order sentinel")
pieces[token] = []
current = token
elif 0 <= token < 256 and current is not None:
pieces[current].append(token)
else:
raise ValueError("Invalid reconstruction target")
if (
len(pieces) < 2
or pieces[max(pieces)]
or any(not v for k, v in pieces.items() if k != max(pieces))
):
raise ValueError("Nonempty spans and empty terminal sentinel required")
seen = []
result = []
for token in encoder[1:-1]:
if FIRST <= token < FIRST + SENTINELS:
if token not in pieces or token == max(pieces) or token in seen:
raise ValueError("Unmatched encoder sentinel")
seen.append(token)
result.extend(pieces[token])
elif 0 <= token < 256:
result.append(token)
else:
raise ValueError("Invalid encoder byte")
if seen != list(range(FIRST, max(pieces))):
raise ValueError("Missing or reordered span")
return result
def display(ids):
# Sentinels can split a UTF-8 character: show raw byte runs using repr,
# never treat a lossy decoder display as an exact reconstructed string.
result = []
buffer = []
for value in ids:
if value < 256:
buffer.append(value)
continue
if buffer:
result.append(repr(bytes(buffer)))
buffer = []
result.append(
{PAD: "<PAD>", BOS: "<BOS>", EOS: "<EOS>"}.get(
value, f"<extra_{value-FIRST}>"
)
)
if buffer:
result.append(repr(bytes(buffer)))
return " ".join(result)
Build the encoder-decoder
Three distinct attention paths
Invariant: the decoder causal test passes and cross-attention ignores input PAD tokens.
Common mistake: a causal mask is copied onto the encoder; display the three masks separately.
File src/preentrainement_denoising/model.py
import math, torch
from torch import nn
from .corruption import PAD, BOS, EOS, VOCAB
class Attention(nn.Module):
def __init__(self, width, heads, dropout):
super().__init__()
if width % heads:
raise ValueError("Width must be divisible by heads")
self.width = width
self.heads = heads
self.query = nn.Linear(width, width)
self.key = nn.Linear(width, width)
self.value = nn.Linear(width, width)
self.output = nn.Linear(width, width)
self.dropout = nn.Dropout(dropout)
def forward(self, query, memory, key_mask, causal=False):
batch, steps, width = query.shape
length = memory.shape[1]
size = width // self.heads
if not key_mask.any(1).all():
raise ValueError("Attention needs at least one real key")
q = self.query(query).reshape(batch, steps, self.heads, size).transpose(1, 2)
k = self.key(memory).reshape(batch, length, self.heads, size).transpose(1, 2)
v = self.value(memory).reshape(batch, length, self.heads, size).transpose(1, 2)
allowed = key_mask[:, None, None, :].expand(batch, 1, steps, length)
if causal:
if steps != length:
raise ValueError("Self-attention causal dimensions differ")
allowed = (
allowed
& torch.ones(
steps, length, device=query.device, dtype=torch.bool
).tril()[None, None]
)
if not allowed.any(-1).all():
raise ValueError(
"Use right padding with a visible BOS, not an empty causal prefix"
)
weights = (
(q @ k.transpose(-1, -2) / math.sqrt(size))
.masked_fill(~allowed, float("-inf"))
.softmax(-1)
)
x = (
(self.dropout(weights) @ v)
.transpose(1, 2)
.contiguous()
.reshape(batch, steps, width)
)
return self.dropout(self.output(x))
def mlp(width, hidden, dropout):
return nn.Sequential(
nn.Linear(width, hidden),
nn.GELU(),
nn.Linear(hidden, width),
nn.Dropout(dropout),
)
class EncoderBlock(nn.Module):
def __init__(self, width, heads, hidden, dropout):
super().__init__()
self.norm1 = nn.LayerNorm(width)
self.attention = Attention(width, heads, dropout)
self.norm2 = nn.LayerNorm(width)
self.feedforward = mlp(width, hidden, dropout)
def forward(self, x, mask):
normalized = self.norm1(x)
x = x + self.attention(normalized, normalized, mask, False)
return x + self.feedforward(self.norm2(x))
class DecoderBlock(nn.Module):
def __init__(self, width, heads, hidden, dropout):
super().__init__()
self.norm1 = nn.LayerNorm(width)
self.self_attention = Attention(width, heads, dropout)
self.norm2 = nn.LayerNorm(width)
self.cross_attention = Attention(width, heads, dropout)
self.norm3 = nn.LayerNorm(width)
self.feedforward = mlp(width, hidden, dropout)
def forward(self, x, mask, memory, memory_mask):
normalized = self.norm1(x)
x = x + self.self_attention(normalized, normalized, mask, True)
x = x + self.cross_attention(self.norm2(x), memory, memory_mask, False)
return x + self.feedforward(self.norm3(x))
class SpanDenoiser(nn.Module):
def __init__(self, config):
super().__init__()
width = config["width"]
self.context = config["context"]
self.embedding = nn.Embedding(VOCAB, width)
self.encoder_position = nn.Embedding(self.context, width)
self.decoder_position = nn.Embedding(self.context, width)
self.dropout = nn.Dropout(config["dropout"])
args = (width, config["heads"], config["mlp_width"], config["dropout"])
self.encoders = nn.ModuleList(
[EncoderBlock(*args) for _ in range(config["encoder_blocks"])]
)
self.decoders = nn.ModuleList(
[DecoderBlock(*args) for _ in range(config["decoder_blocks"])]
)
self.encoder_norm = nn.LayerNorm(width)
self.decoder_norm = nn.LayerNorm(width)
self.apply(self.initialize)
@staticmethod
def initialize(module):
if isinstance(module, (nn.Linear, nn.Embedding)):
nn.init.normal_(module.weight, mean=0.0, std=0.02)
if isinstance(module, nn.Linear) and module.bias is not None:
nn.init.zeros_(module.bias)
def embed(self, ids, position):
if ids.ndim != 2 or ids.shape[1] > self.context:
raise ValueError("No silent sequence truncation")
return self.dropout(
self.embedding(ids)
+ position(torch.arange(ids.shape[1], device=ids.device))[None]
)
def encode(self, ids, mask):
x = self.embed(ids, self.encoder_position)
for layer in self.encoders:
x = layer(x, mask)
return self.encoder_norm(x)
def decode(self, ids, mask, memory, memory_mask):
x = self.embed(ids, self.decoder_position)
for layer in self.decoders:
x = layer(x, mask, memory, memory_mask)
return self.decoder_norm(x) @ self.embedding.weight.T
def forward(self, encoder_ids, encoder_mask, decoder_ids, decoder_mask):
return self.decode(
decoder_ids,
decoder_mask,
self.encode(encoder_ids, encoder_mask),
encoder_mask,
)
def make_batch(rows, device="cuda", pad_encoder=None, pad_target=None):
if not rows or any(not r["encoder_ids"] or not r["target_ids"] for r in rows):
raise ValueError("Empty encoder or target")
ne = max(len(r["encoder_ids"]) for r in rows)
nt = max(len(r["target_ids"]) for r in rows)
pe = ne if pad_encoder is None else pad_encoder
pt = nt if pad_target is None else pad_target
if pe < ne or pt < nt:
raise ValueError("Padding must not truncate")
encoder = torch.full((len(rows), pe), PAD, dtype=torch.long, device=device)
decoder = torch.full((len(rows), pt), PAD, dtype=torch.long, device=device)
labels = torch.full((len(rows), pt), -100, dtype=torch.long, device=device)
em = torch.zeros_like(encoder, dtype=torch.bool)
dm = torch.zeros_like(decoder, dtype=torch.bool)
for i, row in enumerate(rows):
source = row["encoder_ids"]
target = row["target_ids"]
encoder[i, : len(source)] = torch.tensor(source, device=device)
decoder[i, : len(target)] = torch.tensor([BOS] + target[:-1], device=device)
labels[i, : len(target)] = torch.tensor(target, device=device)
em[i, : len(source)] = True
dm[i, : len(target)] = True
return encoder, em, decoder, dm, labels
def target_loss(logits, labels):
active = labels != -100
if not active.any():
raise ValueError("No active target")
values = torch.nn.functional.cross_entropy(
logits[active].float(), labels[active], reduction="none"
)
return values.sum(), int(active.sum()), values
Pretrain, then fine-tune the two branches
Different objective, same reused weights
Invariant: both branches see the same supervised IDs; their per-phase costs are recorded.
Common mistake: loading pretrained weights into both branches erases the baseline; compare their initial hashes.
File src/preentrainement_denoising/train.py
import math, random, time
from pathlib import Path
import torch
from .config import read_json, save_json, sha, seed_all, rng_state, restore_rng
from .model import SpanDenoiser, make_batch, target_loss
from .corruption import BOS, EOS, corrupt_with_sentinels
from .state import digest, equal, save_training_state, export_weights, load_weights
class ExampleSampler:
def __init__(self, size, seed):
self.size = size
self.generator = random.Random(seed)
self.order = []
self.cursor = 0
self.epoch = 0
def take(self, count):
selected = []
while len(selected) < count:
if self.cursor == len(self.order):
self.order = list(range(self.size))
self.generator.shuffle(self.order)
self.cursor = 0
self.epoch += 1
selected.append(self.order[self.cursor])
self.cursor += 1
return selected
def state_dict(self):
return {
"size": self.size,
"order": self.order,
"cursor": self.cursor,
"epoch": self.epoch,
"rng": self.generator.getstate(),
}
def load_state_dict(self, state):
if (
state["size"] != self.size
or sorted(state["order"]) != list(range(self.size))
or not 0 <= state["cursor"] <= self.size
):
raise ValueError("Invalid sampler checkpoint")
self.order = list(state["order"])
self.cursor = state["cursor"]
self.epoch = state["epoch"]
self.generator.setstate(state["rng"])
def identity_config(config):
return {k: v for k, v in config.items() if k != "exercise"}
def initialize(config, seed):
seed_all(seed)
model = SpanDenoiser(config)
root = Path("assets/checkpoints")
save_training_state(
root / f"initial-{seed}.pt",
{
"model": model.state_dict(),
"config": identity_config(config),
"seed": seed,
"tokenizer": {
"bytes": [0, 255],
"pad": 256,
"bos": 257,
"eos": 258,
"sentinels": [259, 290],
},
},
)
export_weights(f"outputs/model-initial-{seed}.json", model)
return digest(model.state_dict())
def supervised_row(page):
return {
"document_id": page["document_id"],
"encoder_ids": [BOS] + list(page["text"].encode()) + [EOS],
"target_ids": list(page["target_text"].encode()) + [EOS],
}
def phase_rows(phase, config):
if phase == "pretrain":
train = read_json("data/generated/train/pages.json")
validation_pages = read_json("data/generated/validation/pages.json")
validation_ids = {p["document_id"] for p in validation_pages}
validation = [
p
for p in read_json("data/generated/validation/corruptions.json")
if p["document_id"] in validation_ids
]
else:
path = (
"data/generated/labels/train-pairs.json"
if phase.startswith("exercise_")
else "data/generated/labels/train.json"
)
train = read_json(path)
validation = [
supervised_row(p)
for p in read_json("data/generated/labels/validation.json")
]
if any(p["split"] != "train" for p in train):
raise ValueError("Training family leakage")
expected = (
4200 if phase == "pretrain" else 1680 if phase.startswith("exercise_") else 840
)
if len(train) != expected or len(validation) != (
900 if phase == "pretrain" else 180
):
raise ValueError("Unexpected phase population")
return train, validation
def lr_at(config, phase, step, total):
peak = (
config["learning_rate_pretrain"]
if phase == "pretrain"
else config["learning_rate_finetune"]
)
warmup = max(1, round(total * config["warmup_ratio"]))
return (
peak * (step + 1) / warmup
if step < warmup
else peak
* (
1
- min(1.0, (step - warmup) / max(1, total - warmup - 1))
* (1 - config["minimum_lr_ratio"])
)
)
@torch.no_grad()
def validation_nll(model, rows, batch_size):
model.eval()
total = 0.0
count = 0
for i in range(0, len(rows), batch_size):
encoder, em, decoder, dm, labels = make_batch(rows[i : i + batch_size])
_, n, values = target_loss(model(encoder, em, decoder, dm), labels)
total += float(values.double().sum())
count += n
model.train()
return {
"documents": len(rows),
"target_tokens": count,
"target_nll_nats_per_token": total / count,
}
def train_phase(config, seed, phase, mode):
if (
Path("data/generated/test").exists()
or Path("data/generated/labels/gold.json").exists()
):
raise ValueError("Learning container must not mount test or global gold")
if phase not in [
"pretrain",
"random",
"transferred",
"exercise_random",
"exercise_transferred",
] or mode not in ["interrupt", "continuous", "resume"]:
raise ValueError("Unknown phase/mode")
seed_all(seed)
train, validation = phase_rows(phase, config)
source = Path("assets/checkpoints") / (
f"pretrain-{seed}-resume.pt"
if phase in ["transferred", "exercise_transferred"]
else f"initial-{seed}.pt"
)
initial = torch.load(source, map_location="cpu", weights_only=False)
if initial.get(
"config", initial.get("identity", {}).get("config")
) != identity_config(config):
raise ValueError("Tokenizer/config changed between phases")
model = SpanDenoiser(config)
model.load_state_dict(initial["model"], strict=True)
initial_digest = digest(model.state_dict())
del initial
model = model.cuda()
optimizer = torch.optim.AdamW(
model.parameters(),
lr=config["learning_rate_pretrain"],
weight_decay=config["weight_decay"],
)
sampler = ExampleSampler(len(train), seed + 100)
corruption_rng = random.Random(seed + 200)
total = (
config["pretrain_steps"] if phase == "pretrain" else config["finetune_steps"]
)
interrupt = (
config["pretrain_interrupt"]
if phase == "pretrain"
else config["finetune_interrupt"]
)
stop = interrupt if mode == "interrupt" else total
identity = {
"phase": phase,
"seed": seed,
"config": identity_config(config),
"source_checkpoint_sha256": sha(source),
"initial_model_sha256": initial_digest,
"training_ids": [p["document_id"] for p in train],
"training_sha256": (
sha("data/generated/train/pages.json")
if phase == "pretrain"
else sha(
"data/generated/labels/train-pairs.json"
if phase.startswith("exercise_")
else "data/generated/labels/train.json"
)
),
"implementation_sha256": {
n: sha("src/preentrainement_denoising/" + n)
for n in ["model.py", "corruption.py", "train.py", "state.py"]
},
}
start = 0
history = []
validation_history = []
root = Path("assets/checkpoints")
path = root / f"{phase}-{seed}-{mode}.pt"
if path.exists():
raise ValueError("Preserve phase checkpoint")
if mode == "resume":
state = torch.load(
root / f"{phase}-{seed}-interrupt.pt",
map_location="cpu",
weights_only=False,
)
if state["identity"] != identity or state["scheduler"] != {
"next_step": state["steps"],
"next_lr": lr_at(config, phase, state["steps"], total),
}:
raise ValueError("Resume identity/scheduler mismatch")
model.load_state_dict(state["model"], strict=True)
optimizer.load_state_dict(state["optimizer"])
sampler.load_state_dict(state["sampler"])
corruption_rng.setstate(state["corruption_rng"])
restore_rng(state["rng"])
start = state["steps"]
history = state["history"]
validation_history = state["validation_history"]
del state
model.train()
begun = time.perf_counter()
torch.cuda.reset_peak_memory_stats()
for step in range(start, stop):
selected = sampler.take(config["batch_size"])
rows = []
for index in selected:
page = train[index]
row = (
{
"document_id": page["document_id"],
**corrupt_with_sentinels(
list(page["text"].encode()),
corruption_rng,
config["noise_density"],
config["mean_span_length"],
),
}
if phase == "pretrain"
else supervised_row(page)
)
rows.append(row)
denominator = sum(len(r["target_ids"]) for r in rows)
lr = lr_at(config, phase, step, total)
for group in optimizer.param_groups:
group["lr"] = lr
optimizer.zero_grad(set_to_none=True)
loss_total = 0.0
for offset in range(0, len(rows), config["microbatch"]):
encoder, em, decoder, dm, labels = make_batch(
rows[offset : offset + config["microbatch"]]
)
summed, n, values = target_loss(model(encoder, em, decoder, dm), labels)
if not torch.isfinite(summed):
raise ValueError("Nonfinite loss")
(summed / denominator).backward()
loss_total += float(values.detach().double().sum())
norm = torch.nn.utils.clip_grad_norm_(
model.parameters(), config["gradient_clip"], error_if_nonfinite=True
)
optimizer.step()
history.append(
{
"step": step + 1,
"document_ids": [r["document_id"] for r in rows],
"spans": [r.get("spans", []) for r in rows],
"target_tokens": denominator,
"target_nll_nats_per_token": loss_total / denominator,
"learning_rate": lr,
"gradient_norm_before_clip": float(norm),
}
)
if (step + 1) % 100 == 0:
observed = validation_nll(model, validation, config["batch_size"])
validation_history.append({"step": step + 1, **observed})
print(
{
"seed": seed,
"phase": phase,
"mode": mode,
**history[-1],
"validation": observed,
},
flush=True,
)
torch.cuda.synchronize()
state = {
"identity": identity,
"steps": stop,
"model": model.state_dict(),
"optimizer": optimizer.state_dict(),
"scheduler": {"next_step": stop, "next_lr": lr_at(config, phase, stop, total)},
"sampler": sampler.state_dict(),
"corruption_rng": corruption_rng.getstate(),
"rng": rng_state(),
"history": history,
"validation_history": validation_history,
}
final = digest(model.state_dict())
if final == initial_digest:
raise ValueError("No learned parameter change")
save_training_state(path, state)
if mode == "resume":
export_weights(f"outputs/model-{phase}-{seed}.json", model)
save_json(
f"outputs/train-{phase}-{seed}-{mode}.json",
{
"seed": seed,
"phase": phase,
"mode": mode,
"identity": identity,
"steps": stop,
"history": history,
"validation_history": validation_history,
"initial_model_sha256": initial_digest,
"final_model_sha256": final,
"parameters": sum(p.numel() for p in model.parameters()),
"checkpoint_sha256": sha(path),
"checkpoint_bytes": path.stat().st_size,
"max_torch_allocated_bytes": torch.cuda.max_memory_allocated(),
"seconds": time.perf_counter() - begun,
"test_visible": False,
},
)
def verify_resume(seed, phase):
root = Path("assets/checkpoints")
a = torch.load(
root / f"{phase}-{seed}-continuous.pt", map_location="cpu", weights_only=False
)
b = torch.load(
root / f"{phase}-{seed}-resume.pt", map_location="cpu", weights_only=False
)
checks = {k: equal(a[k], b[k]) for k in a}
if not all(checks.values()):
raise ValueError("Resume differs: " + str(checks))
save_json(
f"outputs/resume-{phase}-{seed}.json",
{
"status": "PASS",
"all_state_exact": True,
"checks": checks,
"boundary": "complete optimizer update, including corruption and permutation RNG",
},
)
Compare normalization, not just losses
Downstream metrics and coverage
Invariant: 180 test cases per branch, with an explicit status for every format failure.
Common mistake: a format difference is counted as an incorrect field without explaining it; publish raw exact match and normalized fields.
File src/preentrainement_denoising/evaluate.py
import math, time
from pathlib import Path
import torch
from .config import read_json, save_json, sha, seed_all
from .corruption import PAD, BOS, EOS, FIRST, VOCAB, restore_from_target
from .data import FIELDS
from .model import make_batch, target_loss
from .state import load_weights, digest
from .train import supervised_row
from .baseline import normalize_with_rules, parse_canonical, upstream_complete
@torch.no_grad()
def greedy(model, rows, config):
model.eval()
maximum = max(len(r["encoder_ids"]) for r in rows)
encoder = torch.full((len(rows), maximum), PAD, dtype=torch.long, device="cuda")
mask = torch.zeros_like(encoder, dtype=torch.bool)
for i, row in enumerate(rows):
encoder[i, : len(row["encoder_ids"])] = torch.tensor(
row["encoder_ids"], device="cuda"
)
mask[i, : len(row["encoder_ids"])] = True
memory = model.encode(encoder, mask)
decoder = torch.full((len(rows), 1), BOS, dtype=torch.long, device="cuda")
dm = torch.ones_like(decoder, dtype=torch.bool)
done = [False] * len(rows)
outputs = [[] for _ in rows]
probes = []
for step in range(config["max_new_tokens"]):
logits = model.decode(decoder, dm, memory, mask)[:, -1].float()
logits[:, [PAD, BOS]] = float("-inf")
chosen = logits.argmax(-1).cpu().tolist()
if step < 2:
probes.append(
{
"generation_position": step,
"logits": [
None if not math.isfinite(x) else x
for x in logits[0].cpu().tolist()
],
"chosen": chosen[0],
}
)
next_ids = []
next_mask = []
for i, token in enumerate(chosen):
if done[i]:
next_ids.append(PAD)
next_mask.append(False)
else:
outputs[i].append(token)
next_ids.append(token)
next_mask.append(True)
done[i] = token == EOS
if all(done):
break
decoder = torch.cat(
[decoder, torch.tensor(next_ids, device="cuda")[:, None]], 1
)
dm = torch.cat([dm, torch.tensor(next_mask, device="cuda")[:, None]], 1)
return outputs, probes
def decode_generated(ids):
finish = "eos" if ids and ids[-1] == EOS else "length"
content = ids[:-1] if finish == "eos" else ids
valid_bytes = all(0 <= v < 256 for v in content)
raw = bytes(content) if valid_bytes else None
try:
text = raw.decode("utf-8") if raw is not None else None
utf8 = raw is not None
except UnicodeDecodeError:
text = None
utf8 = False
return {
"finish_reason": finish,
"content_bytes_hex": raw.hex() if raw is not None else None,
"byte_only": valid_bytes,
"utf8_valid": utf8,
"text": text,
}
def selected_pages(config):
return [
p
for p in read_json("data/generated/input/pages.json")
if p["split"] == "test" and p["formulation"] in config["labeled_formulations"]
]
@torch.no_grad()
def predict(config, seed, phase):
if Path("data/generated/labels").exists():
raise ValueError("Generation service must not mount target labels")
seed_all(seed)
path = Path(f"outputs/model-{phase}-{seed}.json")
model = load_weights(path, config).cuda().eval()
pages = selected_pages(config)
if len(pages) != 180:
raise ValueError("Expected180heldout pairs")
rows = [
{
"document_id": p["document_id"],
"encoder_ids": [BOS] + list(p["text"].encode()) + [EOS],
}
for p in pages
]
result = []
probes = []
begun = time.perf_counter()
torch.cuda.reset_peak_memory_stats()
for offset in range(0, len(rows), config["generation_batch_size"]):
batch = rows[offset : offset + config["generation_batch_size"]]
generated, actual = greedy(model, batch, config)
if offset == 0:
probes = actual
for row, ids in zip(batch, generated):
decoded = decode_generated(ids)
parsed = (
parse_canonical(decoded["text"], decoded["finish_reason"])
if decoded["text"] is not None
else {
"format_valid": False,
"completion_valid": False,
"record": None,
"reason": (
"NON_BYTE_TOKEN" if not decoded["byte_only"] else "INVALID_UTF8"
),
}
)
result.append({**row, "generated_ids": ids, **decoded, "parsed": parsed})
torch.cuda.synchronize()
save_json(
f"outputs/predictions-{phase}-{seed}.json",
{
"phase": phase,
"seed": seed,
"model_sha256": digest(model.state_dict()),
"model_json_sha256": sha(path),
"test_input_sha256": sha("data/generated/input/pages.json"),
"rows": result,
"first_case_logit_probes": probes,
"seconds": time.perf_counter() - begun,
"max_torch_allocated_bytes": torch.cuda.max_memory_allocated(),
"gold_visible": False,
"decoding": "greedy291tokens; PAD/BOS suppressed; sentinels not suppressed",
},
)
@torch.no_grad()
def nll_rows(model, rows, batch_size):
results = []
probes = []
for offset in range(0, len(rows), batch_size):
batch = rows[offset : offset + batch_size]
encoder, em, decoder, dm, labels = make_batch(batch)
logits = model(encoder, em, decoder, dm)
_, _, values = target_loss(logits, labels)
cursor = 0
for row in batch:
n = len(row["target_ids"])
results.append(
{**row, "target_nll": values[cursor : cursor + n].cpu().tolist()}
)
cursor += n
if offset == 0:
for pos in [0, len(batch[0]["target_ids"]) - 1]:
probes.append(
{
"document_id": batch[0]["document_id"],
"target_position": pos,
"target": int(labels[0, pos]),
"logits": logits[0, pos].cpu().tolist(),
"target_nll": results[0]["target_nll"][pos],
}
)
values = [v for row in results for v in row["target_nll"]]
return {
"documents": len(rows),
"target_tokens": len(values),
"target_nll_nats_per_token": math.fsum(values) / len(values),
"rows": results,
"probes": probes,
}
@torch.no_grad()
def evaluate_reconstruction(config, seed, phase):
if phase not in ["initial", "pretrain"]:
raise ValueError("Reconstruction comparison is before/after pretraining")
seed_all(seed)
model = load_weights(f"outputs/model-{phase}-{seed}.json", config).cuda().eval()
test_ids = {
p["document_id"]
for p in read_json("data/generated/input/pages.json")
if p["split"] == "test"
}
rows = [
r
for r in read_json("data/generated/input/pool.json")
if r["document_id"] in test_ids
]
result = nll_rows(model, rows, config["batch_size"])
generations, probes = greedy(model, rows[:12], config)
examples = []
for row, ids in zip(rows, generations):
try:
restored = restore_from_target(row["encoder_ids"], ids)
valid = True
except (ValueError, IndexError):
restored = None
valid = False
examples.append(
{
"document_id": row["document_id"],
"encoder_ids": row["encoder_ids"],
"target_ids": row["target_ids"],
"generated_ids": ids,
"finish_reason": "eos" if ids[-1] == EOS else "length",
"valid_sentinel_structure": valid,
"restored_ids": restored,
"exact_reconstruction": valid and restored == row["clean_ids"],
}
)
save_json(
f"outputs/reconstruction-{phase}-{seed}.json",
{
"seed": seed,
"phase": phase,
"model_sha256": digest(model.state_dict()),
**result,
"fixed_first12_generations": examples,
"first_case_generation_probes": probes,
},
)
def score_fields(predictions, gold):
correct = dict.fromkeys(FIELDS, 0)
exact = valid = truncated = 0
details = []
for row in predictions:
expected = gold[row["document_id"]]
parsed = row["parsed"]
accepted = parsed["completion_valid"]
fields = {
k: accepted and parsed["record"][k] == expected["target"][k] for k in FIELDS
}
raw = accepted and row["text"] == expected["target_text"]
exact += raw
valid += accepted
truncated += row["finish_reason"] == "length"
for key in FIELDS:
correct[key] += fields[key]
details.append(
{
"document_id": row["document_id"],
"fields_correct": fields,
"raw_exact_match": raw,
}
)
n = len(predictions)
return {
"documents": n,
"valid_completions": valid,
"valid_completion_rate": valid / n,
"raw_exact_match": exact,
"raw_exact_match_rate": exact / n,
"field_correct": correct,
"field_accuracy": {k: v / n for k, v in correct.items()},
"truncated": truncated,
"rows": details,
}
def score_canonical_line(config, seed, phase):
gold = {r["document_id"]: r for r in read_json("data/generated/labels/gold.json")}
prediction = read_json(f"outputs/predictions-{phase}-{seed}.json")
metrics = score_fields(prediction["rows"], gold)
model = load_weights(f"outputs/model-{phase}-{seed}.json", config).cuda().eval()
pairs = [{**p, **gold[p["document_id"]]} for p in selected_pages(config)]
nll = nll_rows(model, [supervised_row(p) for p in pairs], config["batch_size"])
save_json(f"outputs/nll-{phase}-{seed}.json", nll)
save_json(
f"outputs/evaluation-{phase}-{seed}.json",
{
"seed": seed,
"phase": phase,
"metrics": metrics,
"response_nll_nats_per_token": nll["target_nll_nats_per_token"],
"quality_gain_required": False,
},
)
print(
{
"seed": seed,
"phase": phase,
"metrics": {k: v for k, v in metrics.items() if k != "rows"},
},
flush=True,
)
def rule_baseline(config):
gold = {r["document_id"]: r for r in read_json("data/generated/labels/gold.json")}
rows = []
for p in selected_pages(config):
observed = normalize_with_rules(p["text"])
rows.append(
{
"document_id": p["document_id"],
"text": observed["text"],
"status": observed["status"],
"finish_reason": "eos",
"parsed": {
"completion_valid": observed["status"] == "ACCEPT",
"record": observed["record"],
},
}
)
missing = read_json("data/generated/input/invalid.json")
decisions = [
{
"document_id": p["document_id"],
"missing_field": p["missing_field"],
"accepted_upstream": upstream_complete(p["text"]),
}
for p in missing
]
save_json(
"outputs/rules.json",
{
"rows": rows,
"metrics": score_fields(rows, gold),
"coverage": sum(r["status"] == "ACCEPT" for r in rows) / len(rows),
"abstentions": sum(r["status"] == "ABSTAIN" for r in rows),
"missing_cases": decisions,
"missing_rejected": sum(not r["accepted_upstream"] for r in decisions),
"scope": "Upstream rule over controlled synthetic fields; not a guarantee of factual completeness for real texts",
},
)
def compare_branches(config):
rows = {}
for seed in config["reference_seeds"]:
rows[str(seed)] = {}
for phase in ["random", "transferred"]:
report = read_json(f"outputs/evaluation-{phase}-{seed}.json")
training = read_json(f"outputs/train-{phase}-{seed}-resume.json")
pretraining = read_json(f"outputs/train-pretrain-{seed}-resume.json")
continuous = read_json(f"outputs/train-{phase}-{seed}-continuous.json")
precontinuous = read_json(f"outputs/train-pretrain-{seed}-continuous.json")
rows[str(seed)][phase] = {
"metrics": {k: v for k, v in report["metrics"].items() if k != "rows"},
"normalization_target_nll": report["response_nll_nats_per_token"],
"supervised_examples": len(training["identity"]["training_ids"]),
"supervised_presentations": sum(
len(r["document_ids"]) for r in training["history"]
),
"supervised_target_tokens": sum(
r["target_tokens"] for r in training["history"]
),
"continuous_supervised_seconds": continuous["seconds"],
"additional_pretraining_steps": 600 if phase == "transferred" else 0,
"additional_pretraining_target_tokens": (
sum(r["target_tokens"] for r in pretraining["history"])
if phase == "transferred"
else 0
),
"continuous_additional_pretraining_seconds": (
precontinuous["seconds"] if phase == "transferred" else 0
),
}
a = read_json(f"outputs/train-random-{seed}-resume.json")
b = read_json(f"outputs/train-transferred-{seed}-resume.json")
if [r["document_ids"] for r in a["history"]] != [
r["document_ids"] for r in b["history"]
]:
raise ValueError("Supervised batches differ")
save_json(
"outputs/comparison.json",
{
"seeds": rows,
"rules": {
k: v
for k, v in read_json("outputs/rules.json").items()
if k not in ["rows", "missing_cases"]
},
"equal_annotation_budget": True,
"equal_total_compute": False,
"continuous_times_include_validation_and_checkpoint_io": True,
"resume_verification_overhead_not_in_path_cost_but_in_reader_wall_time": True,
"quality_gain_required": False,
},
)
Change the annotation budget
Marginal benefit of transfer
Invariant: the new annotations belong only to training families.
Common mistake: comparing two different test sets distorts the gap; reuse the IDs and hash the list.
File src/preentrainement_denoising/exercise.py
from pathlib import Path
from .config import read_json, save_json, sha
def preserve_main(config):
paths = ["outputs/comparison.json", "outputs/rules.json"]
for seed in config["reference_seeds"]:
for phase in ["initial", "pretrain", "random", "transferred"]:
paths.append(f"outputs/model-{phase}-{seed}.json")
for phase in ["random", "transferred"]:
paths.extend(
[
f"outputs/evaluation-{phase}-{seed}.json",
f"outputs/predictions-{phase}-{seed}.json",
]
)
save_json("outputs/exercise-before.json", {name: sha(name) for name in paths})
def run_label_budget(config):
before = read_json("outputs/exercise-before.json")
if any(sha(name) != h for name, h in before.items()):
raise ValueError("Exercise changed main artifacts")
rows = {}
for seed in config["reference_seeds"]:
rows[str(seed)] = {}
for branch in ["random", "transferred"]:
phase = "exercise_" + branch
report = read_json(f"outputs/evaluation-{phase}-{seed}.json")
training = read_json(f"outputs/train-{phase}-{seed}-resume.json")
main = read_json(f"outputs/train-{branch}-{seed}-resume.json")
if training["initial_model_sha256"] != main["initial_model_sha256"] or len(
training["identity"]["training_ids"]
) != 2 * len(main["identity"]["training_ids"]):
raise ValueError("Initial state or doubled labels changed")
rows[str(seed)][branch] = {
"main": read_json(f"outputs/evaluation-{branch}-{seed}.json")[
"metrics"
],
"double_labels": report["metrics"],
"labeled_pairs": len(training["identity"]["training_ids"]),
"optimizer_steps": training["steps"],
"presentations": sum(
len(r["document_ids"]) for r in training["history"]
),
"initial_model_sha256": training["initial_model_sha256"],
"target_tokens": sum(r["target_tokens"] for r in training["history"]),
}
save_json(
"outputs/exercise/label-budget.json",
{
"seeds": rows,
"protected_sha256": before,
"main_artifacts_unchanged": True,
"same_pretraining_corpus_and_weights": True,
"same180test_cases": True,
"same300updates_fewer_passes_over_more_labels": True,
"quality_gain_required": False,
},
)
Sources
The T5 paper and the authors' repository were consulted for context on the objective. No T5 code or weights are reused. The dependencies actually used are locked in the project. Docker GPU and NVIDIA Container Toolkit document the prerequisites.
Results observed during GPU development
The full walkthrough completed successfully with 15 tests in 898.97 seconds. Two seeds (42 and 43) were trained, with 600 denoising updates followed by 300 SFT updates per branch. Resume comparisons cover the ten training chains, including the exercise: model, optimizer, sampler, RNG, and histories are recovered exactly. Reproducing the result from the public archive remains a separate verification.
Across the 180 normalization test cases, none of the eight neural variants produces a fully exact line, and none correctly recovers the reference. The rules achieve 180/180. Pretraining therefore improves some signals without solving the task: with two supervised formulations, valid complete format rises from 101/180 to 180/180 (seed 42) and from 98/180 to 180/180 (seed 43). Correct material rises from 69 to 131 and from 54 to 145; diameter remains correct in only six cases for each transferred model.
A retained real example: for « Piece K0510, aluminium, diametre 18 mm. », the seed-42 transferred model produces reference=K0024;materiau=aluminium;diametre_mm=26. The format and material are correct, but the other two fields are wrong. This output is enough to show why syntactic compliance is not reliable extraction.
The NLL for reconstructing gaps on 900 records and 9,150 target tokens falls from 5.6601 to 1.0212 (seed 42) and from 5.6971 to 1.0314 (seed 43). Normalization NLL under the target prefix falls from 0.6420 to 0.3425 and from 0.6785 to 0.3387. These conditioned losses and the quality of free outputs answer different questions; their decrease does not contradict the observed errors.
The exercise doubles the number of supervised formulations, from 840 to 1,680 available examples, while keeping 300 updates: each example is therefore revisited less often. It does not provide a uniform gain. For seed-42 transfer, valid format falls to 170/180 and material to 109/180; for seed 43, they reach 180/180 and 146/180. No setting was adjusted to force an improvement. The 60 cases with a missing field are rejected by the upstream rule-based check; they do not prove that the network can abstain.
The additional cost of pretraining is included: 36.13 s then 35.26 s on continuous paths, plus about 16–17 s per SFT. These durations include validation and writing; resumes, generation, and other checks explain a total walkthrough duration far greater. Observed internal allocation peaks are 153,128,448 bytes for denoising and 164,146,176 for SFT; they are not the total VRAM seen by Linux. CPU preparation for other work took place during the walkthrough: no measurement on a fully idle host is claimed.
This closed corpus and its rules are a learning aid for the mechanism. The additional pretraining does not justify replacing the deterministic parser to normalize these records. Two seeds are insufficient to establish business generality.
Development observations
Values recalculated from the raw outputs of this walkthrough. They are not improvement thresholds; the walkthrough from the ZIP recalculates them on the same populations.
{
"status": "PASS",
"documents": 6000,
"test_normalization_cases": 180,
"seeds": [
42,
43
],
"parameters": 3269952,
"metrics": {
"random-42": {
"documents": 180,
"valid_completions": 101,
"valid_completion_rate": 0.5611111111111111,
"raw_exact_match": 0,
"raw_exact_match_rate": 0.0,
"field_correct": {
"reference": 0,
"materiau": 69,
"diametre_mm": 3
},
"field_accuracy": {
"reference": 0.0,
"materiau": 0.38333333333333336,
"diametre_mm": 0.016666666666666666
},
"truncated": 0
},
"transferred-42": {
"documents": 180,
"valid_completions": 180,
"valid_completion_rate": 1.0,
"raw_exact_match": 0,
"raw_exact_match_rate": 0.0,
"field_correct": {
"reference": 0,
"materiau": 131,
"diametre_mm": 6
},
"field_accuracy": {
"reference": 0.0,
"materiau": 0.7277777777777777,
"diametre_mm": 0.03333333333333333
},
"truncated": 0
},
"exercise_random-42": {
"documents": 180,
"valid_completions": 109,
"valid_completion_rate": 0.6055555555555555,
"raw_exact_match": 0,
"raw_exact_match_rate": 0.0,
"field_correct": {
"reference": 0,
"materiau": 59,
"diametre_mm": 8
},
"field_accuracy": {
"reference": 0.0,
"materiau": 0.3277777777777778,
"diametre_mm": 0.044444444444444446
},
"truncated": 0
},
"exercise_transferred-42": {
"documents": 180,
"valid_completions": 170,
"valid_completion_rate": 0.9444444444444444,
"raw_exact_match": 0,
"raw_exact_match_rate": 0.0,
"field_correct": {
"reference": 0,
"materiau": 109,
"diametre_mm": 7
},
"field_accuracy": {
"reference": 0.0,
"materiau": 0.6055555555555555,
"diametre_mm": 0.03888888888888889
},
"truncated": 0
},
"random-43": {
"documents": 180,
"valid_completions": 98,
"valid_completion_rate": 0.5444444444444444,
"raw_exact_match": 0,
"raw_exact_match_rate": 0.0,
"field_correct": {
"reference": 0,
"materiau": 54,
"diametre_mm": 0
},
"field_accuracy": {
"reference": 0.0,
"materiau": 0.3,
"diametre_mm": 0.0
},
"truncated": 0
},
"transferred-43": {
"documents": 180,
"valid_completions": 180,
"valid_completion_rate": 1.0,
"raw_exact_match": 0,
"raw_exact_match_rate": 0.0,
"field_correct": {
"reference": 0,
"materiau": 145,
"diametre_mm": 6
},
"field_accuracy": {
"reference": 0.0,
"materiau": 0.8055555555555556,
"diametre_mm": 0.03333333333333333
},
"truncated": 0
},
"exercise_random-43": {
"documents": 180,
"valid_completions": 178,
"valid_completion_rate": 0.9888888888888889,
"raw_exact_match": 0,
"raw_exact_match_rate": 0.0,
"field_correct": {
"reference": 0,
"materiau": 60,
"diametre_mm": 5
},
"field_accuracy": {
"reference": 0.0,
"materiau": 0.3333333333333333,
"diametre_mm": 0.027777777777777776
},
"truncated": 0
},
"exercise_transferred-43": {
"documents": 180,
"valid_completions": 180,
"valid_completion_rate": 1.0,
"raw_exact_match": 0,
"raw_exact_match_rate": 0.0,
"field_correct": {
"reference": 0,
"materiau": 146,
"diametre_mm": 6
},
"field_accuracy": {
"reference": 0.0,
"materiau": 0.8111111111111111,
"diametre_mm": 0.03333333333333333
},
"truncated": 0
}
},
"rules": {
"documents": 180,
"valid_completions": 180,
"valid_completion_rate": 1.0,
"raw_exact_match": 180,
"raw_exact_match_rate": 1.0,
"field_correct": {
"reference": 180,
"materiau": 180,
"diametre_mm": 180
},
"field_accuracy": {
"reference": 1.0,
"materiau": 1.0,
"diametre_mm": 1.0
},
"truncated": 0
},
"quality_gain_required": false,
"scope": "Data-only exact sampler/corruption replay, all target budgets, weight hashes, full-state resume evidence, CE probes, raw decoding/metrics and observed phase costs; no scalar reimplementation of the full Transformer"
}
