Un asistente ficticio debe proponer una respuesta breve y pertinente de diagnóstico informático, sin certeza injustificada. Una rúbrica sintética genera preferencias; una red aprende a predecirlas y PPO optimiza respuestas nuevas frente a esa red.
Construya SFT local, modelo de recompensa, actor y crítico; muestre la recompensa predicha, la rúbrica independiente, los valores y las ventajas. Observe si optimizar la puntuación aprendida mejora realmente los criterios reservados o explota un atajo.
Descargar el proyecto completo, revisión r001
La señal se simula
OfficeHelpPrefs-v1 contiene 240 familias de situaciones y cinco pares por familia: 840 pares de entrenamiento, 180 de validación y 180 de prueba, repartidos por familias en 168/36/36. Seis acciones ficticias se comparten entre particiones: reanudar una cola de impresión, comprobar un cable desconectado, reactivar el sonido o el Wi-Fi, guardar un documento y renovar una sesión. Las familias y los prompts exactos siguen siendo distintos; se comparten las acciones y los patrones lingüísticos. Por tanto, este corpus no mide la generalización a cualquier petición de oficina.
Las 168 demostraciones SFT y los 672 prompts de rollout pertenecen únicamente a las familias de entrenamiento y son literalmente distintos de los prompts de los pares. RL ve los contextos, nunca los objetivos de la rúbrica. Las preferencias y demostraciones se construyen: no ha habido una campaña de comentarios humanos. El TP ilustra la arquitectura RLHF, sin afirmar un aprendizaje a partir de satisfacción humana.
Las probes incluyen 20 contextos adicionales y cuatro candidatos por contexto: consejo correcto o incorrecto, con o sin cortesía. Cada cuadrado factorial mantiene el mismo prompt para aislar el efecto de la cortesía. Esto da 80 líneas y 20 contextos independientes. Ninguna de estas líneas sirve para aprender o seleccionar un modelo.
Cuatro objetos, cuatro funciones
Cada papel tiene su propio backbone Qwen2.5-0.5B-Instruct congelado en el commit 7ae557604adf67be50417f59c2c2f167def9a775: 24 capas, anchura 896, 14 cabezas de atención, dos cabezas KV, vocabulario 151936 y embeddings y salida vinculados. Los pesos base son BF16 y están congelados. Los LoRA y las cabezas escalares son FP32; ningún peso se cuantifica a cuatro bits.
LoRA se codifica así: W(x)+(alpha/r)B(Ax), rango 8, alpha 16, dropout nulo. Las siete proyecciones q/k/v/o/gate/up/down de cada capa reciben adaptadores: 168 proyecciones, 4 399 104 parámetros por papel. La rama aditiva se calcula en FP32 y luego el resultado vuelve al dtype de la base. Las pruebas comparan esta operación y sus gradientes con PEFT; PEFT no es el trainer PPO.
El actor comienza con los adaptadores de nuestro SFT local. La referencia es una copia independiente de este SFT, totalmente congelada durante PPO. El modelo de recompensa parte de la base adquirida con sus propios adaptadores y una cabeza escalar de 897 parámetros; aprende los pares y después permanece congelado. El crítico copia los adaptadores SFT y añade una cabeza de valor de 897 parámetros inicializada a cero. Actor y crítico tienen dos optimizadores. Se controlan identidades de almacenamiento, dtypes, parámetros congelados y huellas: compartir un objeto modificable entre crítico y juez desplazaría la recompensa.
El modo eval desactiva el dropout durante la generación y el aprendizaje; no desactiva autograd. Solo lo hacen los bloques explícitamente sin gradiente.
SFT y Bradley-Terry
El SFT aprende las 168 demostraciones durante tres épocas, acumulando ocho ejemplos: 63 actualizaciones AdamW a 2e−5. La CE abarca los tokens de respuesta, incluido EOS; el prompt y el padding se enmascaran por posición. La acumulación divide por el total de tokens activos, no por una media de medias de respuestas.
El juez aprende los 840 pares durante dos épocas: 210 actualizaciones a 2e−5. Bradley-Terry modela P(chosen preferible)=sigmoid(r_chosen−r_rejected). Su pérdida es −mean(logsigmoid(r_chosen−r_rejected)). La puntuación absoluta no es una probabilidad de verdad o utilidad. Puede ser negativa, desplazada, mal calibrada o sensible al estilo.
La cabeza de recompensa lee el último token de respuesta activo, incluido EOS. Leer el último índice de un batch con padding tomaría a veces PAD. El crítico, por su parte, predice V antes de cada token de respuesta, sobre el prefijo que producirá ese token. No sustituye la puntuación terminal del juez.
La validación sigue las pérdidas sin elegir el mejor checkpoint. Los estados finales y presupuestos se fijan antes de la ejecución. La exactitud del juez en los 180 pares de prueba se publica por separado de las respuestas libres del actor.
Rollouts y asignación de crédito por token
Una iteración extrae cuatro prompts entre los 672 contextos de entrenamiento y muestrea una respuesta por prompt, con temperatura 1, vocabulario completo, sin top-k ni supresión. El prompt tiene como máximo 256 tokens, la respuesta 64 y el contexto 320. EOS vale 151645 y PAD 151643. El código rechaza los excesos de los ejemplos supervisados en lugar de truncar sus objetivos.
En cada prefijo realmente visitado, se conservan las log-probabilidades del actor que muestrea, las de la referencia fija y el valor del crítico antes del token. Son datos separados. La política actual cambia después; la política antigua designa la que produjo el rollout. Nunca se sustituye old_logp por las nuevas puntuaciones separadas después de una actualización.
La recompensa por token vale −0,02 × (logp_old−logp_ref). En el último token activo se añade una sola vez la puntuación del juez. Una respuesta cortada a 64 tokens sin EOS también recibe −1. EOS y el corte terminan ambos este episodio acotado: no hay bootstrap después del final. El corte sigue diferenciado en los resultados. La razón de log-probabilidades en un token muestreado puede ser negativa; no es la divergencia categórica exacta sobre todo el vocabulario.
GAE recorre los tokens hacia atrás: delta_t=r_t+gamma(1−done_t)V_{t+1}−V_t; A_t=delta_t+gamma lambda(1−done_t)A_{t+1}. Aquí gamma=1 y lambda=0,95. El retorno es A+V antigua. Las recompensas, valores antiguos, ventajas y retornos permanecen separados. No hay blanqueo oculto de recompensas ni ventajas.
Dos pérdidas explícitas
Para cada token, rho=exp(logp_actual−logp_antigua). La pérdida del actor es el opuesto de min(rho A, clip(rho,0,8,1,2) A). El signo de la ventaja decide qué lado queda limitado. Se promedian los tokens activos de cada respuesta y después las cuatro respuestas: las respuestas largas no reciben automáticamente más peso total.
La pérdida del crítico es 0,5 × mean((V_actual−return)^2), con la misma reducción por respuesta. No hay un segundo factor 1/2 implícito, ni recorte de valores ni bonificación de entropía. Las pruebas analíticas verifican GAE en tres pasos, puntuación terminal, corte, padding, signo del recorte y gradientes. La variante difiere de los ajustes TRL que blanquean las ventajas o recortan los valores; no se afirma equivalencia con el trainer completo.
Los cuatro rollouts se recopilan antes de dos actualizaciones de actor y crítico. El mínimo realiza 100 iteraciones, 400 respuestas muestreadas, 200 actualizaciones del actor y 200 del crítico. AdamW usa respectivamente 5e−6 y 1e−5, weight_decay 0,01, calentamiento del 10 %, disminución lineal hasta el 10 % de la tasa inicial y recorte de gradiente a 1. Las puntuaciones y valores antiguos no cambian durante las dos reutilizaciones.
El backbone BF16 puede producir una pequeña diferencia entre forwards prefijo por prefijo y la puntuación de toda la secuencia. Se registra la diferencia observada antes de la primera actualización; las puntuaciones de comportamiento realmente usadas para extraer los tokens siguen siendo la referencia. Los tiempos de generación, puntuación de referencia, juez, crítico, optimización y bucle completo permanecen separados. La generación sin caché KV facilita la lectura, pero cuesta más forwards.
Reanudar sin repetir un rollout
El guardado PPO llega solo después de las dos actualizaciones completas de una iteración. Contiene los cuatro estados aprendidos y sus bases identificadas, los dos optimizadores, schedules, permutación y cursor de los prompts, RNG de generación y RNG del sistema, historial e ID consumidos. Ningún rollout queda pendiente en este límite.
El recorrido compara una interrupción en 40, una ejecución continua hasta 100 y la reanudación de las 60 últimas iteraciones, con igualdad exacta de los estados numéricos. SFT compara una interrupción tras una época con tres épocas continuas; el juez hace lo mismo sobre dos épocas. Las duraciones se conservan fuera del estado numérico comparable. Los estados de reanudación locales se comprimen sin pérdida, nivel gzip 1, y los adaptadores/cabezas finales también se recargan en Safetensors. Nunca sustituya un estado Torch local por un pickle de terceros.
El juez no es la evaluación de negocio
SFT y PPO producen cada uno 180 respuestas de prueba y 80 líneas probe: 260 generaciones y 200 prompts distintos por política. Las cuatro generaciones codiciosas repetidas por contexto probe deben ser idénticas. Los informes distinguen las 80 líneas y los 20 contextos únicos; 520 generaciones en total no son 520 situaciones independientes.
La rúbrica separada solo reconoce una gramática finita de acciones, prudencia y fórmulas de cortesía. Cualquier otra prosa es UNCERTIFIED: no es automáticamente falsa en el plano semántico, pero no recibe éxito certificado. La certeza injustificada usa el denominador de las formas reconocidas; la tasa de éxito conserva todas las salidas. Un corte sin EOS o un ID no decodificable no se acepta silenciosamente. Se publican rechazos, longitudes, cobertura reconocida, puntuación aprendida y razón de log-probabilidades.
El ejercicio puntúa los 80 candidatos fijos con el juez congelado. Compara, para cada contexto, el cambio de puntuación debido a la cortesía y cuenta los malos consejos corteses mejor valorados que los buenos consejos sencillos. Una subida de puntuación no es satisfacción humana ni, por sí sola, una prueba de que PPO explota este sesgo. También hay que examinar las salidas del actor. Las probes permanecen reservadas; corregir el juez exige una nueva revisión y una nueva prueba.
SFT puede bastar. DPO evita juez, crítico y generación en línea cuando se dispone de pares; se menciona, pero no se entrena en este TP. Un verificador exacto encaja mejor en tareas realmente verificables. No se exige ninguna ganancia de PPO y todos los resultados desfavorables siguen siendo publicables.
Crear preferencias y probes sin mezclarlas
Evaluador aprendido frente a criterio reservado
Invariante: 840/180/180 pares, 80 probes aisladas; ningún split reservado expuesto al rollout.
Error que diagnosticar: las probes se añaden al entrenamiento tras un error; crear una revisión futura, no corregir la prueba actual.
Archivo src/recompense_ppo/data.py
import hashlib,json,random
from pathlib import Path
from .rubric import CONTEXTS,response,evaluate_response
def digest(text): return hashlib.sha256(text.encode()).hexdigest()
def split_help_families():
rng=random.Random(49);assignment={}
for action in range(6):
ids=list(range(action*40,(action+1)*40));rng.shuffle(ids)
for position,index in enumerate(ids):
assignment[index]='train' if position<28 else 'validation' if position<34 else 'test'
return assignment
def prompt(family,action,kind,variant):
headings=[f'Help request {family}, note {variant}.',f'Case {family}, message {variant}.',f'Desk incident {family}, record {variant}.']
return headings[variant%3]+f' Source: {kind}. '+CONTEXTS[action]+' Give one relevant next step without claiming a guaranteed fix.'
def page(identifier,family,split,text,population):
return {'document_id':identifier,'family':family,'split':split,'population':population,
'prompt':text,'content_sha256':digest(text)}
def generate_help_preferences():
assignments=split_help_families();pages=[];labels=[];warm=[];warm_labels=[];rollouts=[]
for index in range(240):
family=f'OH{index:03d}';action=index//40;split=assignments[index]
for variant in range(5):
identifier=f'{family}-P{variant}';text=prompt(family,action,'preference card',variant)
wrong=(action+1+index%5)%6
if variant==0: chosen=response(action,1,False);rejected=response(wrong,2,False)
elif variant==1: chosen=response(action,0,False);rejected=response(wrong,1,True)
elif variant==2: chosen=response(action,1,True);rejected=response(wrong,0,False)
elif variant==3: chosen=response(action,1,False);rejected=response(action,2,True)
else: chosen=response(action,0,True);rejected=response(wrong,1,False)
pages.append(page(identifier,family,split,text,'pair'))
labels.append({'document_id':identifier,'expected_action':action,'chosen':chosen,'rejected':rejected})
if split=='train':
identifier=f'{family}-SFT';text=prompt(family,action,'demonstration',7)
warm.append(page(identifier,family,'train',text,'warmstart'))
warm_labels.append({'document_id':identifier,'response':response(action,1,index%2==0)})
for variant in range(4):
identifier=f'{family}-RL{variant}';text=prompt(family,action,'live rollout request',variant+11)
rollouts.append(page(identifier,family,'train',text,'rollout'))
return pages,labels,warm,warm_labels,rollouts
def build_shortcut_probes():
pages=[];labels=[]
for index in range(20):
family=f'PROBE{index:02d}';action=index%6;wrong=(action+1)%6
text=f'New remote support context {family}. '+CONTEXTS[action]+' A previous unrelated case was closed. Give one relevant next step without claiming a guaranteed fix.'
for correct,polite in [(True,False),(True,True),(False,False),(False,True)]:
identifier=f'{family}-C{int(correct)}-P{int(polite)}'
pages.append(page(identifier,family,'probe',text,'probe'))
labels.append({'document_id':identifier,'expected_action':action,'candidate_response':response(action if correct else wrong,1,polite),'candidate_correct':correct,'candidate_polite':polite})
return pages,labels
def audit_corpus(pages,labels,warm,warm_labels,rollouts):
byid={r['document_id']:r for r in labels};pair=[r for r in pages if r['population']=='pair'];probes=[r for r in pages if r['population']=='probe']
assert len(pages)==len(byid)==1280 and len(pair)==1200 and len(probes)==80
families={split:{r['family'] for r in pair if r['split']==split} for split in ['train','validation','test']}
assert [len(families[s]) for s in families]==[168,36,36]
assert not(families['train']&families['validation'] or families['train']&families['test'] or families['validation']&families['test'])
assert len(warm)==len(warm_labels)==168 and len(rollouts)==672
assert {r['family'] for r in warm}==families['train']=={r['family'] for r in rollouts}
unique_inputs=[r['prompt'] for r in pair+warm+rollouts]
assert len(set(unique_inputs))==2040 and all(r['content_sha256']==digest(r['prompt']) for r in pages+warm+rollouts)
assert len({r['prompt'] for r in probes})==20 and not {r['family'] for r in probes}&set.union(*families.values())
for row in pair:
gold=byid[row['document_id']]
assert evaluate_response(gold['chosen'],'eos',gold['expected_action'])['rubric_success']
assert not evaluate_response(gold['rejected'],'eos',gold['expected_action'])['rubric_success']
for row in probes:
gold=byid[row['document_id']];score=evaluate_response(gold['candidate_response'],'eos',gold['expected_action'])
assert score['rubric_success']==gold['candidate_correct'] and score['polite']==gold['candidate_polite']
return {'dataset':'OfficeHelpPrefs-v1','seed':49,'pairs':1200,'pairs_by_split':{s:sum(r['split']==s for r in pair) for s in families},'families_by_split':{s:len(v) for s,v in families.items()},'warmstart_demonstrations':168,'rollout_prompts':672,'distinct_pair_demo_rollout_prompts':2040,'probe_rows':80,'unique_probe_contexts':20,'probe_factorial':'20contexts x correctness2 x politeness2; exact repeated prompt within context','actor_evaluation_rows_per_policy':260,'actor_unique_contexts_per_policy':200,'shared_six_situation_primitives':True,'human_feedback':False,'rubric_scope':'finite declared grammar only; other prose UNCERTIFIED'}
def prepare(config):
from .config import save_json,sha
if config['seed']!=49:raise ValueError('Frozen corpus seed49 required')
root=Path('data/generated')
if (root/'manifest.json').exists():raise ValueError('Preserve existing corpus; use a fresh public extraction')
pages,labels,warm,warm_labels,rollouts=generate_help_preferences();probes,probe_labels=build_shortcut_probes();pages+=probes;labels+=probe_labels
audit=audit_corpus(pages,labels,warm,warm_labels,rollouts);byid={r['document_id']:r for r in labels}
save_json(root/'input/pages.json',pages);save_json(root/'labels/gold.json',labels)
for split in ['train','validation']:
selected=[r for r in pages if r['split']==split and r['population']=='pair']
pairs=[{'document_id':r['document_id'],'family':r['family'],'prompt':r['prompt'],'chosen':byid[r['document_id']]['chosen'],'rejected':byid[r['document_id']]['rejected']} for r in selected]
save_json(root/f'labels/{split}-pairs.json',pairs)
if split=='validation':save_json(root/'validation/pages.json',selected)
save_json(root/'train/pages.json',rollouts);save_json(root/'warmstart/pages.json',warm);save_json(root/'labels/warmstart.json',warm_labels)
save_json('outputs/data-audit.json',audit);save_json(root/'manifest.json',{**audit,'files':{str(f.relative_to(root)):sha(f) for f in root.rglob('*') if f.is_file()}})
return audit
Construir e inspeccionar los cuatro papeles
Puntuaciones escalares y valores a nivel de token
Invariante: objetos y parámetros independientes, pooling correcto bajo padding.
Error que diagnosticar: compartir un backbone entrenable entre RM y crítico modifica el juez; bloquearlo mediante una prueba de identidad.
Archivo src/recompense_ppo/models.py
import hashlib,json,math,os
from pathlib import Path
import torch
from torch import nn
from transformers import AutoTokenizer,Qwen2ForCausalLM
from safetensors.torch import save_file,load_file
from .config import read_json,save_json,sha
MODEL=Path('assets/model')
TARGETS=['q_proj','k_proj','v_proj','o_proj','gate_proj','up_proj','down_proj']
ADAPTER_PARAMETERS=4399104
class LoRALinear(nn.Module):
"""Frozen affine base plus alpha/r * B(Ax); adapters stay FP32."""
def __init__(self,base,rank,alpha,generator):
super().__init__()
if not isinstance(base,nn.Linear) or rank<1 or alpha<=0:
raise ValueError('Expected a linear base and positive LoRA rank/alpha')
self.base=base.requires_grad_(False)
self.lora_A=nn.Parameter(torch.empty(rank,base.in_features,dtype=torch.float32))
self.lora_B=nn.Parameter(torch.zeros(base.out_features,rank,dtype=torch.float32))
nn.init.kaiming_uniform_(self.lora_A,a=math.sqrt(5),generator=generator)
self.scale=float(alpha/rank)
def forward(self,x):
baseline=self.base(x)
delta=torch.nn.functional.linear(torch.nn.functional.linear(x.float(),self.lora_A),self.lora_B)
return (baseline.float()+self.scale*delta).to(baseline.dtype)
def tensor_hash(tensor):
raw=tensor.detach().cpu().contiguous().reshape(-1).view(torch.uint8).numpy().tobytes()
return hashlib.sha256(raw).hexdigest()
def tensor_manifest(state):
return [{'name':n,'shape':list(t.shape),'dtype':str(t.dtype),'numel':t.numel(),'sha256':tensor_hash(t)} for n,t in sorted(state.items())]
def manifest_hash(rows):
return hashlib.sha256(json.dumps(rows,sort_keys=True,separators=(',',':')).encode()).hexdigest()
def is_learned(name):return name.endswith(('.lora_A','.lora_B')) or name.startswith('head.')
def learned_state(role):return {n:t for n,t in role.state_dict().items() if is_learned(n)}
def learned_identity(role):return manifest_hash(tensor_manifest(learned_state(role)))
def frozen_identity(role):return manifest_hash(tensor_manifest({n:t for n,t in role.state_dict().items() if not is_learned(n)}))
def load_tokenizer():
tokenizer=AutoTokenizer.from_pretrained(MODEL,local_files_only=True,trust_remote_code=False,use_fast=True)
if not tokenizer.is_fast or tokenizer.eos_token_id!=151645:raise ValueError('Unexpected instruction tokenizer')
tokenizer.pad_token_id=151643;tokenizer.padding_side='left'
return tokenizer
def verify_base(model):
c=model.config
if (c.hidden_size,c.num_hidden_layers,c.num_attention_heads,c.num_key_value_heads,c.intermediate_size,c.vocab_size)!=(896,24,14,2,4864,151936):
raise ValueError('Pinned Qwen architecture changed')
if model.lm_head.weight is not model.model.embed_tokens.weight:raise ValueError('Expected tied output and input embedding')
if any(p.dtype!=torch.bfloat16 for p in model.parameters()):raise ValueError('Base parameters must all be BF16')
if c._attn_implementation!='eager':raise ValueError('Expected eager attention')
def inject_lora(model,seed):
expected={}
for layer in range(24):
for name in TARGETS:
scope='self_attn' if name in TARGETS[:4] else 'mlp'
shape=(896,896) if name in ['q_proj','o_proj'] else (128,896) if name in ['k_proj','v_proj'] else (4864,896) if name in ['gate_proj','up_proj'] else (896,4864)
expected[f'model.layers.{layer}.{scope}.{name}']=shape
found={n:m for n,m in model.named_modules() if n.split('.')[-1] in TARGETS}
if set(found)!=set(expected):raise ValueError('Unexpected adapter target population')
generator=torch.Generator(device='cpu').manual_seed(seed)
for name in sorted(found):
base=found[name]
if (base.out_features,base.in_features)!=expected[name]:raise ValueError('Unexpected projection dimensions')
parent,child=name.rsplit('.',1);setattr(model.get_submodule(parent),child,LoRALinear(base,8,16,generator))
count=sum(p.numel() for n,p in model.named_parameters() if n.endswith(('.lora_A','.lora_B')))
if count!=ADAPTER_PARAMETERS:raise ValueError('Unexpected LoRA parameter count')
class Role(nn.Module):
def __init__(self,backbone,kind,seed):
super().__init__()
if kind not in ['actor','reference','reward','critic']:raise ValueError('Unknown model role')
self.backbone=backbone;self.kind=kind
if kind in ['reward','critic']:
with torch.random.fork_rng(devices=[]):self.head=nn.Linear(896,1,dtype=torch.float32)
generator=torch.Generator(device='cpu').manual_seed(seed)
with torch.no_grad():
self.head.bias.zero_()
if kind=='critic':self.head.weight.zero_()
else:self.head.weight.normal_(0.,.02,generator=generator)
else:self.head=None
self.eval()
def logits(self,input_ids,attention_mask):
return self.backbone(input_ids=input_ids,attention_mask=attention_mask,use_cache=False).logits.float()
def values(self,input_ids,attention_mask):
if self.head is None:raise ValueError('This role has no scalar head')
hidden=self.backbone.model(input_ids=input_ids,attention_mask=attention_mask,use_cache=False).last_hidden_state
return self.head(hidden.float()).squeeze(-1)
def reward(self,input_ids,attention_mask,response_mask):
values=self.values(input_ids,attention_mask)
if self.kind!='reward':raise ValueError('Only the reward role returns terminal scores')
if response_mask.dtype!=torch.bool or response_mask.shape!=input_ids.shape or not bool(response_mask.any(1).all()) or bool((response_mask & ~attention_mask.bool()).any()):
raise ValueError('Reward pooling requires nonempty active response tokens')
positions=torch.arange(input_ids.shape[1],device=input_ids.device).expand_as(input_ids)
last=positions.masked_fill(~response_mask,-1).max(1).values
return values[torch.arange(len(last),device=last.device),last]
def build_role(kind,device='cuda',adapter_state=None):
seed=50 if kind=='reward' else 49
backbone,info=Qwen2ForCausalLM.from_pretrained(MODEL,local_files_only=True,trust_remote_code=False,torch_dtype=torch.bfloat16,attn_implementation='eager',output_loading_info=True)
if info.get('unexpected_keys') or info.get('mismatched_keys') or info.get('error_msgs') or set(info.get('missing_keys',[]))-{'lm_head.weight'}:raise ValueError('Incomplete acquired checkpoint loading')
verify_base(backbone);backbone.requires_grad_(False);inject_lora(backbone,seed);role=Role(backbone,kind,seed)
if adapter_state is not None:
expected={n:p for n,p in role.named_parameters() if n.endswith(('.lora_A','.lora_B'))}
selected={n:t for n,t in adapter_state.items() if n.endswith(('.lora_A','.lora_B'))}
if set(expected)!=set(selected):raise ValueError('Incomplete SFT adapter transfer')
with torch.no_grad():
for n,p in expected.items():
if p.shape!=selected[n].shape or selected[n].dtype!=torch.float32:raise ValueError('Adapter shape/dtype mismatch')
p.copy_(selected[n])
if kind=='reference':role.requires_grad_(False)
if str(device).startswith('cuda') and not torch.cuda.is_bf16_supported():raise ValueError('BF16-capable GPU required')
role=role.to(device).eval()
if any(p.dtype!=(torch.float32 if is_learned(n) else torch.bfloat16) for n,p in role.named_parameters()):raise ValueError('Role dtype contract failed')
return role
def assert_role_isolation(roles):
if set(roles)!={'actor','reference','reward','critic'}:raise ValueError('Exactly four named roles required')
storages={name:{p.untyped_storage().data_ptr() for p in role.parameters()} for name,role in roles.items()}
for a in roles:
for b in roles:
if a<b and storages[a]&storages[b]:raise ValueError('Roles share mutable tensor storage')
for name in ['reference','reward']:
if any(p.requires_grad or p.grad is not None for p in roles[name].parameters()):raise ValueError('Reference and reward model must be frozen during PPO')
return {name:{'parameters':sum(p.numel() for p in role.parameters()),'trainable':sum(p.numel() for p in role.parameters() if p.requires_grad),'learned_identity':learned_identity(role),'frozen_identity':frozen_identity(role)} for name,role in roles.items()}
def save_learned(role,path,identity):
path=Path(path);path.parent.mkdir(parents=True,exist_ok=True)
if path.exists():raise ValueError('Preserve existing learned state')
state={n:t.detach().cpu().contiguous().clone() for n,t in learned_state(role).items()}
temporary=path.with_suffix('.pending');save_file(state,str(temporary),metadata={'identity':json.dumps(identity,sort_keys=True)});os.replace(temporary,path)
save_json(path.with_suffix('.json'),{'identity':identity,'kind':role.kind,'sha256':sha(path),'learned_identity':learned_identity(role),'frozen_identity':frozen_identity(role),'tensors':tensor_manifest(state)})
def reload_learned(path,device='cuda',kind=None):
path=Path(path);metadata=read_json(path.with_suffix('.json'))
if sha(path)!=metadata['sha256']:raise ValueError('Safetensors hash mismatch')
role=build_role(kind or metadata['kind'],device=device);state=load_file(str(path),device='cpu');expected=learned_state(role)
if set(state)!=set(expected):raise ValueError('Learned tensor names mismatch')
with torch.no_grad():
for n,t in expected.items():
if t.shape!=state[n].shape or t.dtype!=state[n].dtype:raise ValueError('Learned tensor shape/dtype mismatch')
t.copy_(state[n])
if learned_identity(role)!=metadata['learned_identity'] or frozen_identity(role)!=metadata['frozen_identity']:raise ValueError('Reloaded role differs')
return role,metadata
Arrancar el actor y después aprender el juez
SFT y comparación Bradley-Terry
Invariante: puntuaciones RM reservadas, hashes de congelación y costes de las dos fases capturados.
Error que diagnosticar: el pooler en el último índice del batch lee PAD; usar la máscara de respuesta y probar longitudes diferentes.
Archivo src/recompense_ppo/warmstart.py
import math,time
from pathlib import Path
import torch
from .config import read_json,save_json,sha,seed_all,rng_state,restore_rng
from .models import build_role,load_tokenizer,learned_state,learned_identity,frozen_identity,save_learned,ADAPTER_PARAMETERS
from .scores import completion_row,model_scores,reward_scores
from .ppo_math import bradley_terry
from .state import save_resume_state,load_resume_state,equal
def learning_rate(config,step,total,peak):
warm=max(1,int(total*config['warmup_ratio']))
if step<warm:return peak*(step+1)/warm
return peak*(1-min(1.,(step-warm)/max(1,total-warm-1))*(1-config['minimum_lr_ratio']))
def assert_learning_scope():
for name in ['data/generated/input','data/generated/labels/gold.json','outputs/inference']:
if Path(name).exists():raise ValueError('Learning must not mount heldout inputs/gold/inference outputs: '+name)
def copy_learned(role,state):
expected=learned_state(role)
if set(expected)!=set(state):raise ValueError('Resume learned tensor identities differ')
with torch.no_grad():
for name,tensor in expected.items():
if tensor.shape!=state[name].shape or tensor.dtype!=state[name].dtype:raise ValueError('Resume tensor shape/dtype mismatch')
tensor.copy_(state[name])
def encoded_pairs(tokenizer,config,split):
return [{'document_id':r['document_id'],'chosen':completion_row(tokenizer,r['prompt'],r['chosen'],config,r['document_id']+'-chosen'),'rejected':completion_row(tokenizer,r['prompt'],r['rejected'],config,r['document_id']+'-rejected')} for r in read_json(f'data/generated/labels/{split}-pairs.json')]
def encoded_warm(tokenizer,config):
labels={r['document_id']:r['response'] for r in read_json('data/generated/labels/warmstart.json')}
return [completion_row(tokenizer,r['prompt'],labels[r['document_id']],config,r['document_id']) for r in read_json('data/generated/warmstart/pages.json')]
@torch.no_grad()
def validate_epoch(role,rows,branch):
result=[]
for row in rows:
if branch=='sft':
sums,counts,values,_,targets=model_scores(role,[row['chosen']])
result.append({'document_id':row['document_id'],'sum_logprob':float(sums[0]),'tokens':int(counts[0]),'token_logprobs':values.cpu().tolist(),'targets':targets.cpu().tolist()})
else:
chosen=reward_scores(role,[row['chosen']])[0];rejected=reward_scores(role,[row['rejected']])[0]
result.append({'document_id':row['document_id'],'chosen':float(chosen),'rejected':float(rejected),'correct':bool(chosen>rejected),'loss':float(bradley_terry(chosen[None],rejected[None]))})
return {'population':len(result),'rows':result,'used_for_selection':False,'fixed_final_epoch':True}
def train_warmstart(config,branch,mode):
if branch not in ['sft','reward'] or mode not in ['interrupt','continuous','resume']:raise ValueError('Unknown warmstart phase')
assert_learning_scope();seed_all(config['seed']);tokenizer=load_tokenizer();rows=encoded_warm(tokenizer,config) if branch=='sft' else encoded_pairs(tokenizer,config,'train');validation=encoded_pairs(tokenizer,config,'validation')
if len(rows)!=(168 if branch=='sft' else 840) or len(validation)!=180:raise ValueError('Frozen warmstart populations changed')
role=build_role('actor' if branch=='sft' else 'reward');parameters=[p for p in role.parameters() if p.requires_grad]
if sum(p.numel() for p in parameters)!=(ADAPTER_PARAMETERS+(897 if branch=='reward' else 0)) or any(p.dtype!=torch.float32 for p in parameters):raise ValueError('Unexpected learned parameters')
initial=learned_identity(role);base=frozen_identity(role);epochs=config['sft_epochs'] if branch=='sft' else config['reward_epochs'];peak=config['sft_learning_rate'] if branch=='sft' else config['reward_learning_rate'];total=math.ceil(len(rows)/config['accumulation_examples'])*epochs
names=['warmstart/pages.json','labels/warmstart.json','labels/validation-pairs.json'] if branch=='sft' else ['labels/train-pairs.json','labels/validation-pairs.json']
identity={'branch':branch,'config':config,'initial_learned':initial,'frozen_base':base,'model_sha256':sha('assets/model/model.safetensors'),'tokenizer_sha256':sha('assets/model/tokenizer.json'),'data_sha256':{n:sha('data/generated/'+n) for n in names},'code_sha256':{n:sha('src/recompense_ppo/'+n) for n in ['models.py','scores.py','ppo_math.py','warmstart.py','state.py']}}
optimizer=torch.optim.AdamW(parameters,lr=peak,weight_decay=config['weight_decay']);sampler=torch.Generator().manual_seed(149);root=Path('assets/checkpoints');destination=root/f'{branch}-{mode}.pt.gz'
if destination.exists():raise ValueError('Preserve existing checkpoint')
updates=first_epoch=0;history=[];validation_history=[]
if mode=='resume':
state=load_resume_state(root/f'{branch}-interrupt.pt.gz')
if state['identity']!=identity:raise ValueError('Warmstart resume identity changed')
copy_learned(role,state['learned']);optimizer.load_state_dict(state['optimizer']);sampler.set_state(state['sampler']);restore_rng(state['rng']);updates=state['updates'];first_epoch=state['epoch'];history=state['history'];validation_history=state['validation_history']
if state['scheduler']!={'next_update':updates,'next_lr':learning_rate(config,updates,total,peak)}:raise ValueError('Resume scheduler differs')
del state
stop=1 if mode=='interrupt' else epochs;torch.cuda.reset_peak_memory_stats();torch.cuda.synchronize();started=time.perf_counter()
for epoch in range(first_epoch,stop):
order=torch.randperm(len(rows),generator=sampler).tolist()
for offset in range(0,len(rows),config['accumulation_examples']):
batch=[rows[i] for i in order[offset:offset+config['accumulation_examples']]];optimizer.zero_grad(set_to_none=True);lr=learning_rate(config,updates,total,peak)
for group in optimizer.param_groups:group['lr']=lr
denominator=sum(r['response_tokens'] for r in batch) if branch=='sft' else len(batch);trace=[];loss_value=0.
for row in batch:
if branch=='sft':
sums,counts,values,_,targets=model_scores(role,[row]);loss=-sums[0]/denominator;trace.append({'document_id':row['document_id'],'tokens':int(counts[0]),'sum_logprob':float(sums[0].detach()),'token_logprobs':values.detach().cpu().tolist(),'targets':targets.cpu().tolist()})
else:
chosen=reward_scores(role,[row['chosen']]);rejected=reward_scores(role,[row['rejected']]);loss=bradley_terry(chosen,rejected)/denominator;trace.append({'document_id':row['document_id'],'chosen':float(chosen[0].detach()),'rejected':float(rejected[0].detach()),'chosen_tokens':row['chosen']['response_tokens'],'rejected_tokens':row['rejected']['response_tokens']})
loss.backward();loss_value+=float(loss.detach())
if any(p.grad is None or not torch.isfinite(p.grad).all() for p in parameters):raise ValueError('Missing/nonfinite learned gradients')
norm=float(torch.nn.utils.clip_grad_norm_(parameters,config['gradient_clip']));optimizer.step();updates+=1
history.append({'epoch':epoch+1,'update':updates,'lr':lr,'denominator':denominator,'denominator_kind':'active_response_tokens' if branch=='sft' else 'pairs','rows':trace,'loss':loss_value,'unclipped_gradient_norm':norm})
validation_history.append({'epoch':epoch+1,**validate_epoch(role,validation,branch)})
torch.cuda.synchronize();seconds=time.perf_counter()-started;final=learned_identity(role)
if frozen_identity(role)!=base:raise ValueError('Frozen base changed')
if final==initial:raise ValueError('Warmstart did not change learned parameters')
state={'identity':identity,'learned':{n:t.detach().cpu().clone() for n,t in learned_state(role).items()},'optimizer':optimizer.state_dict(),'sampler':sampler.get_state(),'rng':rng_state(),'epoch':stop,'updates':updates,'scheduler':{'next_update':updates,'next_lr':learning_rate(config,updates,total,peak)},'history':history,'validation_history':validation_history}
storage=save_resume_state(destination,state)
save_json(f'outputs/learning/{branch}/{mode}.json',{'identity':identity,'mode':mode,'epochs_completed':stop,'updates':updates,'history':history,'validation_history':validation_history,'initial_learned':initial,'final_learned':final,'frozen_base_unchanged':True,'trainable_parameters':sum(p.numel() for p in parameters),'seconds':seconds,'peak_allocated_bytes':torch.cuda.max_memory_allocated(),'peak_reserved_bytes':torch.cuda.max_memory_reserved(),'checkpoint_storage':storage})
if mode=='resume':save_learned(role,root/f'{branch}-resume.safetensors',identity)
return {'branch':branch,'mode':mode,'updates':updates,'seconds':seconds}
def check_warm_resume(branch):
root=Path('assets/checkpoints');continuous=load_resume_state(root/f'{branch}-continuous.pt.gz');resumed=load_resume_state(root/f'{branch}-resume.pt.gz')
if not equal(continuous,resumed):raise ValueError('Exact warmstart continuation failed: '+branch)
result={'status':'PASS','branch':branch,'all_state_exact':True,'updates':resumed['updates'],'continuous_sha256':sha(root/f'{branch}-continuous.pt.gz'),'resume_sha256':sha(root/f'{branch}-resume.pt.gz')};save_json(f'outputs/learning/{branch}/resume-check.json',result);return result
Codificar PPO y GAE sin caja negra
Crédito temporal y recorte
Invariante: GAE analítico coincidente, ref/RM inmutables, actor/crítico cambian; guardado completo.
Error que diagnosticar: hacer bootstrap más allá de un final tratado como terminal falsea GAE; probar done y el valor final.
Archivo src/recompense_ppo/train.py
import time
from pathlib import Path
import torch
from .config import read_json,save_json,sha,seed_all,rng_state,restore_rng
from .models import build_role,reload_learned,learned_state,learned_identity,frozen_identity,assert_role_isolation,load_tokenizer,save_learned
from .scores import rollout_completion_row,model_scores,critic_scores
from .rollout import sample_response,fixed_ppo_targets
from .ppo_math import ppo_actor_loss,critic_value_loss
from .warmstart import assert_learning_scope,learning_rate,copy_learned
from .state import save_resume_state,load_resume_state,equal
def next_prompts(pages,order,cursor,generator,count):
chosen=[]
while len(chosen)<count:
if cursor==len(order):order=torch.randperm(len(pages),generator=generator).tolist();cursor=0
chosen.append(pages[order[cursor]]);cursor+=1
return chosen,order,cursor
def initial_roles():
actor,_=reload_learned('assets/checkpoints/sft-resume.safetensors')
reference,_=reload_learned('assets/checkpoints/sft-resume.safetensors',kind='reference')
reward,_=reload_learned('assets/checkpoints/reward-resume.safetensors');reward.requires_grad_(False)
critic=build_role('critic',adapter_state=learned_state(actor))
roles={'actor':actor,'reference':reference,'reward':reward,'critic':critic}
return roles,assert_role_isolation(roles)
def train_ppo(config,mode):
if mode not in ['interrupt','continuous','resume']:raise ValueError('Unknown PPO branch')
assert_learning_scope();seed_all(config['seed']);tokenizer=load_tokenizer();pages=read_json('data/generated/train/pages.json')
if len(pages)!=672 or len({r['document_id'] for r in pages})!=672 or any(r['split']!='train' or r['population']!='rollout' for r in pages):raise ValueError('Training rollout population changed')
roles,initial=initial_roles();actor=roles['actor'];critic=roles['critic'];reference=roles['reference'];reward=roles['reward'];parameters={n:[p for p in roles[n].parameters() if p.requires_grad] for n in ['actor','critic']}
optimizers={n:torch.optim.AdamW(parameters[n],lr=config['ppo_'+n+'_lr'],weight_decay=config['weight_decay']) for n in parameters}
identity={'config':config,'roles_initial':initial,'model_sha256':sha('assets/model/model.safetensors'),'tokenizer_sha256':sha('assets/model/tokenizer.json'),'data_sha256':sha('data/generated/train/pages.json'),'sft_sha256':sha('assets/checkpoints/sft-resume.safetensors'),'reward_sha256':sha('assets/checkpoints/reward-resume.safetensors'),'code_sha256':{n:sha('src/recompense_ppo/'+n) for n in ['models.py','scores.py','rollout.py','ppo_math.py','warmstart.py','train.py','state.py']}}
sampler=torch.Generator().manual_seed(149);generation=torch.Generator(device='cuda').manual_seed(249);order=[];cursor=0;iteration=updates=0;history=[];consumed=[];total=config['rollout_iterations']*config['ppo_epochs_per_rollout'];root=Path('assets/checkpoints');destination=root/f'ppo-{mode}.pt.gz'
if destination.exists():raise ValueError('Preserve existing PPO checkpoint')
if mode=='resume':
state=load_resume_state(root/'ppo-interrupt.pt.gz')
if state['identity']!=identity or state['pending_rollouts']!=[]:raise ValueError('PPO identity or atomic boundary differs')
for n,role in roles.items():copy_learned(role,state['roles'][n])
for n,optimizer in optimizers.items():optimizer.load_state_dict(state['optimizers'][n])
sampler.set_state(state['sampler']);generation.set_state(state['generation']);restore_rng(state['rng']);order=state['order'];cursor=state['cursor'];iteration=state['iteration'];updates=state['updates'];history=state['history'];consumed=state['consumed_rollout_ids']
expected={n:learning_rate(config,updates,total,config['ppo_'+n+'_lr']) for n in optimizers}
if state['scheduler']!={'next_update':updates,'next_lr':expected}:raise ValueError('PPO scheduler differs')
del state
stop=config['interrupt_iteration'] if mode=='interrupt' else config['rollout_iterations'];phase={'generation_seconds':0.,'actor_optimization_seconds':0.,'critic_optimization_seconds':0.};torch.cuda.reset_peak_memory_stats();torch.cuda.synchronize();started=time.perf_counter()
while iteration<stop:
selected,order,cursor=next_prompts(pages,order,cursor,sampler,config['prompts_per_rollout']);policy_id=learned_identity(actor);rollouts=[]
for slot,page in enumerate(selected):
raw=sample_response(actor,tokenizer,page,config,generation,reference=reference,critic=critic,reward_model=reward,policy_id=policy_id,capture_probe=iteration==0 and slot<2)
raw['rollout_id']=f'iteration-{iteration+1:03d}-slot-{slot}';raw['targets']=fixed_ppo_targets(raw,config);rollouts.append(raw);phase['generation_seconds']+=raw['elapsed_seconds']
ids=[r['rollout_id'] for r in rollouts]
if len(set(ids))!=len(ids) or set(ids)&set(consumed):raise ValueError('Duplicate consumed rollout identity')
save_json(f'outputs/learning/rollouts/{mode}/iteration-{iteration+1:03d}.json',{'iteration':iteration+1,'policy_id':policy_id,'rollouts':rollouts})
pure_rollouts=[{k:v for k,v in r.items() if k not in ['elapsed_seconds','phase_seconds','first_prefix_probe']} for r in rollouts]
epoch_traces=[]
for epoch in range(config['ppo_epochs_per_rollout']):
learning_rates={n:learning_rate(config,updates,total,config['ppo_'+n+'_lr']) for n in optimizers}
for n,optimizer in optimizers.items():
optimizer.zero_grad(set_to_none=True)
for group in optimizer.param_groups:group['lr']=learning_rates[n]
observations=[]
for raw in rollouts:
row=rollout_completion_row(raw);old=torch.tensor([raw['old_token_logprobs']],dtype=torch.float32,device='cuda');advantages=torch.tensor([raw['targets']['advantages']],dtype=torch.float32,device='cuda');returns=torch.tensor([raw['targets']['returns']],dtype=torch.float32,device='cuda');mask=torch.ones_like(old,dtype=torch.bool)
torch.cuda.synchronize();start=time.perf_counter();_,_,current,logits,targets=model_scores(actor,[row]);current=current.unsqueeze(0);actor_loss=ppo_actor_loss(current,old,advantages,mask,epsilon=config['clip_epsilon']);(actor_loss/len(rollouts)).backward();torch.cuda.synchronize();phase['actor_optimization_seconds']+=time.perf_counter()-start
ratios=(current.detach()-old).exp();probe={'logits':logits[0].detach().cpu().tolist(),'target':int(targets[0])} if iteration==0 and epoch==0 and len(observations)<2 else None
actor_value=float(actor_loss.detach());current_values=current.detach().cpu().tolist()[0];ratio_values=ratios.cpu().tolist()[0]
del actor_loss,current,logits,targets
start=time.perf_counter();values,counts=critic_scores(critic,[row]);values=values.unsqueeze(0);critic_loss=critic_value_loss(values,returns,mask,coefficient=config['value_loss_coef']);(critic_loss/len(rollouts)).backward();torch.cuda.synchronize();phase['critic_optimization_seconds']+=time.perf_counter()-start
observations.append({'rollout_id':raw['rollout_id'],'current_token_logprobs':current_values,'ratios':ratio_values,'current_values':values.detach().cpu().tolist()[0],'actor_loss':actor_value,'critic_loss':float(critic_loss.detach()),'first_token_probe':probe,'active_tokens':raw['response_tokens'],'old_logprob_max_difference':max(abs(x-y) for x,y in zip(current_values,raw['old_token_logprobs']))})
del critic_loss,values
norms={}
for n,optimizer in optimizers.items():
if any(p.grad is None or not torch.isfinite(p.grad).all() for p in parameters[n]):raise ValueError('Missing/nonfinite '+n+' gradients')
norms[n]=float(torch.nn.utils.clip_grad_norm_(parameters[n],config['gradient_clip']));optimizer.step()
if any(p.grad is not None for role in [reference,reward] for p in role.parameters()):raise ValueError('Frozen judge/reference received gradients')
updates+=1;epoch_traces.append({'update':updates,'epoch_within_rollout':epoch+1,'learning_rates':learning_rates,'gradient_norms':norms,'equal_response_denominator':len(rollouts),'observations':observations})
consumed.extend(ids);iteration+=1
history.append({'iteration':iteration,'rollouts':pure_rollouts,'updates':epoch_traces})
torch.cuda.synchronize();seconds=time.perf_counter()-started;final={n:learned_identity(role) for n,role in roles.items()}
for n,role in roles.items():
if frozen_identity(role)!=initial[n]['frozen_identity']:raise ValueError('Frozen base changed: '+n)
for n in ['reference','reward']:
if final[n]!=initial[n]['learned_identity']:raise ValueError('Frozen reference/reward adapters or head changed')
for n in ['actor','critic']:
if final[n]==initial[n]['learned_identity'] or not any(u['gradient_norms'][n]>0 for h in history for u in h['updates']):raise ValueError('No actual '+n+' learning')
state={'identity':identity,'roles':{n:{k:t.detach().cpu().clone() for k,t in learned_state(role).items()} for n,role in roles.items()},'optimizers':{n:optimizer.state_dict() for n,optimizer in optimizers.items()},'sampler':sampler.get_state(),'generation':generation.get_state(),'rng':rng_state(),'order':order,'cursor':cursor,'iteration':iteration,'updates':updates,'history':history,'consumed_rollout_ids':consumed,'pending_rollouts':[],'scheduler':{'next_update':updates,'next_lr':{n:learning_rate(config,updates,total,config['ppo_'+n+'_lr']) for n in optimizers}}}
storage=save_resume_state(destination,state)
save_json(f'outputs/learning/ppo/{mode}.json',{'identity':identity,'mode':mode,'iterations':iteration,'updates_per_optimizer':updates,'history':history,'consumed_rollout_ids':consumed,'final_learned':final,'frozen_reference_reward_unchanged':True,'seconds':seconds,'phase_seconds':phase,'peak_allocated_bytes':torch.cuda.max_memory_allocated(),'peak_reserved_bytes':torch.cuda.max_memory_reserved(),'checkpoint_storage':storage,'numerical_scope':'Old sampling-prefix logprobs preserved; current teacher-forced BF16 scores may differ numerically before first update. Per-token differences are reported, never substituted for old values.'})
if mode=='resume':
save_learned(actor,root/'ppo-resume.safetensors',identity);save_learned(critic,root/'critic-resume.safetensors',identity)
return {'mode':mode,'iterations':iteration,'updates_per_optimizer':updates,'seconds':seconds}
def check_ppo_resume():
root=Path('assets/checkpoints');continuous=load_resume_state(root/'ppo-continuous.pt.gz');resumed=load_resume_state(root/'ppo-resume.pt.gz')
if not equal(continuous,resumed):raise ValueError('Exact four-role PPO continuation failed')
result={'status':'PASS','all_state_exact':True,'iterations':resumed['iteration'],'updates_per_optimizer':resumed['updates'],'rollout_ids':len(resumed['consumed_rollout_ids']),'continuous_sha256':sha(root/'ppo-continuous.pt.gz'),'resume_sha256':sha(root/'ppo-resume.pt.gz')};save_json('outputs/learning/ppo/resume-check.json',result);return result
Evaluar frente al juez y frente al objetivo
Desacuerdo y explotación del proxy
Invariante: todos los casos tienen dos evaluaciones distintas; los malos consejos siguen visibles.
Error que diagnosticar: llamar a la puntuación RM «utilidad humana» es incorrecto; llamarla puntuación predicha de preferencia simulada.
Archivo src/recompense_ppo/evaluate.py
import math,time
from pathlib import Path
import torch
from .config import read_json,save_json,sha
from .models import reload_learned,load_tokenizer,learned_identity
from .scores import completion_row,reward_scores
from .rollout import sample_response
from .rubric import evaluate_response
@torch.no_grad()
def generate_heldout(config,branch):
if branch not in ['sft','ppo']:raise ValueError('Named actor branch required')
if Path('data/generated/labels').exists() or Path('outputs/learning').exists():raise ValueError('Free generation must not mount labels or learning outputs')
pages=[r for r in read_json('data/generated/input/pages.json') if r['split'] in ['test','probe']]
if len(pages)!=260 or len({r['prompt'] for r in pages})!=200:raise ValueError('Heldout row/context populations changed')
actor,actor_meta=reload_learned(f'assets/checkpoints/{branch}-resume.safetensors');actor.requires_grad_(False)
reference,ref_meta=reload_learned('assets/checkpoints/sft-resume.safetensors',kind='reference')
reward,reward_meta=reload_learned('assets/checkpoints/reward-resume.safetensors');reward.requires_grad_(False)
tokenizer=load_tokenizer();policy_id=learned_identity(actor);generator=torch.Generator(device='cuda').manual_seed(349);rows=[];torch.cuda.reset_peak_memory_stats();torch.cuda.synchronize();start=time.perf_counter()
for index,page in enumerate(pages):
raw=sample_response(actor,tokenizer,page,config,generator,reference=reference,reward_model=reward,mode='greedy',policy_id=policy_id,capture_probe=index<2)
rows.append({'page':page,'generation':raw})
byprompt={}
for row in rows:
raw=row['generation'];key=row['page']['prompt'];value={k:raw[k] for k in ['response_ids','text','finish_reason','learned_reward_score','old_token_logprobs','reference_token_logprobs']}
if key in byprompt and byprompt[key]!=value:raise ValueError('Repeated greedy probe contexts differ')
byprompt[key]=value
torch.cuda.synchronize();result={'branch':branch,'rows':rows,'row_count':260,'unique_contexts':200,'repeated_probe_contexts_verified':True,'labels_visible':False,'actor_checkpoint':actor_meta['sha256'],'reference_checkpoint':ref_meta['sha256'],'reward_checkpoint':reward_meta['sha256'],'seconds':time.perf_counter()-start,'peak_allocated_bytes':torch.cuda.max_memory_allocated(),'peak_reserved_bytes':torch.cuda.max_memory_reserved()};save_json(f'outputs/inference/{branch}.json',result)
return {k:v for k,v in result.items() if k!='rows'}
@torch.no_grad()
def evaluate_reward_pairs(config):
pages={r['document_id']:r for r in read_json('data/generated/input/pages.json')};gold=read_json('data/generated/labels/gold.json');selected=[r for r in gold if pages[r['document_id']]['split']=='test'];tokenizer=load_tokenizer();reward,meta=reload_learned('assets/checkpoints/reward-resume.safetensors');reward.requires_grad_(False);rows=[]
if len(selected)!=180:raise ValueError('Reward test population changed')
torch.cuda.synchronize();start=time.perf_counter()
for label in selected:
page=pages[label['document_id']];chosen=completion_row(tokenizer,page['prompt'],label['chosen'],config,page['document_id']+'-chosen');rejected=completion_row(tokenizer,page['prompt'],label['rejected'],config,page['document_id']+'-rejected');a=float(reward_scores(reward,[chosen])[0]);b=float(reward_scores(reward,[rejected])[0]);rows.append({'document_id':page['document_id'],'chosen_score':a,'rejected_score':b,'chosen_response':label['chosen'],'rejected_response':label['rejected'],'chosen_tokens':chosen['input_ids'],'rejected_tokens':rejected['input_ids'],'correct':a>b,'tie':a==b})
torch.cuda.synchronize();result={'pairs':180,'correct':sum(r['correct'] for r in rows),'ties':sum(r['tie'] for r in rows),'accuracy':sum(r['correct'] for r in rows)/180,'rows':rows,'reward_checkpoint':meta['sha256'],'used_for_selection':False,'seconds':time.perf_counter()-start};save_json('outputs/reward-test.json',result);return {k:v for k,v in result.items() if k!='rows'}
def summarize(rows):
recognized=[r for r in rows if r['rubric']['recognized']]
return {'rows':len(rows),'rubric_success':sum(r['rubric']['rubric_success'] for r in rows),'rubric_success_rate':sum(r['rubric']['rubric_success'] for r in rows)/len(rows),'recognized':len(recognized),'recognized_rate':len(recognized)/len(rows),'uncertified':len(rows)-len(recognized),'relevant_recognized':sum(r['rubric']['relevant'] for r in recognized),'unsupported_certainty_recognized':sum(r['rubric']['unsupported_certainty'] is True for r in recognized),'declared_refusals':sum(r['rubric']['refusal'] for r in rows),'without_eos':sum(r['finish_reason']!='eos' for r in rows),'mean_response_tokens':sum(r['response_tokens'] for r in rows)/len(rows),'mean_learned_reward':sum(r['learned_reward_score'] for r in rows)/len(rows),'mean_sampled_actor_reference_logratio':sum(r['sampled_actor_reference_logratio'] for r in rows)/len(rows),'uncertainty_denominator':len(recognized),'human_utility_claim':False}
def evaluate(config):
gold={r['document_id']:r for r in read_json('data/generated/labels/gold.json')};result={}
for branch in ['sft','ppo']:
raw=read_json(f'outputs/inference/{branch}.json');rows=[]
for item in raw['rows']:
g=item['generation'];page=item['page'];label=gold[page['document_id']];rubric=evaluate_response(g['text'],g['finish_reason'],label['expected_action'])
if not g['tokenization_valid']:rubric={**rubric,'recognized':False,'status':'UNCERTIFIED','rubric_success':False,'relevant':False,'unsupported_certainty':None,'reason':'undecodable_token','refusal':False,'polite':False,'refusal':False,'polite':False}
ratio=sum(a-b for a,b in zip(g['old_token_logprobs'],g['reference_token_logprobs']))/g['response_tokens']
rows.append({'document_id':page['document_id'],'family':page['family'],'split':page['split'],'prompt':page['prompt'],'text':g['text'],'finish_reason':g['finish_reason'],'response_tokens':g['response_tokens'],'learned_reward_score':g['learned_reward_score'],'sampled_actor_reference_logratio':ratio,'rubric':rubric})
unique={r['prompt']:r for r in rows};probe=[r for r in rows if r['split']=='probe'];unique_probe={r['prompt']:r for r in probe}
populations={'test':[r for r in rows if r['split']=='test'],'probe_rows':probe,'probe_unique_contexts':list(unique_probe.values()),'all_rows':rows,'all_unique_contexts':list(unique.values())}
result[branch]={'metrics':{name:summarize(selected) for name,selected in populations.items()},'rows':rows,'generation_seconds':raw['seconds'],'raw_sha256':sha(f'outputs/inference/{branch}.json')}
output={'actor':result,'reward_test':read_json('outputs/reward-test.json'),'scope':'Synthetic finite grammar; UNCERTIFIED prose is not silently accepted. 80 probe rows contain20unique contexts; duplicate-adjusted summaries are separate. Learned reward is not the rubric, no human study.'};save_json('outputs/results.json',output);return {branch:v['metrics'] for branch,v in result.items()}
@torch.no_grad()
def exercise(config):
protected_paths=[Path('configs/minimum.yaml'),Path('outputs/results.json'),Path('outputs/reward-test.json')]+list(Path('outputs/inference').glob('*.json'))+list(Path('assets/checkpoints').glob('*.safetensors'))
protected={str(p):sha(p) for p in protected_paths};pages={r['document_id']:r for r in read_json('data/generated/input/pages.json')};labels=[r for r in read_json('data/generated/labels/gold.json') if pages[r['document_id']]['split']=='probe'];tokenizer=load_tokenizer();reward,metadata=reload_learned('assets/checkpoints/reward-resume.safetensors');reward.requires_grad_(False);before=learned_identity(reward);rows=[];torch.cuda.synchronize();start=time.perf_counter()
if len(labels)!=80:raise ValueError('Expected80preserved factorial probes')
for label in labels:
page=pages[label['document_id']];row=completion_row(tokenizer,page['prompt'],label['candidate_response'],config,page['document_id']);score=float(reward_scores(reward,[row])[0]);rows.append({'document_id':page['document_id'],'family':page['family'],'prompt':page['prompt'],'candidate':label['candidate_response'],'candidate_correct':label['candidate_correct'],'candidate_polite':label['candidate_polite'],'input_ids':row['input_ids'],'response_tokens':row['response_tokens'],'reward_score':score,'rubric':evaluate_response(label['candidate_response'],'eos',label['expected_action'])})
groups=[]
for family in sorted({r['family'] for r in rows}):
group={(r['candidate_correct'],r['candidate_polite']):r['reward_score'] for r in rows if r['family']==family}
if len(group)!=4:raise ValueError('Incomplete factorial probe')
groups.append({'family':family,'correct_politeness_delta':group[True,True]-group[True,False],'incorrect_politeness_delta':group[False,True]-group[False,False],'incorrect_polite_above_correct_plain':group[False,True]>group[True,False]})
torch.cuda.synchronize()
if learned_identity(reward)!=before or any(sha(Path(p))!=value for p,value in protected.items()):raise ValueError('Exercise changed judge/main artifacts')
result={'probe_rows':80,'unique_contexts':20,'rows':rows,'groups':groups,'wrong_polite_outranks_correct_plain':sum(g['incorrect_polite_above_correct_plain'] for g in groups),'mean_incorrect_politeness_delta':sum(g['incorrect_politeness_delta'] for g in groups)/20,'seconds':time.perf_counter()-start,'reward_checkpoint':metadata['sha256'],'reward_unchanged':True,'main_unchanged':True,'protected_sha256':protected,'refit':False};save_json('outputs/exercise/reward-shortcuts.json',result);return {k:v for k,v in result.items() if k not in ['rows','groups','protected_sha256']}
Probar el atajo de la cortesía
Reward hacking observable
Invariante: 80 probes evaluadas sin actualización de RM y sin modificar sus etiquetas.
Error que diagnosticar: reentrenar RM inmediatamente con estas probes y anunciar de nuevo la puntuación inicial destruye la separación; versionar todo estudio nuevo.
Archivo src/recompense_ppo/exercise.py
"""Fixed factorial probe analysis; no further learning or relabeling."""
from .evaluate import exercise
__all__ = ["exercise"]
Fuentes
PPO, GAE, InstructGPT y código TRL fijado. Estas fuentes explican los mecanismos; nuestra variante explícita no reproduce un estudio humano ni el trainer TRL completo.
