AI 安全與預訓練
SPP Instills Value Reflection From the First Token, Widening the 3B Model’s OOD Risk Gap to 19 Points
Synthetic Persona Pretraining embeds first-person reflections generated from a “values constitution” into roughly 10% of pretraining documents, then uses post-training to bind that persona to the assistant identity. Newly released full experiments, weights, and data show that early intervention mainly improves value generalization, while resisting conventional jailbreaks may not require training from scratch.
EPFL and other institutions have officially released the [Synthetic Persona Pretraining (SPP) paper](https://arxiv.org/abs/2608.13482) and the complete experimental artifacts, expanding May’s preliminary 1.7B results to 3B parameters and 500B tokens. The method first uses a constitution containing 35 rules covering dignity, safety, integrity, privacy, governance, and other principles to generate first-person value reflections for roughly 10% of pretraining documents. The reflections are placed after a special `<assistant>` token and inserted at different positions within each document, while the model continues to learn both the original text and the reflections using standard cross-entropy loss.
The study compares five data-matched model groups, including an unmodified corpus, removal of harmful documents, reflections added only near the end of training, and reflections introduced from the first token. All models receive the same persona-binding SFT. At the 3B scale, intervention from the first token increases the advantage over midtraining-only models on AIRisk moral dilemmas from roughly 4 points in the smaller-scale experiment to 19 points. The advantage on ConstitutionEval-Hard similarly grows from about 7 points to 14 points. Every SPP variant has a lower jailbreak success rate than the baseline, but the version that adds reflections only near the end of training is also highly effective, suggesting that refusal mechanisms and deep-seated value preferences may be two distinct problems.
The engineering value lies in reproducibility: [GitHub](https://github.com/epfl-dlab/spp) now provides the data-generation, Megatron-LM training, and evaluation pipelines, while [Hugging Face](https://huggingface.co/dlab-spp) hosts five model groups at two scales—including base, instruct, and intermediate checkpoints—as well as approximately 51.4 million reflection examples. However, the results currently extend only to 3B parameters, and the values constitution, synthetic annotator, and model judges could all introduce correlated bias. Moreover, the gains shrink substantially when the post-training format fails to activate the pretrained persona. The next step is to determine whether the effect transfers across different chat templates, RL training, and adversarial fine-tuning, and whether it persists in larger models.