模型訓練
Post-Norm Beats Pre-Norm During Progressive Transformer Depth Growth, Cutting CE by 0.0328 in a 9-Layer Student Model
A Qwen3-8B distillation experiment found almost no difference between the two normalization placements when training the full depth jointly, but Post-Norm clearly outperformed after Transformer blocks were added in stages. The results show that normalization placement should not be chosen independently of the model growth strategy.
Modern decoders typically use Pre-Norm, applying normalization before attention or the MLP to improve gradient stability when initializing a full-depth model. However, new research submitted on August 13 suggests that this convention may need to be reconsidered when a model is not trained with all layers at once, but instead grows from shallow to deep by progressively adding blocks.
The researchers used the 36-layer Qwen3-8B-Base as a frozen teacher and built a student model with nine decoder layers and approximately 3.09 billion parameters. When all nine layers were trained jointly, the validation cross-entropy difference between Pre-Norm and Post-Norm was only 0.0004. Under a three-stage depth-growing curriculum, however, Post-Norm achieved a CE 0.0328 lower than Pre-Norm. Fixed-depth control runs with additional training tokens, matched active-layer token counts, and learning-rate restarts still ended with a CE above the Post-Norm growing model’s 2.7330, indicating that the difference cannot be explained solely by additional compute.
Diagnostic results suggest a possible mechanism: a newly added block receives a residual stream produced by an already-trained prefix. Pre-Norm does not directly constrain the scale of outputs across blocks, and the standard deviation of its per-token RMS drifted from 5.81 to 19.98; Post-Norm kept it at approximately 0.07 to 0.19. The drift was especially concentrated around structural tokens such as document boundaries. Removing the final block from the growing Pre-Norm model changed CE by only 0.02, suggesting that the layer was close to an identity mapping. Removing any block from the growing Post-Norm model increased CE by at least 2.38.
This has direct implications for progressive pretraining, model depth expansion, and distillation schedules: architecture search should treat normalization placement and growth schedule as joint hyperparameters. However, the evidence remains limited to a single Qwen teacher, a nine-layer student, a fixed 3.11-billion-token data shard, and a specific distillation setup. The authors also provide only correlational diagnostics and have not yet directly intervened on residual scale. The next step is to reproduce this reversal in deeper models, under different data distributions, and in pretraining from scratch.