模型訓練系統
Puro-2B Publishes RTX 5090 Pretraining Recipe, With a Reported Cost of $6,891 for 1.4T Tokens
Puro-2B integrates a two-stage data curriculum, blockwise FP8, and MuonH optimization on a consumer-grade GPU cluster. The authors report that a shortened $4,370 training run already surpasses Qwen2-1.5B in average score, although the cost accounts only for accelerator rental-equivalent value.
Tsinghua University’s PACMAN team and collaborators have released Puro-2B, aiming to turn “fully reproducing pretraining” from a large-lab project into an experiment affordable for typical research teams. This is more than a weights release: the Hugging Face collection also provides the model, data, training recipe, and checkpoints at different token budgets under the Apache 2.0 license, enabling researchers to examine how data ordering, optimizers, and low-precision computation jointly affect results.
The model is a dense, decoder-only Transformer with approximately 2 billion parameters. It follows the Qwen3-1.7B configuration but unties the input embedding and output head. Training runs on an RTX 5090 cluster, with matrix multiplications in the main linear layers using blockwise FP8, while master weights, optimizer states, and numerically sensitive operations remain in BF16 or FP32. The system is implemented with Megatron Core and combines MuonH, a learning-rate design, and curriculum model averaging.
In the standard pipeline, 24 GPUs first process 438.84B tokens, after which the cluster scales to 96 GPUs. For the full version, the second stage adds approximately 960B tokens, bringing the total to nearly 1.4T tokens and consuming 22,514 effective GPU-hours. Under the paper’s rental-equivalent calculation, this amounts to $6,891. The shorter uniform-data version processes 918.84B tokens in total at a cost of $4,370. Across 15 mathematics, coding, reasoning, and knowledge benchmarks, the authors report an average score of 57.81 for the full model, above Qwen2-1.5B’s 55.14 but still below Qwen2.5-1.5B’s 60.73. Their fitted cost-scaling curve also indicates that roughly $4,400 is sufficient to surpass the earlier baseline.
What truly deserves attention is the controllable, end-to-end research platform: the team can vary the data curriculum within the same architecture and then track downstream post-training effects, without having to infer causality from weights produced by different vendors. However, “under $7,000” is not the total cost of ownership for building a physical cluster, nor can it be directly extrapolated to larger models. Networking, host systems, storage, electricity, reruns after failures, and labor could still add substantially to the budget. External teams should next reproduce the scaled-down recipe and verify the RTX 5090’s long-duration stability, FP8 numerical error, and the complete provenance of the training data.