模型訓練
Puro-2B Publishes a Consumer-Grade GPU Pretraining Recipe, With Full 2B Model Training Costing About $6,900
A Tsinghua University team trained a 2B-parameter model from scratch using RTX 5090 GPUs, blockwise FP8, and MuonH, and released the weights, data, and code. A lower-budget checkpoint outperformed Qwen2-1.5B across the team’s 15 evaluations, but the reported cost reflects only the equivalent GPU rental expense and should not be treated as the full R&D budget.
Tsinghua University’s PACMAN team has released Puro-2B: a family of 2B-parameter dense language models trained from random initialization on consumer-grade RTX 5090 GPUs, along with an auditable end-to-end pretraining stack. The model uses an architecture configuration compatible with Qwen3-1.7B but does not inherit any Qwen weights. The final version was trained on approximately 1.4 trillion tokens and uses a 28-layer Transformer with a hidden dimension of 2,048, a 4,096-token context window, and untied input and output embeddings.
The cost reduction does not come from a single technique. Training uses blockwise FP8 to reduce compute and memory requirements, together with the MuonH optimizer featuring a hyperball constraint. The experiments also bring standard data ordering, proxy-model-guided data selection, late-stage curriculum ordering, and the averaging of six checkpoints into a single framework. The team reports that the final model consumed 22,514 effective GPU-hours, equivalent to approximately $6,891 at $0.31 per RTX 5090 per hour. Another checkpoint, which completed only half of the second training stage, cost about $4,370 and achieved a higher average score than Qwen2-1.5B across the team’s 15 base-model evaluations. The final model is described as approaching Qwen2.5-1.5B performance.
For small research teams, the reproducibility beyond the weights is what matters most. Puro-Megatron preserves data packing, resumable stage transitions, FP8 checkpoints, distributed optimizer states, and corrupted-data handling. The data-processing code and multiple intermediate checkpoints have also been released. This allows researchers to directly compare data curricula and optimization strategies instead of merely fine-tuning existing models.
However, the reported cost excludes engineering, labor, storage, networking, failed experiments, and post-training. The dataset also uses mixed licensing because of its upstream sources. Puro-2B-Base is an unaligned base model and cannot be used directly as a chat assistant. The next question is whether external teams can reproduce its throughput, cost, and evaluation results under the same hardware and data conditions.