模型訓練基礎設施
Marin Launches Open Training of a 535B-A23B Model, Publicly Trackable Across 11 GB200 NVL72 Systems
Marin has begun training a mixture-of-experts (MoE) model with 535B total parameters and approximately 23B active parameters per token. The project plans to use 11 GB200 NVL72 systems to process roughly 18T to 18.75T tokens. Configurations, scaling experiments, and real-time telemetry are public, but model quality, the long-context strategy, and the final token budget have yet to be finalized.
The Marin community has launched its largest open training experiment to date: a mixture-of-experts model with approximately 535.3B total parameters and 22.76B active parameters. The public plan calls for using 11 GB200 NVL72 systems over roughly three months. The announcement specifies a data run of 18.75T tokens, requiring approximately 2.7×10²⁴ FLOPs. Unlike projects that publish only the final weights, Marin is also making its configurations, data composition, scaling predictions, and Weights & Biases telemetry publicly available. [Training tracking issue](https://github.com/marin-community/marin/issues/8435)
The model contains 48 Transformer blocks. Each layer has 384 routed experts with top-8 routing, plus two shared SwiGLU experts. LatentMoE compresses the 6,144-dimensional token representations to 3,072 dimensions before expert dispatch, reducing all-to-all communication. 64-way expert parallelism operates within each 72-GPU rack, while the 11 racks serve as data-parallel replicas. A capacity factor of 1.15 and three static expert waves are used to constrain communication and compute buffers, at the cost of potentially dropping tokens when the load is imbalanced. [Technical overview and configuration links](https://www.ai-primer.com/engineer/stories/marin-535b-training)
Before the main training run, the team established a four-stage scaling ladder, ranging from a 1.6B-A61M model trained on 48B tokens to a 27.7B-A1.2B model trained on 926B tokens. These preliminary runs consume about 1% of the main experiment’s compute and are used to predict loss, gradient norms, and evaluation trajectories. An earlier experiment used this process to identify gradient growth at large batch sizes, prompting the addition of logit z-loss. If the main model diverges from the predicted curves, maintainers can investigate the data, routing, or core implementation before consuming the full three-month compute allocation.
The main risks remain long-context training and stability at scale. The model is initially trained with 4K sequences because a larger number of sequences improves expert balancing. However, existing tests show token dropping rising from about 7% to approximately 40% when the context is extended to 65K. The team is considering dropless ragged all-to-all, a higher capacity factor, or sequence-level balancing, and plans to experiment with 65K and 262K contexts only during later training stages. Public materials also cite both 18.0T and 18.75T tokens, so the project should currently be viewed as a reproducible training experiment with an open architecture and telemetry—not as a completed model release with established capability results.