開放模型
AMD Opens the Full Instella-MoE Training Pipeline, With the 16B Model Activating Only 2.8B Parameters per Token
AMD has released six Instella-MoE checkpoints spanning pretraining, mid-training, long-context extension, SFT, DPO, and reinforcement learning, along with the training frameworks, data recipes, and inference code. The model uses Gated MLA and FarSkip-Collective to reduce attention and cross-GPU communication costs, but a formal technical report and independent evaluations are still pending.
AMD has updated the Instella project and released the complete Instella-MoE-16B-A3B model pipeline. It is a sparse Mixture-of-Experts model with 16 billion total parameters and approximately 2.8 billion parameters activated per token. It has 64 experts and two shared experts, with the router selecting six experts at a time. The model uses 27 decoder layers, a hidden-state dimension of 2,048, and a vocabulary of 128,896 tokens. Its weights are currently released as BF16 Safetensors.
What stands out in this release is not just the final chat model. AMD is also providing six checkpoints covering pretraining, mid-training, long-context extension, supervised fine-tuning, DPO, and reinforcement learning. This allows developers to compare how capabilities emerge at each stage or redesign post-training from an earlier base model. AMD also says it has released the data mixture, training recipes, the ROCm-based Primus training framework, and the Miles reinforcement learning framework. Compared with releases that provide only final weights, this is closer to a research-ready, fully open model.
Architecturally, Instella-MoE uses Gated Multi-head Latent Attention to compress attention states and control outputs through gating. FarSkip-Collective allows more distant compute tasks to bypass blocked collective communication, aiming to overlap GPU computation with the all-to-all wait times common in MoE systems. The model was trained from scratch on MI300X and MI325X accelerators. Its model card also lists loading paths for Transformers, vLLM, and SGLang, so it is not tied to AMD's own serving interfaces.
However, the currently available benchmark results are primarily official comparisons presented as images, while the formal Instella-MoE technical report is still marked as forthcoming. Multilingual capabilities have not yet been evaluated either. Engineering teams should first verify `trust_remote_code` requirements, memory requirements, and compatibility with non-ROCm platforms, then monitor whether AMD provides a complete training-data inventory, router load-distribution details, the supported long-context length, and fully reproducible scripts.