AI 研究
K2-Horizon Technical Report Details MoVA Attention and Uno Decoding Acceleration
The IFM team submitted the K2-Horizon technical report on November 16, describing its attention sparsification, synthetic data, and decoding acceleration designs. The model family launched in September; this report focuses on a complete technical methodology and test conditions that researchers can examine.
The Institute of Foundation Models (IFM) at MBZUAI submitted the K2-Horizon technical report to arXiv on November 16. It covers the architecture, training, and inference methods for six language models. This is a research-document update for an existing model family: the university announced the release of models ranging from 0.9B to 375B-A23B on September 3, and provided weights, code, and training data. Paper submission record, original launch announcement
The report’s architectural focus is Mixture-of-Value Attention (MoVA), which introduces expert routing to attention’s value projections and selects a subset of projection computations for each token. It preserves the attention computation itself, so it can be combined with methods such as FlashAttention and GQA. The authors report that the 36B-A4B model’s pretraining loss is close to that of a 32B dense model trained on the same data and with the same settings, though the dense model retains a slight advantage. Architecture details
For inference, the report introduces the Uno diffusion adapter, which accelerates generation through block drafting and verification without requiring a separate draft model. In the specified 1K/8K throughput tests, the authors measured the 7B version as approximately 2.2× faster at batch size one and 1.5× faster at batch size 64. These results come from the team’s tests and cannot be directly extrapolated to end-to-end acceleration on arbitrary hardware or agent tasks. Uno test results
For deployers, the model card provides specific integration requirements: 36B-A4B supports a 524,288-token context, and the SGLang recipe was validated on two H200 GPUs. It also requires configuring dedicated inference and tool-call parsers. The model card cautions that the Hugging Face implementation may behave differently during training from native xLLM, including in its auxiliary load-balancing loss. Continued pretraining from the original run should use the native checkpoint and runtime environment. Official model card
The next engineering step is to retest throughput and task success rates with real prompt lengths, concurrency levels, and toolchains. Researchers can use intermediate checkpoints to track capability changes and test whether the architectural gains reproduce under their own evaluation settings. Checkpoint and usage recommendations