llama.cpp b10448 Adds Native Kimi K3 Support, Losslessly Reorders MXFP4 to Avoid a 5.5 TB BF16 Intermediate

llama.cpp b10448 adds support for the Kimi K3 text model. The change goes beyond recognizing a new architecture name: it integrates the model’s specialized inference paths into the GGML graph. K3 interleaves linear KDA with full MLA attention and also features cross-layer residual attention, 896 latent experts with 16 selected per token, a new non-SiLU activation, an MLA output gate, and a different form of KDA decay gate. Because the model’s internal `text_config` still reports the older Kimi Linear architecture, llama.cpp must route by the top-level name instead; otherwise, it applies the wrong model definition.

The key deployment change is quantized-weight conversion. The official checkpoint uses compressed-tensors’ `mxfp4-pack-quantized` format. Its numeric encoding is identical to GGML MXFP4, differing only in the nibble order within each block. The new release therefore performs a direct bit-level reordering, which the author verified produces zero dequantization error, rather than first expanding the weights into roughly 5.5 TB of BF16 data and then requantizing them. The reordering is also performed lazily, preventing the GGUF writer from retaining all converted tensors at once before output.

The chat layer now also supports parsing K3’s XTML-style reasoning, responses, and typed tool calls. It was validated with the full model on eight B200 GPUs, including streamed content, reasoning fields, and stop reasons. One limitation is that the cross-layer residual operator currently has kernels only for CPU and CUDA; Metal and Vulkan fall back node by node, so “loadable” should not be mistaken for good performance across all backends. Areas to watch next include the actual size of community GGUF builds, KDA state-memory requirements, and how quickly non-CUDA kernels are implemented.