llama.cpp 0.3.0 Adds a Hybrid-Attention KV Cache, MTP, and Tensor Splitting for DeepSeek 4

llama.cpp has released version 0.3.0, focusing not on a single new model but on enabling three inference paths that are relatively difficult to implement. The first is dots3-note, a 288B multimodal MoE model that combines 13 layers of dynamic sparse attention (DSA) with 33 layers of sliding-window attention (SWA). DSA selects up to 2,048 positions per operation, and the official maximum context length is 512K. The new release introduces a dedicated DSA-ISWA KV cache for this interleaved architecture. `mtmd` can also process the model’s image, video, and audio inputs. Fixes were additionally made for WebP decoding, `moov` atoms located at the end of video files, and the resizing algorithm to align it with Pillow.

The second addition is MTP (multi-token prediction) support for GLM-4.5-Air. MTP uses additional prediction heads to propose multiple subsequent tokens at once, which are then verified by the main model, allowing it to serve as a draft source for speculative decoding. This eliminates the need to load a separate, complete small draft model. However, the actual acceptance rate and speedup still depend on the prompt, batch size, quantization format, and whether the MTP weights are complete. Because the 0.3.0 release page publishes no performance figures, a fixed multiplier improvement cannot be claimed.

The third addition is `-sm tensor` support for DeepSeek 4. The commonly used layer split approach assigns complete layers to different devices, whereas tensor splitting distributes the weights for an individual operation across multiple GPUs. This can accommodate models that cannot be evenly divided by whole layers while balancing VRAM usage. The tradeoff is the need for cross-device reductions and synchronization, meaning PCIe, RDMA, or backend communication efficiency may become a bottleneck. The new release also fixes multi-sequence rollback and the propagation of split state in the meta-backend, indicating that this path previously still carried correctness risks.

The underlying ggml library has also been updated to version 0.22.0, adding changes such as meta-backend tensor splitting, per-operation Metal kernels, and parallel compilation. For local deployment, this release expands the range of workloads that can run with emerging hybrid-attention architectures, MTP, and massive MoE models. The next step should still be to compare time to first token, generation throughput, KV cache capacity, and multi-GPU communication costs on actual hardware—not merely confirm that a model can be loaded.