推論系統
llama.cpp 0.4.0 Reduces Large-Model Loading Spikes with Lazy Tensor Loading and Adds Support for the New Qwen3.8 Architecture
The new release adds `--lazy-mode`, a quantization working-memory limit, and row-slab streaming, eliminating the need to load extremely large tensors into RAM all at once. It also introduces preliminary support for Qwen3.8-Flash-Next and Nemotron-3-Puzzle, though performance optimizations and compatibility with some hardware remain incomplete.
The focus of llama.cpp 0.4.0 is not merely adding more model names, but reworking how data is moved for large, heterogeneous models. The new `--lazy-mode` loads tensors on demand, preventing all weights from being placed in physical memory immediately at model startup. The quantizer also adds `max_buf_size` and row-slab streaming to limit the temporary memory required when processing a single enormous tensor. This is particularly important for Qwen3.8-Flash-Next: in addition to its 125B-parameter MoE backbone, which activates roughly 6B parameters per token, the model includes an n-gram embedding with approximately 51B parameters. Relying solely on conventional full loading can therefore cause significant RAM spikes.
At the model level, the release adds `qwen4exp`, Nemotron-3-Puzzle-75B-A9B, Nemotron 3.5 DSpark, and nanbeige4.2-3B. The inference core gains DFlash2, n-gram history lookup, per-layer expert routing, and optimizations for KV cache restoration and sequence scanning. The server can use `--kv-unified-per-slot` to set context quotas for individual request slots, reducing the risk that a single long conversation will monopolize shared KV capacity. The multimodal interface also expands support for video input and adds a new tokenization parts API.
Engineering teams should note before upgrading that the session/state format has changed due to KV-cell token tracking, so previously saved states may not remain directly compatible. The project also explicitly labels Qwen3.8 support as “preliminary,” with optimization work still unfinished. The community has reported slowdowns or abnormal output on some AMD HIP devices during long-context, multi-turn generation. Lazy loading can reduce resident memory usage, but it may shift the bottleneck to storage latency and the page cache. Performance must therefore be measured using the actual GGUF file, SSD, backend, and context length; being able to load a model should not be treated as equivalent to running it efficiently.