推論系統
Unsloth 0.1.806 Enables MTP Speculative Decoding by Default, Making Local Inference with Qwen3.8 and GLM-5.3 Up to 2× Faster
Unsloth now enables multi-token prediction by default for Qwen3.8-Flash-Next and GLM-5.3-Flash, while improving its MLX, multi-GPU, and ROCm paths. The official claim of up to 2× higher speed is not accompanied by a complete benchmark matrix, while Qwen’s technical documentation reports gains of 1.3× to 1.7× at low concurrency.
Unsloth has released version 0.1.806-beta, enabling MTP (Multi-Token Prediction) speculative decoding by default for Qwen3.8-Flash-Next and GLM-5.3-Flash. The approach uses an additional draft head to predict several subsequent tokens at once, then passes them to the main model for verification in a single forward pass. Accepted candidates can be emitted directly, while rejected candidates fall back to the main model’s results. This does not change the output after verification; its main benefit comes from reducing the cost of running the full main model once for every token.
Qwen’s deployment documentation recommends using a 2.60 GB shared-Q8_0 draft head that can reuse the main model’s embedding and output projection, saving about 1.3 GB compared with the version that includes its own projection layer. The documentation estimates a speedup of roughly 1.3× to 1.7× at low concurrency and recommends `--spec-draft-n-max 2`. Extending the draft increases the number of candidate tokens, but performance may regress if the acceptance rate falls. This also explains why the release headline’s “up to 2×” claim cannot be extrapolated directly to every prompt, quantization format, or parallel workload.
Deployment still has clear limitations: the standard version of llama.cpp does not yet include qwen4exp’s MTP graph, cross-model tensor reuse, or the corresponding parameters. Users must therefore use Unsloth’s precompiled build, its fork, or an upstream PR that has yet to be merged. The draft head is also located in the MTP subdirectory and must be explicitly specified with `-md`; otherwise, the system may silently fall back to standard inference speed.
The same release also improves long-context caching with MLX on Apple Silicon, VRAM release after model switching, automatic multi-GPU placement, ROCm installation, tool calling, and MCP connectivity. However, most of these points are vendor-reported. Engineering teams should rerun benchmarks using their own context lengths, quantization settings, draft acceptance rates, time to first token, and concurrency levels. In particular, gains measured with a single worker slot should not be treated as equivalent to improvements in server throughput.