推論系統
The Optimization Stack for Running Hybrid Qwen Models on Apple Silicon Remains Incomplete: MLX Conversion Does Not Fully Preserve MTP
Community testing indicates that Apple Silicon inference frameworks still cannot reliably provide built-in MTP speculative decoding, hybrid-state prefix caching, and batched serving at the same time. The problem affects more than benchmarks: it can also force long-running coding agents to repay the prefill cost on every conversational turn.
What local inference on Apple Silicon currently lacks is not a single high-speed kernel, but a complete path connecting model-weight conversion, state caching, and speculative decoding. A recent community investigation uses hybrid models such as Qwen3.8-27B as examples. These models combine an attention KV cache, recurrent state, and built-in multi-token prediction (MTP) heads, but the standard `mlx-lm` conversion and loading workflow does not yet natively handle the full set of MTP weights and execution logic. Existing GitHub discussions also confirm that MTP components may not be preserved through the standard MLX conversion and quantization path. Another reproducible case shows that the `qwen3_5_mtp` type can even fail during server loading because it is not registered.
The alternatives involve trade-offs as well. `vllm-metal` already provides paged KV caching, continuous batching, and partial automatic prefix caching, but its official support matrix still lists hybrid GDN models such as Qwen3.5 and Qwen3.6 as incompatible with automatic prefix caching. For speculative decoding to roll back a rejected draft, it must restore not only the KV cache but also the recurrent state. When validating multiple draft tokens, Metal kernels may also reread the entire cache repeatedly. As a result, some frameworks can demonstrate MTP speedups with short prompts but cannot guarantee equivalent gains for long conversations, shared system prompts, or concurrent multi-agent workloads.
When engineering teams evaluate inference on Macs, they should measure multi-turn time to first token (TTFT), contexts of 32K tokens or more, time per output token (TPOT) after a cache hit, and the MTP acceptance rate—not just single-turn tokens per second. The next developments to watch are whether `mlx-lm` merges native support for preserving MTP weights and batched generation, and whether `vllm-metal` can enable both prefix caching and speculative decoding for hybrid recurrent state. Current conclusions are based primarily on community testing and rapidly evolving documentation, rather than controlled comparisons across hardware models and frameworks.