推論與執行期
llama.cpp b10731 Adds Recurrent State Rollback for Qwen3.8-Flash-Next, Eliminating Repeated Full-State Transfers During MTP Speculative Decoding
llama.cpp fixes an issue in Qwen3.8-Flash-Next that prevented SSM and convolutional states from being rolled back correctly when draft tokens were rejected. Testing on a single RTX PRO 6000 showed code decoding performance rising from 123 to 183 tokens/s, though the results remain limited to a specific quantization, draft head, and single processing slot.
The b10731 prerelease of llama.cpp adds recurrent state rollback support for Qwen3.8-Flash-Next, addressing a path where MTP (multi-token prediction) speculative decoding could paradoxically become slower as more acceleration was applied. MTP first uses a draft head to predict multiple tokens, then passes them to the target model for verification in a single step. If only part of the draft is accepted, the target model must revert to the state immediately before the first rejected token. For the `qwen4exp` architecture, which uses an SSM/recurrent cache, this involves more than simply truncating a conventional KV cache.
Before the fix, llama.cpp marked this case as `SEQ_RM_TYPE_FULL`, causing the complete recurrent state to be serialized to host memory during every speculative decoding cycle. The project notes that although the existing recurrent cache already allocated `n_rs_seq + 1` snapshot planes, the model-specific `build_conv_state_at()` wrote only to the current plane. Enabling rollback directly would therefore produce an inconsistent state in which the SSM state was correct but the convolutional history was not. The new implementation stores a snapshot from one token earlier for every rollback position, covering both delta-net QKV and PLE convolutional states, allowing the correct history to be restored on-device.
A maintainer tested the fix on an RTX PRO 6000 with a single serving slot, `n-max=3`, and Qwen3.8-Flash-Next UD-Q4_K_XL. After the fix, code generation reached 183 tokens/s and general text generation reached 144 tokens/s, up from 123 and 83 tokens/s, respectively. Decoding without a draft head reached 108 tokens/s. These results also show that the previous MTP path for general text was actually slower than direct decoding because the bottleneck was state transfer rather than the draft acceptance rate itself.
The update should not yet be extrapolated to imply a 1.5–2× performance gain across all hardware. Testing covered only one GPU, one quantization, and a single processing slot. Multi-slot concurrency, different `n-max` values, CPU/Vulkan/ROCm backends, and the memory cost of the additional 2.5 GB Q4 draft head could all affect the results. Deployers should also note that b10731 is labeled a prerelease. Before using it in production, they should compare exact token consistency, acceptance rates, device-to-host traffic, and tail latency using fixed prompts.