推論與開源執行環境
DFlash 2 Drafts Tokens in Parallel for Qwen3.8-27B, but vLLM Nightly Currently Has Loading and VRAM Errors
DFlash 2 improves block-diffusion drafting with a candidate-path selector and dynamic convolution, delivering 2.7× to 3.4× the throughput of autoregressive decoding for Qwen3.8-27B in single-request tests. The latest reproducible issues show that vLLM nightly instantiates the wrong decoder layer and temporarily allocates nearly 4.74 GiB of extra memory before sharing vocabulary weights, potentially preventing startup on 24 GB GPUs.
Inco AI’s DFlash 2 does not replace Qwen3.8-27B; it serves as a speculative-decoding draft model with approximately 2B parameters. Conventional speculative decoding first uses a smaller model to propose multiple tokens and then has the target model verify them in a single pass. The DFlash series instead uses block diffusion to predict an entire draft segment in parallel within one forward pass. The new version retains the top 16 candidates at each position, uses a lightweight selector to find a coherent path, and applies two-tap dynamic convolution to prevent accuracy from degrading toward the end of the draft.
Official tests used a single H200 GPU, SGLang, FlashAttention 3, and Qwen’s recommended sampling parameters. For Qwen3.8-27B, the average acceptance length increased from 4.28 with native MTP to 4.80. Depending on the dataset, single-batch throughput was 2.7× to 3.4× that of standard autoregressive decoding. This acceleration does not directly accept the draft output: the target model still verifies it through rejection sampling. Greedy decoding should therefore produce the same tokens, while stochastic sampling preserves the target distribution. Real-world gains still depend on content predictability, batch size, and the draft acceptance rate.
Notably, a vLLM issue report submitted on August 24 reproduced two independent errors on the current main branch and a specified nightly build. First, `DFlashQwen3Model` hard-codes an older class when constructing layers, ignoring the `decoder_layer_cls` override provided by DFlash 2. As a result, `attention_conv` cannot be found while loading weights. The second issue occurs before the target and draft models share their embedding and LM head weights: initialization still allocates a separate temporary 248,320×5,120 BF16 module for each model, consuming about 2.37 GiB apiece. On an RTX 4090 paired with a 15.7 GiB W4A16 target model, the second allocation deterministically causes an OOM error; reducing the context length or KV cache budget does not help.
The reporter successfully started the model after shrinking the temporary vocabulary and applying a two-line layer fix, but an official upstream patch has not yet been merged. For now, deployers should pin a verified SGLang or other inference-engine version. Those using vLLM should treat the PR/nightly path as experimental and first test model loading, peak VRAM usage, sampling consistency, and tail latency under real workloads.