模型與推論架構
Qwen3.8-Flash-Next Uses Block-Sparse Attention and n-gram Parameter Scaling to Preview a New Architecture for Qwen4
Alibaba has released the weights for Qwen3.8-Flash-Next, which uses Qwen Sparse Attention, Gated Residual, and 51 billion n-gram embedding parameters to reduce the computational cost of long contexts. The model natively supports 262,144 tokens, but its performance figures come primarily from official evaluations, and it is not licensed under the commonly used Apache 2.0 license.
Alibaba’s Qwen team has released the experimental Qwen3.8-Flash-Next model, describing its architecture as the foundation for Qwen4. The core language model has 125 billion parameters, with approximately 6 billion activated per token, plus 51 billion bigram/trigram embedding parameters and 4 billion multi-token prediction parameters. Its native context length is 262,144 tokens and can be extended to 1 million.
The main architectural change is the interleaving of Gated DeltaNet and Qwen Sparse Attention (QSA). Instead of selecting attention positions token by token, QSA builds an index using micro-blocks, with each layer reading blocks equivalent to no more than 2,048 tokens. In theory, this prevents long-context costs from increasing proportionally with sequence length. The new Gated Residual mechanism uses a data-dependent read gate and per-branch write gates to control a widened residual stream. The team also treats short n-gram lookup tables as a low-compute scaling axis that can be easily offloaded to main memory, using them alongside MoE, which requires routing and matrix multiplication.
According to official results, Qwen3.8-Flash-Next scores 62.5% on SWE-bench Pro and 81.0% on SWE-bench Multilingual, outperforming the company’s own 27B dense model on both benchmarks. However, the DeepSWE score uses the better result from two agent harnesses, while the SWE-bench Pro baselines were rerun on revised tasks, so the results cannot be compared directly with public leaderboards. Some long-horizon office-work and application-reconstruction tests are also internal benchmarks.
Deployment paths are available for Transformers, vLLM, SGLang, and TokenSpeed, but the new attention mechanism, n-gram embeddings, and MTP all require recent runtime support. Engineering teams should benchmark VRAM usage, time to first token, and block-indexing overhead across different context lengths. They should also note that the model is released under the Qwen Community License 1.0 rather than the more familiar Apache 2.0 license.