推論系統
Ling-3.0-flash INT4 Hits 38.7 tokens/s in Single-System Testing; Using Mainline vLLM May Silently Produce Incorrect Output
A community deployment recipe running on a single DGX Spark uses CUDA Graph and a one-layer MTP speculative decoder to increase Ling-3.0-flash INT4 generation speed from 20.8 to 38.7 tokens/s. More importantly, it finds that mainline vLLM does not yet support the V3 architecture: forcing the model through an older attention path may not raise an error but can produce plausible-looking yet incorrect content.
A newly published deployment recipe shows that Ling-3.0-flash INT4—with 124B total parameters and approximately 5.1B active parameters per token—can fit on a single NVIDIA DGX Spark with 128GB of unified memory. Tests were conducted on the same machine with the same prompt, a single stream, and 512 generated tokens per run. After discarding warm-up runs, the reported figures are three-run averages: eager mode without MTP achieved 20.8 tokens/s, while removing `--enforce-eager` and enabling one speculative token increased performance to 38.7 tokens/s. For comparison, a community Q5_K_M GGUF implementation measured 35.2 tokens/s.
The performance difference is driven primarily by execution scheduling rather than the quantization format. Because Ling activates only a small number of experts at each step, the fixed cost of GPU kernel launches becomes more significant. CUDA Graph can replay a captured execution graph, reducing per-token launch overhead. The checkpoint also includes a one-layer multi-token prediction draft layer, allowing the server to use `bailing_hybrid_v3_mtp` to predict the next token before having the main model verify it.
The deployment risk is more noteworthy than the performance figures. Mainline vLLM currently lacks `BailingMoeV3ForCausalLM`. If V3 weights are forcibly loaded through the older V2.5 class, the program may route KDA weights through a different attention computation path. The service may still start, and its output may remain grammatically fluent, making it difficult for ordinary health checks to detect that the numerical semantics are already incorrect. The recipe therefore requires inclusionAI’s dedicated `vllm-ling-v3` branch.
This remains a small-scale test by a single author on one machine. It has only been validated up to a 16K context, and cold starts reportedly have about a 50% chance of stalling while loading the 24 shards; a watchdog-based retry is currently used as a workaround. Engineering teams should first add regression tests covering known-answer prompts and tool calls, then compare throughput, time to first token, and long-context quality. They should also wait for official support in mainline vLLM and further tuning of MoE kernels specifically for GB10.