推論系統
NInfer Hard-Codes Inference Paths for Two Qwen3.6 Models, Reaching 543 Tokens per Second in Long-Context Decoding on a Single RTX 5090
The open-source NInfer uses custom quantization, weight layouts, fused kernels, and an MTP draft path to sustain long-context decoding at about 543 tokens per second for Qwen3.6-35B-A3B on a single RTX 5090. Its speed comes from deep specialization for the model and GPU, but it currently supports only two checkpoints, lacks continuous batching, and has yet to be independently benchmarked under identical conditions.
The open-source project [NInfer](https://github.com/Neroued/ninfer) has recently sparked a wave of testing in the LocalLLaMA community. Rather than extending llama.cpp or vLLM, its author rewrote a fixed inference pipeline from scratch in C++/CUDA specifically for Qwen3.6-27B and Qwen3.6-35B-A3B, targeting the RTX 5090’s sm_120a architecture. Its best publicly reported result comes from generating 65,536 tokens in a single request: the 35B-A3B model achieved an average decoding speed of 542.8±12.5 tokens per second with a 73% MTP acceptance rate. Shorter structured outputs can reach about 661 tokens per second.
The performance gains come from custom quantization at roughly 4.97 bits per weight, weights rearranged to match kernel access patterns, per-operator optimization, kernel fusion, and a dedicated LM head draft path. The released model artifact is approximately 20.84 GiB. With an INT8 KV cache, it can use the model’s native 262,144-token context window within 32GB of VRAM. The project also supports text, image, and video inputs, along with OpenAI- and Anthropic-compatible HTTP interfaces. The author has also published single-run evaluations on AIME and GPQA-Diamond to check whether quantization causes an obvious collapse in model capability.
Initial community reproductions show the same model reaching about 448 tokens per second, though testers also found that template handling had incorrectly cleared the prefix cache. Other users noted that NInfer’s 27B prefill performance may not outperform vLLM using NVFP4. This helps define the project’s technical value: jointly optimizing the model, weight format, and a single GPU can greatly outperform general-purpose runtimes on some decoding paths, at the cost of portability and serving features. NInfer currently supports only two checkpoints, primarily targets the RTX 5090, and lacks continuous batching. The next things to watch are blind evaluations at equivalent quantization quality, standardized prefill and decode comparisons, and whether its advantage persists under concurrent multi-user workloads.