開源推論
NInfer-3090 Uses ReplaySSM to Reduce Speculative Decoding State Costs, Enabling Concurrent Qwen3.8-27B Inference on an RTX 3090
NInfer’s Ampere branch fits Qwen3.8-27B, MTP3, paged KV, and compatible prefix reuse onto a single 24GB RTX 3090. The author measured aggregate eight-request decoding throughput of 165.33 tokens/s, though the results are limited to a fixed workload with short prompts and long outputs.
NInfer-3090 v0.6 ports its specialized C++20/CUDA inference engine, originally designed for newer-generation GPUs, to the RTX 3090’s `sm_86` architecture, with Qwen3.8-27B as its primary supported model. The key addition is ReplaySSM: during MTP speculative decoding, it treats the recurrent state of the hybrid model as a transaction that can be committed or rolled back, avoiding the need to retain multiple complete copies of temporary state for each concurrent sequence. Combined with INT8 KV, paged KV, CUDA Graphs, and fixed cohorts, the project claims it can keep three-token MTP drafts enabled for eight requests within 24GB of VRAM, instead of disabling speculative decoding because of state growth.
In the author’s sustained tests, each request generated 1,024 tokens. Single-request end-to-end throughput was 70.19 tokens/s, with an average TTFT of 149 ms. With eight concurrent requests, aggregate throughput reached 161.28 tokens/s, while decode-only throughput reached 165.33 tokens/s. Peak VRAM usage was 22,138 MiB, and the MTP acceptance rate fell from 61.13% for one request to 56.84% for eight. This indicates that the system primarily trades per-request latency for higher aggregate throughput: average TTFT increased to 1.215 seconds with eight requests. The engine also provides OpenAI Chat Completions, Responses, and Anthropic-compatible interfaces, and can reuse compatible prompt prefixes.
The model file is neither GGUF nor a Transformers checkpoint, but a proprietary 16.96 GiB `.ninfer` container containing quantized text weights, a vision encoder, MTP, and a proposal head. Deployment is therefore tied to a specific runtime revision. More importantly, the published benchmarks used inputs of roughly 29 to 34 tokens and shared 8K/16K KV pools, measuring short prefill followed by long decode. They cannot be used to directly infer how coding agents would perform with repository contexts spanning tens of thousands of tokens. On Linux, only compilation and startup checks have been completed so far; actual model performance remains unverified. The next things to watch are independent reproductions, TTFT with long prompts, and whether ReplaySSM still saves memory under low acceptance rates and frequent tool calls.