AI 推論系統
Atlas Takes On vLLM With a Pure Rust/CUDA Inference Engine on a Single GB10, Delivering Up to 22.5% Higher Throughput Under High Concurrency
Atlas has published reproducible Qwen3.8-27B-NVFP4 benchmarks showing it outperforming vLLM at concurrency levels from 1 to 128 on DGX Spark. The results cover only one model and a short-input, long-output workload, so they cannot yet be generalized to typical data center deployments.
Atlas Inference has published a new round of DGX Spark benchmarks featuring its open-source inference engine, whose control plane is written in Rust and core is implemented in CUDA. It does not depend on Python or PyTorch in the serving hot path. Instead, it configures specialized kernels for specific combinations of hardware, model architecture, and quantization format, while providing an OpenAI-compatible API. The community edition is licensed under AGPL-3.0, while its Contributor License Agreement allows the team to release a separate enterprise edition.
The benchmark loaded `unsloth/Qwen3.8-27B-NVFP4` on a single NVIDIA GB10 with 121.7 GB of unified memory. Input length was fixed at 128 tokens and output length at 1,024 tokens, with thinking mode disabled and deterministic decoding enabled. Both Atlas and vLLM used multi-token prediction with K=4, while a vLLM configuration without speculative decoding was retained for comparison. Each concurrency level received one warm-up run followed by three timed runs. From C=1 to C=128, Atlas achieved aggregate throughput ranging from 23.59 to 478.11 tokens/s, outperforming the faster vLLM configuration at each level by 1.2% to 22.5%. The largest gap appeared at 128 concurrent requests, suggesting that its scheduler and kernels continue to scale as requests queue up.
The engineering value goes beyond eliminating Python startup overhead. Atlas also integrates a paged FP8 KV cache, Gated DeltaNet, MTP, NVFP4 dequantization, and model recipes into a single inspectable binary deployment path, making it suitable for teams that want to serve multiple agent requests directly from a workstation. However, all figures come from the project's own single-machine testing. The short-input, long-output workload also does not cover million-token contexts, tool calls, prefix-cache hit rates, or tail latency. Its MLPerf 6.1 results are still awaiting publication by MLCommons. The next things to watch are independent reproductions, long-context correctness, and whether the advantage persists on hardware other than GB10.