開源推論系統
Paddock Open-Sources Rust/CUDA Inference Engine Targeting Concurrent AI Agents on a Single NVIDIA GPU
Truespar has released Paddock’s scheduler, paged KV cache, memory management, and custom CUDA kernels, along with OpenAI- and Anthropic-compatible interfaces. The authors’ benchmarks show it outperforming vLLM and SGLang on some single-GPU workloads, but the results have not been independently reproduced, and Paddock does not support tensor parallelism, ROCm, or Metal.
On September 4, Truespar open-sourced Paddock’s internal libraries under a dual MIT/Apache-2.0 license. This is not a wrapper around llama.cpp or a Python inference backend: the Rust service layer, continuous batching scheduler, paged KV cache, radix prefix cache, memory management, and C++ CUDA kernels are all housed in the same repository. `paddock-runner` can directly serve interfaces compatible with OpenAI Chat Completions, Responses, embeddings, audio transcription, and Anthropic Messages. It supports weights in GGUF and safetensors formats, with quantization paths including FP8, NVFP4, MXFP4, Q8_0, and some Q4_K variants.
Its design focuses on serving multiple coding agents or long-running conversations concurrently on a single GPU. In addition to chunked prefill and fair scheduling, Paddock can spill the KV cache into system memory or disk and stream MoE experts that exceed VRAM capacity from RAM. The CUDA kernel pack is separated from the Rust binary through a stable C ABI, so users running a precompiled build need only an NVIDIA driver; CUDA 13.x is required only to compile the kernels locally. The bundled Studio handles model downloads, comparisons, and artifacts, although these peripheral features also expand the code surface requiring audit.
Using NVIDIA `aiperf` to test Qwen3.8-27B FP8 on an RTX PRO 6000, the authors claim that Paddock outperformed vLLM in all 13 scenarios by factors ranging from 1.02× to 1.19×. Against SGLang, it recorded ten wins, two losses, and one tie. These remain vendor-controlled, single-GPU tests. Reddit users have also reported compilation errors on the RTX 5080, failures with some GGUF quantizations, and potential security issues such as unauthenticated loopback access and continuing startup after failures. The compilation error has been patched, but the security implications have not yet been confirmed in a formal advisory. At present, Paddock supports only CUDA on Windows and Linux. Each model must fit on a single GPU; there is no tensor or pipeline parallelism, nor support for ROCm, Vulkan, or Metal. Before deployment, operators should pin the model, quantization, driver, and workload; rerun the published scenarios; and place the API behind a reverse proxy that enforces authentication.