推論基礎設施
Triton 2.72 Fixes Dynamic Batching Starvation and Response Ordering, Upgrades to CUDA 13.4 Inference Stack
NVIDIA Triton Inference Server 2.72.0 fixes an issue that could cause the scheduler to stop dispatching work while requests were still pending and restores the completion-order semantics of `preserve_ordering`. The release also updates its CUDA, TensorRT, and gRPC dependencies, but no TensorRT-LLM image is available, while security warnings remain for vLLM and Ray deployments.
NVIDIA released Triton Inference Server 2.72.0 on August 31, corresponding to the NGC 26.08 container. Rather than highlighting new generative models, this update addresses scheduling and status-reporting issues that directly affect the reliability of production services. The dynamic batcher’s `waiting_consumer_count` could gradually drift from the actual state, causing the scheduler to stop dispatching work even while requests remained in the queue. The new release fixes this starvation condition. Teams using dynamic batching under highly concurrent traffic should include prolonged stress testing in their upgrade validation.
`preserve_ordering` now reserves completion slots when requests enter the queue instead of allocating them only after computation finishes, restoring the output order expected by clients. For model health checks, `TRITONSERVER_ServerModelIsReady` now returns errors other than “model not found,” and models that fail to resolve are correctly marked as `ready=false`. The model-loading path now catches exceptions and handles `invalid_argument` and `out_of_range` consistently, helping prevent invalid model configurations from disrupting the server lifecycle.
For embedded deployments, new C and Python logging callbacks allow Triton messages to be sent directly to an application’s existing observability stack. Backend fixes cover CUDA Graph execution for non-batched TensorRT models, vLLM JSON input parsing, and OpenVINO model generation on ARM SBSA. The build process can also read experimental preset files and limit parallel compilation based on available memory, reducing the risk of out-of-memory errors when building the ONNX Runtime or OpenVINO backends from source.
The 26.08 container includes Triton 2.72, CUDA 13.4.1, and TensorRT 11.2.1.2, with a minimum supported compute capability of 7.5. Before upgrading, users should verify host-driver compatibility and custom backend ABIs. Notably, this release does not provide a TensorRT-LLM backend container. NVIDIA also warns that the vLLM v0 API and Ray contain vulnerabilities, so executors and hosts should not accept untrusted network connections. Other known limitations include the CuPy multithreaded CUDA 13 Device API and vLLM tensor parallelism under explicit model control. This is therefore not an update where teams should simply replace the image and skip regression testing.