推論系統
llm-d Releases 1.0, Strengthening Distributed Inference Operations with Multi-Tenant Traffic Control and Stable APIs
On October 20, llm-d released version 1.0, adding multi-tenant quality of service, coordinated upgrades, and agent session routing to its inference stack. The release provides a stable v1 configuration API, but priority policies still require explicit configuration, and failover capabilities remain on the roadmap.
The llm-d project announced version 1.0 on October 20, shifting the focus of distributed inference toward post-deployment operations: how to share a cluster between interactive requests and bursts of agent traffic, and how to coordinate upgrades across multiple inference components. This Kubernetes-native open-source stack sits above engines such as vLLM and SGLang, handling routing across replicas and resource coordination. IBM’s release notes say that technologies such as prefix-cache routing, KV cache offloading, and prefill/decode disaggregation were already available; this release strengthens integration and operational maturity. IBM release notes
Multi-tenant control is a substantive change in the new release. The router queues requests by tenant or workload identity and priority level, dispatches them by priority, then rotates fairly among tenants at the same level. When backends are saturated, requests can wait at the routing layer, preventing large queues from forming on model servers. This can help protect interactive latency, but the default behavior remains close to first-come, first-served, so deployers must enable the policy themselves. Project technical announcement
For upgrades, the Endpoint Picker adds warm standby. DisaggregatedSet treats prompt-processing and token-generation replicas as a single logical unit, selecting compatible pairs based on version and readiness. The configuration API has also moved to stable v1, giving platform teams a clearer integration interface. Project technical announcement
For agent applications, session routing prioritizes sending follow-up requests back to a replica that retains the KV cache, redistributing them once that replica becomes saturated. The technical benefit is reduced recomputation of the same code, documents, and conversation history on each turn; actual gains still depend on context reuse rates and workload distribution. IBM technical overview
Engineering teams should next validate tail latency under mixed traffic and error rates during upgrades. The project still lists active-active routers, request failover, and migration as future work, so claims of “zero downtime” should be tested against deployment conditions and cannot be assumed to cover every failure scenario. Project roadmap