llm-d 1.0 Strengthens Multi-Tenant Inference Scheduling, Stabilizes APIs, and Adjusts Upgrade Compatibility

The llm-d maintainers announced version 1.0 on October 20, shifting the focus of distributed inference toward ongoing operations after deployment: how the same set of accelerators can serve different tenants, handle bursts of agent requests, and maintain service during updates. Prefix-cache routing, KV-cache offloading, and prefill/decode disaggregation were already available; the main change is bringing these capabilities together in more stable scheduling and configuration interfaces. Official announcement

Multi-tenant control classifies traffic by tenant or workload identifier and priority, maintaining separate queues for each. The scheduler handles priority first, then rotates fairly among tenants at the same priority. When backends are saturated, requests can wait at the routing layer instead of continuing to pile up on model servers. Deployment teams should note that the default behavior remains similar to first-come, first-served: tenant policies must be configured explicitly, and upgrading does not automatically create service-level isolation. Traffic control overview

For agent serving, session state routes subsequent turns to replicas that already have the relevant KV cache; requests are rebalanced when a replica becomes saturated. This is especially useful for agents that repeatedly read the same code, documents, and conversation history, as it can reduce the need to recompute context. IBM’s overview also emphasizes that the release’s value lies in integration and operational maturity under mixed workloads; a single cache-hit rate should not be treated as a cost reduction that applies to every application. IBM technical overview

For migration, the router’s EPPConfig and InferenceObjective APIs advance to llm-d.ai/v1; the latter supports setting objectives across multiple resource pools. The legacy inference.networking.x-k8s.io resource group has been removed and must be replaced with inference.networking.k8s.io. Some kvcache_* metric aliases and the sidecar’s --vllm-port flag have also been removed. These changes may affect deployment manifests, monitoring queries, and startup scripts. Release notes

Engineering teams should next test with interactive requests mixed with bursts of agent traffic, tracking time to first token, wait times for each tenant, and whether cache affinity creates hotspots. Upgrade rehearsals should also verify version compatibility between prefill and decode replicas. The official announcement was dated October 20, while the router component tag was released earlier, on October 16; these should not be treated as the same release date.