推論系統
Ray 2.58 Completes KV-Cache/Token-Aware Routing, but the New Router Remains Alpha
Ray Serve LLM now centralizes prompt tokenization, cache-hit estimation, and replica selection at the ingress, then passes tokens directly to the inference engine. The design can reduce repeated prefill and tokenization, but no updated performance data has been published, and the control plane introduces additional synchronization overhead.
Ray 2.58.0 turns the KV-cache- and token-aware routing previewed in 2.57 into a deployable path. Previously, requests were distributed solely by request count, which could send prompts with reusable prefixes to GPUs lacking the corresponding KV blocks and force the engine to run prefill again. Under the new design, the `LLMRouter` ingress first tokenizes the prompt using the same template as the engine, then compares each replica’s existing KV overlap, pending prefill work, and active decode workload.
Replicas report cache state through vLLM KV creation and eviction events, which Ray then broadcasts to every ingress. The router even includes KV blocks offloaded to CPU in its cache-hit estimates. After selecting a replica, it sends the prompt tokens to the engine over a separate channel, avoiding a second tokenization pass. This is especially useful for agent services that share large system prompts, few-shot examples, or long conversation prefixes, because cache-hit location and queue load can now be weighed in the same decision instead of optimizing only for session affinity.
The tradeoff is that ingress must spend additional CPU resources on tokenization and scoring while maintaining an eventually consistent view of caches across replicas; high-concurrency workloads may require more ingress replicas. The documentation also notes that the KV-aware router remains alpha and requires direct streaming and the NVIDIA Dynamo selection service. For simple traffic with little prefix reuse, the coordination overhead may outweigh the benefits. Version 2.58 also upgrades vLLM to 0.26.0. Operators should replay representative production traffic and compare time to first token, KV-cache hit rate, and ingress CPU usage rather than assuming higher throughput from the architecture alone. The release additionally introduces the experimental gVisor Ray Sandbox and fixes an issue that allowed Serve replicas to bypass token authentication, so upgrade assessments should also cover security and compatibility.