推論基礎設施
SageMaker Preserves KV Cache with Prompt Prefix Routing, Cutting Llama 3.1 70B Time to First Token by Up to 77%
SageMaker real-time inference endpoints now support `PREFIX_AWARE`, which aims to route requests with the same system prompt, documents, or conversation history to the same instance. AWS’s official seven-node tests show substantial gains for long-prefix workloads, although serialization differences, popular prefixes, and cache capacity can still affect real-world performance.
Amazon SageMaker Inference has added prefix-aware routing to address a problem in multi-instance LLM serving: the inference framework may support prefix caching, but the load balancer can still render the cache ineffective. Traditional random or least-outstanding-request routing may distribute requests sharing the same system prompt, RAG documents, or conversation history across different GPUs. Each instance must then repeat the prefill phase instead of reusing previously computed attention key-value tensors.
The new `PREFIX_AWARE` strategy creates a stable mapping based on the beginning of each request, routing shared prefixes to the same instance. Engineers can configure `PrefixLength`: the native Invoke API calculates it using 1,024 to 65,536 bytes, while the OpenAI-compatible interface calculates it using the character count of message text. `ConcurrencyThreshold` can be set from 1 to 1,024. When the target instance is overloaded, the router sacrifices a cache hit and sends the request to a less busy node. Multi-tenant services can also use `X-Amzn-SageMaker-Prefix-Aware-Id` or `prompt_cache_key` to isolate routing groups. The feature can be used alongside inference components and dynamic LoRA adapters.
AWS tested the feature using seven `ml.p5.48xlarge` instances, vLLM, and Llama 3.1 70B. In scenarios with a shared 8,000-token prefix, P50 time to first token fell by 71% to 77%, while P90 fell by 33% to 37%. The KV cache hit rate increased from approximately 25% to 82%, and throughput rose by 15% to 16%. For ShareGPT-style conversations with shorter, variable-length prefixes, the throughput gain was only 1.7% to 2.0%. The routing itself added approximately 1.3 to 1.9 milliseconds of latency.
This does not automatically guarantee faster model inference: prefix caching must still be enabled in the container, and the native API compares raw request bytes directly. Differences in JSON whitespace or field order—or placing sampling parameters near the beginning—can split traffic that should otherwise share a cache entry. Setting the prefix too short can also create hotspots. Operators should tune both parameters using real prompt distributions while monitoring cache hit rate, overflow ratio, TTFT tail latency, and cache warm-up time after scaling events.