AWS adds routing that sends SageMaker AI requests where prompt work is already cached
The feature tracks what model servers actually retain, rather than guessing from requests. AWS reports large speed gains, but its benchmark compares against random routing.
Announced November 16, SageMaker AI’s new KV-cache-aware routing tracks cache insertions and evictions, then directs each request toward a server with the longest reusable token match while accounting for load. AWS’s tests on Llama 3.1 70B Instruct reported up to 97% lower median time to first token and 234% higher throughput than random routing; those figures do not establish an advantage over SageMaker’s existing prefix-aware routing. Teams need vLLM container 0.33.0 or later and at least two instances or model copies, and should validate results on their own workloads.
01
AWS tested seven ml.p5.48xlarge instances with 20 or 100 simulated users and prompts reaching about 64,000 tokens; cache hit rates rose from 3–48% to 80–93%.
02
The routing index stores truncated one-way hashes mapped to servers, not prompts, token IDs, request payloads, or cached model calculations.
03
If lookup is slow, finds no useful match, or matching servers are busy, SageMaker falls back to load-aware routing without the cache benefit.
AI conversations on Amazon SageMaker AI can now reach a server that has already processed much of their history, avoiding repeated work. AWS introduced KV-cache-aware routing for real-time endpoints in a November 16 launch notice, then published a technical breakdown on November 17. The feature considers both cached content and server load.
A language model stores intermediate calculations from processing a prompt in GPU memory, called a key-value, or KV, cache. When another request begins with the same text, the server can reuse those calculations. But that benefit depends on the request reaching a server that still holds them.
SageMaker’s existing prefix-aware routing groups requests by their opening text. It infers where useful work should be cached, without checking what remains in memory. That can miss when a server discards older entries to make room. It can also scatter early conversation turns across servers while their growing history remains shorter than the configured matching length.
The new approach gets updates whenever the model container stores or removes cache entries. SageMaker maintains an index, matches incoming prompts against it, and weighs the longest cached match against how busy each candidate is. Matching uses tokens—the units a model processes—rather than a fixed number of bytes or characters.
AWS says the index stores one-way, truncated hashes mapped to the server or model copy holding the cached work. It does not store prompts, token IDs, request payloads or the cached model calculations themselves.
AWS’s benchmark against random routing
Up to 97% lowerMedian time to first token
AWS reported up to a 97% reduction in median time before output began, compared with random routing.
Up to 234% higherThroughput
AWS reported up to 234% higher throughput with long prompts under load, compared with random routing.
Editorial illustration for AWS adds routing that sends SageMaker AI requests where prompt work is already cached.Source: aws.amazon.com.
For its benchmark, AWS ran Llama 3.1 70B Instruct across seven ml.p5.48xlarge instances. The 30-minute tests used 20 or 100 simulated users conducting multi-turn chats and document question-and-answer sessions, with prompts reaching about 64,000 tokens. Cache hit rates rose from 3–48% with random routing to 80–93% with KV-aware routing.
Those headline improvements are not a measured advantage over prefix-aware routing. AWS says gains were smallest for short prompts, where less initial processing can be saved. Stable shared instructions and moderate traffic can still suit the older approach, which works with any container supporting prefix caching and needs no container changes.
Checking the index adds a few milliseconds for short prompts and more for long ones, mostly from tokenization. If the lookup is slow, finds no useful match, or all matching servers are busy, SageMaker falls back to load-aware routing. The request still runs, but without the cache benefit.
Customers enable cache-event reporting through the SM_VLLM_KV_EVENTS_CONFIG environment variable and set RoutingStrategy to KV_AWARE in the endpoint configuration. Applications keep calling the same APIs. At launch, the feature requires AWS’s vLLM Deep Learning Container version 0.33.0 or later, plus at least two instances or two deployed model copies.
AWS lists support for single-model endpoints and inference components, including LoRA adapters, on GPU instances. The feature carries no additional routing charge; customers still pay for their endpoint instances. AWS says it is available across all Regions where SageMaker AI is available.
Fallbacks are not shown in the SageMaker console or metrics at launch. To check whether routing is helping their traffic, teams can enable detailed observability for cache hit rate and time to first token, then compare performance under their own workload.
Sources
aws.amazon.comAmazon SageMaker AI now supports KV cache-aware routing for real-time inference endpoints - AWS
aws.amazon.comAmazon SageMaker AI real-time endpoints now support KV cache aware routing | Amazon Web Services
Reader comments
Newest comments first. Replies stay oldest first.