SageMaker AI real-time endpoints support KV cache-aware routing
What happened
On 18 November 2026, the AWS Machine Learning Blog reported that Amazon SageMaker AI has introduced KV cache-aware routing (KV_AWARE) for real-time endpoints. The feature forwards requests to the instance with the longest cache match that is not overloaded, based on the cache state reported by the model container. The report does not mention when the feature became available, its scope, or how it relates to existing routing strategies.
Written by AI from the reports · updated 5 h ago
Timeline
Follow the reports to see the event from each side.
- AWS Machine Learning BlogAmazon SageMaker AI real-time endpoints now support KV cache aware routing
Amazon SageMaker AI real-time endpoints now support KV cache-aware routing, a new strategy that routes each request to the instance holding the longest cached match for its prompt. Learn how it works, when to use it compared with prefix-aware routing, and how to enable, invoke, and verify it on single model and inference component endpoints.
Heat of this event
Current heat 5·Peak in the comparable range 6(Nov 19 01:00)·Change over 24 hours in the comparable range –
The trend compares only the same accounts observed without a break, so its range can be narrower than the current heat count. Move the pointer or tap the chart to see each hour; with a keyboard, use the left and right arrow keys.