Skip to content

SageMaker AI real-time endpoints support KV cache-aware routing

What happened

AI digest

On 18 November 2026, the AWS Machine Learning Blog reported that Amazon SageMaker AI has introduced KV cache-aware routing (KV_AWARE) for real-time endpoints. The feature forwards requests to the instance with the longest cache match that is not overloaded, based on the cache state reported by the model container. The report does not mention when the feature became available, its scope, or how it relates to existing routing strategies.

Written by AI from the reports · updated 5 h ago

Timeline

Follow the reports to see the event from each side.

Nov 18
  1. AWS Machine Learning Blog
    Amazon SageMaker AI real-time endpoints now support KV cache aware routing

    Amazon SageMaker AI real-time endpoints now support KV cache-aware routing, a new strategy that routes each request to the instance holding the longest cached match for its prompt. Learn how it works, when to use it compared with prefix-aware routing, and how to enable, invoke, and verify it on single model and inference component endpoints.

Heat of this event

Current heat 5·Peak in the comparable range 6(Nov 19 01:00)·Change over 24 hours in the comparable range –

0246Nov 1901:00Nov 1902:00Nov 1904:00Nov 1905:00

The trend compares only the same accounts observed without a break, so its range can be narrower than the current heat count. Move the pointer or tap the chart to see each hour; with a keyboard, use the left and right arrow keys.