Skip to content
AWS Machine Learning Blog· Dmitry Soldatkin·· 1 d ago

Amazon SageMaker AI real-time endpoints now support KV cache aware routing

Amazon SageMaker AI real-time endpoints now support KV cache aware routing

AI summary

Amazon SageMaker AI real-time endpoints now support KV cache-aware routing, a new strategy that routes each request to the instance holding the longest cached match for its prompt. Learn how it works, when to use it compared with prefix-aware routing, and how to enable, invoke, and verify it on single model and inference component endpoints.

Selection record

Not admittedSum of both 116 < twice the threshold 120

Source tier
Official, first-hand; this tier's threshold is 60
Pre-filter
passed:SageMaker AI新增KV缓存感知路由,属AI推理技术

A model scores each item twice, independently, against one written standard, out of 100. An item is admitted only when the two scores add up to twice the threshold. The threshold is set per source tier.

Source: AWS Machine Learning Blog · aws.amazon.com