AWS Machine Learning Blog· Dmitry Soldatkin·· 1 d ago
Amazon SageMaker AI real-time endpoints now support KV cache aware routing
Amazon SageMaker AI real-time endpoints now support KV cache aware routing
AI summary
Amazon SageMaker AI real-time endpoints now support KV cache-aware routing, a new strategy that routes each request to the instance holding the longest cached match for its prompt. Learn how it works, when to use it compared with prefix-aware routing, and how to enable, invoke, and verify it on single model and inference component endpoints.
Selection record
Threshold 60Official, first-handFirst 58Second 58
Not admittedSum of both 116 < twice the threshold 120
- Source tier
- Official, first-hand; this tier's threshold is 60
- Pre-filter
- passed:SageMaker AI新增KV缓存感知路由,属AI推理技术
A model scores each item twice, independently, against one written standard, out of 100. An item is admitted only when the two scores add up to twice the threshold. The threshold is set per source tier.
Source: AWS Machine Learning Blog · aws.amazon.com