AWS Machine Learning Blog· Sunita Nadampalli·· 6 h ago
Push-based KV transfer to improve TTFT and TPOT for disaggregated inference in vLLM
Push-based KV transfer to improve TTFT and TPOT for disaggregated inference in vLLM
AI summary
vLLM's prefill-decode disaggregation splits LLM inference across engine instances but must transfer the KV cache between them. This post introduces a push-based NIXL connector that moves transfer coordination off the critical path, delivering up to 3x better TTFT and 30% better TPOT in benchmarks on Amazon EC2 P5en instances with AWS EFA.
Selection record
Threshold 60Official, first-handFirst 38Second 34
Not admittedSum of both 72 < twice the threshold 120
- Source tier
- Official, first-hand; this tier's threshold is 60
- Pre-filter
- passed：vLLM大模型推理KV传输优化技术
A model scores each item twice, independently, against one written standard, out of 100. An item is admitted only when the two scores add up to twice the threshold. The threshold is set per source tier.
Source: AWS Machine Learning Blog · aws.amazon.com