AWS Machine Learning Blog· Sunita Nadampalli·· 7 h ago
Improve TTFT for multi-turn agentic workloads with bi-directional KV transfer in vLLM
Improve TTFT for multi-turn agentic workloads with bi-directional KV transfer in vLLM
AI summary
Multi-turn agentic workloads reprocess growing conversation context each turn, driving up latency. See how bi-directional KV cache transfer lets prefill and decode nodes reuse previously computed context across turns, cutting time to first token by up to 1.6x for Qwen3-32B and 3.3x for GLM-5-FP8 on Amazon EC2 P5en instances with EFA.
Selection record
Threshold 60Official, first-handFirst 58Second 58
Not admittedSum of both 116 < twice the threshold 120
- Source tier
- Official, first-hand; this tier's threshold is 60
- Pre-filter
- passed：vLLM多轮Agent KV传输优化，实质AI技术
A model scores each item twice, independently, against one written standard, out of 100. An item is admitted only when the two scores add up to twice the threshold. The threshold is set per source tier.
Source: AWS Machine Learning Blog · aws.amazon.com