AWS implements bidirectional KV transfer in vLLM to optimise multi-turn agent TTFT
What happened
On 20 October 2026, the AWS Machine Learning Blog described a bidirectional KV transfer mechanism it implemented in vLLM. In later turns, decode nodes return already-computed KV to prefill nodes for reuse, avoiding recomputation of the conversation history. The work targets time to first token (TTFT) in multi-turn agent workloads. On a 1P1D deployment of Amazon EC2 P5en instances with EFA, TTFT fell by about 1.6x for Qwen3-32B and about 3.3x for zai-org/GLM-5-FP8 at the longest prompts. This report is currently the only source on the work, with no other developments.
Written by AI from the reports · updated 2 h ago
Timeline
Follow the reports to see the event from each side.
- AWS Machine Learning BlogImprove TTFT for multi-turn agentic workloads with bi-directional KV transfer in vLLM
Multi-turn agentic workloads reprocess growing conversation context each turn, driving up latency. See how bi-directional KV cache transfer lets prefill and decode nodes reuse previously computed context across turns, cutting time to first token by up to 1.6x for Qwen3-32B and 3.3x for GLM-5-FP8 on Amazon EC2 P5en instances with EFA.
Heat of this event
Not enough continuous observations yet to draw a trend.