AWS blog introduces vLLM push-based KV transfer optimisation for TTFT
What happened
On 20 October 2026, the AWS Machine Learning Blog introduced a new push-based NIXL connector in vLLM. The approach moves the coordination and setup of KV transfer off the first-token critical path for disaggregated inference. On a 1P1D deployment on an Amazon EC2 P5en cluster, Qwen3-32B saw up to 3x better TTFT and up to 30% better TPOT. The optimisation has already shipped with vLLM; the report does not mention future plans or validation results on other models or hardware.
Written by AI from the reports · updated 3 h ago
Timeline
Follow the reports to see the event from each side.
- AWS Machine Learning BlogPush-based KV transfer to improve TTFT and TPOT for disaggregated inference in vLLM
vLLM's prefill-decode disaggregation splits LLM inference across engine instances but must transfer the KV cache between them. This post introduces a push-based NIXL connector that moves transfer coordination off the critical path, delivering up to 3x better TTFT and 30% better TPOT in benchmarks on Amazon EC2 P5en instances with AWS EFA.
Heat of this event
Current heat 9·Peak in the comparable range 9（Oct 20 03:00）·Change over 24 hours in the comparable range –
The trend compares only the same accounts observed without a break, so its range can be narrower than the current heat count. Move the pointer or tap the chart to see each hour; with a keyboard, use the left and right arrow keys.