AWS blog introduces vLLM push-based KV transfer optimisation for TTFT
What happened
On 20 October 2026, the AWS Machine Learning Blog introduced a new push-based NIXL connector in vLLM. The approach moves the coordination and setup of KV transfer off the first-token critical path for disaggregated inference. On a 1P1D deployment on an Amazon EC2 P5en cluster, Qwen3-32B saw up to 3x better TTFT and up to 30% better TPOT. The optimisation has already shipped with vLLM; the report does not mention future plans or validation results on other models or hardware.
Written by AI from the reports · updated 2 h ago
Timeline
Follow the reports to see the event from each side.
- AWS Machine Learning BlogPush-based KV transfer to improve TTFT and TPOT for disaggregated inference in vLLM
vLLM's prefill-decode disaggregation splits LLM inference across engine instances but must transfer the KV cache between them. This post introduces a push-based NIXL connector that moves transfer coordination off the critical path, delivering up to 3x better TTFT and 30% better TPOT in benchmarks on Amazon EC2 P5en instances with AWS EFA.
Heat of this event
Not enough continuous observations yet to draw a trend.