Improve TTFT for multi-turn agentic workloads with bi-directional KV transfer in vLLM

Multi-turn agentic workloads build on accumulated context as an agent reasons, invokes tools, and incorporates earlier results. With each new turn, the client resends the conversation history, including system instructions, prior prompts, tool results, and model responses, causing the growing context to be processed repeatedly. In Prefill-decode (P-D) disaggregation , each request is split across specialized nodes: the prefill node processes the prompt and builds its key-value (KV) cache, then the decode node retrieves those KV blocks and generates the response. Conventional disaggregated serving, however, only transfers KV cache in one direction: from prefill to decode node. This one-directional flow creates two inefficiencies for multi-turn workloads. First, after generating a response, the decode node holds KV cache entries for both the prompt and the newly generated tokens. When processing the next turn, the prefill node cannot retrieve those entries, so it must recompute the KV projections for the previous response tokens. Second, the decode node maintains the session state while the prefill nodes don’t, increasing the probability of cache eviction for active sessions on the prefill nodes. On a full cache miss, the prefill node must recompute KV projections for the entire conversation history, including system instructions, prior user requests, tool results, and model-generated responses. As the conversation grows, this redundant work consumes more accelerator capacity and increases time to first token (TTFT). When you build conversational assistants and agentic applications, longer interaction histories can increase latency and consume accelerator capacity through repeated prefill computation. In this post, we show how bi-directional KV cache transfer lets prefill and decode nodes reuse previously computed context across turns, which improves accelerator utilization and helps applications sustain responsive TTFT as conversations grow. In our multi-turn benchmarks on a 1P1D deployment using Amazon Elastic Compute Cloud (Amazon EC2) P5en instances, bi-directional KV transfer reduced TTFT by up to 1.6x for Qwen3-32B and up to 3.3x for zai-org/GLM-5-FP8 (see the Performance data section). Results vary with workload and configuration. The instances are connected with AWS Elastic Fabric Adapter ( AWS EFA ), a low-latency remote direct memory access (RDMA) network used across servers in AWS. On supported instance types , AWS EFA supports GPUDirect RDMA and Neuron Direct RDMA to move KV blocks directly between GPU or AWS Trainium device memory on the prefill and decode nodes. We originally built and validated the feature on AWS Trainium instance clusters connected with EFA and contributed it to vLLM as an accelerator-agnostic capability. The results presented here are from an Amazon EC2 P5en GPU cluster with EFA, showing that the benefits also carry over to GPUs. This post explains the bi-directional KV transfer architecture, router metadata flow, KV-block retention and recompute controls, performance results, and how to reproduce the benchmark. Bi-directional KV transfer Bi-directional KV transfer is a mechanism in which a decode node returns previously computed KV to a prefill node on subsequent turns of a conversation. This avoids recomputing shared context on the prefill node and is governed by a configurable KV recompute threshold. It mitigates both computational-redundancy scenarios described earlier. To make it work, the KV must survive past the turn that produced it and travel in the reverse direction, coordinated by a KV-cache-aware router. NIXL KV connector enhancements for bi-directional transfer We extended the NVIDIA Inference Xfer Library (NIXL) KV connector to let prefill nodes consume remote KV blocks, similar to how decode nodes already consume KV blocks that the prefill node provides. This is not a trivial change, because the two node types are architecturally different: prefill nodes compute KV representations for new prompt tokens and can provide complete prompt KV blocks. Decode nodes lack the capability (or efficiency) to generate complete KV representations and can only serve previously cached KV blocks. As a result, prefill and decode nodes still carry their own roles and logic for computing the metadata of remote and local KV blocks. The actual KV transfer mechanism (a NIXL READ), however, is common to both the P->D and D->P directions. The following diagram first shows the existing flow in blue. The router sends the request to prefill node, prefill node computes the prompt KV and returns its block metadata, and the router forwards the request and metadata to decode node for the standard P→D NIXL READ. The numbered red path illustrates the bi-directional KV transfer additions: (1) decode node retains eligible KV blocks after generation, (2) returns their metadata in the streaming response, and (3) the router caches that metadata for the next prefill request. Prefill node then (4) checks the remotely reusable KV after it applies local cache matches, (5) queues an asynchronous receive when the reusable token count meets the threshold, and (6) performs a D→P NIXL READ before computing the remaining suffix. Figure 1: Standard prefill-to-decode flow (blue) and the bi-directional KV transfer additions (red) KV-cache-aware router vLLM includes connector-side support for bi-directional KV transfer, while coordinating KV reuse across conversation turns requires a KV-cache-aware router. vLLM GitHub provides an example proxy that implements the flow used in this post which can be adapted for your environment. The KV-cache-aware router is a lightweight, stateful control-plane component between the client, prefill, and decode nodes. It maintains an in-memory mapping from the client-provided conversation_id to decode’s kv_transfer_params . The router stores only block-location metadata, not KV tensors, and obtains that metadata from the response stream rather than separately querying decode node. The following diagram shows how the router manages KV metadata across conversation turns. On the first turn, the cache lookup misses and the request follows the standard P→D flow. The router then saves decode’s metadata from the response stream. On later turns, the router retrieves that metadata using the conversation_id so prefill can reuse KV from decode. It then refreshes the metadata after generation for the next turn. Figure 2: KV-cache-aware router metadata flow across conversation turns Block retention TTL Normally a decode instance frees a request’s blocks as soon as generation finishes. Under bi-directional KV transfer mode, it holds them for a configurable lifetime ( decoder_kv_blocks_ttl , default 480 seconds) and publishes their location and expiry. The conversation’s KV, both prompt prefix and generated response, then stays available for the next turn. For a turn that arrives before the time to live (TTL) expires, prefill node reuses the KV. One that arrives later falls back to normal prefill. Because this TTL is fixed and not renewed, tune it to the expected time between turns to balance KV reuse against accelerator-memory usage. A natural next improvement is to extend the heartbeat-renewed lease to the decode side. A decode instance could then reclaim blocks as soon as a conversation goes idle rather than at the end of a fixed timer. KV recompute threshold KV reuse isn’t always faster than local recomputation. For a short reusable prefix, interconnect and coordination overhead can exceed the time needed to recompute the tokens. The kv_recompute_threshold setting, which defaults to 64 tokens, controls this decision. Prefill node recomputes when the token count to compute is below the threshold and pulls KV from decode node when it is at or above the threshold. Deployments can tune this value based on model computation time, KV size, and interconnect performance. Performance data For evaluation, we used Qwen3-32B and GLM-5-FP8 models