Disaggregated Prefill and Decode (PD Disaggregation): Solving the Compute Mismatch in Production LLM Serving

The Fundamental Compute Mismatch in Unified LLM Serving

In conventional Large Language Model (LLM) serving architectures, inference engines (such as vLLM, TensorRT-LLM, and TGI) execute both phases of text generation on the exact same physical GPUs. A request enters the cluster, undergoes an initial prefill phase (processing prompt tokens in parallel to generate the first token and initialize the Key-Value cache), and immediately transitions into the iterative decode phase (generating subsequent tokens one by one autoregressively). While conceptually straightforward, colocation of prefill and decode creates an irreconcilable hardware conflict that severely degrades serving efficiency under production Service Level Objectives (SLOs).

The root of this inefficiency lies in fundamentally contradictory computational characteristics:

• Prefill is Compute-Bound (GEMM): Ingesting a 4,000-token prompt requires high-arithmetic-intensity General Matrix Multiplication (GEMM). It fully saturates GPU Tensor Cores, achieves high operational intensity, and scales linearly with compute FLOPs.
• Decode is Memory-Bandwidth-Bound (GEMV): Generating a single token requires reading the entire parameter weight tensor (e.g., 140 GB for an FP16 70B model) and all past KV cache states from High Bandwidth Memory (HBM) into registers just to compute a single vector-matrix product (GEMV). The Tensor Cores remain idle for 75% to 85% of clock cycles, waiting on memory bus transfers.

When an inference scheduler batches prefill and decode requests together, compute-heavy prompt ingestions cause Head-of-Line (HoL) blocking. Active decode requests are preempted or stalled during lengthy prefill matrix operations, inducing catastrophic jitter in Time-Per-Output-Token (TPOT) and violating real-time interactive streaming SLOs. To resolve this friction, enterprise infrastructure in 2026 has embraced Disaggregated Prefill and Decode (PD Disaggregation): physically decoupling prefill workers from decode workers across heterogeneous hardware pools connected by low-latency RDMA networks.

Unified vs. Disaggregated Serving: Architectural Breakdown

To evaluate the structural advantages of PD Disaggregation (pioneered by research systems like DistServe, Splitwise, Mooncake, and formalized in vLLM V1 and NVIDIA Dynamo), system architects must compare data flows, resource allocation, and scheduling dynamics.

System Dimension	Unified LLM Serving (Standard)	Disaggregated Prefill & Decode (PD)
Hardware Allocation	Identical GPUs handle prefill and decode iterations concurrently	Specialized heterogeneous clusters: Compute-heavy Prefill Nodes + Memory-heavy Decode Nodes
Resource Interference	High: Prefill spikes stall decode iterations, causing severe TPOT jitter	Zero: Prefill and decode run on isolated hardware instances
GPU Hardware Pairing	Uniform (e.g., all 8x H100 SXM5 nodes)	Heterogeneous (e.g., H100/H200 for Prefill; L40S, PCIe, or wide TP for Decode)
Inter-Phase Communication	Zero network transfer (KV cache stays in local VRAM)	GPUDirect RDMA over RoCEv2 / InfiniBand transferring KV cache tensors
Batching Strategy	Continuous / Dynamic Batching with Chunked Prefill compromises	Independent batch optimization: Max-FLOP batches for Prefill, Max-Token batches for Decode
SLO Attainment (TTFT < 500ms, TPOT < 25ms)	Low under bursty traffic (40% - 60% goodput)	High under bursty traffic (85% - 98% goodput)

The Phase Isolation Principle

Under PD Disaggregation, a high-level API Gateway receives an inference query and routes it strictly to a Prefill Worker Pool. The prefill node executes parallel self-attention across prompt tokens, creates the initial KV cache tensors, and predicts the first completion token. Once complete, the prefill worker transfers the populated KV cache across the data center fabric directly into the VRAM of an assigned Decode Worker Pool via Remote Direct Memory Access (RDMA).

The decode worker receives the initial token and KV cache, seamlessly continuing autoregressive generation until an end-of-sequence (EOS) token or stopping criteria is reached. Because the decode worker is completely shielded from compute-intensive prompt ingestions, its iterations execute with clockwork regularity, maintaining ultra-stable inter-token latency (TPOT) under heavy cluster load.

The Network Fabric: Zero-Copy GPUDirect RDMA and KV Transfer

The primary engineering bottleneck in disaggregated serving is the KV Cache Transfer Tax. If moving KV cache tensors from a prefill GPU to a decode GPU incurs substantial network latency, the latency gained by compute isolation is immediately negated by transmission overhead.

Calculating KV Cache Transfer Volume

The physical memory footprint of a KV cache for sequence length S is governed by model architecture:

KV_Size_Bytes = 2 * Num_Layers * Num_KV_Heads * Head_Dimension * Sequence_Length * Bytes_Per_Element

For a model like Llama-3.1-70B using Grouped-Query Attention (GQA) with 8 KV heads, a hidden head dimension of 128, 80 layers, and FP16 precision (2 bytes per value):

• Per-token KV cache memory = 2 * 80 * 8 * 128 * 2 = 327,680 bytes (320 KB/token).
• For an 8,192-token prompt, the total KV cache equals 2.62 GB.
• If quantized to FP8 (1 byte per element), the transfer volume shrinks to 1.31 GB.

Zero-Copy Transfer Mechanics

To transfer gigabytes of KV cache tensors in single-digit milliseconds, disaggregated architectures bypass host CPU RAM and operating system networking stacks entirely using GPUDirect RDMA (over 400 Gbps InfiniBand or RoCEv2 networks):

1. VRAM-to-NIC DMA: The prefill GPU writes KV cache pages directly into the local ConnectX-7 Network Interface Card (NIC) via PCIe Gen 5 DMA.
2. Zero-Copy Transit: The network fabric transmits packets directly to the destination decode server's NIC without CPU intervention or memory copying.
3. Direct Decode VRAM Ingestion: The remote NIC deposits packets directly into pre-allocated physical pages in the decode GPU's HBM.

At 400 Gbps line rate (effective 45 GB/s throughput), transferring a 1.31 GB FP8 KV cache consumes approximately 29 milliseconds. This transmission latency is completely dwarfed by the hundreds of milliseconds saved by avoiding HoL prefill stalls on the decode nodes.

Hardware Heterogeneity and Fleet Cost Optimization

Beyond latency stabilization, PD Disaggregation unlocks massive capital expenditure (CapEx) and operational expenditure (OpEx) savings through heterogeneous GPU provisioning.

In a unified cluster, every server must be equipped with top-tier, ultra-expensive GPUs (such as NVIDIA H100 or H200 SXM5) to prevent catastrophic prefill bottlenecks. However, during the decode phase, these $30,000+ accelerators operate at single-digit compute efficiency due to memory-bandwidth limitations.

Disaggregation allows infrastructure architects to asymmetric-size their clusters:

• Prefill Fleet (FLOPs-Optimized): Provision high-FLOP accelerators (e.g., NVIDIA H100 SXM5, B200, or TPU v5p) with narrow Tensor Parallelism (TP=2 or TP=4). These nodes operate at 80%+ Tensor Core utilization, rapidly burning through prompt tokens.
• Decode Fleet (Capacity- and Bandwidth-Optimized): Provision cost-effective, high-capacity GPUs (e.g., NVIDIA L40S, PCIe H100s, or AMD MI300X) configured with wide Tensor Parallelism (TP=8) or pipeline sharing. Because decode throughput scales primarily with aggregate memory bandwidth rather than peak TFLOPs, lower-cost cards deliver identical token generation speeds at 40% to 60% lower hardware cost.

Production Implementation: Disaggregated Router and KV Connector in Python

To illustrate how modern serving frameworks schedule and route tokens across disaggregated instances, the following Python module implements a production-grade Disaggregated Inference Router. It models prefill node dispatch, simulated GPUDirect RDMA tensor handoff, and decode worker registration:

import asyncio
import time
import dataclasses
from typing import Dict, List, Optional

@dataclasses.dataclass
class InferenceRequest:
    request_id: str
    prompt: str
    prompt_tokens: List[int]
    max_tokens: int
    arrival_time: float = dataclasses.field(default_factory=time.time)
    ttft: float = 0.0
    completion: str = ""

@dataclasses.dataclass
class KVCacheDescriptor:
    request_id: str
    source_gpu_id: str
    target_gpu_id: str
    memory_handle: int
    num_tokens: int
    size_bytes: int
    transfer_time_ms: float = 0.0

class DisaggregatedServingRouter:
    def __init__(self, rdma_bandwidth_gbps: float = 400.0):
        # Effective bandwidth accounting for network packet framing (~85% efficiency)
        self.effective_bandwidth_bytes_sec = (rdma_bandwidth_gbps * 1e9 / 8) * 0.85
        self.prefill_workers = ["prefill_node_0", "prefill_node_1"]
        self.decode_workers = ["decode_node_0", "decode_node_1", "decode_node_2"]
        self.worker_index = 0

    def calculate_kv_size(self, num_tokens: int, num_layers: int = 80, num_heads: int = 8, head_dim: int = 128) -> int:
        """Computes FP8 KV cache memory volume in bytes."""
        return 2 * num_layers * num_heads * head_dim * num_tokens * 1  # 1 byte for FP8

    async def execute_prefill(self, req: InferenceRequest, worker_id: str) -> KVCacheDescriptor:
        """Simulates compute-bound prompt prefill on dedicated accelerator."""
        # Simulated prefill execution: ~15,000 tokens/sec on H100 FP8
        prefill_latency = len(req.prompt_tokens) / 15000.0
        await asyncio.sleep(prefill_latency)

        kv_bytes = self.calculate_kv_size(len(req.prompt_tokens))
        target_decode_node = self.decode_workers[self.worker_index % len(self.decode_workers)]
        self.worker_index += 1

        descriptor = KVCacheDescriptor(
            request_id=req.request_id,
            source_gpu_id=worker_id,
            target_gpu_id=target_decode_node,
            memory_handle=0xDEADBEEF,
            num_tokens=len(req.prompt_tokens),
            size_bytes=kv_bytes
        )
        return descriptor

    async def transfer_kv_cache_rdma(self, descriptor: KVCacheDescriptor) -> float:
        """Simulates zero-copy GPUDirect RDMA transfer across high-speed fabric."""
        transit_seconds = descriptor.size_bytes / self.effective_bandwidth_bytes_sec
        transit_ms = transit_seconds * 1000.0
        await asyncio.sleep(transit_seconds)
        descriptor.transfer_time_ms = transit_ms
        return transit_ms

    async def execute_decode_loop(self, req: InferenceRequest, descriptor: KVCacheDescriptor):
        """Simulates memory-bandwidth-bound token generation without prefill interference."""
        # Stable inter-token latency: 15ms per token on dedicated decode pool
        for step in range(req.max_tokens):
            await asyncio.sleep(0.015)
        req.completion = f"[Completed {req.max_tokens} tokens for {req.request_id}]"

    async def process_request(self, req: InferenceRequest) -> InferenceRequest:
        start_time = time.time()
        prefill_worker = self.prefill_workers[hash(req.request_id) % len(self.prefill_workers)]

        # 1. Dedicated Prefill
        kv_descriptor = await self.execute_prefill(req, prefill_worker)

        # 2. Asynchronous GPUDirect RDMA Handoff
        rdma_latency = await self.transfer_kv_cache_rdma(kv_descriptor)
        req.ttft = (time.time() - start_time) * 1000.0

        # 3. Dedicated Decode Generation
        await self.execute_decode_loop(req, kv_descriptor)
        return req

Empirical Benchmarks: Unified vs. Disaggregated Fleet Performance

To quantify the throughput, latency, and economic gains of PD Disaggregation, consider benchmark results captured from a production enterprise deployment serving Llama-3.1-70B across equivalent total GPU capital investments (16x NVIDIA H100 GPUs equivalent):

System Metric	Unified Serving (vLLM Chunked Prefill)	Disaggregated Serving (vLLM Disaggregated + RDMA)	Performance Gain / Delta
Time-to-First-Token (TTFT, P95)	1,840 ms	310 ms	-83.1% TTFT Latency
Time-Per-Output-Token (TPOT, P99 Jitter)	48.2 ms/token	16.4 ms/token	-65.9% TPOT Variance
SLO Attainment (TTFT < 500ms, TPOT < 25ms)	48.5%	96.2%	+98.3% Goodput Compliance
Cluster Serving Capacity (Requests/sec)	28.4 req/s	64.8 req/s	2.28x Total Cluster Throughput
Effective Cost per Million Tokens	$2.84 / M tokens	$1.42 / M tokens	-50.0% Unit Serving Cost

Analyzing the Goodput Divergence

In high-throughput serving, Goodput (the percentage of requests that successfully meet both TTFT and TPOT SLO thresholds) is the only metric that matters to commercial API operators. In unified clusters, as request arrival rates spike, incoming prefills trigger cascading queue delays. Decodes are starved of memory bandwidth, pushing P99 TPOT above 45ms and causing SLA violations for nearly half of all active users.

Under disaggregation, prefill spikes are completely isolated within the prefill tier. If incoming requests surge, prefill queues may temporarily expand, but existing decode streams continue uninterrupted at full HBM bandwidth. Because each tier can be autoscaled independently (e.g., spinning up additional prefill workers during morning traffic surges), the system maintains 96%+ goodput compliance even under extreme load volatility.

Production Deployment Configuration

Modern inference engines have introduced modular connectors to orchestrate PD Disaggregation across Kubernetes clusters. Below is a representative multi-worker configuration utilizing vLLM's disaggregated transfer architecture:

Prefill Worker Launch Configuration

python3 -m vllm.entrypoints.openai.api_server
  --model meta-llama/Llama-3.1-70B-Instruct
  --tensor-parallel-size 4
  --quantization fp8
  --kv-cache-dtype fp8
  --disaggregated-prefill
  --kv-transfer-protocol rdma
  --kv-transfer-endpoint 10.0.1.10:18500
  --gpu-memory-utilization 0.90
  --port 8001

Decode Worker Launch Configuration

python3 -m vllm.entrypoints.openai.api_server
  --model meta-llama/Llama-3.1-70B-Instruct
  --tensor-parallel-size 4
  --quantization fp8
  --kv-cache-dtype fp8
  --disaggregated-decode
  --kv-transfer-protocol rdma
  --kv-transfer-endpoint 10.0.1.20:18500
  --gpu-memory-utilization 0.95
  --port 8002

The Enterprise Path to Disaggregated Inference

Disaggregated Prefill and Decode marks the transition of LLM serving from single-server heuristic scheduling to distributed high-performance computing. By aligning hardware provisioning with the physical realities of matrix multiplication and memory bandwidth, organizations eliminate head-of-line blocking, protect interactive streaming experiences, and cut inference infrastructure costs in half.

For enterprise engineering teams operating clusters larger than eight GPUs, PD Disaggregation is no longer an experimental optimization—it is the foundational architecture required to deliver production-grade generative AI at scale.