Header Ad Placement

Decode Token Parallelism on Eight H100 Nodes: Up to 1.53x Long-Context DCP Throughput with One Weight Copy, 1.7x with Two

DISTRIBUTED INFERENCE

A decode pool can spread request-local attention across nodes while keeping dense tensor-parallel collectives within NVLink. The extra capacity reduces admission queues, but small workloads still favor replicas and the highest DCP comparison uses a second distributed weight copy.

By Ossam Hashim

KEY FINDINGS

  • Decode Token Parallelism, abbreviated TKNP by the authors, places one model copy in stationary layer blocks across a decode pool. Each request's complete KV history for all layers stays on its assigned node, with heads sharded across that node's GPUs.
  • The current block's owner runs dense projections and MLPs for the batch; all request owners compute their own attention. Only current-token activations and block-boundary states cross nodes during decoding, without cross-node merging of partial attention outputs.
  • On eight decode nodes, single-copy TKNP reaches up to 1.53x the completion throughput of the tested DCP baseline. Optional dual-active-root scheduling raises the comparison to 1.7x, but stores two distributed model copies. The headline does not describe a universal single-copy gain.
  • Data-parallel replicas win when workloads fit their KV capacity. The evaluation targets long-context, high-concurrency dense models on H100/InfiniBand systems, with matched decode GPU counts but a separate fixed prefill pool.

Dense Layers and Attention Want Different Resource Splits

Susav Shrestha, Brad Settlemyer and Narasimha Reddy, from NVIDIA and Texas A&M University, propose Decode Token Parallelism: Memory-Efficient Distributed Attention Parallelism for Scalable LLM Decoding. Their starting point is a mismatch within a decode step: dense layers process newly generated tokens, while attention repeatedly reads every active request's accumulated history.

A larger batch amortizes dense weight reads. It also increases KV-cache capacity and bandwidth needs. In the authors' single-layer Llama 3.1 405B measurements at 32K context, attention becomes nearly 70% of summed component latency at batch 128. Extending tensor parallelism beyond the model's KV-head count does not keep distributing unique history: standard TP begins replicating KV heads instead.

Data parallelism avoids cross-node coordination by keeping independent replicas, but repeats the weights. Sequence-sharded decode context parallelism, or DCP, recovers KV capacity while requiring partial attention results to be combined. In the tested vLLM configuration, expanding DCP also expands the tensor-parallel group, pushing dense collectives onto the inter-node network. TKNP changes that coupling rather than simply making those collectives faster.

The Weight Owner Rotates; the Request Owner Does Not

The base design distributes contiguous layer blocks across nodes. Block boundaries balance weight bytes, including endpoint parameters and precision differences, so nodes retain similar space for KV cache. Equal numbers of layers are not necessarily equal memory budgets.

For a layer, its weight-owning node becomes the active root. Its local tensor-parallel group computes query, key and value projections for the active batch, then scatters projected tensors to request owners. Each owner applies rotary position processing, appends its new KV entries and computes attention against its own history. Complete attention outputs return to the root, which runs output projection, residual and normalization work and the MLP.

The root keeps executing until its layer block ends, then passes hidden and residual activations to the next block's owner. Weights remain stationary. A request also keeps one owner until completion rather than migrating KV on every layer. Attention spans heads within a node and requests across nodes, while dense all-reduces remain within the node's NVLink domain.

This is not sequence sharding across the whole pool. One request's full history for the local head partition must fit within its owner's node. The scheduler checks capacity for input and expected output growth; if no node qualifies, the request remains queued. A hybrid that also shards a single request's sequence is future work, not a capability established by the base design.

Smaller Output Tensors Are Not the Same as Total Traffic Savings

TKNP trades partial-result collectives for scatter/gather of projected current-token data. Under balanced assignment, each owner processes roughly one divided by the number of nodes of the batch. Its returned attention tensor is correspondingly smaller than a full-batch DCP partial-output tensor.

For the paper's batch-128 example on four TP8 nodes, each remote owner's returned output is 128 KiB versus a 512 KiB DCP partial output. But TKNP also dispatches 144 KiB of projected QKV to that owner. DCP exchanges normalization statistics and has its own query and collective communication. The 75% reduction is an individual output-tensor comparison, not a measured 75% reduction in all network bytes.

The volume of these current-token exchanges does not grow with cached context length. Local attention work still does. That distinction makes the design attractive when long histories dominate, without promising that communication disappears or that every context length crosses over at the same batch.

More KV Capacity Changes Who Gets to Run

The authors report that a TP8 replica of quantized Llama 3.1 405B allocates 454 GiB for the model, leaving only 106 GiB of KV space under their measured runtime budget. A 128-request, 32K workload needs 2,016 GiB of BF16 KV before output growth. Repeating weights on every node leaves many submitted requests waiting for later execution waves.

Single-copy TKNP, DCP and pipeline parallelism preserve roughly comparable aggregate KV capacity. At eight nodes, their usable capacity is approximately five times that of data-parallel replicas in this measured configuration. This is a capacity ratio after model and runtime allocations, not a claim that KV tensors become five times smaller.

The DCP comparison therefore matters: TKNP's advantage there is not simply more cache memory. It also changes how distributed execution uses that memory. Conversely, against replicas, both admission capacity and execution efficiency affect the outcome. A larger resident batch can improve total completion while making the already-admitted requests' token intervals look worse than those in a smaller replica batch.

Read Completion Throughput from Workload Release

The experiments use four or eight decode nodes, each with eight 80GB H100 GPUs, plus two fixed prefill nodes. Nodes have eight 400 Gb/s InfiniBand links. Models are Llama 3.1 405B and Nemotron-4 340B, with FP8 weights except selected BF16 parameters and BF16 KV. The implementation uses vLLM with FlashAttention-3.

Fixed-length requests generate 1,024 output tokens. Completion throughput divides all output tokens by elapsed time from workload release until the final request finishes, including admission waiting, execution waves and drain. It is a decode-pool measurement, not a complete prefill-to-answer service benchmark or a total-cluster GPU-efficiency comparison. Prefill configuration stays fixed to isolate decode changes.

On the tested workloads, TKNP reaches up to 2.06x replica throughput; Nemotron's smaller weight footprint gives replicas more KV room and narrows the corresponding maximum to 1.54x. At low concurrency where requests fit in replica memory, DP wins because it avoids per-layer scatter/gather. That negative result belongs in the decision, not in a footnote after a universal speedup claim.

Doubling the decode pool from four to eight nodes for the same 128-request, 32K workload raises single-copy TKNP throughput by 1.81x versus 1.37x for DCP. The authors explicitly include reduced admission waves in that gain. It is not a clean isolated measurement of faster attention kernels.

Two Active Roots Add Overlap and Another Weight Copy

For larger pools, optional scheduling splits the batch into two microbatches with distinct active roots. Workers alternate attention work to fill gaps during the other root's dense computation. Each layer block is stored twice across the pool, so every node holds roughly two divided by the number of nodes of the model weights instead of one divided by that count. KV ownership remains fixed and KV itself is not replicated.

The reported gain is up to 11.1% over the base schedule, taking the eight-node DCP comparison from at most 1.53x to 1.7x. More weight storage reduces KV capacity; smaller dense batches, scheduling and synchronization also cost time. In smaller memory-constrained pools, the authors favor single-copy execution. The overlap option is a tradeoff, not free parallelism.

A mixed-length trace drawn from L-Eval and LongBench ranges from about 8.2K to 126.8K input tokens. Under its tested arrivals, TKNP retains a throughput advantage. Its SLO is arrival-to-completion time divided by output length, at most 100 milliseconds per output token, including queueing. That is not an inter-token-latency threshold. Prefix sharing is disabled throughout, and Nemotron's extended contexts use linear rotary scaling for performance characterization rather than a claim of preserved long-context accuracy.

The practical question is which resource is constraining the decode pool: weight replication, request admission, attention reads or coordination. Our earlier prefill-decode disaggregation analysis gives context for the fixed external prefill service. TKNP addresses the next placement problem inside the decode pool, with results tied to dense models, the tested DCP implementation and the H100 network hierarchy.

Assumptions and limitations

This article analyzes one primary paper, not an independent benchmark reproduction. Gains are workload maxima, not typical or guaranteed improvements. The evaluation uses two dense models, fixed precision choices, H100 nodes and InfiniBand; larger NVLink domains can change the tradeoffs. Small batches and short contexts can lose. Single-request history is not sequence-sharded across nodes in base TKNP. Matched decode GPUs exclude the separate fixed prefill pool, prefix sharing is off, and the dual-root option adds a second distributed weight copy. The toy budget is invented.

Worked Example: Capacity Ratios Depend on Weight Placement

This invented memory budget has four nodes with 100 units each and a 60-unit model. Replicas leave 160 units for KV in total; one distributed copy leaves 340, and two leave 280. The ratios are 2.125x and 1.75x, not the paper's measured fivefold capacity result. Runtime workspaces and fragmentation are deliberately omitted.

from fractions import Fraction

n, cap, weights = 4, 100, 60
replica = n*(cap-weights)
one = n*cap-weights
two = n*cap-2*weights
print(replica, one, two)
print(float(Fraction(one, replica)))
print(float(Fraction(two, replica)))
assert (replica, one, two) == (
    160, 340, 280)

Sources & Provenance

  • Shrestha, Settlemyer and Reddy, "Decode Token Parallelism: Memory-Efficient Distributed Attention Parallelism for Scalable LLM Decoding" - primary paper, Sections 3-6 and Figures 5-12.
  • Section 4.4 supports tensor-size rather than total-network-traffic comparisons; Sections 5.1-5.4 define hardware, completion timing, baselines and the optional replication tradeoff.
  • Related GlobalTechDigest analysis: prefill-decode disaggregation analysis.

Written by Instinct for GlobalTechDigest. Analysis and toy code are original; measured results belong to the cited authors.

Comments

Kubernetes Liveness Is Not Readiness: Why a Health Check Can Amplify an Outage

KUBERNETES FAILURE RECOVERY A failed health check is an observation. Restarting a container and withholding traffic are different responses, and choosing the wrong one can amplify an outage. By Hossam The important question is what failure should trigger A web application loses access to its shared database. Its processes are still running, but requests that require database access fail. A health endpoint reports the dependency problem. Should Kubernetes restart every application container, or stop directing those requests to it? The endpoint alone cannot choose the right recovery action. Kubernetes separates startup, liveness and readiness probes because they answer different operational questions. Startup protects initialization, liveness can trigger container termination and restart handling, and readiness controls eligibility for traffic from matching Services. A green check is meaningful only within the check and response that were designed. This is a documentation-based de...

Editorial Governance & Pages