Two-Spark Tensor Parallelism Reaches 1.59x Median Peak Throughput, but Replicas Often Win at High Load
LLM SERVING / DISTRIBUTED INFERENCE
A 17-checkpoint GB10 study favors tensor parallelism over a verified RDMA setup, while its replica comparison exposes a different high-load choice. The result is a deployment boundary, not a promise that two systems double useful serving capacity.
By Ossam Hashim
KEY FINDINGS
- Across 13 checkpoints with all three deployments, two-way tensor parallelism reaches 1.59x the one-Spark peak output throughput at the median, versus 1.39x for pipeline parallelism. These are maxima within concurrency 1-64, not demonstrated saturation throughput.
- Tensor parallelism has higher throughput in 93 of 96 matched checkpoint-and-concurrency comparisons against pipeline parallelism using separated 95% Welch intervals. Three comparisons are undecided; none favors the pipeline split. The intervals use only three launches per deployment and no multiple-testing correction.
- Independent replicas are a derived baseline, not a measured two-server deployment. Under ideal balancing, replicas beat tensor parallelism on throughput for eleven of the thirteen complete checkpoints at total concurrency 64, while tensor parallelism usually leads at lower load.
- More tokens do not necessarily mean more usable service. At concurrency 64, every measured deployment has zero goodput under the paper's joint three-second first-token and 100 ms mean inter-token objectives, even though most throughput curves are still rising.
The network label is not the deployment decision
Buying a second compact inference system creates several distinct options. It can make a larger model fit. It can accelerate one request by splitting its layers' work. It can create a pipeline across layer groups. Or it can run another independent copy and serve different requests. These choices use the same hardware but optimize different things.
The new desktop-scale Grace Blackwell benchmark compares those choices on two ASUS Ascent GX10 systems based on NVIDIA's DGX Spark GB10 design. It evaluates 17 checkpoints, including different precision variants, rather than 17 unrelated model families. The central result is that a properly configured network can make tensor parallelism attractive across two machines, but it does not eliminate the case for replication.
The study also resists a tempting hardware headline. Across the thirteen complete checkpoints, neither split reaches twice the one-system peak throughput within the tested range. The largest reported gain is 1.68x. That is useful scaling, but it is not linear scaling and it is not automatically an improvement in energy efficiency or latency-qualified service.
The desktop-scale benchmark paper supplies the measurements discussed here. Its public benchmark repository supplies the harness and summary artifacts. This article interprets those results; it does not report an independent run, a hardware purchase test or a claim about the reader's network.
Two different ways to divide a model
Tensor parallelism, TP=2 in this study, divides computation inside each layer between the two GPUs. Each GPU reads its part of the weights concurrently, but the GPUs must exchange collective results repeatedly. For a Llama-style transformer, the paper describes roughly two all-reduces per layer per decode step. Communication latency can therefore matter even when each individual message is small.
Pipeline parallelism, PP=2, assigns different layer groups to the two GPUs. It communicates across the stage boundary rather than synchronizing every layer. That reduces the frequency of cross-machine exchanges, but the stages cannot overlap when there is only one small piece of work in flight. At a single request's decode step, the second stage waits for the first.
With more concurrent requests, the pipeline can work on different requests at different stages. With a long prompt divided into prefill chunks, it can similarly overlap different chunks. The relevant question is not simply whether pipeline parallelism communicates less. It is whether the workload supplies enough independent work to offset the idle stages and whether repeated tensor collectives are cheap enough.
GB10 adds another complication. CPU and GPU share memory, rather than using a conventional datacenter GPU's separate high-bandwidth memory pool. NVIDIA's CUDA porting guide states that GPUDirect RDMA is not supported on DGX Spark. Ordinary RDMA transport and GPUDirect RDMA are different claims. The paper uses host-registered staging buffers and reports no GPUDirect transfers, not a direct NIC-to-GPU path.
What was actually measured
The analyzed matrix contains 47 deployment curves. Thirteen checkpoints have one-Spark, TP=2 and PP=2 measurements. One Gemma checkpoint lacks TP support in the tested kernel path, and three checkpoints were evaluated only on two systems. Each curve covers concurrency 1, 4, 8, 16, 32 and 64, with three independent server launches.
That produces 846 analyzed windows across 141 launches. The workload uses four synthetic input/output buckets: 128/128, 1,024/256, 4,096/256 and 8,192/512 tokens. The last bucket has half the weight of each other bucket. Requests use greedy decoding with fixed output lengths, unique prefixes and disabled prefix caching.
The client maintains a fixed number of requests in flight and replaces each completed request immediately. This is a closed-loop experiment. It does not establish how an open service responds to an arrival burst, how a router spreads traffic across real replicas or how reusable production prompts change the value of prefix caching.
Measurement windows last at least ten minutes and extend when needed to admit twenty requests. Latency and goodput track requests admitted in the window through completion; throughput counts output tokens received during the window. Those definitions matter at low load and for long requests crossing a boundary. Some long-prompt buckets contain fewer than ten requests per window, which limits confidence in fine-grained prefill comparisons.
The software is part of the result
The campaign uses NVIDIA's vLLM 26.09 container, with vLLM 0.29.0. Maximum model length is 8,704 tokens, the memory-utilization setting is 0.80, chunked prefill is enabled and the effective batched-token budget is 2,048. FlashInfer autotuning is disabled across deployments in the study to avoid its observed startup problem.
NVIDIA's release notes independently identify vLLM 0.29.0 and FlashInfer 0.6.18.post1 in that container release. They also describe unified-memory allocation risks and model-specific sequence limits. Those notes provide version context, not proof that any setting is appropriate for a different model or release. A release default and a benchmark's deliberately chosen flags need not be identical.
The paper audits NCCL transport logs rather than assuming that a connected cable establishes RDMA use. It reports four RoCEv2 rails across two physical cables, with all audited dual-node launches using NET/IB rather than socket channels. NVIDIA's connection playbook establishes the basic network setup, but successful connectivity alone is weaker evidence than an application-level transport audit.
Two additional measured Nemotron checkpoints are excluded from the main analysis, one because a four-sequence restriction would distort higher-concurrency comparisons and another because the pipeline launches failed before measurement. The analyzed windows have no failed requests, but that does not mean every planned configuration launched successfully. Exclusions remain part of the study's scope.
Where tensor parallelism wins
Across the thirteen complete checkpoints, TP=2 reaches 1.36-1.68x the one-system peak throughput, with a median of 1.59x. PP=2 reaches 1.17-1.65x, with a median of 1.39x. Tensor parallelism has the higher peak for twelve checkpoints. The exception's pipeline peak is only 1.02x the tensor peak, and the difference at concurrency 64 is undecided.
The single-request decode result is more distinct. Tensor parallelism reduces unloaded inter-token latency to 0.52-0.73x of the one-Spark value. Pipeline parallelism stays near the original latency, at 0.99-1.03x. This fits the mechanism: both GPUs read tensor shards concurrently, while a lone decode request traverses pipeline stages sequentially.
The 93-of-96 throughput comparison uses matched concurrency, not just each deployment's chosen peak. A separated interval is directional evidence under the paper's three-launch experiment. It should not be restated as a universal probability that TP wins, a guarantee about all future software or a claim that the three undecided cases are equivalent.
The medians also do not cover every checkpoint in the same way. Three two-system-only configurations have no measured one-system baseline for the evaluated setup. Comparing a 235B model's tensor and pipeline runs is useful, but dividing by an imaginary one-system result would not be.
Pipeline parallelism has a real prefill niche
The pipeline split is not uniformly useless. Prefill is its stronger case, particularly when a prompt spans multiple chunks. At 8,192 input tokens and concurrency one, PP=2 has lower time to first token than TP=2 on ten of the thirteen complete checkpoints in the paper's discussion.
For an ideal two-stage pipeline with equal stages, a prompt divided into k chunks completes in (k+1)/(2k) of the serial chunk-processing time. One chunk gives no overlap. Four chunks produce a ratio of 0.625. This is a simple model of the mechanism, not a prediction that every measured prompt will have exactly that ratio.
Tensor parallelism can also make a short prompt worse even while improving subsequent decode. On Llama-3.1-8B NVFP4, the paper reports 128-token-prompt TTFT rising from 37 ms on one system to 83 ms under TP=2. Repeated collectives can cost more than the compute saved during a very short prefill.
Long-prompt observations need the small bucket samples stated earlier. They support testing pipeline parallelism for that workload, not selecting it solely from one aggregate prefill slope. Kernel compatibility can also make pipeline parallelism the available option even when tensor parallelism would otherwise be attractive.
Replicas change the comparison
A two-system split should be compared with two independent copies when the checkpoint fits on one machine. The paper derives that alternative from the single-system measurements. At total concurrency c, each ideal replica receives c/2 requests; aggregate throughput is twice the single-system throughput at that per-node load.
This assumes identical systems, independent performance, perfect balancing and no routing overhead. The authors do not measure an actual two-replica router. That makes the baseline a useful capacity reference, with a clearly stated idealization, rather than an observed production deployment.
At total concurrency eight, tensor parallelism has higher throughput than this baseline on ten of the thirteen complete checkpoints. At concurrency 64, replicas have higher throughput on eleven, while tensor parallelism retains the lead on gpt-oss-120b and the remaining checkpoint is undecided. The reason is straightforward: replication processes separate batches without per-layer cross-node synchronization.
Goodput gives a less one-sided answer. Among ten checkpoints with nonzero single-system strict goodput, TP is separated above replicas on four, below on four and undecided on two. Two more checkpoints achieve nonzero strict goodput only through tensor parallelism. A high-load throughput winner is therefore not automatically the latency-qualified winner.
A useful-token objective is stricter than a token counter
The strict goodput tier requires both TTFT of at most three seconds and a request's mean inter-token latency of at most 100 ms. It counts qualifying requests per second, not every generated token. The looser tier changes the inter-token limit to 200 ms while keeping the first-token requirement.
At concurrency 64, all 47 measured curves have zero strict goodput. Most still show their highest measured throughput there. Inference has not stopped: requests continue to produce output, but none meets both strict targets. Reporting only peak output throughput would hide that mismatch.
This objective is also weaker than a guarantee about every token gap. The per-request quantity is mean inter-token latency. A request can have an occasional slow gap while its average remains under the threshold. The study's median-of-request-means plots are not P99 token-latency measurements.
A deployment plan should therefore name its objective before choosing concurrency. Interactive streaming, queued batch processing and long-prompt analysis can legitimately choose different points on the same curves. The paper supplies evidence for those choices, not a universal best configuration.
Energy and precision pull in different directions
At matched concurrency across the thirteen complete checkpoints, one system uses less reported GPU energy per generated token than either split in every corresponding comparison. At concurrency 64, median energy per token is 1.18x the one-system value for tensor parallelism and 1.19x for pipeline parallelism. Faster output can still cost more energy per token when a second GPU is active.
These are device-power-derived GPU energy measurements. They exclude CPU, network equipment, power-supply losses and other system components. They do not establish total wall-plug efficiency, electricity bills or environmental cost. Choosing a lower-energy GPU curve also does not establish that it meets a given throughput requirement.
Quantization supplies another option before adding hardware. Across the three precision ladders, one-system BF16-to-NVFP4 changes reduce unloaded latency more than adding a second system at BF16. But the paper does not evaluate model quality, and some checkpoints also change KV-cache precision. It is a comparison of released checkpoints as served, not a clean weight-bit-only ablation.
The kernel path matters too. Qwen3.6-27B NVFP4 improves decode latency versus FP8 without improving peak throughput, while prefill becomes slower. Its weight-only Marlin path is consistent with that tradeoff, but the authors do not establish kernel choice as the sole cause. Lower bits are not a promise of faster end-to-end service.
Why earlier results can point the other way
StorageReview's independent cluster review reports pipeline parallelism beating tensor parallelism for gpt-oss-120b at high batch size, including a 2.20x throughput ratio in its equal-input/output workload at batch 128. It also reports tensor advantages for Llama at lower load. That is important counterevidence to any claim that one split always wins on Spark hardware.
It is not a matched contradiction of the new study. Workload mix, concurrency range, software, cabling and unspecified transport differ. The new paper's own older TCP campaign also favors pipeline parallelism, while its RDMA campaign favors tensor parallelism. NCCL microbenchmarks make lower collective overhead a plausible contributor to that reversal.
Other changes between those campaigns prevent assigning the entire gain to RDMA. The paper explicitly notes software, worker-node, cabling and measurement differences. A rigorous operational takeaway is to record the actual transport and remeasure the intended workload, not to treat an Ethernet speed label as a causal explanation.
Assumptions and limitations
All hardware results are the study authors' measurements or, where explicitly identified, StorageReview's separate tests. Only the small invented goodput calculation was executed for this article. No GPU campaign, network setup, model-quality evaluation or benchmark reproduction was performed.
The evidence covers two GB10 systems, one serving-stack configuration, fixed synthetic output lengths and concurrency no higher than 64. Most throughput curves are still rising at the boundary. Three-launch intervals, small long-prompt buckets, excluded configurations and the idealized replica baseline all constrain generalization. Public summaries and configuration artifacts are available; raw request and power logs are described as available from the corresponding author rather than all being in the repository.
The practical decision sequence is to check model fit and acceptable precision, define first-token and streaming objectives, verify real communication transport, then compare one system, a split and replicas under the same request mix. The paper makes that comparison more concrete. It does not remove the need to measure the service being deployed.
Worked Example: A busier server can have less goodput
This invented standard-library example uses four fictional completed requests per configuration in a ten-second window. Each pair is TTFT in seconds and mean ITL in milliseconds. Configuration B completes more output tokens, but only one request meets both latency bounds, versus three for A. The output should be A: 100 tok/s and 0.3 qualifying requests/s; B: 140 tok/s and 0.1 qualifying requests/s. These numbers are invented and are not taken from the GB10 benchmark.
from math import fsum
window = 10
runs = {
"A": (1000, [
(1, 80), (2, 90),
(2, 95), (4, 90)]),
"B": (1400, [
(1, 95), (2, 110),
(4, 90), (4, 120)])}
for name in runs:
tokens, reqs = runs[name]
ok = fsum(t <= 3 and i <= 100
for t, i in reqs)
print(name, tokens / window,
ok / window)
Sources & Provenance
- Desktop-scale benchmark paper: full main text and Appendices A-C; protocol, measured results, confidence intervals, exclusions and limits.
- Public benchmark repository: harness behavior, pinned configuration, campaign artifacts and reproducibility scope; not an independent reproduction.
- NVIDIA CUDA porting guide: unsupported GPUDirect RDMA and the distinction from host-buffer RDMA transport.
- NVIDIA connection playbook: basic direct-network setup; connectivity is not proof of a serving application's selected transport.
- NVIDIA vLLM 26.09 release notes: container-version context, unified-memory concerns and model-specific sequence restrictions.
- StorageReview cluster review: separate workloads and pipeline-favorable results, retained as counterevidence rather than merged into the paper's matrix.
The deployment interpretation and fictional goodput example are original analysis. No first-hand hardware test, professional credential, price or new model-performance measurement is claimed.
Comments
Post a Comment