EPD Serving Saves GPU Energy, but Slower Prefill Can Erase the Downclocking Benefit
MULTIMODAL SERVING AND GPU ENERGY
A three-L40S study separates lower watts from lower joules, and shows why first-token queues and strict latency budgets still decide which settings are usable.
By Ossam Hashim
KEY FINDINGS
- A Matter of Stages compares three fused replicas with one encoding, one prefill and one decode GPU. Across 21 paired operating points, separation reduces GPU energy per successful request by 13.3–37.0%, with output-token throughput differing by less than 0.7%. Mean time to first token rises at every compared point.
- Uniform downclocking to an 80% target saves 11.7–18.6% of three-GPU energy across 18 workloads, relative to the default-frequency disaggregated pipeline. These savings are not whole-server electricity savings, and small changes in mean latency do not prove that tail-latency budgets hold.
- Aggressive prefill downclocking can save little energy despite lower power. In one Qwen3-VL workload, aggregate power falls from 766.4 to 702.4 watts while benchmark duration grows from 672 to 728 seconds. Reported energy falls just 0.7%, and mean first-token latency rises from 0.565 to 27.42 seconds.
- Four configuration probes match the measured energy oracle in all 288 positive-margin latency scenarios. That is a result inside a restricted 25-configuration search space. With zero-margin budgets, even six probes find only 76 of 107 scenarios known to contain a feasible configuration.
A Lower Power Reading Is Not the Objective
A multimodal serving pipeline can look more efficient on a power dashboard while making users wait longer and doing almost nothing for energy per completed request. The November 16 preprint A Matter of Stages: Energy-Latency Trade-offs in Multimodal LLM Serving, by Yifan Sun and colleagues, measures that distinction across encoding, prefill and decode rather than treating each GPU as an isolated optimization target.
The useful question is not whether one stage draws fewer watts after its clock is reduced. It is whether the complete service uses fewer joules to finish equivalent work within an acceptable latency budget. A slower upstream stage can keep the entire pipeline alive longer. Downstream GPUs still draw power while waiting, and the queue can continue draining after the final request arrives.
This is also why disaggregation and downclocking need separate baselines. Replacing three complete replicas with a dedicated three-stage pipeline changes where work runs. Reducing clocks within that new pipeline is another intervention. The paper reports benefits for both in its tested settings, but the percentages cannot simply be added or treated as one measured end-to-end deployment saving.
What the Three-GPU Comparison Actually Holds Constant
The experimental server contains four 48 GB NVIDIA L40S GPUs, but only three perform inference and enter the serving-energy calculation. In the fused setup, a round-robin proxy distributes requests among three independent replicas. Each replica performs visual encoding, prompt prefill and token decoding on its own GPU. In the separated setup, the same three-GPU budget becomes one encoder, one prefill worker and one decode worker.
Visual embeddings travel from encoding to prefill, and the resulting key-value state travels from prefill to decode. The testbed uses PCIe-attached GPUs with the NIXL/UCX communication stack. It is not a measurement of a large NVLink fabric, a geographically distributed deployment or an independently autoscaled production fleet.
Both deployments use vLLM 0.30.0, eager execution and disabled prefix caching. Four image-text models enter the deployment comparison: InternVL3-9B, Llama-3.1-Nemotron-Nano-VL-8B-V1, Qwen3-VL-8B-Instruct and LLaVA-1.5-7B. Their visual encoders, image tiling and KV-cache demands differ. Those differences are part of the explanation, not incidental model names.
Across 21 paired points on VisionArena-Chat, disaggregation reduces GPU energy per successful request by 13.3–37.0%. Output-token throughput differs by less than 0.7%. This makes the result stronger than comparing one fused GPU with three separated GPUs, but it does not establish equal user experience: the first-token and subsequent-token latency results diverge.
The First Token Pays for Handoffs and Concentrated Queues
Mean time to first token, or TTFT, increases at every compared operating point. Qwen3-VL rises from 0.128 to 0.274 seconds at four requests per second and from 0.196 to 1.159 seconds at 20 requests per second. At the highest tested loads, InternVL3 rises from 0.355 to 3.024 seconds and Nemotron from 0.343 to 1.024 seconds.
The request now crosses multiple workers before the user sees output. Transfers and coordination add work to that path. The resource layout changes too: each dedicated stage receives the full arrival stream, whereas three fused replicas divide requests among three complete paths. Spare capacity on the encoder cannot automatically process a backed-up prefill queue.
A lower energy number can therefore coexist with a worse interactive experience. An offline chart-analysis queue might accept that exchange; a conversational product with a tight first-response deadline might not. Those are application choices that need explicit latency targets, not a universal recommendation to separate or fuse stages.
Decode Isolation Helps Until Decode Contention Wins
Time per output token, or TPOT, tells a different story. Removing encoder and prefill work from the decode GPU reduces direct interference. Nemotron at five requests per second improves mean TPOT from 37.95 to 26.49 milliseconds, a reported 30.2% reduction. InternVL3 improves by 29.7% at its corresponding five-request load.
LLaVA is the counterexample that prevents a simple isolation-is-always-faster conclusion. Its encoding is relatively light, while its language backbone has a large per-token KV footprint. Concentrating all decoding on one GPU adds cache and execution pressure without removing as much visual-processing interference. TPOT worsens at every tested request rate.
At 12 requests per second, LLaVA's decode-side sampled KV-cache utilization reaches 100%, with 18 preemptions and a 73.2% TPOT increase. Even before preemptions, the penalty grows from 8.4% at two requests per second to 51.9% at ten. The deployment decision depends on what interference disappears and what new bottleneck replaces it.
Energy Accounting Must Include the Queue Drain
The authors sample board power for all three serving GPUs once per second, integrate over the benchmark and divide by successfully completed requests for the per-request metric. The measurement continues until submitted requests finish. Waiting intervals and the final queue drain belong in the energy calculation rather than being discarded as non-compute time.
This is GPU-board accounting, not a wall-plug measurement of the server. Host CPUs, DRAM, storage, cooling and other facility costs are outside the reported three-GPU boundary. NVIDIA's NVML power-usage documentation describes readings for a GPU and its associated circuitry, including memory; it does not turn that reading into a complete machine-energy meter.
That boundary matters especially when communication or CPU-side media handling changes. A lower sum of GPU energy can be useful evidence without proving an equal reduction in an electricity bill or carbon emissions. Production measurement should state the boundary and retain the same one across candidates, ideally adding node-level energy when the deployment decision concerns total operating cost.
Why Encoding Is Often the Wrong Place to Chase the Last Saving
The frequency sweep uses six targets from 90% down to 40%, referenced to the L40S maximum frequency of 2,520 MHz. It changes core frequency while leaving memory frequency at its default. E80 means only the encoder receives the 80% target; Uniform80 applies it to all three stages. These are target settings, not promises that every observed clock or performance metric scales linearly.
For Nemotron on VisionArena at two requests per second, E40 reduces encoder energy from 91.7 to 75.2 kilojoules, an 18.0% local reduction. Yet total GPU energy falls by just 4.5%. At default, prefill and decode together account for most of the pipeline's energy. A meaningful local improvement can remain a small global improvement.
The marginal return also shrinks. For Qwen3-VL at ten requests per second, moving the encoder from E80 to E40 lowers total energy from 445.7 to 442.1 kilojoules, only another 0.8%. The appendix shows the same broad pattern across its six workloads: E80 saves 3.2–3.7%, while E40 saves 3.8–4.3%. Deeper encoding cuts are not a free route to large additional whole-pipeline savings.
Prefill Has a Queueing Cliff, Not a Smooth Discount Curve
Prefill becomes dangerous when a lower processing rate approaches the arrival rate. On Qwen3-VL with VisionArena at 15 requests per second, P70 is the lowest tested prefill-only setting that satisfies both latency budgets in all three repetitions. P60 pushes P99 TTFT to 2.13–2.17 seconds, above the two-second budget.
At P40, mean TTFT jumps from 0.565 to 27.42 seconds. Logged peak waiting requests rise from 17 to 775. Combined GPU power falls from 766.4 to 702.4 watts, but the benchmark lengthens from 672 to 728 seconds. The reported total energy change, 515.0 to 511.3 kilojoules, is only a 0.7% saving. Lower watts conceal a severe service-quality loss.
Matching requests per second does not match prefill work. At the same 15-request rate and P40 setting, Qwen3-VL on Chart-to-text has mean TTFT of 0.767 seconds rather than VisionArena's 27.42 seconds. Image processing, visual-token count and text context change the demand per request. A frequency configuration calibrated on one request mix can fail when the traffic mix changes even if the request counter stays flat.
Moderate Downclocking Looks Useful, but Means Are Not SLO Proof
Across 18 combinations of three models, two datasets and three request rates, Uniform80 saves 11.7–18.6% of GPU energy against the default-frequency disaggregated baseline. Mean TTFT changes range from -1.1% to +2.7%, and mean TPOT increases by 1.4–4.3%. Decode-only D80 saves 8.6–13.0%, with mean TPOT increases of 1.2–4.1%.
These results make moderate configurations sensible starting probes. They do not make 80% a deployment prescription. The study's feasibility test uses P99 TTFT and mean TPOT, checked in every valid repetition. A small mean-TTFT change is a different statistic from a passing P99 constraint.
Nor is the lowest frequency necessarily the lowest-energy point. For Qwen3-VL on Chart-to-text at 20 requests per second, Uniform50 uses 415.1 kilojoules. Uniform40 lowers power further but stretches runtime enough to use 423.5 kilojoules, a 2.0% increase, while mean TTFT rises from 1.00 to 31.69 seconds. The energy curve and the feasible operating region must both be measured.
Four Probes Match a Measured Oracle, Not Every Possible Configuration
The paper turns its stage observations into a budgeted search. Default is measured first, followed by Uniform80. A bisection-like heuristic moves among frequency levels using measured energy improvement and latency feasibility. It can switch to decode-only or encoder-only exploration when uniform settings violate an SLO. Default and infeasible measurements both consume the probe budget.
The search space has 25 candidates: Default plus six frequencies for each of four modes. It excludes arbitrary combinations such as independently lowering encoding and decode by different amounts while leaving prefill unchanged. The oracle is simply the lowest-energy feasible configuration within those measured candidates. It is not a proof of global optimality across all possible controls.
Under positive margins on both latency budgets, four probes match that oracle in all 288 scenarios. Under the fixed two-second P99 TTFT and 50-millisecond mean TPOT budgets, 17 of 18 workloads contain a feasible configuration. Two probes find feasible choices for all 17; five probes are needed to close every measured oracle gap.
Zero-margin budgets expose the search's limit. Of 162 scenarios, only 107 contain any feasible candidate. Six probes find feasible settings in 76 of those 107, leaving 31 misses. Budgets are derived from Default averages, while feasibility requires every valid repetition to pass, so Default itself can fail a zero-margin threshold. The authors also identify non-monotonic feasibility as a reason the heuristic overlooks candidates.
Deployment Advice Should Preserve the Experimental Boundary
Related evidence argues against turning this preprint into a universal architecture verdict. Revisiting Disaggregated Large Language Model Serving for Performance and Energy Implications evaluates two A100 GPUs, Llama-3.2-3B and synthetic simultaneous arrivals under several KV-transfer paths. Its frequency experiment finds no energy advantage for its disaggregated settings over its colocated baselines. That is a different hardware, workload and accounting context, not a direct replication that this new result overturns.
Current vLLM engineering documentation also distinguishes E+PD from fully separated E/P/D. Its encoder-disaggregation material describes connector-based embedding transfers and separate KV transfer for prefill/decode. A November vLLM engineering post discusses metadata-only forwarding, caching and newer execution paths. The L40S paper uses vLLM 0.30.0, eager execution and no prefix cache, so its percentages should not be transplanted into those newer paths without measurement.
A useful production trial would keep topology, request replay and completion criteria explicit, then compare fused and separated layouts under the same resource budget. Inside a selected layout, start with a conservative frequency candidate, record actual clocks, measure all serving GPUs through queue drain and check the required latency percentiles across repetitions. Recheck when media mix, output length, cache behavior or arrival patterns change. The lesson is to optimize completed work under a latency budget, not the lowest power trace.
Assumptions and limitations
This is an unreviewed preprint's reported evaluation, not an independent reproduction. It tests three L40S serving GPUs in one server, fixed 1E+1P+1D allocation, selected image-text models and two workloads. Appendix D explicitly excludes alternative scales, partial fusion, mixed multi-stage frequency settings and online adaptive control. Some aggressive settings already produce request accumulation and framework crashes; 40% is the sweep's lower bound, not a certified safe floor.
The energy boundary covers the three serving GPU boards and the full measured completion interval, not total facility energy or the unused fourth GPU. Uniform80 savings are relative to the default disaggregated configuration, whereas the deployment savings compare disaggregated serving with three fused replicas. Reported means, tail constraints, successful-request denominators and valid-repetition feasibility should remain separate. The worked example uses invented constant-average-power totals and supplied latency observations solely to demonstrate arithmetic and selection rules.
Worked Example: A Smaller Power Bill per Second Can Still Cost More per Request
This invented arithmetic example is not a reproduction of the paper, a queueing simulator or a recommended clock setting. Each candidate completes 1,000 requests. Baseline uses 480 kJ, moderate uses 403.2 kJ and deep uses 504 kJ. Moderate saves 16.0%, but deep uses 5.0% more energy than baseline despite drawing less average power. A 2.0-second P99 TTFT budget and 50-millisecond mean TPOT budget reject the deep candidate. The code also checks a repetition set whose average P99 TTFT is exactly 2.0 seconds but whose worst repetition fails; averaging away that failure would change the decision.
from math import isclose
# name, watts, seconds, TTFTs, TPOTs
rows = [
("baseline", 800, 600,
(1.90, 1.95, 1.92),
(40, 41, 40)),
("moderate", 640, 630,
(1.94, 1.98, 1.96),
(42, 43, 42)),
("deep", 560, 900,
(8.0, 8.5, 8.2),
(48, 49, 48)),
]
completed = 1000
base_j = 800 * 600
choices = []
for row in rows:
name, watts, secs, tt, tp = row
joules = watts * secs
good = (max(tt) <= 2.0
and max(tp) <= 50)
saving = 100 * (1-joules/base_j)
per_req = joules / completed
print(name)
print("kJ:", joules / 1000)
print("J/request:", per_req)
print("Saving %:", saving)
print("Feasible:", good)
if good:
choices.append((per_req,
name))
print("Selected:", min(choices)[1])
reps = (1.9, 2.0, 2.1)
avg = sum(reps) / len(reps)
assert isclose(avg, 2.0)
print("Mean P99:", avg)
print("All pass:", max(reps) <= 2.0)
Sources & Provenance
- Sun and colleagues, A Matter of Stages: Energy-Latency Trade-offs in Multimodal LLM Serving, main sections 1–7: deployment comparison, stage sweeps, measurement boundary and budgeted-search results.
- Full v1 PDF and appendices A–D: complete model sweeps, all-valid-repetition feasibility, search procedure and explicit limitations.
- NVIDIA NVML power-usage documentation: GPU-board and associated-circuitry scope of the power reading; this is not a node-level energy measurement.
- vLLM Disaggregated Encoder documentation: E+PD and E/P/D examples, encoder-cache connectors and subsequent prefill-to-decode KV transfer.
- vLLM, Encoder Disaggregation in vLLM: Scaling Multimodal Serving with EPD, November 11: newer metadata, caching, transport and execution-path context, not an energy replication of this study.
- Li and colleagues, Revisiting Disaggregated Large Language Model Serving for Performance and Energy Implications, methods and results: contrasting two-A100 text-only benchmark and energy findings; IBM Research's publication record corroborates its scope.
All paper measurements are attributed to their authors. Interpretation, operational checks and the executable arithmetic example are original analysis. No paper figure is reproduced, and no energy or latency result is presented as a guarantee for another deployment.
Comments
Post a Comment