When a 12.3x TTFT Gap Hides a 4.6% Throughput Difference: Lessons from a One-L4 Study
LLM SERVING AND BENCHMARK DESIGN
A controlled comparison of older vLLM and SGLang releases shows why first-token latency, queue growth and cache design cannot be read as the same result. Its strongest lesson is a measurement protocol, not a ranking of today's engines.
By Ossam Hashim
KEY FINDINGS
- In a one-L4 study of vLLM 0.6.3 and SGLang 0.4.1, matching chunked prefill reduces a throughput ratio from 1.22x to 1.046x at one tested load. The reported 78% gap removal uses logarithms; it is not a 78% throughput increase.
- At that matched load, the median time-to-first-token ratio remains 12.3x. One engine falls behind arrivals while the other roughly keeps pace, so a small capacity difference accumulates into a large waiting-time difference.
- Matching does not erase every advantage. SGLang retains significant p95 first-token advantages of 2.44x and 6.43x in two other comparisons with similar achieved throughput. Request ordering is a hypothesis, not an isolated cause.
- These are authors' measurements on one GPU, one model and old releases, not our tests or a current engine leaderboard. Current vLLM V1 documentation says chunked prefill is enabled by default whenever possible.
The Number on the Chart Includes a Queue
A first token arriving twelve times later does not mean that a GPU executes the same prompt twelve times more slowly. Time to first token, or TTFT, starts when the client sends a request. It includes admission waiting as well as prompt processing. Once arrivals outstrip service, the waiting component can dominate.
Muhammad Faizan Anwar and Sina Keshvadi examine that distinction in A Controlled Experimental Evaluation of Configuration and Cache Architecture in LLM Serving Engines. Their experiment changes a scheduling setting while leaving another engine unchanged as a control. That makes the paper useful for engineers evaluating benchmark claims, even though its specific software pairing is no longer the pairing a current default deployment would use.
What Was Actually Compared
The authors serve Llama-3.1-8B-Instruct in bfloat16, with a 16,384-token context, on one NVIDIA L4 with 24 GB of memory. They compare vLLM 0.6.3's V0 engine with SGLang 0.4.1. Prefix caching is enabled in both. Autoscaling is disabled and each engine runs as one replica.
Their synthetic agent graph contains a planner, four workers and an aggregator. Workers wait for the planner and simulated tool delays; the aggregator waits for the workers. Prompts contain an exact shared prefix and a unique suffix. Bursty graph arrivals and fixed random seeds let the authors send paired traces to both engines. Across the study they report 99 runs and 63,288 measured requests.
This is a cache-and-scheduling experiment, not a coding-agent accuracy evaluation. Random token prompts provide exact control over sharing, but omit natural prompt-length distributions, near-duplicate tool schemas and meaningful task success. That distinction matters when applying the result to a production agent service.
One Default Changes the Experiment
On this hardware and context length, the tested vLLM release does not chunk prefill by default; the tested SGLang release uses 2,048-token chunks. A default-versus-default chart therefore changes cache design and scheduling together. It cannot assign the entire measured gap to a radix tree rather than block-based prefix reuse.
In the matched arm, the authors enable chunked prefill in vLLM with a 2,048-token iteration budget. SGLang's resolved configuration stays the same. They read effective settings from startup logs rather than trusting launch flags alone. This is important because defaults are conditional on release, GPU memory and context length.
A long uninterrupted prefill can hold up ongoing decode work. Chunking offers opportunities to mix the two phases, but it does not remove their resource cost. In this study, enabling chunking at one high-load condition raises vLLM's median time per output token from 112 to 125 milliseconds even as first-token latency improves. An operator optimizing only TTFT could miss that streaming trade-off.
Read the 78% Claim on Its Own Scale
At 0.28 agent graphs per second, equivalent to a nominal 1.68 requests per second, the reported SGLang-to-vLLM throughput ratio falls from 1.22 under defaults to 1.046 with chunking matched. Table 5 calls that a 78% removal of the gap.
The fraction is computed on a logarithmic scale: log(default ratio divided by matched ratio), divided by log(default ratio). It measures how much of a multiplicative separation disappears. It does not say vLLM becomes 78% faster, that GPU cost falls 78%, or that every workload has this decomposition. Other tested TTFT gaps lose 30-67% on the same scale.
This care also applies to the headline latency number. At the matched 0.28 condition, the median TTFT ratio is 12.3, while the throughput ratio is 1.046. Both are seed-paired geometric-mean ratios. They describe different consequences of nearly the same service-capacity boundary, not contradictory hardware measurements.
A Small Shortfall Can Grow for Ten Minutes
The paper reports approximately 1.66 completed requests per second for SGLang and 1.58 for vLLM at that matched operating point. These rounded absolute values need not reconstruct the published ratios exactly. Relative to nominal arrivals, SGLang roughly keeps pace while vLLM does not. A backlog grows on the slower side.
The authors express the effect with a fluid approximation. If one engine sustains a reference rate and another falls short by a fraction delta, the request arriving halfway through a run of length T gains roughly delta divided by one minus delta, multiplied by T/2, in waiting time. Longer measurement windows can therefore produce larger latency ratios without a new change in prompt-compute speed.
The model is explanatory, not a production predictor. Across seven qualifying runs, predicted and measured median TTFT correlate at 0.85 on the log scale, but predictions overestimate by a factor of 1.6 on average. The agent graph partly throttles itself: delayed planners release workers later, and delayed workers release aggregators later. A fully open arrival approximation misses that feedback.
The Residual Advantage Is Real but Unattributed
It would be equally wrong to conclude that matching one flag makes the engines equivalent. At 0.12 graphs per second, both matched engines keep pace and have very similar median TTFT, yet SGLang's p95 TTFT is 2.44x better. With 8,192-token prompts, achieved throughputs are close and both fall behind nominal load by similar amounts, while SGLang's p95 remains 6.43x better.
The second condition is not an unsaturated test. Similar throughput there makes unequal queue growth a less convincing explanation for the tail gap, but it does not make the queue disappear. The paper's confidence interval for that 6.43x ratio is wide, from 1.83 to 22.55, based on three paired repetitions.
Request ordering is a plausible explanation: this SGLang configuration prioritizes long cached-prefix matches, whereas the tested vLLM configuration admits in arrival order. But attention kernels, overlapped scheduling and memory-budget semantics also differ. Matching a chunk budget does not match these mechanisms. An ordering ablation would be needed before calling the residual a proven benefit of longest-prefix scheduling.
More Sharing Does Not Prove a Better Cache
A separate default-configuration sweep finds the median gap narrowing and then reversing as prefix sharing increases. At high sharing, vLLM is faster in that sweep. That observation challenges a simple story in which more reuse must always increase SGLang's advantage.
It does not prove that block caches are inherently better. The sharing sweep was not repeated with chunking matched. High reuse also shrinks the uncached prefill work, changing how much the scheduling difference matters. The authors label this explanation as post hoc and call for a matched sharing sweep. Cache layout and scheduling still need separate tests.
A Better Benchmark Acceptance Checklist
Before using an engine comparison to size a fleet, save effective configuration logs alongside the model revision, precision, GPU type, context limit and engine version. Record allocated KV capacity rather than assuming similarly named memory fractions mean the same thing. In the paper, the two engines' memory-fraction options do not even bound identical categories of memory.
Then report offered load, achieved throughput, run length, completion rate, median TTFT, p95 TTFT and decode timing together. A high completion rate alone is insufficient: requests may eventually complete after substantial queueing. A single median is insufficient too: an intermittent stall can harm the tail well before most requests slow down.
Compare both equal offered-load and comparable-utilization operating points. Pair traces by seed, keep a control configuration and repeat the run near the saturation knee, where bursts can decide whether a queue starts growing. Avoid treating a non-significant difference from three repetitions as proof of equality.
Finally, separate two questions. A default-deployment comparison can tell an operator what those particular resolved defaults do. A mechanism claim, such as cache architecture causing a speedup, needs controlled interventions. Neither question is answered by attaching an architectural name to the fastest bar.
Use Today's Documentation, Not Yesterday's Flag Recipe
The current vLLM optimization guide says V1 enables chunked prefill by default whenever possible and prioritizes decode work before using the remaining token budget for prefill. It also describes a tuning trade-off: smaller budgets can improve inter-token latency, while larger ones can improve TTFT. The study's 2,048-token value is an experimental control, not a universal deployment recommendation.
Current SGLang tuning guidance likewise treats longest-prefix-match scheduling as an option for shared-prefix workloads and notes its scheduling overhead. That supports testing the policy, not assuming it wins every queue shape. The useful next experiment is to reproduce a matched comparison on your installed releases and representative traffic, not copy a historical launch command or declare a permanent winner.
Assumptions and limitations
This article analyzes one primary study and selected official documentation, not the full serving-systems literature. All experimental results are attributed to the authors; we did not run their GPU workloads or independently audit the artifact's logs. The linked artifact provides a route to re-analysis, not evidence that we performed it.
The study's defaults and matched arms ran on separately provisioned deployments. Unchanged SGLang measurements shifted by up to 11.7%, and there is no direct equivalent bound for vLLM. The effects are much larger than that control drift in the central comparisons, but small residual differences deserve caution. Three repetitions, wide intervals, 64 uncorrected paired tests, one GPU/model setup and synthetic prompts limit generalization. The appendix also says image digests were obtained from published image pages rather than recorded at run time.
No result here establishes current-release superiority, production cost savings, an application accuracy gain or a guaranteed latency target. The practical conclusion is narrower: measure the effective configuration and queue state before explaining a latency gap as a cache-architecture win.
Worked Example: The Same Shortfall, a Longer Wait
This invented fluid-queue illustration is not a reproduction of the authors' traces. Suppose arrivals are exactly 2 requests per second and an engine serves 1.92, a 4% shortfall. The midpoint waiting estimate is 6.25 seconds for a 300-second run and 12.5 seconds for a 600-second run. Adding an assumed 0.5-second baseline produces illustrative TTFT ratios of 13.5 and 26.0. The code checks that arithmetic. Real dependency graphs, bursts and admission controls change the outcome.
from fractions import Fraction as F
arrival = F(2)
service = F(48, 25)
delta = 1 - service / arrival
baseline = F(1, 2)
assert delta == F(1, 25)
for seconds in (300, 600):
wait = delta / (1 - delta)
wait *= F(seconds, 2)
ttft = baseline + wait
ratio = ttft / baseline
print(seconds, float(wait),
float(ratio))
Sources & Provenance
- Primary paper: A Controlled Experimental Evaluation of Configuration and Cache Architecture in LLM Serving Engines, Muhammad Faizan Anwar and Sina Keshvadi, arXiv:2611.05991, November 4, 2026. Sections 3-9, Table 5 and Appendix A support the setup, decomposition, queue model, residuals and limits.
- Authors' artifact repository: llm-serving-config-eval. Includes the stated reproduction workflow and experimental materials; linked for inspection, not independently reproduced here.
- Official vLLM Optimization and Tuning guide, accessed November 16, 2026. Supports the current V1 default and token-budget trade-off, not the paper's measured speedups.
- Official SGLang Hyperparameter Tuning guide, accessed November 16, 2026. Supports the current scheduling-policy option and overhead caution.
The queue example and benchmark acceptance checklist are explanatory analysis. They are not additional measurements or claims that the paper tested every recommended control.
Comments
Post a Comment