VelServe Reports 2.4-2.9x Tail-SLO Goodput, Not a 2.9x Faster Model
LLM SERVING AND TAIL LATENCY
A scheduler can make token delivery steadier without increasing raw token throughput. VelServe's reported gains depend on per-request tail targets, measured headroom and a real cost to first tokens.
By Ossam Hashim
KEY FINDINGS
- On its Mixed workload, VelServe reports 2.4-2.9x the strongest baseline's SLO goodput for Qwen3-8B and an expert-sliced DeepSeek-V4-Flash setup. Goodput here means the highest arrival rate with at least 90% of requests meeting both deadlines, not a model-speed or raw-throughput multiplier.
- Its two main controls work together: reserve iteration time for measured prediction errors and host stalls, then serve short remaining prefills first. A smaller budget alone protects token gaps but can leave first tokens waiting.
- In the full-model, eight-GPU disaggregated test, attainment rises to 35.1%, versus 15.6% and 14.8%. That is an improvement, but every system remains far below the paper's 90% goodput threshold at that load.
- Calibration is not a guarantee. Long prompts still time out, one irreducible stall causes 7.6% misses in a longer test, and standard throughput is nearly the same across schedulers. Baseline ports, modified model execution and metric choices limit generalization.
The Metric Changes the Meaning of Faster
VelServe: Taming Decode Jitters in LLM Inference with Tail-Calibrated Scheduling studies an awkward serving problem: an engine can generate enough tokens on average while repeatedly interrupting individual streams. Time to first token, or TTFT, captures the initial wait. Time between tokens, or TBT, captures the gaps after generation starts. Mean time per output token, or TPOT, can hide a few long pauses among many short gaps.
The paper scores each request against two conditions. Its first token must arrive within the larger of one second and three times the prompt's prefill time on an idle engine. Its own 99th-percentile token gap must be at most 150 milliseconds. SLO attainment is the share satisfying both. SLO goodput is the highest arrival rate retaining at least 90% attainment, interpolated between tested rates.
Those definitions are not interchangeable with completed tokens per second or with the 99th percentile pooled over everyone's tokens. A pooled tail can hide an unlucky request among many smooth ones. Conversely, buffered text reading may tolerate pauses that a speech pipeline cannot. The separate study Revisiting Service Level Objectives and System Level Metrics in Large Language Model Serving explains why neither average token speed nor a gap metric alone fully describes human waiting. VelServe's strict target is a chosen service contract, not a universal measure of experience.
Why Chunking Does Not Remove Every Stall
Chunked prefill divides a long prompt across iterations, mixing prompt work with active decodes. That prevents one full prefill from monopolizing the GPU, but a token-count cap does not directly bound wall-clock duration. A late-context chunk, prediction error or host stall can still push an iteration past the gap target.
Every active stream waits through that iteration. A stall is therefore shared across the batch and accumulated across a long answer. VelServe calls these overruns decode jitters. Its reported Medha implementation plans no iteration over the target, yet 1.36%-3.97% of measured iterations overrun across the three model settings in Table II. Planning right up to a predicted deadline leaves little protection against error.
Changing queue order solves a different problem. With first-come-first-served prompts, a short arrival waits behind a long prompt's remaining chunks. Shrinking every chunk can protect decoding while stretching that wait. This is why a useful scheduler needs both an iteration-time policy and a policy for whose prompt work gets admitted.
Measured Headroom, Not an Arbitrary Safety Constant
VelServe fits iteration cost using fixed overhead, prefill tokens, decode count and a context-dependent term. It treats the remaining error, including host stalls, as a residual. The planned batch must fit the TBT target after reserving a residual margin and token-delivery allowance.
The background calibrator refits the cost model every 500 iterations using the last 8,000 iterations. Its tail statistic uses up to 200,000 recent iterations containing decodes. Importantly, these are the scheduler's own iterations. A margin measured under another budget or on an idle engine describes a different mixture of work.
The margin is a high-ranked observed residual rather than a fitted tail distribution. Stalls too large for even the minimum budget are treated as irreducible rather than forcing useful work toward zero. The budget can fall immediately but needs repeated evidence before rising. A minimum budget triggers an infeasibility flag instead of silently claiming the target is met.
That loop matters empirically. At 1.0 request per second on Qwen3-8B's Mixed workload, online calibration reports 99.6% attainment. Freezing the margin at its 9.4-ms offline value yields 47.6%; carrying a 29.3-ms margin from another workload yields 83.1%. These are ablations of the evaluated setup, not suggested constants for a different server. On DeepSeek-V4-Flash, a hand-tuned fixed margin matches VelServe, so the evidence supports automated selection rather than claiming static tuning can never work.
Shortest Remaining Prefill Pays for the Margin
Reserving time slows prefill. VelServe counters that cost by admitting prompts with the least uncached work remaining, preempting longer prefills at chunk boundaries. When first-token deadlines scale with prompt length, longer jobs have more absolute slack for short jobs to use.
The paper's supporting scheduling argument assumes linear, context-free iteration cost and a fixed decode trajectory. Real queue order changes when prompts become decodes, so the theorem is not an unrestricted optimality result for the whole server. The experiments, rather than that simplified argument, carry the full-system claim.
Size estimation also depends on attention architecture. A token deep in a long context is not necessarily as cheap as one near the beginning. VelServe tests whether token counts adequately describe work and otherwise uses the fitted time model. Forcing token-only estimates on DeepSeek-V2-Lite reduces reported attainment at 0.75 request per second from 99% to 84%.
With prefix caching, it sizes waiting work by the uncached suffix, while keeping the deadline based on the full prompt. This is not the invention of shortest-prefill-first scheduling. The merged SGLang shortest-prefill-first change also prioritizes remaining uncached tokens. VelServe's question is how that order interacts with a measured tail budget.
What the Headline Experiment Actually Runs
The colocated experiments use one H100 NVL with 94 GB of memory and a 400-W cap, vLLM 0.30.0, CUDA 13.0 and PyTorch 2.13. The Mixed workload is synthetic: 75% of prompts are 128-2,000 tokens, 20% are 8,000-32,000 and 5% are 64,000-120,000. Arrivals are Poisson, with outputs of 64-512 tokens.
Qwen3-8B and DeepSeek-V2-Lite use real weights. The full DeepSeek-V4-Flash checkpoint does not fit on one GPU, so this arm retains 64 of 256 routed experts per layer and slices the router. The authors argue that per-GPU compute and weight traffic match a four-way expert-parallel shard. The generated text does not match the full model, and output lengths are fixed. This is a serving-cost experiment, not an answer-quality evaluation of the complete model.
Table D.1 reports P99-TBT SLO goodput of 1.54 versus 0.65 requests per second for Qwen3-8B and 1.29 versus 0.44 for the sliced V4 setup, each against Medha, the strongest evaluated baseline. The rounded ratios are 2.4x and 2.9x. DeepSeek-V2-Lite remains above the threshold at the highest tested rate, so its at-least-4.4x result is a lower bound on that sampled range, not a measured peak.
Every system runs in the same vLLM build. Sarathi and SGLang policies are ported; Medha and Niyama are the authors' implementations with documented choices. Limited checks against official Sarathi and Niyama artifacts support the direction, but do not establish equivalence for every scheduling decision or current release. This is a comparison of specified policies and configurations, not a blanket ranking of today's products.
Disaggregation Improves, but Does Not Pass
The separate disaggregated test uses the full DeepSeek-V4-Flash on eight RTX PRO 6000 GPUs in one node: four for prefill and four for decode, with KV transfer over NIXL. Each system serves the same 1,200 Mooncake requests over 20 minutes at a mean of one request per second.
Newly arriving KV does not eliminate all decode work. The joining request's final prompt token still needs processing. In this setup, that changes the execution path from decode-only graph replay to many kernel launches. A join lasts roughly 170-185 ms, beyond the 150-ms target, and expert-parallel synchronization spreads the pause across ranks.
Current vLLM data-parallel documentation confirms that these MoE ranks must align forward passes. Its agentic-serving write-up already describes synchronizing prefill admission cadence across ranks. VelServe does not originate coordination; it derives an admission period from the gap allowance, grouping ready joins into shared epochs. Urgent first-token deadlines can open an earlier window.
Attainment reaches 35.1%, versus vLLM's 15.6% and Niyama's 14.8%. Calling that twice as much goodput would be misleading: none reaches 90% attainment at this load. The gate makes slow joins less frequent, not shorter. Median TTFT also rises from 2.38 seconds under vLLM to 3.25 seconds. A separate 300-request gate-off/gate-on comparison supports the mechanism, but it is one pair of windows without run-to-run variance. Its numbers should not be mixed with the main 1,200-request comparison.
Where the Policy Loses
Appendix D reports nearly equal raw output throughput at the same offered load. VelServe can have higher median inter-token latency than vLLM because it runs smaller iterations. Its P99 TTFT is not generally lower, because long prompts wait. The gain is smoother service under a defined joint target, not faster completion of every request.
Long prompts are not protected unconditionally. In one 40-minute Qwen3-8B run, 19.1% of the longest class times out under VelServe. With Mooncake-shaped lengths, V4 goodput falls to less than half its Mixed-workload value. Rearranging nearly late prompts does not recover the lost prefill capacity.
The tail bound has important conditions. It assumes the calibrated overrun rate also holds for the iterations requests actually experience, within the reference load. Big batches can give more requests exposure to an overrun than an iteration-weighted statistic suggests. Distribution shifts take time to enter the window. In a ten-minute ablation, one irreducible host stall produces 7.6% misses, exceeding the calibration's own miss target.
Urgent openings spend extra decode-stall allowance to protect first tokens, so the colocated analytical bound does not carry over to the disaggregated gate. That gate is tested on one node, split, model, engine and trace. The paper also gives inconsistent worst-gap goodput values for Qwen3-8B and V2-Lite between Tables D.1 and D.2; this article uses the separately identified P99-TBT headline comparison, not those conflicting strict-variant values.
A Reproduction Plan That Tests the Claim
Start with client-observed per-token timestamps and retain request identity. Score each request's own gap percentile alongside its first-token deadline, mean TPOT, maximum gap and completion status. Count every arrival, including timeouts, and separate prompt-length classes. A single pooled histogram cannot reconstruct each request's own P99.
The current vLLM production-metrics documentation exposes TTFT, inter-token latency, per-request TPOT, queue and phase-time histograms. They are useful operational signals, but the paper's client-side records answer the request-specific question. Compare offered load and completed tokens as well as SLO capacity, so a metric gain is not mistaken for more compute throughput.
Then vary workload and disturbance separately: prompt mix, cached suffixes, response lengths, host stalls and admission timing. Freeze a previously successful margin as a control. Test how quickly calibration recovers after a shift and how often long requests time out. For disaggregation, log join cost and urgent openings; a gate is unnecessary when joins already fit the chosen bound.
Assumptions and limitations
This is an analysis of one new primary study, with official engine documentation and related metric research used for context. Performance numbers are author-reported; no independent benchmark, code audit or production deployment was performed. Appendix C says the authors will release the scheduler, baselines, configurations and run data; this article does not claim a verified usable release.
Most main rate cells use a 120-second arrival window after warm-up, with two seeds and three for the main sliced-V4 comparisons; longer and trace runs provide additional, narrower evidence. Threshold interpolation, lower bounds, model slicing, baseline reimplementations and prompt-scaled deadlines affect the meaning of every ratio. A deployment with fixed deadlines, different stall behavior or an overloaded long-prompt population may make a different trade-off.
Worked Example: A Margin Changes the Plan, Not the GPU
This invented arithmetic uses a 150-ms gap target, a 5-ms delivery allowance and a 40-ms margin. Only 105 ms remains for predicted iteration work. A proposed 130-ms iteration fits without the margin but is rejected with it; a 100-ms plan plus the same margin and allowance fits at 145 ms. The example illustrates an admission check, not VelServe's regression, empirical quantile, queue order or proof. A 60-ms unexpected stall could still break that plan.
from fractions import Fraction as F
target = F(150)
margin, delivery = F(40), F(5)
room = target - margin - delivery
print(room)
plans = (F(130), F(100))
for predicted in plans:
total = predicted + margin
total += delivery
print(total, total <= target)
assert room == 105
Sources & Provenance
- Primary study: VelServe: Taming Decode Jitters in LLM Inference with Tail-Calibrated Scheduling, Ishaan Datta and colleagues, arXiv:2611.18392, November 2026. Main Sections II-IV and Appendices A-L checked. Table D.1 supports the P99 headline; Appendix K separates full-model disaggregated results and gate ablation; D/G/H/I/L provide costs and controls.
- vLLM data-parallel documentation. Supports synchronization behavior of MoE data/expert-parallel ranks, not the paper's measured join latency or comparative results.
- vLLM agentic-serving write-up. Its admission-cadence section shows the existing coordination mechanism and the first-token trade-off; it is not independent verification of VelServe.
- SGLang shortest-prefill-first change, pull request 40024. Merged September 18, 2026; supports remaining-uncached-token prioritization as an existing policy.
- vLLM production-metrics documentation. Supports available histogram categories; does not replace request-keyed client timestamps.
- Revisiting Service Level Objectives and System Level Metrics in Large Language Model Serving. Used for the distinction between token gaps, buffered reading and user idle time, not as a replication of the new scheduler.
The reproduction checklist and numerical admission example are explanatory analysis. No universal fairness, latency, cost-saving or model-quality improvement is asserted.
Comments
Post a Comment