The Burst That Outlasts Itself: Metastable Collapse in Prefix-Cached Agent Serving and the Parking Brake That Ends It
LLM SERVING
Give a vLLM server five minutes of extra agent traffic and it may never come back on its own - the hit ratio falls to 10%, throughput halves, and the backlog keeps growing for over an hour after the burst is gone.
By Ossam Hashim
KEY FINDINGS
- In paired runs sharing identical base arrivals, a 5-minute burst (2.5x total arrivals) left a vLLM server persistently degraded in 9 of 12 pairs - hit ratio ~0.10 vs 0.94-0.95 healthy, throughput ~0.4 turns/s, backlog past 100 sessions - with zero pairs degraded in the reverse direction (sign test p = 0.0039).
- The collapse is metastable, not ordinary overload: with prefix caching disabled the same absolute burst drains away, and in 7 of 11 degraded runs every injected burst session had already finished while the hit ratio stayed at 0.10-0.12.
- A first-order drain condition, calibrated on cache-disabled runs, puts per-session cold cost at ~78 engine-seconds and break-even near 0.013 sessions/s; the measured bracket is 0.010-0.014, and no collapsed cache re-formed within two hours of observation.
- Recovery requires acting on admitted sessions, not just new ones: admission-only throttling recovered 1 of 7 matched collapsed runs, while suspending sessions against a KV-pool-derived limit recovered all 7 within 6 minutes, and a demand budgeter recovered 17 of 17 collapsed runs across experiments.
- The failure reproduces on SGLang (which absorbs smaller triggers before collapsing) and on Qwen3-32B replaying SWE-bench agent trace shapes; a 128 GiB CPU offload tier delays the collapse but does not prevent it.
A Five-Minute Storm, a Seventy-Minute Hangover
Agent sessions are closed loops: the model calls a tool, the tool output is appended, and the grown conversation goes back to the model, tens to hundreds of times per task. Serving this economically depends on prefix caching - a hit costs an incremental prefill of a few hundred to a few thousand tokens, while a miss recomputes a 10-40K-token context. On an MI300X serving Qwen3-8B, Huang and Liu measure a mid-session turn at ~2 seconds of prefill on a miss versus ~0.15 seconds on a hit (arXiv abstract page). Their experiment is brutally clean: two runs of the same vLLM server receive the same base sessions at the same times, and one additionally gets a 5-minute burst of extra arrivals at minute 15. During the burst the hit ratio dips, as expected. What is not expected is the aftermath: the base load returns to a level the server was handling comfortably (~70% of cache-warm capacity), yet the hit ratio settles near 0.1, turn throughput falls to about 0.4 turns/s, and the backlog grows past 100 sessions for the remaining 70 minutes. In 9 of 12 pairs only the burst run degraded; in none did only the no-burst twin. The trigger is long gone; the damage stays.
Why the Cache Never Comes Back
The sustaining mechanism is specific to stateful agent serving. Because a session cannot issue its next turn until the previous one returns, slower service means more sessions alive at once (Little's law). Every active session needs its prefix resident to hit. When service slows, the combined working set outgrows the KV pool, LRU evicts each session's prefix before its next turn, every turn becomes a full recomputation - and service slows further. Three controls separate this from plain overload. Ten of twelve no-burst twins at the same base load never degraded (the other two slid in spontaneously after 25-30 minutes, which is the metastable signature: rare large fluctuations can also push the system over). With prefix caching disabled at matched sub-capacity load, the same absolute burst produces a backlog that then shrinks in every run - there is no second regime to fall into, only work to catch up on. And it is not the burst's own sessions clogging the pipe: in 7 of 11 degraded runs the injected sessions had all completed, yet the hit ratio stayed at 0.10-0.12. Where RidgeFill predicts which cache contents to reconstruct and FastKVComp compresses what the pool must hold, this work asks what happens when the pool simply cannot hold the working set - and the answer is a self-locking failure.
The Drain Equation
In the degraded regime nearly every turn misses, so the engine runs at its no-cache capacity, and the question of escape becomes arithmetic. If every turn misses, serving one session costs S, the sum of per-turn cold costs; the regime can drain only if arrivals bring less work than the engine completes. Calibrating the cold-cost curve independently on cache-disabled runs gives S of about 78 engine-seconds for the synthetic workload, so break-even sits near 0.013 sessions/s - well below the loads at which collapse is observed. When the authors collapse the server and then lower the arrival rate, the estimated remaining cold work falls at 0.006 sessions/s, stays flat around 0.010, and grows again at 0.014 and 0.018 - bracketing the model's prediction. No run's cache re-formed while observed. The healthy regime has its own condition: the cache-warm population must fit the pool. Between the two conditions, the outcome depends on history - which is why rare events (a burst, a restart, an unlucky arrival cluster) rather than average load decide when a server falls.
Parking Beats Throttling
Getting out is the harder problem, and the natural remedy fails. Throttling new admissions does nothing for the sessions already inside, which keep cycling through a cache that cannot hold them: admission-only caps recovered just 1 of 7 matched collapsed runs (2 of 8 overall, after 21-23 minutes), and four runs stayed under a 0.1 hit ratio to the end. The fix is to suspend admitted sessions at turn boundaries against a limit derived from the KV pool, so parked sessions stop issuing turns, their prefixes may be evicted, and the survivors start hitting again. A session-count cap with parking recovered all 7 matched runs within 6 minutes; a demand budgeter - holding the summed context-plus-output-budget of admitted sessions in a 0.75C/0.95C hysteresis band, with thresholds fixed before the experiments and no per-workload tuning - recovered all 7 within the first 5-minute window and 17 of 17 collapsed runs across the 8B, 32B and threshold-sensitivity experiments, restoring a 0.93-0.95 hit ratio (exact paired p = 0.031 against admission-only). Latency-driven AIMD sits in between at 5 of 7 after 14-25 minutes. The budgeter's demand figure is deliberately conservative - it double-counts shared prefixes and bounds the active working set, not physical residency - which is exactly why it needs no tuning. Controllers restore efficiency, not capacity: on the 32B trace-shape runs the queue kept growing slowly under control because load slightly exceeded cache-warm completion rate, and draining it needs more capacity or load shedding.
Where the Argument Bends
Engine design moves the threshold rather than removing the failure: SGLang absorbed a 2x two-minute burst in 4 of 4 runs but collapsed under an extra 1.5x for five to ten minutes in 6 of 6, and in matched-pair runs at higher load all 4 burst runs degraded while all 4 twins stayed healthy. The 32B evidence is weaker - at one load a longer no-burst run degraded on its own after ~55 minutes, and at a higher load every run degraded with or without a trigger - so history dependence there rests on the lighter configuration. The threats section is admirably blunt: synthetic sessions use random tokens, the trace replay preserves lengths and timing but not content, and there is one GPU type, one replica, finite observation windows, and no production evidence. A CPU offload tier only delays collapse. Independent corroboration arrived after submission: Lu et al. report the same failure on prefill-decode-disaggregated SGLang, where a 10-minute surge cut goodput by ~88% for the following hour, sustained by the same population-cache feedback. For operators the diagnostic is cheap: GPU utilization stays high and latency alarms fire, but the prefix hit ratio plus sessions-in-system versus what the KV pool holds identifies the regime directly - and parking, not patience, is the way out.
Assumptions and limitations
All measurements in this article are the authors' own, from their vLLM/SGLang experiments on MI300X GPUs; we did not re-run any serving experiment. Results describe the tested engines, models, pool sizes and workloads - synthetic random-token sessions and SWE-bench trace shapes - not production fleets. The toy example is an invented simplification for intuition; its parameters echo the paper's regime but its numbers are not measurements.
Worked Example: One Load, Two Stable States
The bistability fits in ten lines of arithmetic. Take an invented but paper-flavored setup: sessions of 25 turns, 5 seconds of think time, 0.15 seconds per turn when the cache hits and 2 seconds when it misses, 21K tokens per session against a 400K-token pool. Little's law gives the concurrency each regime implies; the pool decides whether that regime is self-consistent.
from statistics import mean
# Toy: two regimes at one offered
# load (invented sim, paper-shaped).
lam = 8.0 # sessions per minute
think = 5.0 # s between turns
warm_r = 0.15 # s per turn on a hit
cold_r = 2.0 # s per turn on a miss
ctx = 21_000 # tokens per session
pool = 400_000 # KV pool, tokens
turns = 25
def steady(r):
# Little's law concurrency
n = lam*turns*(think + r)/60
return n, n*ctx <= pool
n_w, ok_w = steady(warm_r)
n_c, ok_c = steady(cold_r)
print('warm: %.1f sessions,' % n_w,
'pool fits:', ok_w)
print('cold: %.1f sessions,' % n_c,
'pool fits:', ok_c)
if ok_w and not ok_c:
print('bistable: hits keep the')
print('pool warm; misses')
print('evict, so every turn')
print('misses again')
print('mean concurrency:',
round(mean([n_w, n_c]), 1))
Sources & Provenance
- Huang & Liu, "Bursts That Outlast Themselves: Persistent Degradation in Prefix-Cached Agent Serving" - arXiv abstract page (2611.13412)
- The same paper - full HTML text (Sections 3-6, paired-run tables, drain calibration, threats to validity)
- Related GlobalTechDigest coverage: RidgeFill and FastKVComp
Written by Instinct for GlobalTechDigest. Analysis, framing and toy code are original; all measured results are the cited authors'.
Comments
Post a Comment