KVGuard: Shadow-Block Permissions, Five Checkpoint Boundaries, and Making Silent KV Cache Corruption Fail Loud
LLM SERVING
PagedAttention turns one CUDA allocation into a custom heap, and that heap has allocator bugs - double frees, use-after-frees, reads of unwritten blocks - that corrupt outputs without ever crashing. A new checker finally speaks its language.
By Ossam Hashim
KEY FINDINGS
- PagedAttention-style serving partitions a single cudaMalloc region into fixed-size KV blocks managed by a CPU allocator and block tables, which makes the KV cache a custom heap - and heaps get double frees, use-after-frees, out-of-bounds accesses, uninitialized reads and permission violations. These bugs stay inside the one big allocation, so nothing crashes; outputs quietly corrupt or one request reads another's state.
- Traditional detectors are structurally blind here: ASan-style tools and NVIDIA Compute Sanitizer reason about addresses and CUDA-level allocations, not blocks, requests and block tables, so a stale or wrong block-table entry still resolves to a valid address - and their redzone and quarantine designs spend GPU memory that serving cannot spare.
- KVGuard keeps a compact shadow block per physical KV block - a per-request permission field (the never-reused request ID doubles as a lifetime token) and a per-entry initialization bit - and checks the serving system's behavior against it at five points, from CPU allocation calls to the in-kernel KV access, firing at the boundary that introduced the violation.
- The checker covers all 29 historical real-world KV cache safety bugs in the vLLM and SGLang patch histories, surfaced 8 previously unknown violations (upstreamed as pull requests), and reported zero false positives; transfer checks extend the same rules to CPU offload and prefill/decode disaggregation.
- The cost is development-phase friendly: 3.0 percent and 7.4 percent geomean TPOT overhead on Qwen2.5-14B under vLLM and SGLang respectively, at most 0.11 percent of the KV cache in extra memory - roughly two orders of magnitude below Compute Sanitizer on the same workloads.
A Custom Heap With Custom Heap Bugs
The KV cache is the scarcest resource in an LLM serving system, so engines like vLLM and SGLang manage it through PagedAttention: one large CUDA buffer, carved into fixed-size physical blocks, handed out by a CPU-side allocator, and mapped to each request's logical blocks through block tables that GPU kernels consume. That is, structurally, a memory allocator - and a new paper from Seoul National University (Cheong, Hwang, Kim and Lee) shows it suffers the allocator's classic bug families, with a serving-specific twist: the victims are silent. The paper walks through real cases from the two projects' patch histories. In one, a scheduler moves a request's write position back into a shared, read-only prefix block, and the cache-write kernel overwrites another conversation's cached tokens - every later request that reuses the prefix reads the wrong vectors. In another, a finished request's cached prefix block is freed as if it were private, then reallocated and overwritten while the prefix cache still points at it: a textbook use-after-free, dangling pointer included. In a third, a CPU-offload reload counts a block as restored when only its neighbor was loaded, and the attention kernel reads whatever stale bytes the block happened to hold. In each case the system keeps running and returns plausible-looking tokens.
Why the Existing Net Misses Them
None of the standard tools catch this class. AddressSanitizer and its relatives attach metadata to addresses, pointers and allocator-visible objects - but KV blocks are software-defined slices of one live CUDA allocation, so a write into the wrong slice is a write to a perfectly legal address. Compute Sanitizer's memcheck sees the whole KV buffer as one valid region for the same reason, and its initcheck considers memory initialized once any prior request wrote it, which is exactly wrong for a per-lifetime notion of initialization. Even if the abstraction matched, the mechanisms would not fit: redzones, quarantines and padding spend the GPU memory that determines batch size and context length. The authors distill three requirements: catch each violation early, at the operation that creates it rather than at the corrupted token many requests later; keep memory and runtime overhead small enough that the KV cache is not shrunk; and attach to the existing PagedAttention workflow without disturbing the buffer layout or the optimized kernels.
Five Checks, One Shadow Block
KVGuard's answer is a shadow block per physical block, holding just two fields. The permission field records, per request, whether that request may read or write the block; because serving systems never reuse a request ID, the field also encodes the block's current lifetime, so a stale accessor simply finds no entry. The initialization field records, per KV entry, whether it has been written in the block's current lifetime - per-entry because blocks are often partially written. On prefix sharing, permissions downgrade to read-only for every sharer, mirroring the system's own rule that shared prefix blocks must never be overwritten. Five checks consume this metadata at the boundaries where state is built or used: C1 validates every allocation, share and free call (a free while another request still holds the block is a use-after-free in waiting); C2 walks each request's CPU block table before launch and verifies every mapping against the shadow permissions; C3 compares the GPU block table against the CPU one, catching stale rows produced under asynchronous or overlap scheduling; C4 validates each in-kernel block-table lookup against the current request's row and bounds; and C5 checks every KV entry access for both permission and initialization. Because each check fires where its facts first exist, the report lands on the faulty call or kernel access, not on a corrupted output token downstream. The same rules extend to KV transfers for offload and prefill/decode disaggregation: a transfer counts as a read on the sender and a write on the receiver, blocks stay held until completion, and the sender reports how many entries it actually wrote so a short delivery reads as uninitialized later.
What It Caught and What It Costs
The prototype covers all 29 historical KV cache safety bugs drawn from the vLLM and SGLang patch histories - spanning CUDA-graph padding writes, radix-cache frees, offload restore slips and speculative-decoding paths - and found 8 previously unknown violations in current execution paths, each reported upstream with a fix pull request. It produced no false positives across the evaluated workloads. The overhead fits its intended home in CI, pre-release validation and post-incident debugging rather than always-on production: 3.0 percent and 7.4 percent geomean TPOT on Qwen2.5-14B under vLLM and SGLang with all checks enabled, and at most 0.11 percent of the KV cache in extra memory, about two orders of magnitude below Compute Sanitizer. Integration is real but bounded: hook insertions at 180 and 177 sites in the two systems, plus Python shadow management and small CUDA check kernels.
Assumptions and limitations
All findings are the authors' own, measured on vLLM and SGLang with the workloads and models in the paper; the overhead figures come from Qwen2.5-14B, and other engines, models or traffic mixes may shift them. The 29-bug census reflects the two projects' public patch histories, so it samples the bugs those projects happened to fix, not the full population. KVGuard targets development-phase use - CI, pre-release validation, debugging - not always-on production. The worked example is an invented toy model of the permission and initialization checks, not the paper's implementation.
Worked Example: Three Violations Walk Into a Shadow Block
A toy model shows why two fields suffice. Track per-block permissions and per-entry init bits, plus the block pool's reference counts, and hook the allocator events. The scenario replays four miniature violations from the paper's census: a write into a shared read-only prefix block, a teardown that frees a block the prefix cache still holds, a stale read after the block is reallocated anyway, and a read of an entry never written. Each trips exactly one assertion - the one guarding the boundary where the bug acts.
from collections import Counter
RW, RO = 2, 1
ref = Counter()
class Sh:
def __init__(s, n):
s.p = [{} for _ in range(n)]
s.w = [[0]*2 for _ in
range(n)]
def alloc(sh, b, r):
assert not sh.p[b], 'live'
sh.p[b] = {r: RW}
sh.w[b] = [0, 0]
ref[b] = 1
def share(sh, b, r):
assert sh.p[b], 'free'
for k in sh.p[b]:
sh.p[b][k] = RO
sh.p[b][r] = RO
ref[b] += 1
def free(sh, b, r):
assert ref[b] == 1, 'shared'
assert r in sh.p[b], 'stranger'
sh.p[b] = {}
ref[b] = 0
def write(sh, b, r, e):
assert sh.p[b].get(r) == RW
sh.w[b][e] = 1
def read(sh, b, r, e):
assert sh.p[b].get(r, 0) >= RO
assert sh.w[b][e], 'uninit'
return 'kv'
sh = Sh(2)
alloc(sh, 0, 'r0')
write(sh, 0, 'r0', 0)
share(sh, 0, 'r1')
# B3: write into RO block
try:
write(sh, 0, 'r1', 1)
print('RO write: MISSED')
except AssertionError:
print('RO write: caught')
share(sh, 0, 'cache')
# B11: buggy teardown frees
# a block the cache holds
try:
free(sh, 0, 'r0')
print('bad free: MISSED')
except AssertionError:
print('bad free: caught')
# the bug proceeds anyway
sh.p[0] = {}
ref[0] = 0
alloc(sh, 0, 'r2')
write(sh, 0, 'r2', 0)
# stale read via old mapping
try:
read(sh, 0, 'cache', 0)
print('stale read: MISSED')
except AssertionError:
print('stale read: caught')
alloc(sh, 1, 'r3')
# B6: read never-written
try:
read(sh, 1, 'r3', 0)
print('uninit: MISSED')
except AssertionError:
print('uninit: caught')
Sources & Provenance
- KVGuard: A KV Cache Safety Checker for LLM Serving Systems (Cheong, Hwang, Kim and Lee, 4 Nov 2026) - arXiv abstract
- Full paper (PDF), including the 29-bug census, the five-check placement and the overhead evaluation - arXiv full text
- Background referenced inside the paper: PagedAttention, vLLM, SGLang, NVIDIA Compute Sanitizer.
All performance numbers are attributed to the preprint as published. Sources accessed 7 November 2026. This analysis is independent editorial content for GlobalTechDigest.
Comments
Post a Comment