BellowsKV Architecture: Block-Shared Cache Latents, Outlier-Free Low-Bit Storage, and Single-Decision Eviction for Training-Free 13.8x KV Compression

INFERENCE SYSTEMS

A Rice University measurement study becomes a serving method: adjacent transformer layers cannot share their caches, but they can share what the caches are computed from. One latent per block, quantized without rotation, keeps about 98% of full-cache accuracy at 13.8x compression.

By Ossam Hashim

KEY FINDINGS

• The KV cache now rivals the weights: on Llama-3.1-8B, one 131K-token sequence carries a 16.8 GB fp16 cache against 16 GB of weights; a batch of eight holds 134 GB of cache (paper's figures).
• Adjacent layers cannot reuse each other's cache: neighboring keys and values are near-orthogonal (cosine similarity about zero), and even an oracle-chosen shared key raises perplexity significantly.
• The redundancy lives in the inputs: hidden states feeding adjacent layers reach 0.87 cosine similarity on Llama and 0.82 on Qwen, increasing with depth.
• The shared latent quantizes cleanly: every coordinate mixes channels from all n layers, so no outlier channel sets the scale; relative INT4 error falls from 0.156 (raw cache) to about 0.10, with no rotation pass at read time.
• Layers of a block agree on which tokens matter (Spearman 0.76 to 0.80), so one eviction decision replaces n per-layer decisions; shrinking the latent's dimension is the one compression axis that fails.
• BellowsKV-Quant retains about 99% of full-cache accuracy at 3.6x and 97-98% at 13-14x compression across Llama-3.1-8B, Mistral-7B and Qwen2.5-7B; the strongest prior training-free cross-layer method retains at most 84% at 16x (authors' reported LongBench results).

The Cache That Outgrew the Weights

Long-agent workloads changed the memory balance of LLM serving. Weights load once and serve every sequence in a batch; each sequence's KV cache is private, re-read at every decoding step, and grows with both context length and batch size. The paper's arithmetic makes it concrete: a single 131K-token Llama-3.1-8B sequence carries 16.8 GB of cache in 16-bit precision, slightly more than the model itself after grouped-query attention already shrank it fourfold. Eight such sequences hold 134 GB of cache against 16 GB of shared weights. The established compression toolbox works inside one layer at a time: quantize the entries, evict low-importance tokens, or cut the channel dimension. Each family approaches a floor near 4x. Quantization degrades sharply below two bits and needs outlier machinery; eviction is irreversible once a later turn needs the dropped token; rank reduction removes coordinates attention still uses. Compressing 134 GB by 4x leaves 33 GB. The paper targets an order of magnitude, which means changing what is stored, not only how coarsely. This blog covered the same order-of-magnitude storage pressure from the index side in the LEANN analysis; BellowsKV attacks the live-memory side.

Share the Input, Not the Cache

The obvious cross-layer move, letting one layer reuse its neighbor's cache, fails on measurement. Adjacent keys and values have near-zero cosine similarity, because each layer applies its own projection matrices to the shared input. The redundancy sits one step earlier: the hidden states those projections consume are highly similar across adjacent layers, a consequence of the residual stream. BellowsKV therefore stores neither layer's cache. For each block of n adjacent layers it fits one projection offline, in closed form from activation statistics, mapping the block's hidden states to a single latent vector per token. At decode time each layer reconstructs its own keys and values from that latent through its own up-projection. The name is literal: a bellows folds a large volume into a small one and expands it again on demand. No retraining, no fine-tuning; the method is training-free, which separates it from architectures that learn to share a cache on billions of tokens.

Three Measurements, Three Operators

The design keeps the latent at full dimension and spends the compression budget on the two axes the measurements say are safe. First, quantization: because every stored coordinate is already a linear mixture of channels from all n layers, the latent has the flat dynamic range that rotated caches achieve, without paying a rotation at every read. BellowsKV-Quant applies uniform integer quantization with one scale per 128 adjacent coordinates and nothing else. Second, eviction: adjacent layers rank token importance with Spearman correlation between 0.76 and 0.80, so BellowsKV-Evict computes a single importance decision per block from all n layers' scores and applies it to the shared latent once. Third, the forbidden axis: under a fixed memory budget, reducing the latent's dimension loses far more accuracy than reducing its bits, matching a recent position paper's argument that quantization beats rank reduction at matched budgets. The two main operating points pair block size with precision: n=2 with INT8 and n=4 with INT4, measuring 3.6x and 13.8x compression with all metadata counted.

Reading the Retention Table

The evaluation reports per-task retention of full-cache accuracy on LongBench's eight tasks under the official harness, full-sample, with every ratio counting all metadata. On the strongest tier, BellowsQ with n=4 and INT4 measures 13.84x and retains 98.0% on Llama-3.1-8B and 98.1% on Qwen2.5-7B on average. Pushing to INT3 at 17.65x still holds 98.4% and 97.8%, and a 3-bit key / 2-bit value variant reaches 20.47x at 97.6% on Llama. The comparison rows show why the shared representation matters more than the operator: prior training-free cross-layer methods factorize the caches themselves, and at 13-16x their retention collapses to 84% for xKV and 54% for CommonKV in the paper's tables. At a serving level the arithmetic is the point: the 134 GB eight-sequence cache becomes roughly 9.7 GB at 13.8x, which converts directly into larger batches or longer contexts on the same GPU.

Assumptions and limitations

All accuracy and compression figures are the authors' reported results on LongBench across three 7-8B models; this analysis did not independently re-run the evaluation. Retention on larger models, MoE architectures and multimodal caches is not established by the paper. BellowsKV-Evict's results are narrower than Quant's and eviction remains irreversible for multi-turn regret cases. The official repository was published alongside the paper and is still minimal at the time of writing. The worked example is a simplified illustration of the measurement logic, not the paper's method.

Worked Example: Why the Latent Quantizes Flat

This toy block of two 8-dimensional layers reproduces the paper's first two measurements in miniature: hidden states stay similar while projected keys diverge, and a mixed (latent-like) representation suffers less uniform INT4 error than a raw cache with one outlier channel. The numbers are illustrative, not the paper's pipeline.

import random, math
random.seed(42)
d = 16
n = lambda: random.gauss(0, 1)
def vec():
    return [n() for _ in range(d)]
def cos(a, b):
    dd = na = nb = 0.0
    for x, y in zip(a, b):
        dd += x * y
        na += x * x
        nb += y * y
    return dd / math.sqrt(na * nb)
def matvec(W, x):
    out = []
    for row in W:
        s = 0.0
        for w, v in zip(row, x):
            s += w * v
        out.append(s)
    return out
h1 = vec()
h2 = []
for x in h1:
    h2.append(x * 0.9 + 0.1 * n())
W1 = [vec() for _ in range(d)]
W2 = [vec() for _ in range(d)]
k1 = matvec(W1, h1)
k2 = matvec(W2, h2)
c_h = cos(h1, h2)
c_k = cos(k1, k2)
print("cos(h1,h2):", round(c_h, 2))
print("cos(k1,k2):", round(c_k, 2))
raw = k1[:]
raw[0] *= 25
H = [[1]]
for _ in range(4):
    N = []
    for r in H:
        N.append(r + r)
    for r in H:
        N.append(r+[-v for v in r])
    H = N
lat = matvec(H, raw)
def qerr(x, skip, bits=4):
    lo, hi = min(x), max(x)
    s = (hi-lo) / (2**bits - 1)
    errs = 0.0
    mags = 0.0
    for i, v in enumerate(x):
        if i == skip:
            continue
        q = round((v-lo)/s)*s+lo
        errs += (v - q) ** 2
        mags += v * v
    re_ = math.sqrt(errs / len(x))
    rv = math.sqrt(mags / len(x))
    return round(re_ / rv, 2)
print("int4 err raw:", qerr(raw, 0))
e_lat = qerr(lat, -1)
print("int4 err latent:", e_lat)

Assumptions and limitations All accuracy and compression figures are the authors' reported results on LongBench across three 7-8B models; this analysis did not independently re-run the evaluation. Retention on larger models, MoE architectures and multimodal caches is not established by the paper. BellowsKV-Evict's results are narrower than Quant's and eviction remains irreversible for multi-turn regret cases. The official repository was published alongside the paper and is still minimal at the time of writing. The worked example is a simplified illustration of the measurement logic, not the paper's method.

Sources & Provenance

• BellowsKV paper (arXiv, October 26, 2026) - mechanism, measurements, Table 1 retention figures
• MorrisZJ/BellowsKV repository - official implementation, released with the paper
• BellowsKV full text (HTML) - Section 2 observations and operator details

Analysis is based on the primary sources listed above; benchmark claims are attributed to their authors and not independently verified.