Header Ad Placement

Draft KV Quantization Saves 23.95% of Combined Cache Bytes, but the Prototype Is Slower

LLM INFERENCE AND MEMORY

A six-pair study finds that quantizer layout changes which attention tensors need more bits. Its packed prototype also shows why smaller stored caches do not establish faster or exact speculative decoding.

By Ossam Hashim

KEY FINDINGS

  • Across six target/draft pairs, K4V4 quantization cuts draft KV-cache bytes by 66.20% and combined target-plus-draft cache bytes by 23.95%. These are cache savings, not reductions in total GPU allocation or model weights.
  • Mean acceptance moves from 57.93% to 57.14%, a 0.79-percentage-point drop. The equal-model confidence interval crosses zero, but that does not prove equivalence; Qwen3 and SmolLM2 have resolved pair-level losses.
  • In the packed Qwen2.5 prototype, draft storage falls by 67–71%, yet latency rises 18–33% versus BF16 speculative decoding and peak allocated GPU memory does not fall. BF16 speculative decoding is itself slower than target-only decoding in these runs.
  • Target BF16 caches preserve the mathematical correction rule, not implementation correctness. Sequential-verifier checks pass, while the batched prototype matches target-only sequences on only 2–5 prompts per setting, including failures with an unquantized draft.

Two Caches Create Two Different Optimization Questions

Standard two-model speculative decoding uses a small draft model to propose candidate tokens and a larger target model to check them. That creates a second set of attention keys and values alongside the target cache. Reducing the draft's stored state could make this arrangement easier to fit in memory without changing the target model's weights or cache. The new draft-cache study measures that opportunity, then exposes the distance between a storage result and a working serving improvement.

The useful conclusion is conditional. Its K4V4 layout preserves much of the draft's measured ability to propose acceptable tokens while cutting physical cache storage. But the released packed prototype is slower, does not reduce peak allocation, and has unresolved batched-verification mismatches. Those are central results, not implementation footnotes that can be hidden behind an acceptance average. The study provides evidence for a representation choice and a list of remaining systems problems, not a production-ready acceleration claim.

Define the Boundary Before Reading the Percentages

Gupta and Li quantize the draft cache only. The target cache and verification computation remain in BF16. K4V4 means four payload bits for each key element and four for each value element. It does not mean four-bit weights, a four-bit target, or one quarter of the entire inference footprint. Cache memory sits beside weights, activations, temporary buffers and other allocations that this intervention does not promise to shrink.

The combined-cache saving equals the draft-cache saving multiplied by the draft's share of the two original BF16 caches. A compact draft paired with a much larger target offers less total headroom. In the six-pair study, that share ranges from 17.2% to 50.0%, and combined savings range from 11.30% to 33.17%. The 23.95% headline is an equal-model macro average. It is not a universal discount that can be applied to every model pair or GPU budget.

What the Main Quality Experiment Actually Runs

The central evaluation covers six pairs from Qwen2.5, Qwen3, Llama 3, OLMo 2 and SmolLM2, ranging from a 360M draft to an 8B target. It uses 576 FineWeb-Edu prompts, 96 per pair, with 1,024 input tokens and just 16 generated tokens. Decoding is greedy, batch size one, and each round proposes up to four draft tokens. Every prompt is evaluated with BF16, K8V4, K4V8 and K4V4 draft-cache configurations.

Most quality measurements use fake quantization: values are quantized and dequantized, then stored back in BF16 tensors. That measures the effect of quantization error on predictions, while bytes are accounted for analytically. It cannot measure the runtime of compressed storage. The main cross-family acceptance results also use a sequential verifier that makes one target call per emitted token. That is useful for checking proposal quality and correctness, but it does not demonstrate fewer target calls or speculative speedup.

The Geometry Behind the Bit Split

Attention keys help determine which previous positions receive weight; values carry the information combined using those weights. A simple quantizer gives one scale to each token's key vector within a KV head. A large channel-specific value can then determine the scale for other channels in that token, potentially wasting resolution. The paper instead follows KIVI's asymmetric grouping idea: keys share quantization statistics by channel over a block of tokens, while values share them across the channels of one token and head.

The stronger layout uses affine groups with a stored minimum and scale. Keys are grouped over 32 tokens, and the most recent 128 keys remain BF16 in the default configuration. Values use per-token affine groups with no residual window. These details are part of the method. Assigning the same nominal bit widths to a different grouping, scale convention or residual policy is not a faithful reproduction of its reported operating point.

A Broad Comparison Is Not a Single-Variable Intervention

With the naive per-token quantizer, K8V4 beats K4V8 by 10.87 acceptance points on average. With the grouped-key layout, the contrast becomes minus 0.25 points, and every pair's central gap is below one point. That undermines a universal rule that keys must always get more bits. But the two campaigns also differ in value quantizer, BF16 residual window and prompts. Their large average contrast is not a clean measurement of changing only the key grouping.

The controlled Qwen2.5 3B/1.5B test holds the prompts, value quantizer, decoding loop and residual policy fixed, with no residual window in either arm. Changing the key quantizer from per-token symmetric to grouped per-channel affine moves the K8V4-minus-K4V8 contrast from plus 28.70 to minus 0.33 points. For the more aggressive K4V2-minus-K2V4 split, it changes from plus 15.05 to minus 3.66 points. This isolates the key quantizer as a package of geometry and scale choice, not geometry independently of symmetric-versus-affine quantization.

The Independent Retrieval Probe Still Has a Key-Bit Preference

The controlled acceptance tests use the batched verifier, whose exactness is limited. A separate 16K associative-passkey task avoids speculative verification entirely. It hides 16 similar access-code records in filler text and scores candidate answers by length-normalized likelihood. On 192 Qwen2.5 examples, the naive per-token K4V2 and K2V4 configurations retrieve 166 and 10 answers. Grouped keys improve those counts to 189 and 158.

The key-quantizer change therefore shrinks the bit-split gap from 81.25 to 16.15 percentage points, but does not erase it on Qwen2.5. Llama and Qwen3 show much smaller or no grouped-key gap in the same probe. This matters for the interpretation: the study refutes an unconditional precision rule, not every possible asymmetry between keys and values. It also does not turn a partly synthetic retrieval test into a general reasoning or generation-quality benchmark.

Read Acceptance as a Proposal Metric

Acceptance counts the fraction of proposed draft tokens that the target accepts, excluding correction and bonus tokens. The six-pair K4V4 average declines from 57.93% to 57.14%. The paired difference is minus 0.79 points, with a 95% interval from minus 1.63 to plus 0.04. A zero-crossing interval is compatible with a modest loss; it does not certify that compression is harmless or equivalent to BF16.

The pair-level picture is also useful. Qwen3 loses 1.67 points, with an interval from minus 3.12 to minus 0.30. SmolLM2 loses 1.85 points, with an interval from minus 3.21 to minus 0.54. Four other pair intervals include zero. An acceptance rate does not directly measure text quality or latency, and averaging it across pairs can hide a cost relevant to a particular deployment. The target's verification rule is the intended protection for output semantics, subject to the actual verifier implementing it correctly.

Longer Runs Add Evidence Without Removing the Scope Limits

For Qwen2.5 3B/1.5B, the PG-19 long-context experiments use 36 prompts at 16K and 24 at 32K. Combined-cache savings are about 31%; acceptance changes are minus 0.94 and minus 0.73 points. Only the 16K interval excludes zero. These longer inputs increase the fraction of keys eligible for compression relative to the BF16 tail, but the quality experiment still has a small prompt count and one model pair.

The separate 256-token continuation study provides a more demanding generation horizon than the main 16-token runs. Across its six context/seed/residual settings, K4V4 acceptance changes range from minus 0.59 to plus 0.33 points, all with zero-crossing intervals. The settings share prompts and baselines, so they are not independent replications. All 576 configuration-level evaluations in this sequential-verifier quality study reproduce target-only greedy outputs. That is affirmative correctness evidence for that path, not a transfer of correctness to the different packed batched path.

A Four-Bit Payload Is Not a Seventy-Five-Percent Cache Saving

The implementation stores quantization scales and minima in BF16, plus the BF16 recent-key window. A 32-element key group has two 16-bit metadata values, adding one bit per quantized key element. A value group with head dimension 128 adds one quarter bit per element. Uncompressed tail keys cost more again. These overheads explain why dividing nominal payload width by 16 overstates the reduction.

Physical packed storage nevertheless confirms a substantial saving. For a Qwen2.5-1.5B cache with 28 layers, two KV heads and head dimension 128, the 32,768-token BF16 cache occupies 896.00 MiB. The packed K4V4 representation occupies 260.20 MiB, saving 635.80 MiB. The accounting includes packed payload, metadata and residual keys, but excludes temporary kernel workspace. This is a concrete tensor-storage result, not a measured decrease in the peak allocation of a complete serving process.

The Peak Occurs Before the Smaller Representation Exists

The end-to-end prototype first builds a BF16 draft cache during prefill and then packs it. Its highest allocation can therefore occur before the persistent compressed representation provides any benefit. The reported peak allocated GPU memory is 8.87 GiB for BF16 speculative decoding and both packed variants at 1K, and 10.76 GiB for all three at 16K. Target-only decoding, measured before loading the draft, peaks at 5.98 and 7.69 GiB respectively.

This is why persistent bytes and peak allocated bytes must be separate columns. A compression method can succeed at the former while a conversion phase dominates the latter. A streaming prefill path or another implementation change might alter that lifetime, but the paper does not measure such a fix. It would be misleading to convert the 67–71% packed draft-byte reduction into a claim that the prototype fits proportionally larger models, contexts or concurrent batches.

The Runtime Result Goes in the Opposite Direction

At 1K context, mean time for a 256-token continuation is 7.10 seconds for target-only decoding and 9.68 seconds for BF16 speculative decoding. Packed K4V4 takes 12.77 seconds with no residual window and 12.87 seconds with the 128-key window. At 16K, target-only takes 8.65 seconds, BF16 speculation 14.54 seconds, and packed variants 17.20 and 17.46 seconds. These timings include prefill and exclude warmup and extra sequential audits.

Relative to BF16 speculation, the packed variants are 31.9–32.9% slower at 1K and 18.3–20.1% slower at 16K. A kernel diagnostic supports the distinction between viable storage and fast execution: directly reading packed data takes 0.708 milliseconds per layer at 32K, better than rebuilding BF16 first at 1.448 milliseconds, yet far slower than native BF16 attention at 0.103 milliseconds. The paper does not attribute every end-to-end cost to that kernel, and the measurements are preliminary, not production throughput or concurrency results.

An Unchanged Correction Rule Is Not an Exact Implementation

Speculative-decoding algorithms are designed to preserve target behavior through verification and correction. Leaving the target cache in BF16 keeps draft quantization from changing the mathematical target correction rule. It does not prove that a batched implementation handles cache rollback, positions and numerical behavior exactly. This study's audit makes that boundary unusually visible, including failures when the draft cache is still BF16.

In the 1K packed-prototype campaign, BF16 speculation matches target-only greedy sequences on 3 of 24 prompts; packed residual-zero and residual-128 variants match on 4 and 5. At 16K, all three configurations match on only 2 of 12. A mismatch means at least one output token differs. The authors do not know the cause and do not assume harmless near ties. A storage-versus-speed comparison can still reveal engineering problems, but these timings cannot establish a correct lossless accelerator. The sequential path's successful checks must remain separate from these unresolved batched results.

Useful Defaults Are Not Universal Optima

The default residual window is 128 keys, but the study finds no clear acceptance benefit from making that window longer. Removing it raises draft-byte savings from 66.33% to 70.56% in the cited Qwen control. The 256-token tests likewise give no resolved gain from residual 128 over zero. That does not prove all residual windows are wasteful; it means this one is a configuration choice, not an experimentally established optimum.

The precision-allocation appendix offers another negative result. At a three-bit budget, a per-layer policy profiled directly on speculative acceptance performs 0.48 points worse than a quality-profiled policy, with a reported interval excluding zero, and has higher KL. Those policy experiments have their own narrow model and verifier scope. They do not support the assumption that a more task-specific search objective automatically delivers a better allocation than a simpler draft-quality measure.

What to Measure Before Turning This Into a Serving Claim

An evaluation should keep four gates separate: representation quality, persistent compressed bytes, peak allocation over the full request, and correct end-to-end latency or throughput. The paper clears meaningful parts of the first two. Its current prototype does not clear the latter two. Correctness checks should compare complete emitted token sequences with an independent target-only path, including an unquantized-draft baseline, before calling a speed result lossless.

Only after those checks would it make sense to test fused packed-cache attention, incremental packing during prefill, larger batches, concurrent requests and latency service objectives. These are proposed next measurements, not results reported here. The repository distinguishes frozen outputs from experiments requiring reruns, and notes that some released launchers now use a different sequential-verifier protocol from the historical batched runs. Replaying archived results, rerunning the current launcher and independently validating a new serving integration are different activities.

Assumptions and limitations

This article analyzes the complete preprint, including its verifier audit and appendices, plus released artifact documentation. It does not independently rerun the model experiments, reproduce their GPU timings or audit all implementation code. All performance and quality measurements are author-reported. The main experiment uses greedy batch-one decoding, mostly short continuations, and six model pairs; long-context and packed-system results are narrower. Stochastic sampling, maximum concurrency and production throughput are not established.

The upstream KIVI, speculative-decoding and QuantSpec sources provide design context, not independent confirmation of this new study. QuantSpec studies a different self-speculative design, so its speed claims cannot be borrowed to fill this prototype's missing speedup. The controlled key-quantizer intervention changes both grouping and symmetric/affine convention. Batched-verifier mismatches limit the interpretation of acceptance comparisons that used it, while the separate sequential and verifier-free probes remain useful within their protocols. Smaller stored tensors do not by themselves imply lower peak memory, faster decoding, unchanged text outputs or a safe deployment.

Worked Example: Storage Savings Versus Peak Allocation

This invented cache-lifetime example excludes weights, activations and workspace. The target cache is 12 MiB and the draft BF16 cache is 8 MiB. Compressing the draft to 3 MiB reduces persistent combined cache storage from 20 to 15 MiB, a 25% saving. If conversion allocates the packed draft before freeing the original, the modeled overlap peak becomes 23 MiB. Even a zero-overlap conversion would still have encountered the original 20 MiB prefill state. These are illustrative numbers, not the paper's measurements or a prediction of an actual allocator.

from fractions import Fraction

target = 12
draft = 8
packed = 3
before = target + draft
after = target + packed
saved = 1 - Fraction(after, before)
overlap = before + packed
prefill_peak = max(before, after)
print("persistent", before, after)
print("saved_percent", 100 * saved)
print("overlap_peak", overlap)
print("no_overlap_peak",
      prefill_peak)
assert after == 15
assert saved == Fraction(1, 4)
assert overlap == 23
assert prefill_peak == 20

Sources & Provenance

The gate-based interpretation and executable cache-lifetime example are original analysis. Reported memory, accuracy, acceptance and latency numbers belong to their cited experiments. No personal hardware test, professional credential or new serving benchmark is claimed.

Comments

Kubernetes Liveness Is Not Readiness: Why a Health Check Can Amplify an Outage

KUBERNETES FAILURE RECOVERY A failed health check is an observation. Restarting a container and withholding traffic are different responses, and choosing the wrong one can amplify an outage. By Hossam The important question is what failure should trigger A web application loses access to its shared database. Its processes are still running, but requests that require database access fail. A health endpoint reports the dependency problem. Should Kubernetes restart every application container, or stop directing those requests to it? The endpoint alone cannot choose the right recovery action. Kubernetes separates startup, liveness and readiness probes because they answer different operational questions. Startup protects initialization, liveness can trigger container termination and restart handling, and readiness controls eligibility for traffic from matching Services. A green check is meaningful only within the check and response that were designed. This is a documentation-based de...

Editorial Governance & Pages