Header Ad Placement

FastKVComp: Fused Hadamard-Quantization Kernels, Searchable Lossless Codecs, and Making KV Compression Pay at 300 Gbps

LLM SERVING

Compressing the KV cache before shipping it across a disaggregated serving cluster only helps if the codec is faster than the network it relieves. A new data path shows the original pipeline was quietly failing that test - and fixes both ends of it.

By Ossam Hashim

KEY FINDINGS

  • Disaggregated serving turns the KV cache into a communication payload, and compression only pays when encode plus decode time is smaller than the transfer time saved. On modern 100-400 Gbps interconnects that inequality leaves almost no room for a slow codec - yet the original KVServe path ran its lossless stage below 100 GB/s.
  • FastKVComp rebuilds the execution path without touching compression policy: the same randomized Hadamard transform, head-importance mixed precision and min-max quantization, but fused into a single tiled TileLang kernel that never materializes the intermediate tensor - a 3.94 to 5.85x speedup on the transform-quantization stage.
  • The fixed nvCOMP lossless backend is replaced by a searchable one built from composable LC primitives (tuple reordering, bit shuffling, zero-oriented encoding), chosen by an offline Pareto search over encode latency, decode latency and compressed size; the selected pipeline trades ratio (6.34-6.36x versus nvCOMP's 8.85-9.17x) for 6.0-7.4x faster encoding.
  • End to end on two RTX 5090s over a 306.5 Gbps host-staged path, FastKVComp improves output throughput 1.83-2.04x over raw transfer and cuts job completion time 45.3-51.1%, while the original KVServe - despite shrinking payloads 9.4-10.1x - leaves most of the saving on the table through execution overhead.
  • On a two-node 200 Gb/s InfiniBand deployment (RTX PRO 6000 prefill, H100 decode) the gains hold up across the wire: 30.2-38.6% lower job completion time and 44.2-62.6% higher throughput than raw transfer, with compression throughput of 139 GB/s on the prefill GPU against the original 35.

An Inequality That Bites at High Bandwidth

The logic of KV compression in disaggregated serving is a race: the time spent encoding and decoding must be smaller than the transfer time saved by sending fewer bytes. For years that race was easy to win, because networks were the slow part. But as serving clusters adopt 100-400 Gbps interconnects and RDMA data paths, the saved transfer time shrinks while codec costs stay put - and the sign of the inequality can flip. A team at the Institute of Computing Technology, Chinese Academy of Sciences (Wu, Liu, Ma and colleagues) measured the popular KVServe pipeline and found its execution path, not its compression policy, was the problem: transformation and quantization ran as separate GPU kernels round-tripping a full intermediate tensor through HBM, and the nvCOMP lossless stage stayed below 100 GB/s. Their answer, FastKVComp, keeps KVServe's semantics intact and optimizes only how a selected strategy executes. That framing matters: every ablation compares against identical compression policies, so the gains are attributable to the data path alone.

Fusing the Frontend, Searching the Backend

The first fix fuses the randomized Hadamard transform, quantization-statistics computation and low-bit conversion into one tiled TileLang kernel. Each KV tile is loaded once, transformed values are consumed on-chip instead of materialized, and the quantized output lands directly in a preallocated all-layer buffer; the receiver's reconstruction operator fuses the inverse chain the same way. The measured effect: the transform-quantization stage drops from 27.3-34.4 ms to 5.9-6.9 ms. The second fix treats the lossless codec as a searchable component rather than a fixed library. Using the LC framework's composable GPU primitives, the authors evaluate two- and three-stage pipelines offline on real quantized KV buffers and keep the Pareto-efficient set, selecting TUPL8_1 - tuple reordering - followed by bit shuffling and zero-oriented redundancy encoding. That pipeline compresses less than nvCOMP ANS (6.34-6.36x against 8.85-9.17x) but encodes 6.0-7.4x faster (0.44-0.53 ms versus 3.18-3.27 ms) and decodes about 4.9x faster, wrapped in a native GPU runtime with device-pointer APIs so the online path never touches LC's file-based CLI. After the frontend fusion, lossless coding falls from 31.7-35.1% of the encoding path to 6.1-8.2% - and the pipeline, once chosen, stays fixed at runtime.

What the Measurements Say

On the local rig - two RTX 5090 GPUs trading KV over a host-staged NCCL shared-memory path measured at 306.5 Gbps, serving Llama-3.1-8B and Qwen2.5-7B on LongBench workloads with up to 20K-token inputs - FastKVComp beats raw uncompressed transfer by 1.83-2.04x in output throughput and cuts job completion time by 45.3-51.1%. The comparison that stings is against the original KVServe: it already reduces KV volume 9.4-10.1x, yet its fragmented execution offsets most of that saving, and FastKVComp nearly doubles throughput over it. The remote deployment tells the same story over 200 Gb/s InfiniBand between an RTX PRO 6000 prefill node and an H100 decode node: 30.2-38.6% lower completion time, 44.2-62.6% higher throughput, with the full encode path down from 30.6-37.6 ms to 6.4-7.4 ms. The ablations carry the design lesson. Fused-only nearly matches the full system on the fast local path, while LC-only stays within 2.1% of the original - a cheaper codec alone just moves more bytes. And fused-only versus full FastKVComp differs by at most 1.3%, which the authors read correctly: the right lossless backend depends on the communication regime, not on compression ratio. On a slower network, nvCOMP's higher ratio can win; at 300 Gbps, latency wins.

Assumptions and limitations

All results are the authors' own, from a preliminary implementation evaluated on two 7-8B models (Llama-3.1-8B, Qwen2.5-7B) over three LongBench datasets with up to 20K-token inputs and 50 requests per configuration, on specific hardware (RTX 5090 pair over PCIe Gen5, RTX PRO 6000 plus H100 over 200 Gb/s InfiniBand); other models, context lengths, codecs and interconnects will shift the numbers, and the authors note the best lossless backend is regime-dependent. Quality preservation inherits KVServe's compression policy and is not re-evaluated in this paper. The worked example is an invented toy model using the paper's measured constants, not its implementation.

Worked Example: When the Codec Eats the Saving

The paper's break-even rule compresses into a few lines: the margin is the transfer time saved minus the codec cost, and its sign decides whether compression helps at all. The toy below uses the paper's measured order-of-magnitude constants - a 9x-ratio pipeline at 35 ms codec cost against a 6.3x-ratio one at 7 ms - and sweeps network bandwidth. On a slow link both pipelines pay. At 38 GB/s (roughly the measured PCIe path) only the fast one does. At 100 GB/s the slow pipeline is deep underwater while the fast one still clears the bar.

import statistics as st

# Table 1 throughput, GB/s
T = {
    'PRO6000': (34.773, 139.218),
    'H100': (27.659, 91.606),
}
gains = [f / o
         for o, f in T.values()]
print('throughput gains:',
      [round(g, 2) for g in gains])
print('mean gain:',
      round(st.mean(gains), 2))

def margin(d, r, bw, cost):
    saved = (d - d / r) / bw
    return saved - cost

D = 1.0  # GB KV payload
for bw in [10, 38.3, 100]:
    m_o = margin(D, 9.0, bw, 0.035)
    m_n = margin(D, 6.3, bw, 0.007)
    print(bw, 'GB/s margin ms:',
          round(m_o * 1e3, 1),
          round(m_n * 1e3, 1))

Sources & Provenance

  • Beyond Compression Ratio: High-Throughput KV Compression for Disaggregated LLM Serving (Wu, Liu, Ma, Chen, Tan and Tao, 4 Nov 2026) - arXiv abstract
  • Full paper (PDF), including the pipeline search, component breakdowns and InfiniBand deployment - arXiv full text
  • Background referenced inside the paper: KVServe, TileLang, LC, nvCOMP, Mooncake.

All performance numbers are attributed to the preprint as published. Sources accessed 7 November 2026. This analysis is independent editorial content for GlobalTechDigest.

Comments

Kubernetes Liveness Is Not Readiness: Why a Health Check Can Amplify an Outage

KUBERNETES FAILURE RECOVERY A failed health check is an observation. Restarting a container and withholding traffic are different responses, and choosing the wrong one can amplify an outage. By Hossam The important question is what failure should trigger A web application loses access to its shared database. Its processes are still running, but requests that require database access fail. A health endpoint reports the dependency problem. Should Kubernetes restart every application container, or stop directing those requests to it? The endpoint alone cannot choose the right recovery action. Kubernetes separates startup, liveness and readiness probes because they answer different operational questions. Startup protects initialization, liveness can trigger container termination and restart handling, and readiness controls eligibility for traffic from matching Services. A green check is meaningful only within the check and response that were designed. This is a documentation-based de...

Editorial Governance & Pages