Header Ad Placement

RidgeFill: Closed-Form Cache Prediction, Exact Prefix Rows, and Faster Prefill Without Training

INFERENCE SYSTEMS

Ask a 27B model to read a 120,000-token prompt and the first thing your GPU buys is 128 seconds of silence. RidgeFill's wager is that most of that wait is redundant: predict what the late layers would have cached, keep exact the rows where prediction is weakest, and run the full model everywhere it actually matters.

By Ossam Hashim

KEY FINDINGS

  • RidgeFill retrofits the cross-layer cache prediction of YOCO and DeepSeek-V4.1-Flash - where the projection is trained into the model - onto models never trained for it. Each late layer gets an affine map from the split layer's hidden state, fitted once in closed form by ridge regression over a single calibration pass. No weights change, no token is dropped, and every decoding step still runs the full model.
  • On Qwen3.8-27B in the authors' llama.cpp fork on one A100, time to first token improves 1.19x to 1.31x at 16K to 120K prompt tokens, depending on whether the 1.2 GiB of int8 maps sit in VRAM or stream from host memory. The cost is memory: +1.6 to +3.0 GiB over stock resident, +0.27 to +1.73 GiB streamed.
  • Quality deltas are small and honestly bounded: RULER moves -1.2 and -1.4 points and LongBench v2 -1.4, with 95% intervals that include zero yet do not rule out losses of a few points. On Qwen3.8-Flash-Next in vLLM the speedup is 1.50x to 1.60x at 9K to 32K tokens, needle retrieval holds at 99.8 against the exact model's 100.0, and novel-token perplexity rises 2.65% [-0.02%, +4.93%].
  • The deeper contribution may be the methodology. Loss averaged over all tokens hides the fill's error, so the authors localize it with needle retrieval inside the filled region, loss on tokens that depend on distant context, and splice tests that swap filled keys or values for exact ones. The culprit differs by architecture: the keys on Llama-3.1-8B, half of it in their scale alone, and the values plus the recurrent state on Qwen3.5-9B.
  • The boundary is sharp. With no training at all, the fitted map lands within 0.03 nats of the released SwiftKV checkpoint's long-document loss and beats POP's operating point while skipping more layers - but on Llama-3.1-8B, which lacks QK-norm, the fill is worse than no fill at all on 32 of 36 long documents, with a worst case of +8.19 nats on 128K of source code.

The Wait Before the First Token

For a long prompt, prefill is the wait before the response, and the user sees nothing until it completes. Before the first output token, a decoder runs every prompt token through every layer; at 120,000 tokens on Qwen3.8-27B, the authors measure 128.10 seconds of that on one A100. Decoding gets the engineering attention because it is the meter running on every generated token, but prefill is what makes a 100K-token document feel like a dial-up connection. This blog has covered two other levers on the same problem: FastKVComp compresses what the cache stores, and KVGuard prices what it costs to keep the cache honest. RidgeFill pulls a different lever entirely - do not compute most of the cache in the first place. The idea has a pedigree: YOCO and DeepSeek-V4.1-Flash train a projection that predicts late-layer states from an early one. The obvious objection is that you have to train it in. RidgeFill's claim is that you do not.

A Ridge Map Where a Trained Projection Used to Be

The construction is disarmingly plain. A deployment picks two numbers: a split layer S and a tail length T. Layers below S run exactly for every token. The last T prompt tokens - the tail - run every layer exactly, and so does every generated token. For the remaining prefix, each late layer's input is predicted from the split layer's hidden state by an affine map, a per-layer matrix and vector fitted by ridge regression, solved in closed form from one pass of the unmodified model over calibration text. The layer's own projections then run unchanged on that prediction. On the 27B deployment - a 64-layer hybrid of 48 Gated DeltaNet and 16 attention layers - the split sits at layer 16, the maps for the 48 late layers are stored as int8 with one scale per 128 weights (1.2 GiB total), and the tail is the last 4,096 prompt tokens. Not every prefix row is predicted: a probe-ranked set of high-attention rows, special tokens, and lexical-net rows run exactly, 28 to 31% of the prefix on the timing prompts. Three deployments are shown: maps resident in VRAM, maps streamed from host memory, and a vLLM configuration whose optional quality mode also keeps exact the rare token classes that carried the most damage in development. Every selector is a rule a server can apply with no training.

Speed Numbers With Intervals Attached

The headline table is careful about what it measures. Time to first token is the warm prompt-processing time, median of three requests, stock and RidgeFill on the same physical GPU. At 16,384 prompt tokens the stock fork takes 12.97 seconds and RidgeFill 10.38 resident - a 1.249x speedup - rising to 128.10 against 100.01 seconds at 120,000 tokens, 1.281x. Streamed maps trade a little speed for most of the memory back: 1.192x to 1.300x across the same range at +0.27 to +1.73 GiB instead of +1.6 to +3.0 GiB. The authors also plot the measured speedup against the ceiling implied if time were proportional to multiply-accumulates, and discuss the shaded gap rather than hiding it. Agreement testing goes beyond benchmarks: on 100 LongBench documents per length, scoring 51,200 positions after the prompt, top-1 agreement with the unmodified model is 94.64% at 16K context with mean KL 0.0894 [0.0399, 0.1556]. Benchmark movements come with intervals that include zero - which the authors read correctly: no detectable catastrophe, no proof of zero cost.

Where the Error Lives, and Where It Breaks

The diagnostic work is what separates this from a speed claim. Averaged loss looks fine; targeted probes do not. Splice tests - swap filled keys for exact ones, then filled values - put the error in the keys on Llama-3.1-8B, with half of it in their scale alone, and in the values and recurrent state on Qwen3.5-9B. That architectural dependence has teeth: Llama has no QK-norm, and at a 3/8-depth split RidgeFill's novel-token loss lands 0.50 nats above the identity fill's, worse than no fill at all on 32 of the 36 documents of 64K tokens or longer, hitting +8.19 nats against exact on 128K tokens of source code. At half depth the ordering reverses, which is a clue about where prediction is safe, not a rescue of the shallow split. Against trained competition the showing is remarkably close: within 0.03 nats of the released SwiftKV checkpoint's long-document loss, and non-inferior to POP's operating point on filled-region retrieval at a 3-point margin while skipping more layers, with lower novel-token loss in point estimate (+0.030 against +0.373 on Llama, +0.023 against +0.044 on Qwen3-8B). A closed-form map should not be trading blows with a trained projection, and the authors document exactly where it does and does not.

Assumptions and limitations

All measurements are the authors' own, on their llama.cpp fork and vLLM configuration, on one A100 PCIe 40 GB for the 27B runs, with median-of-three warm timings; absolute numbers will move with engine, GPU, and prompt mix. The quality intervals include zero but do not rule out losses of a few points, and the authors say so. Maps are per-model artifacts: each model needs its own calibration pass, and the maps cost memory to hold or bandwidth to stream. The Llama-3.1-8B failure at shallow splits is a demonstrated boundary, tied to the absence of QK-norm, and long-document behavior is corpus-dependent - source code was the worst case measured. The reference-study contrasts cover three 8 to 9B models at one split and tail each. The worked example below is an invented toy on synthetic data, illustrating only why a fitted linear map beats reusing the wrong layer's state; every speed and quality figure above is the preprint's own measurement, not this blog's.

Worked Example: The Fill as a Least-Squares Problem

The mechanism in miniature: pretend a model has 8 hidden coordinates, and each late-layer coordinate is an unknown scaled, shifted copy of the split-layer coordinate plus noise. The identity fill - reuse the split layer's state as-is - is the naive retrofit. The ridge fill estimates the scale and shift of every coordinate from 400 calibration samples, in closed form, no gradient step anywhere. On 4,000 fresh samples the identity fill's relative error is 0.3174 and the fitted map's is 0.0247 - 12.9x lower, because the map learned what the late layer actually does with the state instead of assuming it does nothing. Invented data, invented numbers; the shape of the argument is the paper's.

import random

random.seed(11)
D = 8        # hidden dims
LAM = 1e-3   # ridge penalty

# ground truth: each late coord is
# a scaled, shifted copy of the
# split-layer coord, plus noise
a = [0.4 + 0.15 * i
     for i in range(D)]
b = [(-1) ** i * 0.3
     for i in range(D)]

def sample():
    x = [random.gauss(0,
                      1 + i * 0.2)
         for i in range(D)]
    y = [a[i] * x[i] + b[i]
         + random.gauss(0, 0.05)
         for i in range(D)]
    return x, y

calib = [sample()
         for _ in range(400)]

# closed-form ridge, one coord at a
# time: w = cov / (var + LAM)
ahat, bhat = [], []
for i in range(D):
    xs = [c[0][i] for c in calib]
    ys = [c[1][i] for c in calib]
    n = len(xs)
    mx = sum(xs) / n
    my = sum(ys) / n
    var = sum((v - mx) ** 2
              for v in xs) / n + LAM
    cov = sum((xs[k] - mx)
              * (ys[k] - my)
              for k in range(n)) / n
    w = cov / var
    ahat.append(w)
    bhat.append(my - w * mx)

test = [sample()
        for _ in range(4000)]

def ridge_fill(v, i):
    return ahat[i] * v + bhat[i]

def ident_fill(v, i):
    return v

def rel_err(fill):
    num = den = 0.0
    for x, y in test:
        for i in range(D):
            p = fill(x[i], i)
            num += (p - y[i]) ** 2
            den += y[i] ** 2
    return (num / den) ** 0.5

ridge = rel_err(ridge_fill)
ident = rel_err(ident_fill)
print(f"identity: {ident:.4f}")
print(f"ridge:    {ridge:.4f}")
r = ident / ridge
print(f"ratio: {r:.1f}x")

Sources & Provenance

  • RidgeFill: Training-Free Late-Layer Cache Fill for Faster Prefill (Johnston and Smith; 6 Nov 2026) - arXiv abstract
  • Full paper: method, deployment configurations, all evaluation tables, splice studies, derivations - arXiv full text
  • Related coverage on this blog - FastKVComp and KVGuard

All speed and quality figures are the preprint's own measurements as published, on the authors' forks and hardware. Sources accessed 10 November 2026. This analysis is independent editorial content for GlobalTechDigest.

Comments

Kubernetes Liveness Is Not Readiness: Why a Health Check Can Amplify an Outage

KUBERNETES FAILURE RECOVERY A failed health check is an observation. Restarting a container and withholding traffic are different responses, and choosing the wrong one can amplify an outage. By Hossam The important question is what failure should trigger A web application loses access to its shared database. Its processes are still running, but requests that require database access fail. A health endpoint reports the dependency problem. Should Kubernetes restart every application container, or stop directing those requests to it? The endpoint alone cannot choose the right recovery action. Kubernetes separates startup, liveness and readiness probes because they answer different operational questions. Startup protects initialization, liveness can trigger container termination and restart handling, and readiness controls eligibility for traffic from matching Services. A green check is meaningful only within the check and response that were designed. This is a documentation-based de...

Editorial Governance & Pages