StrataFlux Architecture: Selective Precision Streaming, Fetch Lead/Lag Scheduling, and Turning Host Memory Into an Accuracy Tier
LLM INFERENCE
A new system from Cornell and CMU treats quantization error as data to be fetched, not fate: keep 2-bit weights and KV on the GPU, stream residual precision for only the channels and tokens that matter this step, and hide every transfer inside the residual stream's own schedule.
By Ossam Hashim
KEY FINDINGS
the accuracy ceiling of low-bit inference is a scheduling problem, not a storage one
- StrataFlux stores quantized weights and KV on the GPU as the resident base and keeps the missing precision (quantization residuals) in host RAM or on SSD, fetching it selectively for the channels and tokens the current step actually uses.
- Weight refinement picks the input channels with the largest activation magnitudes per 1,024-channel chunk; a ridge-regularized "refinement fitting" step lets fetched columns compensate for the columns left behind.
- Fetch lead and fetch lag decouple selection from application across whole Transformer blocks: selection runs on an earlier residual-stream activation, and refinements land several blocks late, buying transfer time that projection-local overlap (DecDEC-style) cannot.
- KV refinement nests sub-interval codes inside the resident KVarN codes and selects about 0.78 percent of token groups by attention weight, adding 4 bits per selected key and value element.
- Reported results: 2-bit Llama-3.1-8B perplexity falls from 9.46 to 8.40 while keeping 95.4 percent of the quantized base's throughput on an RTX 5090, and Qwen3-8B's AIME accuracy climbs from 11.0 to 47.1 percent with 2-bit weights and KV at unchanged GPU memory.
The False Choice Between Bits and Bytes
Quantization and offloading have always been presented as opposite answers to the same capacity wall. Quantize and you keep everything in fast memory but throw away precision; offload and you keep precision but pay transfer latency in the middle of generation. StrataFlux's authors argue the choice is an artifact of deciding the representation before deciding the placement. A 2-bit code does discard information, but the discarded part - the residual between the true weight and its dequantized approximation - is itself just data, and data can live in a cheaper tier. The GPU keeps the compact base for bandwidth; host RAM and SSDs keep the refinement bits; and the system streams refinement only where the current computation can feel it. The insight that makes this practical is the same one behind sparse attention and salient-weight methods: importance in these models is violently uneven, and it is input-dependent, so a static representation is the wrong unit of design.
Residuals Are the Refinement Currency
For a projection, the error of a quantized weight matrix applied to an input is exactly the product of the input with the residual matrix. Because each residual column's contribution scales with its activation magnitude, the largest few activations identify which columns are worth fixing this step. StrataFlux selects top-k channels per 1,024-channel chunk separately for the query/key/value, attention-output, gate/up, and down projections, fetches those columns' 4-bit refinement values from the backing tier, and adds their products to the quantized output. A subtlety the paper calls refinement fitting adjusts the stored refinement values with a ridge-regularized least-squares objective so that the k fetched columns partially compensate for the columns that were not fetched - calibration that knows selection will be sparse. For MLPs there is a second, neuron-wise scheme that bundles gate, up, and down refinements per selected neuron; the authors report neither wins universally, so both ship.
Fetch Lead, Fetch Lag, and the Residual Stream as a Scheduler
Prior dynamic-refinement work fetched columns only once the projection's input existed and needed the result before that projection ended, confining all overlap to a single kernel's execution window. StrataFlux stretches the interval across blocks. Fetch lead applies the projection's normalization to the residual stream s blocks early, selects channels, and starts the transfer before the base computation even reaches that projection. Fetch lag applies attention-output and down-projection refinements to the residual stream s blocks after their nominal addition point, on the argument that a refinement added slightly late still reduces divergence from the full-precision model. Nsight Systems traces in the paper show the effect concretely: at s = 1 with k = 32 on AWQ 3-bit weights, block execution drops by 30.7 percent against projection-local scheduling, because the refinement work disappears into neighboring blocks' compute. The shift s is tuned per projection set under a configurable slowdown budget - slower 2-bit bases hide refinement almost for free, while faster 3-bit kernels need the full one-block shift.
KV Gets the Same Treatment
The KV cache version nests refinement inside the resident quantizer's own codes: each KVarN code's quantization interval is subdivided, and the sub-interval indices become refinement bits stored in the backing tier. Selection uses attention itself - the current query's softmax weights against the resident low-bit keys score each token group, and roughly 1 in 128 groups gets its keys and values refined, fused into the weighted sum so the correction lands inside the attention kernel. Because selection depends on the very scores being computed, a small non-overlappable region remains; contiguous group fetches and the low selection rate keep it cheap. On unified-memory platforms the same machinery runs against a RAM cache filled by background SSD reads, and a skip-on-miss policy means a late transfer degrades gracefully to base precision instead of stalling the decode loop.
Reuse and the Miss You Never Wait For
The system's quietest result is that selections repeat. Measured on Llama-3.1-8B, the channels and token groups picked at one step overlap heavily with the picks of the next several dozen steps, so a modest cache of fetched refinement bits - 128 MiB each for weights and KV on DGX Spark - serves future steps without new transfers. That converts refinement streaming from a per-step latency risk into an amortized bandwidth cost, and it is what lets the same design span an RTX 5090 with pinned host RAM and a MacBook Air M3 reading refinement bits off an SSD. The implementation is end-to-end inside llama.cpp with CUDA and Metal backends, evaluated across RTX 5090, 5070 Ti, and RTX PRO 6000 cards plus DGX Spark and the M3 MacBook Air, on Llama-3.1-8B, Llama-3.3-70B, Qwen3-8B/32B/30B-A3B, Qwen3.5-9B, and DeepSeek-V2-Lite.
What the Numbers Say
The headline figures, all vendor-reported: on 2-bit Llama-3.1-8B-Instruct weights, WikiText-2 perplexity drops from 9.46 to 8.40 while decode keeps 95.4 percent of the quantized base's throughput (234.4 versus 245.8 tokens per second) on an RTX 5090; at the same 3.9 GB VRAM budget, llama.cpp's BF16 partial offloading manages 5.1 tokens per second. Weight-only refinement lifts Qwen3-8B's AIME score from 19.2 to 46.7 percent on YAQA 2-bit weights, and the full system with 2-bit weights and KV raises it from 11.0 to 47.1 percent at unchanged GPU memory. At 128K context, 3-bit weights with 2-bit KV plus refinement matches the perplexity and speed of vLLM running 4-bit AWQ weights with an 8-bit KV cache while using about half the GPU memory. KV-only refinement on BF16 weights recovers near-BF16 PG-19 perplexity and RULER accuracy from 2-bit keys and values. The pattern across tables is consistent: the accuracy recovered costs single-digit percentages of throughput, because almost every transfer is hidden, reused, or skipped.
Assumptions and limitations
All performance and accuracy figures are the authors' own measurements on their llama.cpp implementation, and gains are largest exactly where the technique has the most headroom - very low-bit bases and reasoning benchmarks with heavy quantization damage; at 3-4 bits the recovered margin narrows, some configurations show small regressions, and the approach presumes decoding is memory-bound enough to leave compute idle for refinement kernels. The 44-table appendix suggests a design space that is still being mapped rather than settled, and results on unified-memory platforms depend on SSD and RAM-cache behavior that will vary by device. StrataFlux is explicitly a provisional system name, and no open-source release is linked from the paper at the time of writing.
Worked Example: Paying for Precision Only Where It Counts
The mechanism in miniature: a small projection, a crude 2-bit-style quantization, and a refinement pass that fetches residual columns only for the three largest-magnitude activations. The script measures how much of the quantization error a handful of fetched columns cancels - the same uneven-importance effect StrataFlux exploits at 8B-parameter scale, and the reason selective streaming can afford to exist.
import random
random.seed(7)
D, K = 16, 3
W = [[random.gauss(0, 1)
for _ in range(D)]
for _ in range(8)]
def q2(v):
return round(v * 2) / 2
Q = [[q2(w) for w in r] for r in W]
R = [[w - q for w, q in zip(r, s)]
for r, s in zip(W, Q)]
x = [random.gauss(0, 1)
for _ in range(D)]
def proj(M):
return [sum(a * b
for a, b in zip(r, x))
for r in M]
y = proj(W)
yq = proj(Q)
err0 = sum((a - b) ** 2
for a, b in zip(y, yq))
sel = sorted(range(D),
key=lambda i: -abs(x[i]))
top = sel[:K]
yr = []
for q, r in zip(yq, R):
a = sum(x[i] * r[i]
for i in top)
yr.append(q + a)
err1 = sum((a - b) ** 2
for a, b in zip(y, yr))
cut = 100 * (1 - err1 / err0)
tot = sum(v * v for v in x)
kept = sum(x[i] ** 2 for i in top)
print(f"base MSE: {err0:.4f}")
print(f"refined: {err1:.4f}")
print(f"error cut: {cut:.0f}%")
print(f"fetched {K}/{D} columns")
sh = 100 * kept / tot
print(f"|x| share kept: {sh:.0f}%")
Sources & Provenance
- Primary source: StrataFlux: Memory-Hierarchy-Native Low-Bit LLM Inference (Hyun, Zhao, Seo, Abdelfattah; Cornell University and Carnegie Mellon University), arXiv abstract and arXiv full text, submitted November 5, 2026.
- Related analysis on this blog: the Bevy analysis covers a complementary attack on offloading stalls for MoE experts, and the BitNet analysis covers the extreme low-bit end of the weight spectrum.
- All throughput, perplexity, and benchmark figures above are attributed to the paper's evaluation sections and tables; this article's worked example is an independent toy illustration, not a reproduction of the paper's experiments.
Analysis by Ossam Hashim for GlobalTechDigest. Published November 6, 2026.
Comments
Post a Comment