Muir Architecture: Sensitivity-Allocated Gating for Linear Attention
SEQUENCE MODELING
Muir re-prices every gate in a gated delta-rule linear attention layer by how much the state actually depends on it, then spends the savings on a wider feed-forward network.
By Ossam Hashim
KEY FINDINGS
- GDN-2 spends one-sixth of every layer's parameter budget on full-rank erase and write gate projections; Muir's analysis shows state updates are far less sensitive to erasure than to writes, so the erase gate can be low-rank and the saved parameters reinvested in the feed-forward network.
- Slow-decaying state rows accumulate much larger norms than fast rows and dominate unweighted readout, forcing stronger write-time erasure; a matched-filter readout gate combining a static channel profile with dynamic per-token modulation re-weights rows and damps that interference.
- Standard chunkwise training algorithms break down numerically at the large chunk sizes wide-matrix accelerators need, so Muir trains with a boundary-pivoted chunkwise kernel that splits relative decays without division or full-chunk 3D decay tensors.
- Across 166M to 1.34B parameters, Muir lowers validation NLL by 18.0 to 20.4 millinats over GDN-2 and 25.0 to 36.2 millinats over Transformer, while matching GDN-2 throughput on 8 TPU7x chips at 0.5M tokens per step.
- At 1.34B, Muir extrapolates 2,560x from its 4K training length to 10M-token streams inside a 25 MB float32 state without perplexity drift and holds near-100% passkey retrieval to 1M tokens with static readout, where a Dynamic YaRN Transformer diverges past 8K and would need a 2.1 TB bfloat16 KV cache at 10M.
Where the parameters go in a gated delta rule
Linear attention compresses a variable-length context into a fixed-size key-value state, giving linear-time training and constant-memory inference. Delta-rule recurrences fight the cross-key interference that a finite state invites by erasing the state's current prediction for an incoming key before writing a new value. Gated DeltaNet-2 (GDN-2) is the sharpest version of this idea: channel-wise decay, plus decoupled erase and write gates so every state row and value channel updates on its own terms. That expressiveness has a price. GDN-2 allocates a full-rank d-by-d projection to the erase gate and another to the write gate, while the decay and output gates make do with low-rank bottlenecks. Those two dense matrices consume 2d-squared parameters, one-sixth of the 12d-squared layer budget, and force the feed-forward network to shrink by more than 25 percent to keep the layer iso-parameter with a Transformer. Muir, from researchers at UT Austin and Google, asks whether that allocation matches what the state actually needs, and finds that it does not.
Why erasure tolerates a low-rank gate
The paper's first result is an asymmetry: state updates are far less sensitive to erasure than to writes. Key activations, sharpened by unit normalization, concentrate key energy onto a sparse subset of channels, so the erase operator touches only a narrow slice of the state on each step. Value writes behave differently, scaling all channels at once, so the write gate needs its full-rank resolution. Parameterizing the erase logit through a biased low-rank bottleneck also regularizes the variance of oblique erasures rather than injecting high-rank noise into the state. Muir acts on this by fusing the decay, erase, write, and output gates into a single sensitivity-allocated low-rank projection. The erase gate gets the small rank its sparse footprint justifies, the write gate keeps enough rank to steer every channel, and the parameters freed by the fusion are reinvested in a wider feed-forward network. The authors attribute the bulk of Muir's validation NLL gain over GDN-2 to this reallocation plus the FFN expansion.
Slow rows hoard magnitude
The second result concerns readout. Because state rows decay at different rates, a slow-decaying row accumulates value writes over hundreds to thousands of tokens while a fast row forgets quickly. The paper derives a closed-form scaling law for row norms and shows the slow rows grow much larger, so an unweighted readout is dominated by old, slow-row content. Worse, the model compensates by learning stronger write-time erasure just to suppress the interference, which prematurely flushes older associations across all channels. The fix comes from classical signal processing. Weighting each query channel by a matched-filter readout gate, proportional to the row's signal survival and inversely proportional to its accumulated energy, provably minimizes retrieval error at every lag. Muir implements this as a static channel profile derived from the horizon-averaged signal survival, plus a dynamic per-token modulation, so readout behaves like a correlation receiver instead of a raw sum. The payoff is concrete: static matched-filter readout alone holds near-100 percent passkey retrieval out to 1M tokens, a regime where GDN-2 collapses past 4K to 8K.
A kernel that survives big chunks
Channel-wise decay breaks the standard chunkwise training algorithm. To compute within-chunk interactions with matrix multiplication, the classic construction divides keys by cumulative decay prefixes and solves a lower-triangular system per chunk; at the large chunk sizes needed to saturate wide matrix units, those decay prefixes underflow and the division explodes. GPU kernels dodge this by solving small diagonal sub-blocks sequentially in SRAM at chunk sizes matched to 16-by-16 TensorCores, but accelerators with 64-by-64 and larger matrix units need far bigger chunks to stay busy. Muir's boundary-pivoted chunkwise kernel sidesteps the instability by splitting relative decays at chunk boundaries rather than dividing through the full cumulative prefix, avoiding both the division and any full-chunk 3D decay tensor. On 8 TPU7x chips at 0.5M tokens per step and 1.34B parameters, the kernel lets Muir match GDN-2 throughput despite the extra readout gate, reach its 100B-token validation loss 1.2x faster, and train 2.7x faster than a Transformer at 32K context.
What the numbers say
From 166M to 1.34B parameters, Muir lowers validation NLL by 18.0 to 20.4 millinats over GDN-2 and 25.0 to 36.2 millinats over Transformer, equivalent to 1.8 to 2.0 percent and 2.5 to 3.6 percent perplexity reductions respectively, with commonsense accuracy up by as much as 1.9 points. The dynamic matched-filter readout drives the gain on extractive recall, and at 1.34B Muir beats both baselines on multi-hop and multi-needle retrieval. The extrapolation results are the most striking. Trained at 4K, Muir streams 10M tokens, a 2,560x extension, without perplexity drift, living inside a 25 MB float32 recurrent state. A Dynamic YaRN Transformer diverges past 8K on the same test and would need a 2.1 TB bfloat16 KV cache to represent 10M tokens at all. Constant memory is doing real work here, not just asymptotic bookkeeping.
Assumptions and limitations
All benchmark numbers are the authors' own, measured on models up to 1.34B parameters trained on 100B tokens; behavior at frontier scale is untested. Throughput claims are specific to TPU7x hardware and the custom kernel, and the perplexity gains, while consistent, are single-digit percentages. The passkey and streaming tests probe retrieval, not general reasoning at extreme lengths. The worked example is a toy illustration of row-norm imbalance, not the paper's estimator.
Worked Example: Matched-Filter Readout Restores Balance
A small simulation shows why row-norm imbalance matters. Three state rows decay at 0.9, 0.99, and 0.999 per step while unit keys arrive round-robin. After 400 writes we query for the newest key and measure signal-to-noise: the fresh key's own contribution against the decayed residue of everything else. Unweighted readout lets the big slow rows swamp the answer; dividing each row by its norm, the matched-filter idea in miniature, lifts SNR roughly fourfold.
import math
D = 16
A = (0.9, 0.99, 0.999)
F = [1 + 0.13 * i for i in range(D)]
T = 400
def norm(v):
return math.sqrt(
sum(x * x for x in v))
def kvec(t):
v = [math.sin(t * f) for f in F]
n = norm(v)
return [x / n for x in v]
S = [[0.0] * D for _ in A]
for t in range(T):
r = t % len(A)
a = A[r]
k = kvec(t + 1)
row = S[r]
for i in range(D):
row[i] = a * row[i] + k[i]
ns = [norm(s) for s in S]
print("row norms:",
[round(n, 1) for n in ns])
q = kvec(T)
r = (T - 1) % len(A)
def readout(w):
tot = 0.0
for j in range(len(A)):
dot = sum(q[i] * S[j][i]
for i in range(D))
tot += w(j) * dot
return tot
raw = readout(lambda j: 1.0)
mf = readout(lambda j: 1.0 / ns[j])
s_raw = sum(x * x for x in q)
s_mf = s_raw / ns[r]
print("snr raw:", round(
s_raw / abs(raw - s_raw), 3))
print("snr matched:", round(
s_mf / abs(mf - s_mf), 3))
Sources & Provenance
- Muir: Linear Attention with Sensitivity-Allocated Gating and Matched-Filter Readout (Gangavarapu, Tabak and Schaefer, UT Austin and Google, 5 Nov 2026) - arXiv abstract
- Full paper (PDF), including the sensitivity analysis, matched-filter derivation, kernel design and all scaling experiments - arXiv full text
All performance numbers are attributed to the preprint as published. Sources accessed 6 November 2026. This analysis is independent editorial content for GlobalTechDigest.
Comments
Post a Comment