SenseNova-Loop-9B Architecture: Shared-Weight Looped Decoding, Two-Stream mHC Residual Streams, and Hybrid Linear-Full Attention for 256K-Context Agentic Work

MODEL ARCHITECTURE

SenseTime's open SenseNova-Loop-9B-Preview runs 56 decoder-layer executions while storing only 32 layers of weights. The mechanism is a shared-weight looped block stabilized by manifold-constrained hyper-connections, and it ships as an execution configuration, not just a checkpoint.

By Ossam Hashim

KEY FINDINGS

What the release establishes

• SenseNova-Loop-9B-Preview, released October 30, 2026 under Apache-2.0, executes 56 decoder layers per forward pass from 32 distinct layers: layers 4 through 27 run twice with shared weights, written as B(B(h)).
• Two-stream manifold-constrained hyper-connections (mHC) carry a pair of residual streams through the looped section, with learned read/mix/write mappings and Sinkhorn normalization (20 iterations) constraining the stream-mixing map; mHC routing parameters are stored in FP32 even though the backbone is BF16.
• Attention is hybrid: each four-layer group stacks three linear-attention layers and one full-attention layer, and the loop boundaries align with complete groups so the pattern is identical on both passes.
• The card documents training at sequence lengths up to 262,144 tokens over text, images, code, and tool trajectories, targeting long-context and agentic workflows; MTP-based inference acceleration is available, with Dspark and Dflash2 still in training.
• The loop range, pass count, and mHC parameters are part of the checkpoint's execution configuration - an inference engine must load and reproduce them, which is why support lands as custom modeling code first (a llama.cpp feature request is open) rather than as a generic format.
• At roughly 9B stored parameters and a reported ~14.6B-parameter inference footprint, the model trades extra compute per token for a smaller weight set - a depth-for-parameters exchange, not a free lunch.

Why Loop a Decoder

The recurring tension in model design is that capability tracks depth, but depth is what you pay for twice: once in parameters and again in serving memory. Conditional-compute designs - the family our Mixture-of-Depths analysis covered - attack this by spending compute selectively per token. SenseTime's release, covered by TechNode on November 2, takes the orthogonal route: spend the same weights repeatedly. A contiguous block of the decoder simply runs twice, so the network's effective depth grows while the checkpoint does not. This is the recurrent-depth idea made operational. The hard part was never the looping itself but keeping a looped network trainable and its inference reproducible, and that is where the two other mechanisms in this release - hybrid attention alignment and two-stream mHC - do the real work.

Anatomy of the Loop

The decoder is 32 layers with hidden dimension 4,096, feed-forward dimension 12,288, 16 query heads against 4 KV heads, and a 248,320-token vocabulary. Execution order is fixed: layers 0-3 run once, the block spanning layers 4-27 runs twice with identical weights, and layers 28-31 plus the output head close the pass. That yields 56 layer executions against 32 stored layer weight sets - the card's "~9B total, ~14.6B inference parameters" framing. The attention layout is what makes the loop clean. Full attention and linear attention interleave in four-layer groups (three linear, one full). Because the loop range [4, 28) contains exactly six complete groups, pass two sees the same attention pattern as pass one; nothing about the masking or the linear-attention state handling changes between iterations. Linear attention is also what makes a 256K-token training context tractable at this scale, with full-attention layers providing the global mixing steps.

Two-Stream mHC: Keeping the Loop Stable

Naively looping a block risks degrading into either washed-out representations or runaway residuals, and mHC is the release's answer. Inside the looped section, two residual streams run alongside the layer computations. Each layer reads from the streams through learned mappings, mixes information across the two streams, and writes its output back into the residual state through learned mappings as well. The stream-mixing map is constrained by Sinkhorn normalization - 20 iterations of alternating row and column normalization - which projects it toward a doubly stochastic matrix: mixing stays a weighted reallocation of stream content rather than an arbitrary linear map that could amplify or erase a stream over two passes. Notably, the card exports the mHC routing parameters in FP32 while the backbone ships in BF16 - a signal that the mixing dynamics are precision-sensitive in a way the bulk matrix multiplications are not. The worked example below reproduces both the execution arithmetic and the Sinkhorn constraint in miniature.

Systems Consequences

For serving stacks, the release is a reminder that "a model" is increasingly an execution schedule plus weights. The loop range, pass count, and mHC configuration live in the checkpoint's execution configuration, and any compatible runtime must reproduce that order exactly - hence the custom Transformers modeling code (checked environment: Transformers 5.18.0, PyTorch 2.11.0, CUDA 13.0) and the pending llama.cpp port request. Weight-bandwidth economics also shift: the KV cache and activation traffic behave like a 56-layer model, while weight reads behave like a 32-layer one, which is precisely the regime where memory-bound decode serving benefits. Documented acceleration is MTP today, with Dspark and Dflash2 speculative paths still training. Per the card, practical speedups depend on workload, hardware, and serving configuration, and the published benchmark table is vendor-produced - treat it as a claim awaiting independent replication, as the early third-party benchmark write-ups have begun to attempt.

Assumptions and limitations: the Sinkhorn example uses an arbitrary 2x2 mixing matrix to demonstrate the normalization dynamics; the model's actual learned maps are larger and trained, not exponential-initialized. All architectural figures (loop bounds, layer counts, dimensions, Sinkhorn iterations, parameter totals) come from the model card; benchmark results are vendor-reported and the technical report is marked TBD at publication time. No independent run of the model was performed for this piece.

Worked Example: Loop Arithmetic and the Sinkhorn Constraint

Two computations define this architecture. First, the execution schedule: layers 0-3 once, layers 4-27 twice, layers 28-31 once. Second, the mHC mixing map: a learned matrix pushed toward doubly stochastic form by Sinkhorn iterations, so stream mixing redistributes rather than amplifies. The script below checks both, with the release's own loop bounds and iteration count.

import math

def norm(r):
    s = sum(r)
    return [x / s for x in r]

def sinkhorn(m, iters):
    e = lambda r: [math.exp(x) for x in r]
    m = [e(r) for r in m]
    for _ in range(iters):
        m = [norm(r) for r in m]
        m = [list(c) for c in zip(*m)]
        m = [norm(c) for c in m]
        m = [list(r) for r in zip(*m)]
    return m

sched = list(range(4))
sched += list(range(4, 28)) * 2
sched += list(range(28, 32))
print("executions:", len(sched))
print("distinct:", len(set(sched)))

m = sinkhorn([[2.0, 0.5], [0.5, 2.0]], 20)
for r in m:
    print(["%.4f" % x for x in r])
sums = [round(sum(r), 6) for r in m]
print("row sums:", sums)

Assumptions and limitations: the Sinkhorn example uses an arbitrary 2x2 mixing matrix to demonstrate the normalization dynamics; the model's actual learned maps are larger and trained, not exponential-initialized. All architectural figures (loop bounds, layer counts, dimensions, Sinkhorn iterations, parameter totals) come from the model card; benchmark results are vendor-reported and the technical report is marked TBD at publication time. No independent run of the model was performed for this piece.

Sources & Provenance

• SenseNova-Loop-9B-Preview model card (released Oct 30, 2026) - architecture, specifications, deployment environment
• OpenSenseNova/SenseNova-Loop repository - project code and release notes
• TechNode release coverage (Nov 2, 2026) - announcement context
• MindStudio benchmark write-up (Nov 1, 2026, third-party) - early independent numbers
• llama.cpp support request - inference-engine portability status

Analysis is based on the primary sources listed above; benchmark claims are attributed to their producers and not independently verified.