HELIX Architecture: Operator-Granularity Partitioning, Profile-Guided DP Search, and Freeing Accelerators From Fixed Roles

INFERENCE SYSTEMS

Every disaggregated serving system answers "where does attention run?" before the workload arrives. HELIX asks a better question: given this model, this batch, this hardware mix - what placement minimizes latency? Then it searches.

By Ossam Hashim

KEY FINDINGS

• Disaggregated LLM serving fixes roles in advance: prefill/decode pools, then attention/FFN (A/F) splits. But operator bottlenecks shift with batch size and context - decode GEMMs move from memory-bound to compute-bound as concurrency grows - so the best split for one workload is wrong for the next.
• HELIX (arXiv Nov 4, IBM Research + TU Dresden) treats placement as a per-operator search problem: profile each stage on each device, model transfer costs across links, then find the latency-minimal partition with dynamic programming over a frontier state that tracks residual fan-out/fan-in dependencies and per-device memory.
• Accelerators join as subgraph executors inside one vLLM engine via torch.compile-compatible backends and piecewise compilation - no per-device serving engine, no predefined model partition - which is what lets hardware with partial operator coverage participate at all.
• Numbers (single-step decode, simulated systems): up to 3x lower latency than A/F disaggregation for dense and sparse transformers and 3.5x for hybrid state-space models; vs a single H100, 3.14-6x (Llama-3.1-8B) and 4.48-7.3x (Bamba-9B) at small batches; Mixtral-8x7B, which does not fit on one H100, runs 5-9.8x faster than a 2xH100 baseline.
• The search rediscovers known splits at the edges and beats them in the middle: around batch 64 the optimal GPU+3DCIM partition converges to the classic A/F split, and in three-way systems (GPU + 3DCIM + DRAM-PIM) it finds placements no named strategy can express - attention on DRAM-PIM, all projections on 3DCIM - worth 1.8-3.5x over A/F.

Disaggregation's Fixed-Role Problem

Serving systems have been carving LLMs along ever-finer lines: first prefill from decode, then attention from FFN. Each cut encodes a bottleneck assumption. Attention is memory-bound (KV reads grow with context), FFN amortizes weights across a batch - until the batch grows enough to flip the FFN compute-bound, or the context grows enough to make KV fetching dominate everything. Analytical work confirms the optimal attention-to-FFN device ratio drifts with KV-cache occupancy and request count. A fixed split is a bet placed before the cards are dealt. Worse, the A/F architecture hardwires the bet into the software: each operator class runs as an independent network-connected engine, so supporting a new accelerator means porting an entire serving engine to it - a non-starter for emerging hardware like processing-in-memory that covers only a few operator types.

Placement as a Search, Not a Split

HELIX moves the decision one layer down, into the engine's data plane. A model becomes a DAG of stages in topological order, where residual connections create fan-out/fan-in structures; each stage has an admissible device set, a profiled compute cost per device, and a memory footprint; each edge has a transfer cost (bytes over directed link bandwidth, plus a startup latency). The objective is the finish time of the last stage. Straightforward DP over (stage, device) fails because a fan-in stage's cost depends on arrivals from all its producers, not just the previous stage - so the state extends to the "frontier": the set of stages between a residual fan-out and its fan-in, resolved jointly, with link-availability tracked so a residual transfer overlaps the intermediate stage's compute. Memory is a resource dimension of the state, keeping partitions inside each device's budget after runtime headroom. The search runs once at engine initialization, per model, regime, and hardware combination.

The Engine Path

The partition is only useful if the engine can execute it. HELIX intervenes inside vLLM's compilation path: the graph is intercepted and compiled piecewise, subgraphs are dispatched to per-device torch.compile-compatible backends, and each accelerator runs as a subgraph executor with its own profiling - not as an independent serving engine. That inversion is the enabling move for emerging hardware: a DRAM-PIM unit that only accelerates attention, or a 3D compute-in-memory chip that only does GEMMs, can join the system by exposing the operators it covers. The design targets tightly interconnected single-node systems, not datacenter fabrics with device pools.

What the Numbers Say

The evaluation spans three model families (dense Llama-3.1-8B, sparse Mixtral-8x7B, hybrid state-space Bamba-9B) and GPU + accelerator combinations built from profiled latencies (H100, B200, an AttAcc-style DRAM-PIM for attention, a 3DCIM for GEMMs). Against single H100 at small batches: 3.14-6x for Llama, 4.48-7.3x for Bamba; Mixtral (too big for one GPU) gains 5-9.8x over 2xH100. Against A/F specifically: Bamba gains 1.7-3.2x because the search offloads ALL GEMMs - including the mixer projections that dominate its runtime - which no attention/FFN rule would touch; Llama and Mixtral gain 2.2-1.1x depending on batch and context, shrinking as KV fetching dominates. Around batch 64 the search converges to the A/F split itself, and beyond it pulls GEMMs back to the GPU as transfer costs overtake compute gains. In three-way systems it finds placements no baseline can express (projections on 3DCIM, attention on DRAM-PIM), beating A/F by 1.8-3.5x and AttAcc-style by 1-8.7x - while neither baseline is optimal across the full workload range. Swapping the H100 for a B200 shifts the optimal placements again, which is exactly the point: the answer is workload- and hardware-dependent, so the system must search, not guess. End-to-end HELIX-vLLM runs over full decoding graphs confirm the direction on complete generation throughput.

Assumptions and limitations

All results are the authors'. Accelerator numbers come from simulators and profiled kernel latencies (PerfDatabase plus in-house measurements), not silicon; the cost model's roofline assumptions are validated in the paper's appendix, but single-step estimates exclude some elementwise and normalization costs, and RoPE is assumed co-located with attention on the PIM base die. The focus is decode; prefill partitioning is claimed applicable but not the evaluation target. Systems are tightly coupled single-node; multi-node fabrics are out of scope. Bamba-9B's 4K context cap means longer-context results are extrapolations of a less attention-bound model. Search is at initialization - it does not adapt online as KV occupancy drifts within a serving session. Model and hardware names are as reported in the preprint.

Worked Example: Four Stages, Two Devices

A four-stage decode chain shows why search beats splits. GPU and a GEMM accelerator (CIM) price each stage; attention only fits the GPU; crossing a link costs 1. The A/F rule puts only the FFN on the CIM (total 15.5). Enumerating every admissible placement finds 13.5 - projections on the CIM too, paying one extra transfer to save more compute than it costs.

import itertools
stages = ["qkv", "attn",
          "out", "ffn"]
gpu = [3, 8, 2, 4]
cim = [1, None, 1, 1.5]
XFER = 1
def cost(place):
    tot = 0
    for i, d in enumerate(place):
        c = (gpu, cim)[d][i]
        if c is None:
            return None
        tot += c
        if i and place[i-1] != d:
            tot += XFER
    return tot
best = None
for m in range(16):
    pl = [m>>i&1 for i in (0,1,2,3)]
    c = cost(pl)
    if c:
        if not best or c < best[1]:
            best = (pl, c)
af = (0, 0, 0, 1)
print("A/F:", cost(af))
print("optimal:", best[1], best[0])
on = [stages[i] for i in range(4)
      if best[0][i]]
print(" CIM stages:", on)

Sources & Provenance

• HELIX: HEterogeneous LLM Inference with FleXible and Fine-Grained Partitioning (Clerico et al., IBM Research Zurich/Silicon Valley and TU Dresden, 4 Nov 2026) - arXiv abstract
• Full paper (HTML), including the DP frontier construction, profiling methodology and all partition figures - arXiv full text

All performance numbers are attributed to the preprint as published. Sources accessed 5 November 2026. This analysis is independent editorial content for GlobalTechDigest.