Bevy Architecture: Runahead Inference, Truth-Over-Prediction Expert Fetching, and Why the Cache Betrays the Prefetcher

INFERENCE SYSTEMS

The GPU idles while experts arrive; the link idles while the GPU computes. Prefetching fills the link with guesses - and the bigger your expert cache, the worse the guesses get. Bevy's answer: stop predicting, start scheduling.

By Ossam Hashim

KEY FINDINGS

• Expert offloading lets large MoE models run on local hardware by keeping only some experts in GPU memory, but conventional batching stalls the GPU at every layer while missing experts are fetched. The standard remedy, prefetching, predicts which experts come next and fetches them during compute.
• A new paper (arXiv Nov 3) shows prefetching degrades precisely when it should help: as the expert cache grows, needed experts make up a smaller and harder-to-predict share of the nonresident candidates. A cache holding 30% of experts already contains 54.1% of what the next layer needs, and 51-64% of prefetched experts go unused in live runs.
• Bevy replaces prediction with runahead inference: a request whose experts are cached runs ahead into later layers, revealing its real expert needs early; the link then fetches only experts known to be needed while other requests compute.
• Running requests separately repeats reads of shared weights (GPU busy time grows 1.6-1.7x), so Bevy's self-balancing scheduler batches ready requests per layer (lowest-layer-first) and regroups opportunistically as expert availability changes.
• Numbers across two local platforms (Apple M6 Mac mini, Jetson AGX Thor), two Q4_0 MoE models, and two workloads: highest decode throughput in all 40 configurations, 1.01-1.38x over the fastest prefetching baseline and 1.29-3.46x over llama.cpp; up to 1.55x across cache sizes. Adding prefetching on top of Bevy does not help at all.

The Idle-Link Bargain

Local MoE inference lives on a trade: expert weights stay on SSD, a subset sits in GPU memory as an expert cache, and every layer's gate picks the experts it needs. Miss, and the GPU waits for a fetch over PCIe; hit, and the link sits idle through the compute. Batching concurrent requests does not change the alternation - it just makes everyone wait together. Prefetching attacks the GPU-idle half of the bargain by guessing future expert choices and streaming them in during compute. Three families of predictors exist (next-layer gate replay, similar-token expert maps, and variants), and they all share one assumption: that the link's idle time is free capacity waiting to be filled with useful fetches.

Why the Cache Betrays the Prefetcher

The paper's trace-driven study breaks that assumption. With an LFU expert cache, prefetch precision falls as the cache grows, for all three predictor families. Two effects compound. First, coverage: a 30% cache already holds 54.1% of the experts the target layer needs, so needed experts become a minority of the nonresident candidates a prefetcher must pick from. Second, selection: the cache preferentially retains the experts that are easy to predict - randomizing which experts are cached reproduces only part of the precision drop, meaning the remaining candidates are intrinsically harder to call. Live measurements match the simulation: prefetching baselines fetch 1.4-1.5x as many experts per step as plain demand fetching, and 51-64% of those prefetches are never used. Idle bandwidth filled with wrong guesses is not free; it delays the fetches that matter.

Runahead: Truth Over Prediction

Bevy's runahead inference exploits what prefetching lacks: ground truth about expert needs. When several requests wait at a layer, any request whose experts happen to be cached runs the MoE immediately, proceeds through attention, and reaches the next layer's gate - which reveals exactly which experts it will need, while the link is still busy with the others' fetches. Those known-needed fetches queue behind the current ones, so the link stays busy with certainty instead of probability. Computation and fetching overlap without a single prediction. The catch is real: separated requests stop sharing weight reads, so GPU busy time rises 1.6-1.7x. Runahead only wins if the GPU is given enough batching to claw that back.

Scheduling Is the Whole System

That is why Bevy's scheduler is the system, not an accessory. Compute runs lowest-layer-first: prioritizing the most lagging ready job naturally clusters requests at the same layer into one batched group, adapting as cache hits scatter and regroup them. Swap in progress-blind FIFO and the step lengthens 12.3% - single-job groups rise from 56% to 67% of batches, and the extra GPU busy time (19.2 ms) swamps the idle time saved (2.0 ms). Fetch order matters less but still counts (global shortest-remaining-time costs 1.7%). Grouping is opportunistic rather than static: against every fixed split of six requests, Bevy's adaptive grouping is 0.9-4.3 ms per step faster than the best static choice at each point. Under the hood it is llama.cpp modified to build per-job ggml graphs, reused and CUDA-graph-captured, with LFU eviction and an expert-pack SSD layout read by eight threads.

What the Numbers Say

Across 40 configurations (2-16 concurrent requests, Chat and Agentic Coding workloads, Qwen3.6-35B-A3B and Gemma4-26B-A4B at Q4_0, on an M6 Mac mini and a Jetson AGX Thor), Bevy has the highest decode throughput in every one: 1.01-1.38x over the fastest baseline, 1.29-3.46x over llama.cpp. The margin grows with concurrency (1.15-1.38x at four or more requests) and with cache size (1.40-1.55x at 50-60% cached), while the prefetching baselines decay toward llama.cpp as their precision collapses. The mechanism audit is the cleanest part: llama.cpp never overlaps the two resources (GPU idle 60% of each step), prefetchers overlap 36-40% but still leave the GPU waiting 57-59% of the step, and Bevy keeps link and GPU busy together 77% of the step with the GPU waiting only 10%. Most telling: prefetching layered on top of Bevy adds nothing - with few requests the prefetches are wasted, and with many, the link has no idle time left to fill.

Assumptions and limitations

All results are the authors', measured on two unified-memory platforms with Q4_0-quantized models and SSD-resident experts; discrete-GPU desktops with faster PCIe, larger caches, or different quantizations may shift the trade-offs. The prefetch-precision study is trace-driven (ShareGPT, eight concurrent requests, one model) and live runs agree, but other traffic mixes could differ. Baselines were reimplemented in the same engine with shared fetch infrastructure, which is fair but not identical to their native stacks. Throughput gains are decode-side; prefill behavior and single-request latency are not the story here. The 1.01x low end at two requests shows runahead needs concurrency to bite. Model and platform names are as reported in the preprint.

Worked Example: One Step, Two Schedules

A stripped-down decode step shows why overlap beats guessing. Two requests, four layers: per layer each request needs 10 ms of compute and a 6 ms expert fetch. Serial batching alternates resources - 128 ms per step, GPU idle 37.5%. Runahead overlaps the resources, so the step floor is the larger of total GPU work (80 ms) and total link work (48 ms): 80 ms, GPU idle zero. The gap is the whole argument.

import itertools
L, C, F = 4, 10, 6
R = 2
gpu = R * L * C
link = R * L * F
serial = R * L * (C + F)
over = max(gpu, link)
is_ = 100 * (serial - gpu) / serial
io = 100 * (over - gpu) / over
print("serial:", serial, "ms")
print(" GPU idle %.1f%%" % is_)
print("runahead:", over, "ms")
print(" GPU idle %.1f%%" % io)

Sources & Provenance

• Accelerating Offloading-Based MoE Inference by Interleaving Concurrent Requests (Yu, Gim, Lee, Zhong; SNU and Yale, 3 Nov 2026) - arXiv abstract
• Full paper (HTML), including the prefetch-precision study, scheduler ablations and step-time breakdowns - arXiv full text

All performance numbers are attributed to the preprint as published. Sources accessed 5 November 2026. This analysis is independent editorial content for GlobalTechDigest.