Header Ad Placement

FastAFD's Decode Advantage Falls to 0.99-1.17x When the Baseline Uses Tuned Wide EP

LLM SYSTEMS / SERVING ARCHITECTURE

Separating attention from MoE experts can help, but counting every GPU and tuning the colocated baseline changes the answer. FastAFD's rack-scale study explains the tradeoff, with controlled speculation and an ideal hardware ceiling kept separate from measured results.

By Ossam Hashim

KEY FINDINGS

  • On GB200 NVL72, the paper reports 0.99-1.17x per-GPU steady-state decode throughput against the best measured wide-EP configuration across its main standard-decoding workloads, not the 1.30-2.36x obtained against EP4.
  • Attention-only throughput excludes the dedicated expert GPUs. All-GPU throughput includes them; that accounting difference can turn a large-looking gain into a modest improvement or no improvement.
  • Speculative experiments control acceptance near 1.8 useful tokens per step. They measure the cost of verification and draft execution under that assumption, not natural acceptance on a representative application benchmark.
  • The headline 2.9% prediction error covers 65 main-study configurations up to their measured throughput peaks. Errors are larger beyond those peaks and in extended speculative experiments.

Why the baseline changes the story

Attention-FFN disaggregation sounds like a direct answer to an obvious mismatch. Attention keeps request-specific key-value state and reads it repeatedly. A mixture-of-experts feed-forward network holds shared expert weights and processes routed tokens. Why force both into the same GPU allocation if their bottlenecks differ?

The FastAFD study asks a harder question: what remains after the colocated system has also been allowed to scale intelligently? Its comparison is not simply split versus unsplit. It is a tuned split deployment versus a tuned wide expert-parallel deployment on the same rack-scale NVLink platform. That makes the result more useful than a large speedup against a convenient but narrow baseline.

The FastAFD paper reports standard-decoding gains of 1.30-2.36x against single-node EP4. Against the best measured wide-EP configuration for each model and context, that range becomes 0.99-1.17x. This is still a worthwhile result in the favorable cases, but it does not justify a blanket claim that separating attention and experts doubles serving efficiency.

Wide EP distributes expert weights over more GPUs while keeping attention data-parallel. Each GPU consequently stores a smaller expert shard, leaving more memory for request state. AFD can free that expert-weight memory from attention GPUs entirely, but it is competing with a baseline that has already recovered much of the available benefit. The older vLLM wide-EP explanation describes this memory-versus-communication tradeoff; it is background, not a second measurement of FastAFD.

What actually runs on the rack

The study uses one GB200 NVL72 rack with 18 four-GPU nodes in a 72-GPU NVLink domain. It evaluates three FP8 MoE checkpoints: MiniMax-M2.5, Qwen3-235B-A22B, and Qwen3.5-397B-A17B. Their different expert sizes and attention structures are part of the experiment, not interchangeable details.

FastAFD assigns one four-GPU node to the expert pool and varies the number of attention nodes. Attention GPUs retain normalization, QKV and output projections, routing, request state, and sampling responsibilities. Expert GPUs receive routed activations, execute the expert computation, and send results back. The split is at the MoE boundary, not at the boundary between prompt processing and token generation.

Prefill-decode disaggregation is a different partition. It separates the initial prompt pass from subsequent decoding. FastAFD instead divides work inside decoding and could operate within the decode pool of a larger prefill-decode system. Mixing those definitions would make both the mechanism and the measured scope unclear.

The colocated comparison sweeps EP4, EP8, EP16, EP32, and EP64. In the main standard-decoding cases, the best baseline sits at EP16 or EP32, not automatically at the largest possible group. Wider parallelism frees memory but also changes communication costs. An operator should therefore tune the baseline rather than assume either the narrowest or widest configuration is best.

Decode throughput is not an online service guarantee

Each main workload has a fixed 4K, 8K, or 16K prompt. The experiment prefills first, then measures steady-state decoding with occupied request slots and 1,024 output tokens per request. Prefill is excluded. Batches are chosen around available KV capacity, with alignment adjustments where they improve the measured kernel behavior.

These conditions answer a capacity question: how efficiently can each layout produce tokens during a saturated decode window? They do not establish time to first token, request completion latency under bursty arrivals, queueing behavior, or an application-level latency SLO. A deployment with better saturated decode throughput can still need separate admission and latency testing.

Appendix A also describes the measurement machinery rather than leaving throughput as an unexplained counter. FastAFD derives its rate from the decode period and batch size. The vLLM measurement uses streaming probes and valid decode windows, with explicit rules for fallback windows and exclusion of burst-delivered probes. These rules matter when comparing systems with asynchronous host and device work.

The result is not a claim that every row has the same confidence interval or that every observed difference is statistically decisive. In particular, near-parity ratios deserve cautious language. A 0.99x result is a slight regression in the reported measurement, not a hidden win rounded inconveniently.

Three models, three different outcomes

MiniMax-M2.5 is close to parity in standard decoding. Appendix A reports about 4,118 versus 4,098 tokens per second per allocated GPU at 4K, and about 1,234 versus 1,208 at 16K. Those are small gains compared with the result obtained by choosing EP4 as the baseline. Its relatively light expert work leaves less computation for AFD to hide.

Qwen3 retains the clearest standard-decoding advantage. At 16K, the best measured FastAFD layout uses an 11:1 attention-to-expert node ratio and reports roughly 1,452 tokens per second per allocated GPU, against 1,240 for EP32. That is the approximately 1.17x result. At shorter contexts the gain is smaller, so the maximum should not be presented as the outcome for every request shape.

Qwen3.5 shows why memory savings alone are insufficient. Its hybrid full and linear attention structure permits a larger batch on the attention pool, but the split runtime also repeats attention-side projection work across microbatches. The reported advantage is about 1.16x at 4K and 1.10x at 8K, then falls to 3,955 versus 3,982 tokens per second per GPU at 16K, approximately 0.993x.

This is an especially useful counterexample to the claim that freeing more attention-side memory must improve the system. Batch capacity is one factor. Step duration and the fraction of GPUs reserved for experts are two others. A favorable number in one column cannot settle the architecture decision.

Count the expert GPUs before calling it a speedup

The paper decomposes all-GPU speedup into batch gain, step-time gain, and an allocation factor. If r attention nodes share one expert node, the allocation factor is r/(r+1). The attention pool emits the tokens, but the expert node is still part of the hardware bill.

Attention-only speedup intentionally leaves that allocation factor out. It answers how much each attention GPU benefits, which can inform a hypothetical design where experts move to another device type. It does not say the expert computation is free. In the current homogeneous GPU experiment, all-GPU throughput is the relevant normalized result for comparing the allocated GPU resources.

For example, the main speculative Qwen3 4K row reports about 1.70x on the attention-only denominator but about 1.13x when all GPUs count. Its 2:1 node allocation makes the difference predictable: two of every three nodes hold the attention-side request state. Both ratios can be correct while supporting very different headlines.

Nor is a tokens-per-GPU result automatically a tokens-per-dollar or tokens-per-watt result. Moving expert work to a separate accelerator changes hardware, power, networking, and operational costs. Those would need their own measurements. Removing expert GPUs from the GPU denominator does not remove the replacement devices from a real system budget.

How two microbatches hide expert work

Within one request stream, the next layer needs the current layer's expert output. Simply placing experts elsewhere would add communication without removing that dependency. FastAFD uses two microbatches so one can execute attention while the other is being processed by the expert pool, then their roles switch.

The overlap only works while the expert stage fits under the corresponding attention stage. Adding attention nodes gives one expert node more routed tokens to process. Initially that can improve the allocation factor without lengthening the step much. Beyond a coverage boundary, expert work becomes exposed and step time rises. The best ratio is therefore a workload-dependent balance, not a constant setting.

Hybrid attention makes the accounting more demanding. The paper takes the maximum of attention and expert time within each layer, then sums those layer costs. Averaging all attention layers first could let spare time in one layer appear to cover a dependency in another layer, even though the actual execution order does not allow it.

The runtime also removes avoidable gaps. Its fused MegaMoE path combines communication and expert work, and a one-step-ahead coordinator prepares the next decode plan while the current GPU graph runs. The ablations show that these details matter. Their gains are comparisons between runtime variants, not additional multipliers to apply on top of the paper's headline end-to-end ratio.

Speculation changes provisioning, with controlled acceptance

The main speculative comparison uses draft depth three, meaning four verified positions per step, and holds useful output near 1.8 accepted tokens per request per step. Appendix D clarifies that draft and target models both run. The acceptance is controlled, so the experiment is not a natural draft-quality evaluation over an application distribution.

Verification reuses prefix KV reads while increasing the expert token work. This shifts the best allocation toward more expert capacity. MiniMax's main ratios change from 9:1-17:1 in standard decoding to 4:1-8:1 under depth-three speculation. Qwen3 moves from 5:1-11:1 to 2:1-5:1. A split tuned for ordinary decoding should not be assumed suitable after speculation is enabled.

For the main depth-three cases, the all-GPU advantage over the correspondingly speculative wide-EP baseline is about 1.02-1.13x. Same-engine self-speedup is a different comparison. Reporting FastAFD's improvement over its own non-speculative mode would not establish an equally large advantage over a speculative baseline.

Deeper speculation is not automatically better either. Appendix D includes depth-seven results where MiniMax at 16K and 32K is slightly below the tuned baseline on the all-GPU metric. Increasing verification work without a corresponding useful-token gain can consume the very capacity that the architectural split was meant to recover.

Read the prediction model's denominator too

A useful contribution is the prediction model, which estimates candidate splits from hardware information and colocated profiles rather than fitting parameters to FastAFD runs. It combines attention-stage timing, repeated auxiliary work, and remote expert service. Appendix C describes calibration constants fitted from colocated EP4 profiles; zero-refit does not mean no profiling or no fitted constants anywhere.

The widely summarized 2.9% mean error applies to 65 main-study deployments up to each workload's measured best ratio. That scope is narrower than all configurations in the appendices. Across main-study post-peak configurations, mean errors are 10.2% for standard decoding and 6.6% for speculation. The extended standard table contains a 43.38% error for a Qwen3.5 configuration beyond its peak.

The extended speculative results also matter: the maximum in-scope error reaches 16.81% for a depth-seven Qwen3 case. Thus within 3% is not a per-run guarantee across arbitrary draft depths and ratios. The model is a way to narrow a deployment search, with known operating boundaries, rather than a replacement for validating a selected production configuration.

The main-study model-selected ratio retains 95.3-100% of measured peak standard throughput and 96.5-100% in the speculative cases. Those selection results are useful precisely because they describe the practical consequence of prediction error. They still depend on the measured candidate set and the rack's finite topology.

The 2.48x ceiling is not a measured LPX result

The paper considers moving experts onto non-GPU accelerators and then an idealized case where remote expert latency approaches zero. The latter also removes the need to repeat attention-side work for the two-microbatch overlap scheme. Its modeled ceiling reaches 2.48x in speculative decoding.

That is an architectural bound, not a measured end-to-end result on an installed heterogeneous rack. NVIDIA's LPX architecture article independently describes pairing GPUs for attention with LPUs for expert execution. It supports the relevance of studying that partition, but it does not validate FastAFD's ideal assumptions or supply its missing full-system cost comparison.

There is also an evidence-version issue. The FastAFD repository README retains earlier 1.35-1.45x throughput language and a vLLM 0.19.0 alignment environment. The paper's reproducibility appendix describes later measurement presets and vLLM 0.31.0, with Qwen3.5 rows using 0.33.0. These are not interchangeable instructions for reproducing one frozen experiment. Pin the relevant code and configuration rather than merging convenient numbers from different artifacts.

A separate vLLM AFD plugin is explicitly experimental and has its own model, backend, and version limits. Its published NPU performance example uses forced expert balancing that changes outputs. That example is useful context for why topology and benchmark scope must be disclosed, not independent confirmation of FastAFD's correctness or GPU speedup.

Assumptions and limitations

All hardware results above are attributed to the FastAFD authors. No GB200 rack, expert kernel, checkpoint alignment, or online-serving experiment was reproduced for this article. The executed example is arithmetic with invented inputs and cannot estimate communication, load imbalance, memory allocation, or model quality.

The central measurement is saturated steady-state decoding on specified checkpoints and contexts. It excludes prefill and does not establish production tail latency, request-level reliability, or cost efficiency. Controlled speculative acceptance is not a promise about natural acceptance. Model error summaries have explicit scope boundaries, and the heterogeneous ceiling relies on ideal assumptions.

For an operator, the practical sequence is to tune colocated wide EP first, compare using every allocated device, check the exact model and context, then validate candidate splits under the intended traffic and output-quality requirements. The study's strongest lesson is a measurement discipline: compare the best relevant alternatives and keep the denominator visible.

Worked Example: A faster attention pool can still lose overall

This invented arithmetic example illustrates the paper's denominator, not its hardware model. A batch gain of 1.10 and step-time gain of 1.20 produce an attention-only ratio of 1.32. With two attention nodes per expert node, all-GPU speedup is only 0.88. At four attention nodes per expert node it is 1.056, and at eight it is about 1.173. The last two rows unrealistically keep the step-time gain fixed: in a real system, increasing the ratio can expose expert latency. The code deliberately does not choose a deployment for you.

from fractions import Fraction

batch = Fraction(11, 10)
step = Fraction(6, 5)
attention = batch * step
print("attention", float(attention))
for r in (2, 4, 8):
    share = Fraction(r, r + 1)
    speed = attention * share
    print(r, round(float(speed), 3))

Sources & Provenance

  • FastAFD paper: main sections 1-8 and appendices A-F, including complete throughput, residual, profiling, speculative, ablation, and reproducibility records.
  • FastAFD repository: public implementation, earlier README claims, alignment environment, and experiment entry points; not treated as a frozen copy of the paper's evaluation.
  • vLLM wide-EP background: explanation of expert sharding, KV capacity, communication, and overlap; its older benchmark is not combined with FastAFD results.
  • vLLM Blackwell optimizations: separate vendor-project background on how kernel, precision, and deployment choices change a baseline; no cross-paper speedup calculation.
  • vLLM AFD Plugin: experimental scope, backend distinctions, and explicit forced-balancing benchmark limitations.
  • NVIDIA LPX architecture: vendor description of heterogeneous attention/expert placement, not independent validation of the paper's ideal ceiling.

The headline uses the paper's tuned-baseline all-GPU result. Larger EP4-relative, attention-only, same-engine, and ideal modeled ratios are labeled separately rather than promoted as interchangeable system speedups.

Comments

Kubernetes Liveness Is Not Readiness: Why a Health Check Can Amplify an Outage

KUBERNETES FAILURE RECOVERY A failed health check is an observation. Restarting a container and withholding traffic are different responses, and choosing the wrong one can amplify an outage. By Hossam The important question is what failure should trigger A web application loses access to its shared database. Its processes are still running, but requests that require database access fail. A health endpoint reports the dependency problem. Should Kubernetes restart every application container, or stop directing those requests to it? The endpoint alone cannot choose the right recovery action. Kubernetes separates startup, liveness and readiness probes because they answer different operational questions. Startup protects initialization, liveness can trigger container termination and restart handling, and readiness controls eligibility for traffic from matching Services. A green check is meaningful only within the check and response that were designed. This is a documentation-based de...

Editorial Governance & Pages