Kernel Engineering Under Physical Non-Idealities: Floorsweeping, Wave-Equalized Launches, and the DVFS Confounder Behind H100 Clustering Gains

GPU SYSTEMS

An NUS team shows the H100 you programmed is not the H100 you have: fabrication binning quietly removes whole SMs from cluster reach, and power limits rewrite the speedup math. Accounting for both recovers up to 5.30% on FlashAttention 4.

By Ossam Hashim

KEY FINDINGS

Floorsweeping, the binning step that disables defective silicon to salvage a die, is invisible to the ISA but visible to cluster placement: on the H100 SXM5 it removes GPC-level clustered execution capacity, cutting SMs reachable by clusters from 132 to 124 (paper's measurement).
Cluster placement also drifts at runtime: the paper's microbenchmark records only 120 unique SM IDs per kernel iteration but 124 across iterations, so the effective SM budget oscillates launch to launch.
Fewer usable SMs can add a full execution wave: a launch that needs one wave at 132 SMs can need two at 120, erasing the data-movement savings DSMEM clustering was supposed to buy.
The fix is arithmetic, not hardware: wave-aware clustering picks per-kernel between TPC-level (size 2) and GPC-level (size 4/8) clusters so the thread-block count stays wave-equalized under both the full and reduced SM limits.
On the Hopper FlashAttention 4 forward kernel (whose baseline does not cluster at all), wave-aware clustering improves throughput by up to 5.30% across 2K-64K sequence lengths (authors' reported results).
The second confounder is power: under TDP limits, reduced data movement lets DVFS sustain higher SM clocks, so part of a memory optimization's gain is really a frequency gain; the same power wall caps throughput scaling on Tenstorrent's Wormhole n300d.

The Abstraction That Hides the Silicon

Kernel engineers work through stacked abstractions: language, compiler, ISA, microarchitecture. Each layer hides what sits below, and mostly that is the point. This paper, from the National University of Singapore, documents what the last layer hides: the physical state of the chip itself. Two effects matter. Floorsweeping disables defective units after fabrication and sorts dies into SKUs, so two chips sold under one name can present different resource counts to cluster scheduling. And the thermal design power budget governs, through dynamic voltage and frequency scaling, the clock the chip can sustain while your kernel runs. Neither appears in any programming guide, yet both can decide whether an optimization works. The case study is distributed shared memory on Hopper. DSMEM lets thread blocks in a cluster read each other's shared memory across a per-GPC network-on-chip, so tiles are loaded once and broadcast instead of refetched. Clusters of size 2 map within a texture processing cluster; sizes 4 and 8 require whole GPC-level groupings, and the GPU carries eight GPCs.

Floorsweeping Eats the Cluster

Here is the catch the vendor documentation abstracts away: cluster placement is constrained by the physical organization of the SMs, and floorswept GPCs do not have enough live SMs to host a size-4 or size-8 cluster. The paper measures the consequence directly by logging logical SM IDs during a clustered microbenchmark. At GPC-level cluster sizes, the SMs available for clustered execution drop from the H100 SXM5's 132 to 124, and within any single iteration only 120 unique SM IDs appear. Which SMs sit out varies between launches. Why does losing 8 to 12 SMs out of 132 matter? Wave quantization. A kernel's thread blocks execute in waves sized by the number of SMs that can host them. A launch of 121 to 132 blocks runs in one wave at 132 SMs but needs two waves at 120, and the second wave runs nearly empty. The throughput cliff from an extra wave can be larger than everything DSMEM saved on redundant global loads, so a kernel tuned on paper can regress on silicon.

Wave-Equalized Launches

The paper's response is to make the wave count explicit and choose cluster geometry around it. A launch is wave-equalized when its block count produces the same number of execution waves under both the full 132-SM and the reduced 120-SM limits; the paper derives the condition as a ceiling-function identity over tile counts for GEMM and over batch, head and sequence tiles for FlashAttention. The wave-aware clustering strategy then evaluates each kernel configuration and selects TPC-level or GPC-level clustering so the launch stays wave-equalized while still capturing DSMEM reuse. The results land where the theory says they should. On GEMM the strategy picks the best clustering per shape. On FlashAttention 4's Hopper forward kernel, whose stock implementation uses cluster size 1, wave-aware clustering matches or beats the baseline at almost every configuration from 2K to 64K sequence length, with gains up to 5.30%. The point is not the percentage; it is that the winning move was counting waves against the chip's real, floorswept floor plan rather than its nominal one.

The DVFS Confounder

The paper's second contribution is an explanation for speedups that looked anomalous. Under a TDP cap, sustained SM frequency floats with power draw. A clustering change that reduces data movement reduces energy per iteration, and DVFS converts that headroom into a higher sustained clock. Part of the measured throughput gain is therefore a frequency effect, not a locality effect, and the paper shows the two can be disentangled by logging clocks alongside performance. The same physics shows up on a very different machine: on Tenstorrent's Wormhole n300d, a dual-chip card with 64 Tensix cores at 1 GHz and 64 TFLOP/s peak per chip, throughput scaling flattens as more cores activate because the power budget forces the frequency down. Two vendors, two architectures, one lesson: kernel performance models that stop at the microarchitecture miss the variables that increasingly set the ceiling. Kernel engineers who care about these margins, the audience of this blog's Humming kernel analysis, now have one more hidden variable to log.

Assumptions and limitations

All performance figures are the authors' reported measurements on their H100 SXM5 units and Wormhole n300d card; floorsweeping outcomes vary per chip, so the 124/120 figures describe their silicon, not a universal constant. This analysis did not independently re-run the kernels. The FlashAttention 4 baseline referenced is the Hopper forward kernel as used in the paper. Wave-equalization conditions are derived for GEMM and FlashAttention tile shapes; other kernels need their own derivation. The worked example is a simplified illustration, not the paper's methodology.

Worked Example: Counting Wasted Waves

This toy calculator reproduces the wave-quantization effect: for thread-block counts near the 120/132 boundary, it prints the wave count at both SM limits and flags launches where floorsweeping adds a nearly empty wave. Illustrative arithmetic, not the paper's harness.

import math
def waves(b, sms):
    return math.ceil(b / sms)
print("B    w132 w120 verdict")
penal = 0
for b in range(115, 135):
    a = waves(b, 132)
    c = waves(b, 120)
    if c > a:
        v = "wasted wave"
        penal += 1
    else:
        v = "equalized"
    print(b, a, c, v)
print("penalized:", penal, "of 20")

Assumptions and limitations All performance figures are the authors' reported measurements on their H100 SXM5 units and Wormhole n300d card; floorsweeping outcomes vary per chip, so the 124/120 figures describe their silicon, not a universal constant. This analysis did not independently re-run the kernels. The FlashAttention 4 baseline referenced is the Hopper forward kernel as used in the paper. Wave-equalization conditions are derived for GEMM and FlashAttention tile shapes; other kernels need their own derivation. The worked example is a simplified illustration, not the paper's methodology.

Sources & Provenance

Physical non-idealities paper (arXiv, October 27, 2026) - mechanism, SM measurements, FlashAttention 4 and Wormhole results
Paper full text (HTML) - wave-equalization derivations and microbenchmark detail
FlashAttention repository - baseline kernel referenced by the study

Analysis is based on the primary sources listed above; performance claims are attributed to their authors and not independently verified.

RLiD Architecture: Decision-Plane Reinforcement Learning, Single-Forward-Pass Rollouts, and Deleting the Zero-Mean Noise Token-Level RL Carries

TRAINING SYSTEMS Token-level RL credits every generated token for an action the environment alone executes. RLiD trains on the decision itself: one forward pass reads a compiled state and action menu, and the gradient lands on executable options only. By Ossam Hashim KEY FINDINGS Token-level RL for language agents assigns an action's advantage to the entire generated response - reasoning chains, formatting, and malformed content included - yet the environment executes only the parsed action. Variation in how the same action is expressed adds zero-mean variance to policy-gradient estimates, and unparseable outputs waste interactions. RLiD (Reinforcement Learning in the Decision plane, arXiv Nov 2; UIUC, NUS, Zhejiang, UPenn, Rochester) compiles the observation, an optional compact memory, and a labeled action menu into the input; a large decision model (ldm) picks an action in a single forward pass. No autoregressive generation during rollouts, no parsing, and every ...

GlobalTechDigest

Search This Blog