Header Ad Placement

One-Third of a Speculative Draft Tree Is Low-Probability Branches: Eight Learned Modes Cut Them, Reaching Up to 7.8x Lossless Speedup at Batch Size 1

INFERENCE SYSTEMS

One-pass drafters propose 15 tokens at once, but ranking the verification tree by a product of per-position distributions stitches rival continuations into "chimeras" - spliced prefixes the target takes with under 1% probability, 27-33% of a 64-node tree - and Multi-Future Drafting's eight learned modes shrink that waste into up to 7.8x lossless speedup at batch size 1.

By Ossam Hashim

KEY FINDINGS

  • Beihang and Canva researchers prove the failure is structural: barring ties, no product-ranked draft tree can hold two futures that disagree in m positions without also holding 2^(m-1)-1 chimeras - prefixes spliced from different futures - however well the drafter is trained. Measured as nodes whose last step the target takes with under 1% probability, chimeras make up 27-33% of a 64-node tree, and the tree's best path passes through one in 61-65% of rounds.
  • The repair is a latent variable, not the chain rule: K=8 learned mode embeddings at the drafter's input draft eight coherent blocks in the same forward pass, and a mixture tree keeps the B best prefixes across all modes, each scored by its best mode. Cost: K+1 extra vectors of the drafter's width, about 0.2 ms per round in their SGLang engine, no extra drafting pass.
  • On four Qwen targets from 4B to 27B (two hybrid thinking models), five tasks, both temperatures, batch size 1: MFD is the fastest method everywhere - up to 7.8x over autoregressive decoding, 4.8-7.6% over the strongest identically trained single-mode tree, 9-16% over the released DARTree on the same prompts, and 45-46% over the fastest released drafter inside SGLang.
  • The edges are explicit: the lead is a small-batch one (it falls behind at batch 8-16 unless a default-mode variant steps in), a drafter that already conditions on drafted tokens gains little from modes, and even mixture trees still hold 22-24% chimeras.

The Chimera in the Tree

Speculative decoding lets a cheap drafter propose tokens that the target verifies in parallel, losslessly. The fastest drafters today - DFlash and its descendants - draft a whole block of 15 positions in one forward pass, giving one distribution per position, and DDTree builds the verification tree by ranking prefixes on the product of those distributions. Huang, Kang and colleagues show what that product actually buys (arXiv abstract page). Under tree verification, the expected accepted length is the probability mass the tree covers, so an optimal tree holds the most probable prefixes. But per-position distributions do not know which future each position belongs to. Their running example: after "Let's break" in a GSM8K context, Qwen3-4B continues with one phrasing at probability 0.61 or another at 0.37; the product-ranked tree's best path jumps between the two, 9 of its 15 nodes lie on spliced prefixes, and verification accepts 2 tokens. They name the spliced prefixes chimeras - their definition is a prefix the target never produces, and their measurement counts nodes whose last step the target takes with under 1% probability, which bundles in merely unlikely "dead" steps - and the counting is unforgiving: holding two futures that disagree in m positions forces, barring ties, 2^(m-1)-1 chimeras into the tree, because the product cannot tell real futures from their recombinations. Measured over 628 prompts on Qwen3-4B and Qwen3-8B, chimeras occupy 27-33% of a 64-node tree's nodes and the best path crosses one in 61-65% of rounds. The error of a product is the total correlation between positions, which compounds with depth - so no amount of training fixes a drafter of this kind.

Modes as a Latent Future

The classical repair is the chain rule: condition each drafted position on the tokens before it, as autoregressive drafters and concurrent block-drafter follow-ups do. MFD takes the other classical tool - a latent variable that names the future, given which the positions are independent. K learned mode embeddings are added to the drafter's input; each mode drafts its own block in the same single pass, trained winner-take-all so each mode specializes on the blocks it covers best (weight 0.9 for the best-covering mode, the remainder split across the rest). At inference, every prefix is scored by its best mode, and an exact merged best-first search keeps the B highest-scoring distinct prefixes across all modes - nodes shared by modes that follow the same future are paid for once. With K=1 it reduces to DDTree. The overhead is K+1 vectors of the drafter's width and, measured, about 0.6 ms per round in eager execution and 0.2 ms in their SGLang engine. Placement matters: moving the modes later through the drafter shrinks the gain monotonically, and a per-mode map of the output - the design of output-side mixture heads - gives nothing; input injection delivers about three times the gain of a full-rank output expansion with three orders of magnitude fewer parameters, because in a one-pass drafter positions coordinate only through attention.

The Numbers at Batch Size 1

The evaluation pins the variables a practitioner cares about: Qwen3-4B and Qwen3-8B plus the hybrid Qwen3.5-4B and Qwen3.6-27B thinking models; GSM8K, MATH-500, HumanEval, MBPP and MT-Bench at temperatures 0 and 1; timing back-to-back against an autoregressive baseline on one RTX PRO 6000. Against D-PACE, the strongest single-mode tree trained with identical data, schedule and compute, three-epoch MFD is 4.8-7.4% faster at T=0 and 5.7-7.6% at T=1 - and the single-mode drafter saturates: three epochs leave D-PACE unchanged or slightly worse while MFD keeps improving, the lead growing to 6.2-12.2% when both are fine-tuned on the full 1.42M-prompt data. Against the released DARTree, which restores conditioning by correcting logits along each branch, MFD trained on the same prompts is 9-16% faster - DARTree's per-depth correction costs 2.6-2.7 ms per round of tree building against roughly 0.35 ms for MFD. Inside SGLang 0.5.22 with tree verification in the target's CUDA graphs, MFD is 45-46% faster than the fastest released drafter at batch size 1. The mechanism shows up directly in coverage: MFD's trees cover 8.3% more of the target's probability mass at 64 nodes and 10.3% more at 256 - 64 mixture nodes cover nearly as much as 256 single-mode nodes. Where the target genuinely forks, MFD accepts 10-12% more tokens; elsewhere only 2-4%, which is the signature of a method that wins by hedging rather than by being a better point predictor. Where The Burst That Outlasts Itself traced a serving failure under prefix-cache load and RidgeFill cut the prefill bill, this is the decode-side counterpart.

Where the Argument Bends

The authors are unusually plain about the boundaries. Every headline number is batch size 1: with as many requests in flight as the batch size, MFD leads by 10-18% at batch 1-2 and 3-11% at batch 4, but falls behind DDTree with the released drafter from batch 8 and behind SGLang's own engine paths from batch 16, because its draft cost grows with the batch while its acceptance length does not - their MFD-D variant, which switches to a single-mode default path from three requests, is the fastest configuration they timed at every batch size, but that fallback concedes the mixture's edge exactly where serving is busiest. Modes and conditioning close the same gap, so on a drafter that already conditions on drafted tokens, modes add little. Mixture trees still hold chimeras - 22-24% of nodes at B=64 against the product tree's 27-33% - because modes can disagree about a fork too. And the whole study lives in one model ecosystem (four Qwen targets, DFlash-derived drafters) on one GPU; drafting time caps K at 8-16, and the learned gate picks the winning mode for only 38% of blocks at K=4 and 10% at K=32, which is why the gain is collectible by a tree and not by committing to a chain.

Assumptions and limitations

All results are the authors' own, from their reported runs on four Qwen targets with DFlash-derived drafters at batch size 1 on one RTX PRO 6000; we re-ran nothing. Speedups are relative to the stated baselines and budgets, the large-batch regime favors other configurations, and the worked example is an invented enumeration illustrating the paper's counting argument, not a measurement.

Worked Example: A Chimera Census

The theorem's counting is concrete enough to check by enumeration. On an invented toy where two 6-token futures disagree at every position, the number of possible splices dwarfs the two real futures.

from itertools import product

# Toy chimera census (invented
# numbers; illustrates the paper's
# counting argument).
m = 6
S = set(product('AB', repeat=m))
real = {('A',)*m, ('B',)*m}
chim = S - real
print('disagreeing positions:', m)
print('possible splices:', len(S))
print('real futures:', len(real))
print('chimeras:', len(chim))
# At 50/50 per position every
# splice ties, so a top-B tree is
# mostly chimeras.
B = 8
print('top-%d tree:' % B,
      B - 2, 'of', B, 'chimeras')

Sources & Provenance

  • Huang, Kang et al., "Multi-Future Drafting: Mixture Block Drafters for Tree-Verified Speculative Decoding" - arXiv abstract page (2611.13017)
  • The same paper - full text via the abstract page (setup, Tables 1-4, appendices on batch scaling and ablations)
  • Related GlobalTechDigest coverage: The Burst That Outlasts Itself and RidgeFill

Written by Instinct for GlobalTechDigest. Analysis, framing and toy code are original; all measured results are the cited authors'.

Comments

Kubernetes Liveness Is Not Readiness: Why a Health Check Can Amplify an Outage

KUBERNETES FAILURE RECOVERY A failed health check is an observation. Restarting a container and withholding traffic are different responses, and choosing the wrong one can amplify an outage. By Hossam The important question is what failure should trigger A web application loses access to its shared database. Its processes are still running, but requests that require database access fail. A health endpoint reports the dependency problem. Should Kubernetes restart every application container, or stop directing those requests to it? The endpoint alone cannot choose the right recovery action. Kubernetes separates startup, liveness and readiness probes because they answer different operational questions. Startup protects initialization, liveness can trigger container termination and restart handling, and readiness controls eligibility for traffic from matching Services. A green check is meaningful only within the check and response that were designed. This is a documentation-based de...

Editorial Governance & Pages