Scout Before You Route: A Frozen 4B Prefill Prices Each Query's Reasoning, Saving 9-33% at Matched Accuracy
SYSTEMS
Routers price every candidate model as if each query cost the same to answer, but a reasoning model's output can stretch by an order of magnitude between problems. A small frozen scout that only reads the prompt can predict that length - and the prediction is where the money is.
By Ossam Hashim
KEY FINDINGS
- The headroom is real and large. With five reasoning candidates routed over nine benchmarks, perfect per-query cost knowledge would cut spending 29-61% on eight of them at matched accuracy. With non-reasoning candidates on the same problems the headroom shrinks 1.3-2.3x and the achievable saving up to about 30x - per-query cost prediction is a reasoning-model game, because output length is what varies.
- A linear probe on the prefill activations of a frozen Qwen3-4B - mean and last-token states from eight layers, a logistic regression for success probability, a ridge regression for log output length - captures 40-69% of that headroom: 9.4 to 33.0% saved at matched accuracy against pricing each model at its median length, on eight of nine datasets.
- Predicted cost beats every reimplemented alternative. Against ZeroRouter-style pricing, MixLLM-style text embeddings, a kNN estimator and prompt-feature GBMs, the probe is ahead in 69 of 72 dataset-estimator comparisons (50 significantly, none significantly behind). With only 20 labelled problems it keeps 51-81% of its full-data saving while external estimators save at most 10.3%.
- Length is mostly a property of the problem, not the model - but difficulty is not length. On math and code benchmarks harder problems run longer, so pricing from predicted success alone nearly matches the cost probe; on knowledge and mixed-format benchmarks they decouple, and success-only pricing falls behind by 18.4 points on MMLU-Pro and 22.6 on BIG-Bench Extra Hard.
- Deployment numbers hold up: onboarding a new candidate takes 10-50 labelled problems (two fitted parameters), the probe transfers without labels to four of seven unseen benchmarks, one probe on mixed traffic from all nine benchmarks saves 22.4% beyond per-benchmark median pricing, and the returns stop growing at about a 2B scout - its forward pass runs $0.002-0.007 per 1,000 queries on one H100.
The Median-Length Fiction
A router picks one candidate per query, trading predicted correctness against predicted cost. The success side of that trade has absorbed most of the research attention; the cost side is usually a constant - each model charged as if every query produced the same number of tokens. That fiction was tolerable when models answered in a bounded paragraph. A reasoning model does not: the same endpoint burns a few hundred tokens on an easy problem and tens of thousands on a hard one, at output prices that differ by 2x across the candidates in this study (0.08 to 0.26 dollars per million output tokens, billed averages over OpenRouter calls). The oracle experiment sets the stakes: hand the router each query's actual cost in advance and it spends 29-61% less on eight of nine datasets, accuracy held constant. On the ninth - BigCodeBench, whose library-usage tasks have short, similar solutions - there is almost nothing to capture (10.0%), and no estimator finds it. The authors also show why earlier work could miss this: on RouterBench-style pools of non-reasoning models, per-query cost adds little, which this paper reproduces - the headroom with thinking disabled or instruct-only candidates is much smaller, and much of the residual variation is degenerate output (one 8B model looping to its 16k cap on 9-12% of math problems) that prompts predict poorly. Enabling reasoning is itself the trade: deepseek-v4-flash gains 2.8 to 22.1 points of accuracy for 11-33x the output tokens.
An Actuarial Scout
The method is deliberately unglamorous. A small frozen LLM - Qwen3-4B in the main experiments - reads the prompt once and generates nothing. From eight layers' activations, two linear probes per candidate predict the two numbers the routing rule needs: the probability that candidate answers correctly, and the log of its mean output length, exponentiated with a Duan smearing correction and priced at the candidate's billed token rates. No fine-tuning, no verifier, no fallback pass: the query is answered exactly once by whichever candidate maximizes value times predicted success minus predicted cost. The authors call the scout actuarial - it prices each policy from the applicant. The comparison that matters is against text-based estimators that never see activations: a same-size Qwen3 embedding model with the same probes loses by 9.7 to 23.1 points on the four benchmarks where the probe wins most, and a fully fine-tuned DistilBERT or Qwen3-0.6B loses to the same 0.6B model kept frozen with a linear probe - consistent with fine-tuning distorting good pretrained features rather than creating them.
Where the Money Is
At matched accuracy the router saves 9.4% (CodeContests) to 33.0% (SuperGPQA) against median pricing, capturing two-fifths to over two-thirds of the oracle headroom on seven datasets. On LiveCodeBench it reaches 90% of deepseek-v4-flash's accuracy at 49% of the cost of calling it on every query - median pricing needs 69% for the same. The saving is not merely a price-gap artefact: charging every candidate the same token rate while keeping actual lengths still leaves 13.3-36.3% saved. What it needs is a genuinely useful cheap candidate: remove gpt-oss-20b at low effort and most of the saving disappears (27.8% to 6.5% on LiveCodeBench). And it is not one provider's quirk: adding Qwen3-32B, GLM-4.7-Flash and Nemotron-3-Super-120B to the pool, and dropping deepseek-v4-flash entirely, still leaves 12.4-29.5% saved on the three datasets tested. Just as notable is what the cost estimate decides. Against a random mix of candidates - the baseline any router must beat - median pricing with the same success predictions fails to beat the mix at all on MMLU-Pro or AIME: without per-query cost, routing those benchmarks does not pay. At the very top of the accuracy frontier the cost estimate stops mattering, since every estimator's best policy sends everything to the strongest candidate.
Difficulty Is Not Length
Why does the probe work where success-based pricing fails? On most math and code datasets, empirical difficulty explains the bulk of log length - harder problems get longer answers from every candidate - so pricing from predicted success alone nearly matches the dedicated cost probe (within 3.2 points on LiveCodeBench, and actually ahead on Omni-MATH). On MMLU-Pro, SuperGPQA and BIG-Bench Extra Hard, difficulty explains less than a fifth of log length: a question's subject and format, not its hardness, set the answer's shape. There the cost probe pulls away - by 18.4 points on MMLU-Pro and 22.6 on BIG-Bench Extra Hard - and on the knowledge datasets it is the only estimator that captures meaningful headroom at all. BIG-Bench Extra Hard exposes a different limit: with 23 tasks of fixed, text-visible formats, plain text embeddings identify the task and with it the length, so text-based estimators tie the probe given full training data. The probe's residual edge is where a router actually lives - 20 to 100 labels, new models, mixed traffic - and where the text estimators fall behind again.
Assumptions and limitations
All results are the authors' own, on their candidate pools, prices and benchmarks: five reasoning candidates from two model families called through OpenRouter (three further families on three datasets), nine benchmarks, billed-average token prices - prices and even output lengths vary across providers for the same open-weight model, and the authors pin one candidate to a single provider for that reason. Headroom on a new dataset is not predictable from simple length and price statistics; it must be measured with each candidate's actual lengths on a labelled sample, and a benchmark unlike the training sources still needs some labels (the transferred router saves little on Omni-MATH and BIG-Bench Extra Hard). Capture is lowest on knowledge datasets (0.40 on MMLU-Pro), leaving room for better cost predictors; the probe also predicts differences between candidates worst. The cost probe does not lift the top of the accuracy frontier - that needs better success prediction - and telling apart two strong candidates of similar price remains open (at most 7.0% saved within the gpt-oss-120b-high versus deepseek-v4-flash pair, against an 11-36% oracle). The worked example is an invented toy with made-up lengths and prices; every saving figure above is the authors' measurement, not this blog's.
Worked Example: The Mispriced Easy Query
The routing rule is one line of arithmetic: pick the candidate maximizing value times predicted success minus predicted cost. The toy below gives two candidates a history of lengths, then routes one easy query two ways - pricing each candidate at its median length, and at this query's predicted length. Median pricing overprices the strong model on a query it would answer briefly and misroutes to the cheap one; per-query pricing routes it back. Invented numbers, illustrative only.
from statistics import median
cands = [
{'name': 'cheap', 'p': .70},
{'name': 'strong', 'p': .95},
]
rates = {'cheap': .13,
'strong': .26}
hist = {'cheap': [600, 800, 1000],
'strong': [4000, 9000,
15000]}
def price(n, t):
return t * rates[n] / 1e6
def pick(V, tok):
def s(c):
n = c['name']
t = tok[n]
return V*c['p']-price(n, t)
return max(cands, key=s)['name']
med = {}
for n, h in hist.items():
med[n] = int(median(h))
easy = {'cheap': 500,
'strong': 4000}
q = pick(.005, easy)
m = pick(.005, med)
print('per-query:', q)
print('median :', m)
Sources & Provenance
- Scout Before You Route: Predicting Task Success and Token Count from a Small LLM Prefill (9 Nov 2026) - arXiv abstract
- Full paper (PDF): probe construction, nine-dataset tables, deployment and transfer studies - arXiv full text
All savings figures are the preprint's own measurements at billed OpenRouter prices as published. Sources accessed 9 November 2026. This analysis is independent editorial content for GlobalTechDigest.
Comments
Post a Comment