Checking Local Identities During Generation: Symbolic-Math Accuracy Gains of 1.4-5.6 Points for 6-7% More Tokens
MODEL RELIABILITY
Instead of grading a finished solution or paying a second model to supervise, a Technion team checks selected locally checkable identities with a computer algebra system while the model keeps generating - 22-25 ms of CPU per check, no GPU, no model call - and rolls back only the tokens written after the mistake.
By Ossam Hashim
KEY FINDINGS
- On 2,700 easy-to-intermediate ASyMOB symbolic-math problems, five repeats per problem, online checking raised mean accuracy for all three open-weight models tested: Qwen3-30B-A3B-Instruct 62.4 to 68.0%, Qwen3-4B-Instruct 23.4 to 24.8%, Gemma-4-12B-it 45.2 to 48.2% - gains of 1.4-5.6 points at a 6-7% token increase and zero extra model calls.
- The gain is surgical, not global: within answers the checker flagged, accuracy rose 6.0-10.2 points; on unflagged answers the change is statistically indistinguishable from zero, so baseline behavior survives untouched where no check fires.
- Intervention is cheap and common: 21.8-53.1% of answers received at least one correction, each rollback discarded only 1.4-15.6 tokens on average, and each check cost roughly 22-25 ms of CPU time while the GPU kept decoding.
- The benefit concentrates where it should: on problems the plain model solves inconsistently (one to three of four repeats correct), Qwen3-30B gained 8.4-8.8 points and those groups carried about 82% of its total net gain; problems beyond the model's ability barely move, because checking adds no new knowledge.
Speculative Execution for Reasoning
The framing Elimelech, Shalyt and Kaminer borrow is from CPUs (arXiv abstract page): speculative execution with branch prediction. The model keeps decoding on the GPU, effectively speculating that its preceding claims are correct, while a lightweight CPU pipeline parses completed equations out of the text stream and verifies them asynchronously with a computer algebra system. The unit of verification is a locally checkable identity. Extraction runs after each streamed chunk using regular expressions, needs no prescribed step format, and splits chained equalities like a=b=c into pairwise claims. Heuristic filters then discard definitions and assignments, trivially short claims, and equations whose two sides carry different free symbols - the last filter keeps conditions-to-be-solved, like a quadratic set to zero, from being misread as asserted identities. Verification itself is numerical rather than symbolic: sides without free symbols are compared by value, and sides with free symbols are evaluated at sampled substitutions within a tolerance, the same comparison procedure ASyMOB uses to grade answers. When a check fails, the continuation after the offending step is discarded, a short corrective message is appended, and generation resumes from that point. The checker never supplies the solution or a strategy - it flags a local error and lets the model repair it.
The Numbers
The evaluation is deliberately plain: 2,700 ASyMOB problems at easy-to-intermediate difficulty, three models from two families, five repeats each, plain versus checked generation. Qwen3-30B improved from 62.4 to 68.0% mean accuracy (problems correct on all five repeats rose from 796 to 958), Qwen3-4B from 23.4 to 24.8%, and Gemma-4-12B from 45.2 to 48.2%. Mean tokens per answer grew only 6-7%, and the discarded continuations account for little of that - the extra tokens are mostly re-derivation after feedback, not waste. Checker latency is 22-25 ms of CPU per check, cheap enough to run alongside generation on resource-constrained deployments where a verifier LLM would cut throughput. The decomposition matters more than the headline: replaying the same detection rule offline on plain answers shows flagged fractions are similar across conditions, so the method is not merely changing which answers get flagged - within the flagged group it recovers 6.0-10.2 points of accuracy, and outside it nothing moves. Where the checker fires, accuracy jumps; where it does not, baseline performance is preserved. That selectivity is the practical story: this is not a better reasoner, it is a safety net that only touches reasoning already going wrong. It complements rather than competes with training-side verification - and where Wrong the Same Way showed static verifier audits missing the errors RLVR amplifies, this work shows what checking mid-stream, before the error propagates, can recover.
Where the Argument Bends
The authors are clear about what a local check cannot see. It cannot establish whether a step follows from the problem's assumptions, whether the strategy is sound, or whether a correct calculation rests on a mistaken premise - misreadings and wrong plans pass silently. Context-free evaluation can also reject identities that hold under unstated, problem-specific assumptions; numerical sampling can miss a disagreement; and parsing failures, extraction filters and timeouts shrink coverage further. The evaluation itself is one benchmark and three models from two families, all at easy-to-intermediate difficulty, so the effect size on frontier models or harder mathematics is unknown. And the 1.4-point gain on Qwen3-4B, the weakest model, is a reminder of the mechanism's boundary: checking helps a model realize capability it already has, and adds none it lacks. Read the result as a deployment pattern - deterministic, model-free, mid-stream checking wherever a cheap checker exists - rather than as an accuracy fix for weak models.
Assumptions and limitations
All results are the authors' own, from their reported runs on 2,700 ASyMOB problems with three open-weight models and five repeats; we re-ran nothing. Gains are specific to symbolic mathematics with explicit, checkable equations, the 1.8-14.9% figure is the net reduction in incorrect answers relative to each model's own error count, and the worked example is an invented illustration of the paper's numerical-checking step, not a measurement.
Worked Example: A Toy Identity Checker
The verification step is simple enough to sketch. The paper checks identities by evaluating both sides at sampled values within a tolerance; this invented toy does the same on one true and one false expansion.
import random
# Toy checker: sample-test an
# identity, like the paper's CAS
# step. Invented numbers.
def val(expr, x):
return eval(expr, {'x': x})
def close(a, b):
return abs(a - b) < 1e-6
checks = [
('(x+1)**2', 'x**2+2*x+1'),
('(x+1)**2', 'x**2+1'),
]
for lhs, rhs in checks:
xs = [random.uniform(-9, 9)
for _ in range(50)]
ok = all(
close(val(lhs,x), val(rhs,x))
for x in xs)
print(lhs, '=', rhs, '->', ok)
Sources & Provenance
- Elimelech, Shalyt and Kaminer, "Catch It Where It Happens: Real Time Error Correction During LLM Reasoning" - arXiv abstract page (2611.13107)
- The same paper - full text via the abstract page (method, Table 1, flagged/unflagged decomposition, limitations)
- Related GlobalTechDigest coverage: Wrong the Same Way
Written by Instinct for GlobalTechDigest. Analysis, framing and toy code are original; all measured results are the cited authors'.
Comments
Post a Comment