Header Ad Placement

Checking Local Identities During Generation: Symbolic-Math Accuracy Gains of 1.4-5.6 Points for 6-7% More Tokens

MODEL RELIABILITY

Instead of grading a finished solution or paying a second model to supervise, a Technion team checks selected locally checkable identities with a computer algebra system while the model keeps generating - 22-25 ms of CPU per check, no GPU, no model call - and rolls back only the tokens written after the mistake.

By Ossam Hashim

KEY FINDINGS

  • On 2,700 easy-to-intermediate ASyMOB symbolic-math problems, five repeats per problem, online checking raised mean accuracy for all three open-weight models tested: Qwen3-30B-A3B-Instruct 62.4 to 68.0%, Qwen3-4B-Instruct 23.4 to 24.8%, Gemma-4-12B-it 45.2 to 48.2% - gains of 1.4-5.6 points at a 6-7% token increase and zero extra model calls.
  • The gain is surgical, not global: within answers the checker flagged, accuracy rose 6.0-10.2 points; on unflagged answers the change is statistically indistinguishable from zero, so baseline behavior survives untouched where no check fires.
  • Intervention is cheap and common: 21.8-53.1% of answers received at least one correction, each rollback discarded only 1.4-15.6 tokens on average, and each check cost roughly 22-25 ms of CPU time while the GPU kept decoding.
  • The benefit concentrates where it should: on problems the plain model solves inconsistently (one to three of four repeats correct), Qwen3-30B gained 8.4-8.8 points and those groups carried about 82% of its total net gain; problems beyond the model's ability barely move, because checking adds no new knowledge.

Speculative Execution for Reasoning

The framing Elimelech, Shalyt and Kaminer borrow is from CPUs (arXiv abstract page): speculative execution with branch prediction. The model keeps decoding on the GPU, effectively speculating that its preceding claims are correct, while a lightweight CPU pipeline parses completed equations out of the text stream and verifies them asynchronously with a computer algebra system. The unit of verification is a locally checkable identity. Extraction runs after each streamed chunk using regular expressions, needs no prescribed step format, and splits chained equalities like a=b=c into pairwise claims. Heuristic filters then discard definitions and assignments, trivially short claims, and equations whose two sides carry different free symbols - the last filter keeps conditions-to-be-solved, like a quadratic set to zero, from being misread as asserted identities. Verification itself is numerical rather than symbolic: sides without free symbols are compared by value, and sides with free symbols are evaluated at sampled substitutions within a tolerance, the same comparison procedure ASyMOB uses to grade answers. When a check fails, the continuation after the offending step is discarded, a short corrective message is appended, and generation resumes from that point. The checker never supplies the solution or a strategy - it flags a local error and lets the model repair it.

The Numbers

The evaluation is deliberately plain: 2,700 ASyMOB problems at easy-to-intermediate difficulty, three models from two families, five repeats each, plain versus checked generation. Qwen3-30B improved from 62.4 to 68.0% mean accuracy (problems correct on all five repeats rose from 796 to 958), Qwen3-4B from 23.4 to 24.8%, and Gemma-4-12B from 45.2 to 48.2%. Mean tokens per answer grew only 6-7%, and the discarded continuations account for little of that - the extra tokens are mostly re-derivation after feedback, not waste. Checker latency is 22-25 ms of CPU per check, cheap enough to run alongside generation on resource-constrained deployments where a verifier LLM would cut throughput. The decomposition matters more than the headline: replaying the same detection rule offline on plain answers shows flagged fractions are similar across conditions, so the method is not merely changing which answers get flagged - within the flagged group it recovers 6.0-10.2 points of accuracy, and outside it nothing moves. Where the checker fires, accuracy jumps; where it does not, baseline performance is preserved. That selectivity is the practical story: this is not a better reasoner, it is a safety net that only touches reasoning already going wrong. It complements rather than competes with training-side verification - and where Wrong the Same Way showed static verifier audits missing the errors RLVR amplifies, this work shows what checking mid-stream, before the error propagates, can recover.

Where the Argument Bends

The authors are clear about what a local check cannot see. It cannot establish whether a step follows from the problem's assumptions, whether the strategy is sound, or whether a correct calculation rests on a mistaken premise - misreadings and wrong plans pass silently. Context-free evaluation can also reject identities that hold under unstated, problem-specific assumptions; numerical sampling can miss a disagreement; and parsing failures, extraction filters and timeouts shrink coverage further. The evaluation itself is one benchmark and three models from two families, all at easy-to-intermediate difficulty, so the effect size on frontier models or harder mathematics is unknown. And the 1.4-point gain on Qwen3-4B, the weakest model, is a reminder of the mechanism's boundary: checking helps a model realize capability it already has, and adds none it lacks. Read the result as a deployment pattern - deterministic, model-free, mid-stream checking wherever a cheap checker exists - rather than as an accuracy fix for weak models.

Assumptions and limitations

All results are the authors' own, from their reported runs on 2,700 ASyMOB problems with three open-weight models and five repeats; we re-ran nothing. Gains are specific to symbolic mathematics with explicit, checkable equations, the 1.8-14.9% figure is the net reduction in incorrect answers relative to each model's own error count, and the worked example is an invented illustration of the paper's numerical-checking step, not a measurement.

Worked Example: A Toy Identity Checker

The verification step is simple enough to sketch. The paper checks identities by evaluating both sides at sampled values within a tolerance; this invented toy does the same on one true and one false expansion.

import random

# Toy checker: sample-test an
# identity, like the paper's CAS
# step. Invented numbers.
def val(expr, x):
    return eval(expr, {'x': x})

def close(a, b):
    return abs(a - b) < 1e-6

checks = [
  ('(x+1)**2', 'x**2+2*x+1'),
  ('(x+1)**2', 'x**2+1'),
]
for lhs, rhs in checks:
    xs = [random.uniform(-9, 9)
          for _ in range(50)]
    ok = all(
      close(val(lhs,x), val(rhs,x))
      for x in xs)
    print(lhs, '=', rhs, '->', ok)

Sources & Provenance

  • Elimelech, Shalyt and Kaminer, "Catch It Where It Happens: Real Time Error Correction During LLM Reasoning" - arXiv abstract page (2611.13107)
  • The same paper - full text via the abstract page (method, Table 1, flagged/unflagged decomposition, limitations)
  • Related GlobalTechDigest coverage: Wrong the Same Way

Written by Instinct for GlobalTechDigest. Analysis, framing and toy code are original; all measured results are the cited authors'.

Comments

Kubernetes Liveness Is Not Readiness: Why a Health Check Can Amplify an Outage

KUBERNETES FAILURE RECOVERY A failed health check is an observation. Restarting a container and withholding traffic are different responses, and choosing the wrong one can amplify an outage. By Hossam The important question is what failure should trigger A web application loses access to its shared database. Its processes are still running, but requests that require database access fail. A health endpoint reports the dependency problem. Should Kubernetes restart every application container, or stop directing those requests to it? The endpoint alone cannot choose the right recovery action. Kubernetes separates startup, liveness and readiness probes because they answer different operational questions. Startup protects initialization, liveness can trigger container termination and restart handling, and readiness controls eligibility for traffic from matching Services. A green check is meaningful only within the check and response that were designed. This is a documentation-based de...

Editorial Governance & Pages