Header Ad Placement

Sable Proves a 4,096-Token Qwen2.5-32B Integer Run in 92.5 Seconds, Not an Identical BF16 Computation

VERIFIABLE INTEGER INFERENCE

A single-GPU prover checks an exact quantized transformer with large auditor-oriented proofs. The benchmark excludes cold compilation, the integer model loses some accuracy, and the current proof is not zero-knowledge.

By Ossam Hashim

KEY FINDINGS

  • Sable attests a committed integer function, including normalization, softmax, rounding and token selection. Its bit-exact claim concerns that function, not equivalence to the BF16 checkpoint from which its weights were derived.
  • On one RTX PRO 6000 Blackwell GPU, the paper reports 92.5 seconds for all 64 Qwen2.5-32B layers over 4,096 total tokens. Those tokens are a prompt followed by sixteen generated tokens, not 4,096 newly generated tokens. The reported time is a warm proving run.
  • That proof is 134.8 MiB and takes 18.8 seconds to verify with a compiled core on one CPU core. It uses 36.8 GiB of GPU and 101 GiB of host memory. The implementation targets auditors who can handle substantial verification work, not a tiny proof for every chat response.
  • Exactness has an accuracy and privacy cost. The integer model's WikiText-2 greedy tokens agree with BF16 at 86.3% of positions, and its perplexity is 8.3% higher. The proof reveals trace evaluations and is not zero-knowledge; closed-weight and multi-tenant uses need protections not supplied by the main proof.

First Define the Computation Being Proved

A cryptographic proof of inference needs an exact statement. It is not enough to say that approximately the right model probably produced a plausible answer. Tomer Solberg's Sable: A Small-Field Sumcheck Argument for Bit-Exact Inference of Large Language Models defines a particular integer transformer and proves that it produced the claimed sequence from the stated inputs.

The function, written F_O in the paper, fixes rounding, scaling, normalization, attention and sampling behavior. A deterministic conversion maps a BF16 checkpoint into the integer model. Once converted, that model is a different function, with its own accuracy and runtime properties.

This distinction is the central deployment question. A proof can establish faithful execution of the agreed integer model while its answer differs from the original floating-point model. The verifier must accept the committed weights and circuit definition as the intended service. Proving an unwanted or inaccurately converted function does not make it the right function.

Why Small-Field Arithmetic Needs Integer Bounds

Sable expresses model operations as polynomial constraints and lookup relations over a 31-bit prime field. Small-field arithmetic is convenient for the prover's GPU, but modular arithmetic alone does not preserve the intended integer meaning.

An integer mismatch can disappear after reduction modulo the field prime. The constraint design therefore pairs arithmetic relations with range and table checks. Its wrap-free arguments establish equivalence to the intended integer identities under the specified bounds. Correct bounds are part of the proof statement and construction, not an optional numerical-safety optimization.

Lookups also require careful batching. The system splits them into bounded chunks so that lookup counts stay within the prime-field conditions used in the soundness analysis. A commitment and a successful algebraic check are useful only when the surrounding range, lookup and challenge assumptions match the relations being claimed.

The Prover Trades Recalculation for Storage

The prover processes the model in two main passes. It first runs forward, constructs each layer's witness, commits to it and counts lookup multiplicities. It retains the residual streams between layers in host memory rather than keeping every expanded witness resident on the GPU.

The proof pass then walks backward from output claims, recomputing layer witnesses from stored inputs. Batched sumchecks, lookup arguments and a multilinear commitment connect the claimed computation to committed tensors and weights. Recomputing costs work, but keeps the largest live witness closer to a layer's size than a whole-model trace.

The implementation places its transcript on the GPU and replays repeated layer work through CUDA graphs. These details are part of the measured system. They do not imply graph replay fits every shape: the paper's larger Qwen3-32B runs disable it because captured graphs do not fit in device memory.

What the 92.5-Second Result Measures

The main Qwen2.5-32B experiment covers all 64 layers on one 96GB RTX PRO 6000 Blackwell Server Edition GPU. Its 4,096-token workload contains a prompt plus sixteen greedy generated tokens, and the proof covers stopping behavior. Four 4K sequences can share one proof; a single 16K sequence is another tested shape.

Mean warm proving times are 92.5 seconds for one 4K sequence, 253.1 seconds for four 4K sequences and 397.5 seconds for one 16K sequence. Equal total token counts do not imply equal attention work: one long sequence has more causal attention entries than several shorter sequences.

The timed runs assume kernels are compiled and the model's weight commitment already exists. With an empty cache, the first 4K proof takes 1,530 seconds. A fresh process also pays warm-up for pinned memory and graph capture before the reported repeated warm runs. The headline is therefore not a cold-start latency promise.

For the same Qwen2.5 integer computation, proving takes roughly 41-50 times the authors' serving-engine time. This denominator matters. The paper gives a larger factor, about ninety, against its quoted SGLang FP8 comparison at 4K. Neither comparison means proving accelerates ordinary inference; both quantify substantial extra work needed to attest it.

The extension to Qwen3 is also measured, not merely proposed. At 4K, reported proving times are 121.8 seconds for dense Qwen3-32B and 84.7 seconds for Qwen3-30B-A3B. The denser model's larger shapes have additional memory and timing caveats. These are architecture-specific results, not a universal cost per parameter.

Proof Size, Privacy and Trust Anchors Remain Visible

The main 4K Qwen2.5 proof is 134.8 MiB and verification takes 18.8 seconds on one CPU core with a compiled verification core. Across the tested models and shapes, proof sizes reach roughly 106-477 MiB and verification reaches about 56 seconds. This is useful evidence for an audit workflow, but a long way from a small, instant client check.

The main proof is not zero-knowledge. It reveals evaluations of the computation trace; a batch also exposes its prompts to each verifier. The paper warns that repeatedly reopening committed closed weights could eventually leak them. Open-weight auditing and confidential multi-tenant serving therefore have different requirements, even if they use the same arithmetic core.

Recursive compression and zero-knowledge outer layers are directions, not finished properties of this implementation. Sable implements a BLAKE3 constraint component and reduces verification hashing, but that is not a completed compact, private recursive proof of the entire model run.

Its security claim is conditional too. The stated bound models BLAKE3 as a random oracle and assumes valid in-range committed model data. The round-by-round error parameter is about 2^-103.4 with proof-of-work contributions; the theorem's adversarial bound also depends on the number of oracle queries. A bare claim of unconditional 103-bit safety would discard those qualifications.

Accuracy Is Separate from Execution Integrity

For Qwen2.5-32B on WikiText-2, the integer model's 86.3% greedy-token agreement with BF16 is not an accuracy score. It measures agreement between two implementations. Its perplexity is 8.3% higher than BF16, compared with smaller increases in the paper's simulated FP8 and INT8 comparisons.

Task scores lose less in some tests: the paper reports a 1.4-point MMLU decrease and a 0.6-point GSM8K decrease for this integer model. That does not cancel the next-token discrepancy. It shows that different evaluations reveal different consequences of the same conversion.

Wider query and key values recover part of the perplexity gap in an attribution experiment, but those variants are not provable as implemented. On Qwen3, adjusting the precision split between existing query and key formats improves accuracy without increasing proof cost. Neither result establishes that exact inference and floating-point behavior already coincide.

Sable's contribution is an executable proof of a precisely specified model, with costs exposed. The engineering standard for deployment must keep execution integrity, model quality, confidentiality, verifier trust anchors and total latency as separate checks. Passing one does not imply the others.

Assumptions and limitations

This is a primary-paper analysis, not an independent benchmark or cryptographic audit. Reported proofs concern the specified integer model, not bitwise BF16 equivalence. Warm timing excludes compilation and precomputed weight commitment; workloads include prompts and sixteen generated tokens. The main proof is large and not zero-knowledge. Security relies on the theorem's random-oracle, commitment, range and query-budget assumptions. Accuracy attribution variants and proposed recursive or privacy layers must not be mistaken for supported end-to-end configurations.

Worked Example: A Modular Check Can Hide an Integer Mismatch

This invented example uses a toy prime of seventeen. The integer claim twenty equals three is false, but both sides have the same residue. A bound restricting values below the prime rejects the misleading lift. This illustrates why ranges matter; it is not Sable's field, protocol or a proof of its gadgets.

from math import prod

prime = 17
lhs = prod([4, 5])
rhs = 3
mod_ok = (lhs-rhs) % prime == 0
in_range = 0 <= lhs < prime
print(lhs == rhs, mod_ok, in_range)
assert mod_ok
assert not in_range
assert lhs != rhs

Sources & Provenance

Written by Instinct for GlobalTechDigest. Analysis and the labeled toy example are original; measurements and theorem claims belong to the cited author.

Comments

Kubernetes Liveness Is Not Readiness: Why a Health Check Can Amplify an Outage

KUBERNETES FAILURE RECOVERY A failed health check is an observation. Restarting a container and withholding traffic are different responses, and choosing the wrong one can amplify an outage. By Hossam The important question is what failure should trigger A web application loses access to its shared database. Its processes are still running, but requests that require database access fail. A health endpoint reports the dependency problem. Should Kubernetes restart every application container, or stop directing those requests to it? The endpoint alone cannot choose the right recovery action. Kubernetes separates startup, liveness and readiness probes because they answer different operational questions. Startup protects initialization, liveness can trigger container termination and restart handling, and readiness controls eligibility for traffic from matching Services. A green check is meaningful only within the check and response that were designed. This is a documentation-based de...

Editorial Governance & Pages