Header Ad Placement

MEM-SQL Raises Mean SQL Accuracy to 50%, but Combining Verified Queries Still Falls to 29.7%

TEXT-TO-SQL AND QUERY HISTORY

A history-aware benchmark rewards reusing old computations. Its harder lesson is that finding relevant SQL does not prove that the pieces answer a new question together.

By Ossam Hashim

KEY FINDINGS

  • MEM-SQL reports 50.0% mean execution accuracy across six backbones on 1,600 HARE requests, up from 37.1% without history. The 12.9-percentage-point gain accompanies reported average reductions of 76.6% in input tokens and 68.8% in pipeline latency. This is a comparison with a no-history pipeline, not proof that memory alone causes every saving.
  • Memory stores whole-query templates, SELECT-block components and schema concepts. A layer is accepted only when every retrieval atom assigned to it has a match above its threshold. That checks similarity coverage for the generated atoms, not semantic correctness or complete coverage of the user's intent.
  • Mean accuracy falls from 56.7% with one historical source to 29.7% with several. For multi-source structural changes, all sources are retrieved in 90.8% of requests, yet reported accuracy is only 9.9%. Retrieval and correct composition are separate problems.
  • The evaluation keeps verified memory fixed and does not test schema drift. It also compares execution results, which can match for the wrong reason. A production design still needs current schema and business-rule checks, result validation and a cautious fallback when evidence is incomplete.

History Is a Computation Cache, Not Just Another Example

MEM-SQL: Memory-Empowered Text-to-SQL, by Xiaoqi Wang and colleagues, examines a setting that isolated question-to-SQL tests miss. An analyst repeatedly queries the same database. Yesterday's verified query may already contain the right join, a difficult intermediate calculation or the business meaning of a column. Reconstructing all of it from a schema for each new request wastes work.

The November 13 preprint introduces HARE, a benchmark built to label that reuse. It contains 1,600 new requests derived from a fixed pool of 823 verified historical queries across Spider, BIRD, Spider 2.0 and BEAVER. The central question is not whether retrieving something related helps. It is which parts of the old computation remain valid after the user changes the request.

The distinction is practical. Changing a year in a proven revenue query is unlike asking for a new cohort comparison built from several queries. Both requests can mention the same tables and have high semantic similarity. Only one may be a safe slot substitution. A system that treats them identically can produce SQL that runs, returns plausible numbers and answers the wrong question.

HARE Separates Rewording from New Query Structure

HARE uses six single categories and six two-category combinations. Rephrasing preserves the complete SQL. Parametric changes preserve the query structure while changing allowed values. Component-parametric changes reuse an intermediate computation with new values while allowing different outer logic. Structural changes, shared joins and shared concepts test reuse at increasingly different boundaries.

These labels are not interchangeable with the storage layers. Appendix D makes that explicit: a memory template can expose slots for aggregates, windows and grouping keys, whereas HARE's primitive parametric category has stricter rules about preserving structure. A reusable SELECT-block item can carry both join and clause information. Calling every retrieved template a previously verified answer therefore overstates what was verified.

The benchmark has 1,300 single-category requests and 300 composite requests, with 50 in each composite category. All historical sources for a request come from the same database. GLM5.3-Flash generates candidates under specified edit rules; Claude Opus 5.5 reviews semantic alignment and reuse constraints, with bounded repairs. Parsing, duplicate checks and successful nonempty execution are also part of construction.

That is controlled, model-assisted benchmark construction, not a sample of unfiltered production analyst traffic. Appendix A even records examples whose demonstrated edits depart from the stated generation rules. The labels make controlled comparisons possible, but they should not be treated as a perfect account of every real reuse case or every generated annotation.

What the Three Memory Layers Retain

Whole-query templates preserve tables, joins and clause layout while replacing selected values and expressions with typed slots. A request can retrieve a complete pattern and fill it rather than generate the query from scratch. The stored description tells the retriever what the pattern does; the structured payload retains the SQL that the generator needs.

Components operate at the SELECT-block level, including outer queries, common table expressions, subqueries and set-operation branches. They expose a join path and clause structure that can be placed inside different outer logic. The architecture thus distinguishes remembering how to connect rows from remembering the final question those rows once answered.

Schema concepts map business terms to columns, expressions and relationships. A metric definition is different from a request-specific threshold. The extraction prompt tries to preserve a fixed definition while dropping old ranking, output and incidental values. That is an important boundary: a clinical normal range or a revenue definition must not become a freely changeable parameter merely because it contains a number.

Memory admission also checks more than proximity in embedding space. The paper describes a high-similarity merge only when stored structures or mappings are equivalent; Appendix B adds same-layer and schema-version restrictions, with grain checks for components. If a candidate does not merge, a reviewer can inspect nearby items for coverage before inserting it. These are admission checks for the evaluated static store, not evidence that the system continuously detects production schema changes.

A Worst-Covered-Atom Gate Is Better Than One Average Score

At inference, a decomposer turns a question into retrieval atoms for each layer and a checklist of answer requirements. Each atom gets its own nearest-neighbor budget. The gate takes the best score for each atom, then checks the lowest of those best scores against the layer threshold. A strong match for one requirement cannot numerically cancel a poor match for another.

Appendix B uses two candidates per template atom with a 0.45 threshold, eight per component atom with a 0.50 threshold, and two per schema atom with a 0.50 threshold. The equation uses a strict greater-than comparison. An empty atom set fails the layer gate rather than automatically passing it. These are study settings, not calibrated universal probabilities of correctness.

The gate accepts layers independently, then combines their evidence. It does not require every layer to pass before trying memory. Nor does it compare retrieved SQL against an external proof of the requested business meaning. If the decomposer omits a requirement, or two descriptions are similar despite different aggregation scopes, all recorded atoms can clear their thresholds and the final query can still be wrong.

The retrieval ablation supports a narrower claim. On a separate 240-question calibration set, requirement-specific retrieval raises source-linked micro recall from 56.67% to 94.58% relative to using the full question within each layer. Flat retrieval reaches 96.25% but returns 16.48 items per question instead of 10.26. The layered design keeps a smaller evidence bundle; it does not beat flat retrieval on every accuracy measure.

When Schema Access Becomes Optional

On-demand Schema Linking, or OSL, first tries memory when at least one layer contributes. If the evidence bundle is empty, it retrieves schema immediately. If the generator declares the selected evidence insufficient, schema linking supplies missing context before synthesis. This avoids paying for a broad schema investigation on every request that can already be answered from verified history.

The memory generator is instructed to name missing tables, columns, joins or definitions instead of guessing them. Appendix D illustrates a driver-finishing-time question for which the cache contains a qualifying-session query and a birth-year concept, but not the required results table and join. Related racing language does not make those items sufficient. This is a prescribed abstention example, not a measured guarantee that the model abstains whenever it should.

The pipeline checks syntax and table references before a database dry run. The memory route permits one repair after a validation error; a second failure or a generation error ends that attempt. The schema-linking route permits two repairs within an overall limit of three SQL-generation calls. A dry run can expose an invalid identifier or syntax error, but cannot by itself establish that a valid aggregation implements the intended denominator.

The Headline Gain Mixes Representation and Routing

The main comparison reports mean execution accuracy rising from 37.1% to 50.0%, with average token and latency reductions of 76.6% and 68.8%. Those are averages across six backbones. Pipeline latency includes recorded reused decomposition and schema-linking costs, excludes evaluation time, and is compared within each backbone because serving conditions differ. It is not evidence that one backbone's absolute latency is faster than another's under equal hardware.

Direct Generation has no query history. The three memory baselines in the main table use unconditional schema linking, while MEM-SQL uses OSL. Representation, retrieval and routing therefore differ together. The result is a system comparison, not an isolated causal estimate for a single memory layer or for skipping the schema.

The controlled routing comparison in Section 6 is more revealing. Holding each baseline's memory and retrieval fixed, OSL lowers input tokens and latency for every memory design on average. Accuracy does not move uniformly. BIRD-History loses accuracy on five of six backbones; on Qwen3.8-27B it falls from 45.3% to 42.4%. A-MEM with OSL reaches 58.6% on Gemini4-Argon, above MEM-SQL's 57.5%. The claim that MEM-SQL leads every memory baseline applies to the main unconditional-baseline table, not all reported settings.

The architecture's own ablations are also mixed. Removing templates costs 5.3 accuracy points on average, supporting their value. Flat retrieval matches or exceeds MEM-SQL on five backbones, though it uses more tokens on all six. The design offers a cost-accuracy tradeoff rather than an unqualified dominance result.

Several Correct Sources Can Still Produce One Wrong Query

Table 6 is the sharpest operational warning: mean execution accuracy is about 56.7% for single-source requests and 29.7% for multi-source requests. The lower figure is not the score for every composite category. Source count and reuse category are different partitions, and structural changes can require several sources even within one category.

For multi-source structural changes, retrieving every historical source in 90.8% of cases still yields only 9.9% accuracy. On parametric-plus-shared-join requests with complete source retrieval, accuracy falls from 81.5% for one source to 46.7% for multiple sources. Bringing all the ingredients into context does not establish how their grains, populations and intermediate results should be combined.

Routing does not reliably recognize that difficulty. The paper reports incomplete evidence for 51.7% of composite memory-route requests, yet only 2.7% of composite requests escalate to schema linking. These percentages have different denominators; they should not be subtracted to count missed escalations. They nevertheless show that memory-first routing can remain confident precisely where composition is difficult.

The failure can also survive additional retrieval. Joining two individually valid aggregates at incompatible grains can multiply rows. Summing a customer-level total after attaching several order rows can double-count it. Retrieving the missing source may be necessary, but checking keys, grouping scope and the requested population is a separate synthesis task.

Wrong-Answer Percentages Are Not Failure Rates for All Requests

The paper's error analysis classifies incorrect memory-route answers using the first applicable category in a fixed order. Its 10.8% execution-failure share for MEM-SQL, compared with 3.2% for A-MEM, is a share of wrong answers in that analysis. It should not be described as 10.8% of all 1,600 queries failing to execute.

Aggregation and grouping remain a large error class even when history reduces table-selection errors. This fits the composition evidence: knowing the correct tables and joins does not settle whether to count rows, count distinct entities, average per customer or compute a weighted ratio. An automatic first-applicable-error taxonomy also is not an independent human causal audit of every failure.

Execution accuracy itself has limits. Appendix B compares columns by position and rows as multisets, preserving duplicates, and checks row order when the reference has an outer ORDER BY. It treats SQL NULL, an empty string and the string 'None' as distinct. Execution failures and queries exceeding 660 seconds count as wrong. These details matter when another evaluator or dashboard reports a seemingly comparable score.

A result can still match on the tested data for an unintended reason. Spider's evaluation documentation warns about such false positives, and its test-suite evaluator compares behavior across database variants to probe semantic equivalence more strongly. That supporting work explains a measurement risk; this article does not claim that MEM-SQL was tested with that stronger protocol or estimate how many of its answers would change.

What a Deployment Review Should Add

The useful deployment lesson is to treat history as versioned evidence, not permanent authority. Preserve the query's database, schema version, business definitions, output grain and provenance. Recheck mappings when a table changes or a fiscal calendar is redefined. HARE explicitly does not evaluate schema drift, and its memory remains fixed, so those checks are an engineering recommendation rather than a demonstrated MEM-SQL capability.

Separate inexpensive slot adaptation from multi-source composition in monitoring. Record whether every requested obligation has support, whether each join preserves the intended cardinality, whether a metric's denominator is correct and why the schema route was or was not used. A cosine threshold is a retrieval policy, not a permission to ignore missing evidence.

Evaluate with difficult cases that make wrong SQL disagree with the reference: duplicated join keys, entities with no matching rows, NULLs, tied ranks and unequal group sizes. Use controlled, read-only evaluation environments and inspect query plans and cost limits before treating a generated query as production-ready. These suggestions follow from the metric and composition failure modes; they are not additional benchmark results reported by the authors.

BIRD and Spider 2.0 provide wider context for why these boundaries matter: real database tasks can depend on values, documentation and large schemas. HARE measures a particular verified-history setting derived from those sources. Its 50.0% score is not a new BIRD or Spider 2.0 leaderboard result, and a history-heavy workload should not be used to promise the same savings for a cold-start database.

Assumptions and limitations

This is a source-based analysis of the November 13 MEM-SQL preprint. The main paper and full PDF Appendices A-D, including protocol, supplementary results and prompt templates, were read. The HTML rendering ends partway through Appendix A, so it is not a complete substitute for the PDF. Study results are author-reported, not independently reproduced here. No code-release status, production security guarantee or live-system deployment claim is inferred from the paper.

HARE uses model-assisted generation and review, a fixed verified history pool and a selected mix of reuse categories. It does not test evolving memory or schema drift. Main-table comparisons differ in both history representation and schema-routing policy; the controlled OSL comparison gives a narrower view of routing. Cross-backbone means, conditional error shares and source-count subgroups have different denominators. The worked example is explanatory arithmetic, not a trained retriever or a semantic verifier.

Worked Example: One Strong Match Must Not Hide a Missing Requirement

This invented example represents two retrieval atoms. Their best similarity scores are 0.94 and 0.42, with a threshold of 0.50. Averaging them gives 0.68 and would pass a mean-score gate; the worst-covered-atom rule rejects the layer because 0.42 is too low. If the second score rises to 0.51, the layer passes. This illustrates the strict threshold rule only. It does not show that the retrieved SQL has the right meaning, that the decomposer listed every requirement or that the combined query is correct.

from fractions import Fraction as F
scores = [F(94,100), F(42,100)]
limit = F(1,2)
mean = sum(scores) / len(scores)
gate = min(scores) > limit
print(float(mean), gate)
scores[1] = F(51,100)
print(min(scores) > limit)
assert mean == F(68,100)
assert gate is False
assert min(scores) > limit
assert not (F(50,100) > limit)

Sources & Provenance

  • MEM-SQL HTML, November 13, 2026. Main system design, HARE construction, headline results and limitations; HTML appendix is incomplete.
  • MEM-SQL full PDF, arXiv:2611.18338v1. Complete Appendices A-D, retrieval settings, scoring rules, conditional error denominators and prompt examples.
  • Spider evaluation documentation. Explains why matching one database's execution result can yield a false positive despite different SQL semantics.
  • Spider test-suite evaluator. Supporting context for evaluating denotations across database variants; not used to re-score MEM-SQL in this article.
  • BIRD benchmark project. Context on value-grounded database tasks and benchmark updates, not independent validation of HARE's measurements.
  • Spider 2.0 project. Context on enterprise workflows and schemas; its task scores are not interchangeable with HARE execution accuracy.

The worst-covered-atom example and deployment-review recommendations are original explanations. Relevant history can reduce repeated work; it cannot decide on its own whether a new combination means what the user asked.

Comments

Kubernetes Liveness Is Not Readiness: Why a Health Check Can Amplify an Outage

KUBERNETES FAILURE RECOVERY A failed health check is an observation. Restarting a container and withholding traffic are different responses, and choosing the wrong one can amplify an outage. By Hossam The important question is what failure should trigger A web application loses access to its shared database. Its processes are still running, but requests that require database access fail. A health endpoint reports the dependency problem. Should Kubernetes restart every application container, or stop directing those requests to it? The endpoint alone cannot choose the right recovery action. Kubernetes separates startup, liveness and readiness probes because they answer different operational questions. Startup protects initialization, liveness can trigger container termination and restart handling, and readiness controls eligibility for traffic from matching Services. A green check is meaningful only within the check and response that were designed. This is a documentation-based de...

Editorial Governance & Pages