StaleBench Holds Rank 1 Fixed, Yet Reader Accuracy Differs by 65.6 Points Without Dates
AGENT MEMORY AND EVALUATION
A controlled memory benchmark separates finding the current statement from assembling a context that lets a reader use it, and shows where dates, order and retention change the result.
By Ossam Hashim
KEY FINDINGS
- On 468 test facts, two policies return the identical statement at rank 1 for every query, yet qwen2.5:14b scores 32.7% versus 98.3% when reading their top-five contexts without dates. The corresponding glm-4.6 gap is 68.4% versus 97.9%, or 29.5 percentage points.
- The difference is in the rest of the context: the retain-everything control includes stale versions, while symmetric read/write keying usually removes that contradiction. This is evidence about those contexts and readers, not a universal 65.6-point benefit from deleting memory.
- Showing stored years closes the Qwen pair at 98.3% and reduces the GLM pair's gap to 1.5 points. Reversing presentation order also changes the sign of timestamp reranking's benefit on Qwen. Prompt construction is part of the evaluated system.
- Retention needs a reliable temporal signal too. Out-of-order ingestion can leave one confidently stale statement with zero contamination. On multi-valued facts, one-slot retention also loses coverage of values that remain simultaneously valid.
A Correct First Hit Can Still Produce a Wrong Answer
Memory evaluation often stops after asking whether the right item was retrieved. The StaleBench paper asks what happens when an agent stores several versions of a changing fact and a reader receives them together. A question naming an entity but not its answer can match both the old and new statements. Promoting the current one to rank 1 does not remove its superseded neighbor from the prompt. The reader still has to decide which version to use.
The paper's clearest experiment holds the first hit fixed. Its read-path-only policy and symmetric policy return the same statement at rank 1 on every anchored test query. Both score 98.9% on current@1. Their no-date Qwen reader accuracies are 32.7% and 98.3%. That 65.6-point gap is possible because a ranking metric inspects one position while the reader consumes five. It is a warning about an incomplete measurement boundary, not proof that retrieval accuracy is useless or that retention always wins.
What Current Means in This Benchmark
StaleBench derives its headline track from TempLAMA's Wikidata-based temporal facts. It selects 892 fact keys that are single-valued in every year and change at least once in the 2010-2020 window, then splits them by key hash into 424 validation and 468 test facts. Statements are template verbalizations of the recorded values; consecutive years with one value become one run. The source's validity information defines the current value in the final year, rather than a similarity score defining its own gold label.
This is a frozen historical benchmark. Its questions use words such as currently, but the answer means current in the benchmark's final year, not current in the world today. The distinction matters because the underlying facts concern real public figures and organizations. A benchmark can isolate supersession without becoming a live source of biographical truth. TempLAMA originally probes temporal knowledge in language models; StaleBench moves the question into a store that has already ingested multiple versions.
Separate Stale Answers from Misses and Mixed Contexts
The frozen evaluator distinguishes three top-one outcomes. A current result names the right fact and a currently valid value. A stale result names the right fact but an outdated value. A miss points to another fact or returns nothing. Current@5 asks whether any current version appears among the first five. Contamination@5 asks whether that same context contains both a current and a stale version of the queried fact.
Contamination is not a complete quality score. A context containing only stale statements has zero contamination because it contains no current statement to contradict them. An empty context also has zero. A policy could improve this metric by deleting the answer. The paper therefore pairs it with current@k and separately reports reader accuracy and store footprint. A useful dashboard needs all of these perspectives, particularly when a policy trades contradictions for omissions.
The Cheap Fix Really Does Improve Retrieval
On the anchored test split, cosine store-all retrieval has current@1 of 46.4%, with stale results on 53.0% of queries. Reranking its top two similar hits by timestamp raises current@1 to 92.5%. That is a large, useful retrieval improvement, and it should not disappear from the story just because a later stage behaves poorly. The method is simple and does not require deciding whether two differently worded statements describe the same fact before writing them.
The limitation is that reranking changes order rather than membership. Its top-five context remains contaminated on 97.4% of queries, the same as cosine store-all. With dates omitted and the reported descending-relevance presentation, Qwen reader accuracy falls from 55.8% for cosine to 32.9% after reranking. GLM instead rises from 61.1% to 67.7%. The sign differs across readers, and neither change resembles the 46.2-point retrieval improvement. Those outcomes belong to the full prompt-and-reader pipeline.
What the Controlled Pair Isolates, and What It Does Not
The two keying policies share the same read path: resolve the query to a fact key and rank that key's records by recency. One retains all versions; the other also merges versions on the write path. In the anchored reader experiment, their first returned statement agrees on all 468 queries, verified per query rather than inferred from equal aggregate scores. Contamination@5 is 99.8% for the retain-everything member and zero for the symmetric member.
That controls rank 1, but context composition still differs in more than one textual detail. Removing older versions also makes room for statements about other facts. The policy-track gap therefore measures what those policies actually hand the reader, not an isolated numeric dose of contamination. The paper addresses that limitation with constructed contexts that add a stale statement to an otherwise current-only prompt. Those manipulations carry the narrower causal claim about stale neighbors; the two policy contexts establish why first-hit metrics can miss an important end-to-end difference.
Order Can Dominate the Size of the Penalty
With constructed current-only contexts, Qwen scores 99.1%. Adding one stale statement while placing the current statement last reduces accuracy to 85.0%. Putting the same two statements in the opposite order, current first, reduces it to 31.4%. GLM also suffers from the added stale statement, but its two orders are much closer: 63.7% and 59.2%, compared with 98.7% on current-only contexts. The direction of harm agrees; strong positional sensitivity is much more pronounced for Qwen.
The paper then reverses how an identical retrieved top-five is presented. On Qwen, timestamp reranking changes from 32.9% to 70.3% reader accuracy, while cosine moves only from 55.8% to 56.4% in aggregate. Relative to its own baseline, reranking goes from a 22.9-point loss to a 13.9-point gain. That ablation runs on Qwen only. It supports testing both prompt orders on a deployment's own reader, not declaring that timestamp reranking is intrinsically harmful or that reversing order always fixes it.
Trustworthy Dates Are a Competing Remedy
Showing the stored year changes the controlled pair dramatically. Qwen's retain-everything member rises from 32.7% to 98.3%; its symmetric member remains at 98.3%. GLM reaches 95.7% and 97.2%, leaving a 1.5-point difference. In the constructed worst-order two-statement condition, surfaced years raise Qwen from 31.4% to 92.5% and GLM from 59.2% to 93.8%. The dates give readers evidence they otherwise had to infer from conflicting undated assertions.
This result cuts against a simple sales pitch for retention. If validity dates are available and trustworthy, exposing them can recover much of the reader performance without destroying the history. Retention may still reduce storage and prompt cost, or fit a slot budget, but it is one route to resolving contradiction rather than the only one. Conversely, a generic creation timestamp is not automatically a validity date. A real deployment must establish what a timestamp means before treating newer as true.
Neither Arrival Order nor the Clock Is Automatically Reliable
The clean benchmark streams facts as they become true. Its ordinary overwrite rule therefore treats the latest arrival as current. Appendix E breaks that assumption independently of timestamps. Permuting arrival order drops the gazetteer and write-path oracle to 47.2% current@1, while timestamp reranking stays at 92.5%. Corrupting timestamps instead leaves arrival-based overwrite unchanged but drops reranking from 92.5% to 48.4%. Each family reads a different temporal signal.
A more targeted experiment makes a stale predecessor arrive after the final version for a fraction of test facts. At a 25% late-stale fraction, gazetteer overwrite falls from 97.2% to 72.9%. Its contamination remains zero: the store can be consistently wrong rather than visibly contradictory. A newer-timestamp-only overwrite variant preserves its clean 97.2% under this arrival corruption, but loses that protection if the clock is corrupted. Choosing a policy requires a trust model for event time, ingestion time and corrections, not a blanket preference for keeping only one version.
Same-Fact Detection Is Part of the Cost
Retention must decide whether an incoming statement supersedes an existing fact. Exact names make that look easy. In the anchored condition, gazetteer keying reaches 97.2% current@1, close to the 97.9% write-path oracle. When later mentions systematically use mechanically derived short forms, gazetteer keying falls to 16.9%; threshold-based keying reaches 51.3%. A fuzzy linker using the canonical inventory but no alias table reaches 71.4%.
Applying the fuzzy resolver to the query as well as incoming statements raises current@1 to 83.1% on that mechanical-short-form track. The read-only and symmetric variants tie there, but their contamination rates are 98.3% and 3.4%, and they hold 995 versus 467 slots. The two paths solve different problems: query resolution improves which fact is selected, while write-side merging reduces the conflicting versions available afterward. The paper calls the extra read-side use free in resolver quality, not a measurement that entity resolution consumes no compute, maintenance or engineering effort.
The Inventory and the Query Are Optimistic Resources
The basic keying ladder receives an inventory covering the test subjects. Queries always use the full subject name even when stored updates use short forms, making read-side resolution the easier direction. Expanding the inventory from 468 to 2,995 entries increases the symmetric fuzzy policy's anchor-free deficit against the write-path oracle from minus 0.4 to 19.7 percentage points. Applying the resolver twice does not remove the open-world entity problem.
Natural aliases qualify the mechanical result further. A character-trigram matcher cannot derive an unrelated nickname from a legal name. With real Wikidata aliases and Wikipedia redirects, the claim that symmetric fuzzy matching beats an alias-equipped write-only linker no longer holds in its strong form; the point estimates favor the alias table, but the reported differences are not statistically significant. Resolving on both paths still helps each resolver relative to its own write-only version. The defensible recommendation is to test both path symmetry and entity-resolution coverage, rather than presenting either as a substitute for the other.
One Slot Can Delete a Still-Valid Answer
The larger track includes 2,904 test facts, of which 2,436 have multiple simultaneously valid values. Its top-one score accepts any currently valid value. That can make a one-slot policy look successful even after it discards another answer that remains true. Set recall@5 measures the fraction of distinct currently valid values whose statements reach the context, exposing a cost that the headline single-valued track cannot express.
Anchored, the write-path oracle's set recall is 89.0%, versus 94.0% for recency-weighted store-all. Retention can win top-one freshness and contradiction measures while losing coverage. A multi-employer relationship, several team memberships, or another set-valued field should not be compressed using a single-valued supersession rule without explicit semantics. Where history itself is an audit record, destructive overwrite also removes evidence that may need to remain available. Version invalidation with preserved provenance is a different design choice from deletion.
Evaluate the Returned Text, Then the Reader
The paper's separate deployed-systems track scores returned memories by source attribution, by their actual text, and by a reader's answer. That is a useful evaluation pattern because a transformed or merged memory need not say what an attached source identifier suggests. The study itself warns that this is its narrowest systems evidence: 150 facts, a particular harness and model choices, and one of the four mechanisms is reimplemented rather than a pinned release. It should not become a universal product ranking.
For a practical evaluation, log the returned text, its source and validity information, the context order, and the reader output together. Report stale answers separately from misses, and keep a current-answer coverage measure beside contamination. Test a current-only context, a stale-only context, both orders of a conflicting pair, dates on and off, and a late-arrival stream. These checks are deductions from the paper's failure modes. They are not additional benchmark results or a guarantee that a memory system passing them is production-safe.
Assumptions and limitations
This article analyzes the StaleBench preprint and released documentation; it does not independently reproduce its reader or deployed-system experiments. Headline reader claims concern two models, a fixed top-five context, short answers capped at 32 tokens, English templates and a historical dataset. Three facts nest a superseded value inside the gold string, limiting the reader metric's attainable score to 99.4%. The anchored track is deliberately near-saturated on retrieval current@1, and the retain-everything member of the controlled pair is a diagnostic control, not a recommended memory architecture.
Mechanical aliases, complete subject inventories and full-name queries are controlled resources, not a complete model of conversation. Pronouns, ambiguous corrections and discourse-dependent reference remain outside the central instrument. The larger set-valued and natural-alias tracks add useful pressure without closing those gaps. Released cached outputs can regenerate reported model-call results, but replaying those outputs is different from independently rerunning a current model service. Publication of this benchmark does not establish live-world freshness for any application.
Worked Example: Zero Contradiction Can Mean Confidently Wrong
This invented example uses one fictional entity and a single-valued team field. A 2020 update to Blue arrives before a delayed 2018 Red report. Arrival-only overwrite returns Red: zero contamination, wrong current answer. Keeping both versions returns a current first hit and a contradiction. Filtering by the maximum validity year returns Blue alone. The code assumes trustworthy years and does not resolve ambiguous updates or simultaneous valid values. It illustrates why the temporal assumption must be named beside the metric, not how an LLM reader will answer.
from collections import OrderedDict
stream = [(2020, "Blue"),
(2018, "Red")]
gold = "Blue"
last = stream[-1][1]
ranked = sorted(
stream, reverse=True)
all_vals = [v for _, v in ranked]
latest = [ranked[0][1]]
def metrics(values):
hit = bool(values and
values[0] == gold)
has_cur = gold in values
has_old = any(v != gold
for v in values)
return int(hit), int(
has_cur and has_old)
cases = OrderedDict([
("arrival", [last]),
("keep_all", all_vals),
("validity", latest)])
for name, values in cases.items():
print(name, metrics(values))
assert metrics([last]) == (0, 0)
assert metrics(all_vals) == (1, 1)
assert metrics(latest) == (1, 0)
Sources & Provenance
- Murshed, StaleBench: Isolating Stale Retrieval in Agent Memory, and Pricing the Fix, full main paper and appendices B, C, E, I, L, M, R, S and W for task, reader, temporal, alias, scale and reproduction limits.
- StaleBench repository: policy definitions, result overview, harness and reader-call cache descriptions, not an independent replication.
- StaleBench release contract: dataset configurations, separate answer keys and opaque statement identifiers, with a warning that the anchored track is a diagnostic rather than a leaderboard.
- Frozen StaleBench evaluator: the exact current/stale/miss/contamination metric definitions in its published scoring function.
- StaleBench archived release record: dataset/artifact version and distribution scope.
- Dhingra and colleagues, Time-Aware Language Models as Temporal Knowledge Bases, ACL Anthology record: upstream temporal-knowledge task and the distinct model-level motivation for TempLAMA.
All measured accuracies and percentage-point gaps are attributed to the StaleBench author. The deployment checks and fictional late-arrival example are original analysis. No real person's current affiliation is asserted here, and no model reader was run for this article.
Comments
Post a Comment