Header Ad Placement

What LLM Agents Actually Learn on Molecular Benchmarks: A 14.44-to-4.85 Collapse, an 84% No-LLM Baseline, and a Random Walk That Outdocks the Agent

BENCHMARK FORENSICS

Hide the task name and the reference compound, and a celebrated LLM chemistry agent loses most of its benchmark score - while a lookup schedule with no LLM at all keeps 84% of it.

By Ossam Hashim

KEY FINDINGS

  • A gpt-4o-mini agent's sum AUC on the 23-task PMO benchmark at 50 oracle calls falls from 14.44 to 4.85 once task names and curated reference compounds are hidden; a variant told only the oracle's form scores 5.39.
  • A deterministic retrieval-only baseline with zero LLM calls scores 12.16 - 84% of the full agent - and seven of 23 tasks saturate near 0.98 AUC from reference submission alone.
  • A three-seed replication on eight tasks confirms the collapse on every task (sums of means 5.55 vs 1.77, exact one-sided sign test p = .0039).
  • On a new 50-pair protein-targeted docking benchmark, a fixed single-edit random walk beats the tested agent on docking improvement and composite score, winning 33 of 50 pairs (three-seed composite 0.636 +/- 0.008).
  • A retrieval-free population agent still exceeds REINVENT at the matched 50-call budget (7.39 vs 7.06), so useful blinded optimization is possible - the headline gains are just not measuring it.

The 84 Percent Agent With No LLM

Recent papers report that LLM agents crush classical baselines on molecular optimization at tiny oracle budgets: on the Practical Molecular Optimization (PMO) benchmark at 50 calls, SEISMO reports a summed top-1 AUC of 18.12 against REINVENT's 7.06. Qin, Luo, Xu and Chen ask a ruder question in a new paper (arXiv abstract page): how much of that is optimization, and how much is recall? Fourteen of the 23 PMO task identifiers contain a drug or target name, and some published pipelines feed curated reference molecules straight into the prompt - MT-Mol's prompt includes the top-100 ZINC references per task. So the authors build a retrieval-only baseline: submit the reference compound, up to five seeds, and trivial one-atom RDKit variants of the best molecule on a fixed deterministic schedule, with zero LLM calls. It scores 12.16, which is 84% of the full gpt-4o-mini agent's 14.44. On albuterol, celecoxib, drd2 and four other tasks, reference submission alone saturates the task at roughly 0.98 AUC. The full agent adds only 2.27 points over this deliberately narrow policy - and the authors frame that margin as an upper bound on what the entire LLM loop contributes.

Where the Points Actually Come From

The controlled decomposition is a five-condition ladder. C1 is the standard setup with task name, oracle description and reference compounds; C4 removes identity entirely; C5, the clever one, states only the oracle form - "ECFP4 Tanimoto similarity to a hidden reference, maximize similarity" - with no name and no references. Moving from C1 to C5 costs 9.05 AUC points (14.44 to 5.39), which accounts for 94.4% of the full 9.59-point C1-to-C4 gap; oracle-form information contributes the remaining 0.54. Both identity-blinded conditions land below REINVENT's 7.06. A three-seed replication on eight representative tasks shows C1 beating C4 on all eight (exact sign test p = .0039). The honest counterweight: blinded search is not nothing. C4 still edges the published REINVENT value on albuterol_similarity (0.367 vs 0.358) and ranolazine_mpo (0.358 vs 0.297), so scalar-feedback optimization contributes real, smaller gains once retrieval is closed.

A Random Walk That Outdocks the Agent

The second analysis moves from PMO's ML-based oracles to docking, the regime where several agent papers claim the largest wins. The authors build ProteinLeadOpt-50: 50 starter-target pairs across 10 proteins, scored by a composite of AutoDock Vina improvement, QED and starter similarity at a 30-call budget. The baseline is deliberately dumb - a fixed random policy that adds one of eight small substituents (F, Cl, Br, CF3, OMe, Me, OH, NH2) at an aromatic hydrogen position. In the matched run it posts the strongest docking improvement, success rate and composite score of all four methods, losing only slightly on QED (0.77 vs 0.78-0.80), and it beats the LLM agent head-to-head on 33 of 50 pairs. Across three seeds the walk holds composite 0.636 +/- 0.008 and docking improvement -1.11 +/- 0.04 kcal/mol. The mechanism is embarrassingly simple: adding Cl, Br or CF3 at an aromatic position often improves a Vina score by 0.5-1.5 kcal/mol through volume and lipophilicity terms, while preserving the scaffold and keeping similarity in the 0.85-0.95 band where the penalty never fires. One documented trajectory starts from a JAK2 ligand at -8.3 kcal/mol and gains -1.2 kcal/mol with a single chlorine inside nine calls. The walk wins on 7 of 10 targets; the agent's one target win (F2) comes where the aromatic-substitution trick helps least.

The Control That Keeps the Critique Honest

A pure demolition would overreach, and the authors know it. Their positive control is MolLEO-v4, a population-based genetic algorithm with an LLM batch-mutation operator, structured memory and adaptive planning, run under strict blinding of task names, descriptions and references. It reaches 7.39 at the matched 50-call AUC budget - above REINVENT's 7.06 - and at 100 calls sums 10.24 +/- 0.54 against REINVENT's published 8.72, beating a same-backbone blinded SEISMO implementation on 16 of 23 tasks. The constructive reading: once entity-level retrieval is closed, the real signal shifts to feedback-driven and population-level search. Their two recommendations are cheap enough that there is no excuse to skip them - report C1, C5 and C4 side by side so identity access, oracle-form access and scalar feedback are separated, and report a random-walk floor on any docking-based composite so the agent-floor gap measures search rather than oracle permissiveness.

Where the Argument Bends

Scope discipline matters here, and the paper mostly keeps it. The PMO collapse is measured on one backbone (gpt-4o-mini) inside one ReAct-style LeadOpt-Agent loop; the 2.27-point margin over retrieval-only is an upper bound for that loop, not a law about all agents. The random-walk result covers exactly the Vina-plus-composite-plus-30-call combination, and three seed-level observations produce wide intervals ([0.611, 0.661] for composite). The code audit of public SEISMO, MolLEO and AgentDrug snapshots documents task-information exposure, not intent - and SEISMO's own v3 already reports a generic no-description condition at 8.50, so the field is partly self-correcting. Still, the structural echo with our recent coverage is hard to miss: MCPHallu showed evaluation harnesses manufacturing capability claims, and Goodhart in the Planner showed measured competence collapsing when the metric is gamed. Molecular optimization now has its own instance, with the cleanest numbers yet.

Assumptions and limitations

All benchmark figures in this article are the authors' own measurements as reported in the paper; we did not re-run any oracle, agent or docking job. Results describe the tested combinations - one gpt-4o-mini LeadOpt-Agent loop on PMO at 50 calls, and one Vina-composite oracle at 30 calls on ProteinLeadOpt-50 - not LLM chemistry agents in general. The toy benchmark in the worked example is invented and illustrative; its numbers are chosen for arithmetic clarity and are not measurements.

Worked Example: A Five-Task Toy Benchmark

A tiny invented benchmark makes the mechanic concrete. Five tasks, three with names a pretrained model would recognize. When identity is visible the "agent" recalls those answers; blinded, it falls back to weak local search capped at 0.35 per task. The headline sum hides which part is which - exactly the authors' point.

from statistics import mean

# Toy: an invented 5-task benchmark.
# Three task names are memorizable.
score = {'albuterol': 0.98,
         'celecoxib': 0.98,
         'drd2': 0.98,
         'jnk3': 0.40,
         'qed': 0.10}
mem = {'albuterol', 'drd2',
       'celecoxib'}

def run(see_id):
    out = []
    for t, s in score.items():
        if see_id and t in mem:
            out.append(s)
        else:
            out.append(min(s, 0.35))
    return out

full = sum(run(True))
blind = sum(run(False))
print('full sum ', round(full, 2))
print('blind sum', round(blind, 2))
share = (full - blind) / full
print('identity share:',
      round(share * 100), '%')
print('mean blind task:',
      round(mean(run(False)), 3))

Sources & Provenance

Written by Instinct for GlobalTechDigest. Analysis, framing and toy code are original; all measured results are the cited authors'.

Comments

Kubernetes Liveness Is Not Readiness: Why a Health Check Can Amplify an Outage

KUBERNETES FAILURE RECOVERY A failed health check is an observation. Restarting a container and withholding traffic are different responses, and choosing the wrong one can amplify an outage. By Hossam The important question is what failure should trigger A web application loses access to its shared database. Its processes are still running, but requests that require database access fail. A health endpoint reports the dependency problem. Should Kubernetes restart every application container, or stop directing those requests to it? The endpoint alone cannot choose the right recovery action. Kubernetes separates startup, liveness and readiness probes because they answer different operational questions. Startup protects initialization, liveness can trigger container termination and restart handling, and readiness controls eligibility for traffic from matching Services. A green check is meaningful only within the check and response that were designed. This is a documentation-based de...

Editorial Governance & Pages