Header Ad Placement

MCPHallu: Structural Triggers, Claim-Graph Scoring, and the Unreachable Goals MCP Agents Refuse to Recognize

AI AGENTS

Most agent benchmarks score whether the task got done and diagnose failures afterward. MCPHallu builds the failure into the task - four hallucination subtypes with distinct structural triggers - and finds that frontier agents' weakest skill is admitting a goal cannot be reached.

By Ossam Hashim

KEY FINDINGS

  • MCPHallu evaluates LLM agents operating through the Model Context Protocol on 358 tasks across five domains, running against 36 production MCP servers in isolated containers rather than synthetic mocks, with 14 frontier agents from eight vendors measured over 5,012 trajectories.
  • Every task is constructed around a structural trigger that makes one of four hallucination subtypes the dominant failure risk: Branch Collapse (mutually exclusive branches gated on a live tool output), Unreachable Goal (no available tool can satisfy the goal), Tool Misuse (overlapping schemas with distinct semantics, plus curated confusion tools), and Context Forgetting (a value retrieved early must survive same-domain distractors).
  • Scoring is mostly deterministic: executable verification and claim-graph dependency checks dominate every subtype, with the LLM judge contributing only 20 to 30 percent of each score. A per-task claim graph encodes the correct trajectory's dependency structure, so plan-structure violations are scored separately from wrong final answers.
  • Current MCP agents remain vulnerable: the average overall score is 0.642 on a 0-to-1 scale, the best model reaches 0.728, and expert humans score 0.88 on a stratified 40-task sample. Unreachable Goal is the hardest subtype for 12 of 14 models - average score 0.458, only 23.2 percent of runs passing - because agents keep issuing tool calls or claim success instead of recognizing infeasibility.
  • Subtype rankings barely transfer: a model's rank shifts by 6.9 of 13 positions on average across subtypes, and Branch Collapse is uncorrelated with Unreachable Goal across models (r = -0.12), which argues against ranking agents by a single aggregate score.

A Benchmark Built Around the Failure

Tool-use benchmarks usually measure task completion and, when they diagnose at all, diagnose after the fact. A team from Peking University, Meituan and HKUST (Jiang, Guo, Bai and colleagues) takes the opposite route: each MCPHallu task is a tuple of instruction, tool set, target subtype, difficulty, ground truth and an ordered claim graph, designed so that a specific hallucination subtype is the dominant risk while the correct behavior stays verifiable. The trigger shifts the task's failure distribution toward its target, which makes subtype-level analysis meaningful at the population level even when individual long trajectories mix several errors. The environment is deliberately real. Thirty-six production MCP servers run in isolated Docker containers across general search, analytics, productivity, finance and coding; ground truth comes from dynamic scripts executed against live servers for stateless tasks and from post-execution state assertions for stateful ones. The claim graph does double duty: at construction time it encodes the structural trigger, and at evaluation time it checks whether the agent's tool sequence honored the required dependency order - a median of 7 calls for Branch Collapse and Context Forgetting tasks, 2 for Tool Misuse.

Four Ways an Agent Hallucinates

The taxonomy splits along the agent loop. On the reasoning axis, Branch Collapse violates a data-conditioned constraint: the task branches on a live tool output and the agent follows the wrong branch or conflates outputs across branch instances. Unreachable Goal violates a capability constraint: no available tool can satisfy the request, yet the agent continues acting or presents an infeasible end state as achieved. On the execution axis, Tool Misuse exploits overlapping tool schemas with distinct semantics, baiting the agent with curated confusion tools and catching fabricated, swapped or invalid arguments. On the memory axis, Context Forgetting retrieves a key value early, then buries it under same-domain distractors until the final answer uses a stale, approximate or substituted value. Each subtype carries a distinct detection signal in the trajectory - wrong-branch calls, continued action instead of an infeasibility report, calls to confusion tools, a final answer differing from the retrieved value - which is what makes the four-way split scorable rather than rhetorical.

Scoring That Barely Asks a Judge

Every task is scored along up to three dimensions, and the deterministic ones carry the weight. For Branch Collapse and Context Forgetting, executable outcome verification counts for 60 percent, dependency satisfaction for 20, and the rubric judge for 20. For Unreachable Goal and Tool Misuse, the structural component counts for 70 and the judge for 30. The Unreachable Goal score has a notably humane shape: an agent that declares infeasibility within a probing window of two tool calls earns full credit, the score decays linearly if the declaration comes later, and a run that never declares scores zero - so checking feasibility before giving up is rewarded, while indefinite flailing is not. Indiscriminate refusal does not game it either: on Tool Misuse tasks, trajectories that invoke neither the correct tool nor a confusion tool pass only 8.0 percent of the time.

What Fourteen Agents Revealed

The headline numbers are sobering. The best model averages 0.728, the field averages 0.642 with a 47.8 percent pass rate, and bootstrap confidence intervals are tight enough (half-width about 0.03 to 0.04) that the gaps mean something. Unreachable Goal stands out: hardest for 12 of 14 models, with a 0.458 average and a 23.2 percent pass rate - agents would rather keep calling tools or fabricate completion than say the goal cannot be met, a failure mode the authors call false completion. Two more trajectory-level patterns get names: tool indiscipline (reaching for confusion tools) and state non-commitment (failing to carry retrieved values forward). The most decision-relevant finding may be the rank instability: subtype leaderboards shuffle by nearly 7 of 13 positions on average, and the two reasoning subtypes are essentially uncorrelated across models, so a single aggregate ranking conceals exactly the differences a deployment decision depends on.

Assumptions and limitations

All results are the authors' own, measured on 14 agents from eight vendors over 5,012 trajectories against 36 production MCP servers; scores carry bootstrap intervals of roughly 0.03 to 0.04 half-width, and the human 0.88 reference comes from a stratified 40-task sample, not the full benchmark. The four subtypes are representative rather than exhaustive, subtypes can co-occur within one trajectory, and the LLM judge still contributes 20 to 30 percent of each subtype score. The release instantiates tool selection for Tool Misuse; invocation-level variants are described as future extension. The worked example is an invented toy model of the paper's dependency check and Unreachable Goal scoring rule, not the benchmark's implementation.

Worked Example: Dependency Order and the Infeasibility Clock

Two of MCPHallu's scoring mechanisms compress into a few lines. First, a claim-graph dependency check: each claim lists its upstream claims, and a trajectory violates the structure when a call appears before one of its dependencies. Second, the Unreachable Goal decay: full credit for declaring infeasibility within the probing window, linear decay afterward, zero for never declaring. The toy trace below calls merge before fetch_a - one structural violation - then shows how the infeasibility clock prices late admissions.

import re

DEPS = {
    'search': [],
    'fetch_a': ['search'],
    'fetch_b': ['search'],
    'merge': ['fetch_a', 'fetch_b'],
}

TRACE = """
call(search) -> ok
call(fetch_b) -> ok
call(merge) -> ok
"""

pat = r'call\((\w+)\)'
steps = re.findall(pat, TRACE)

def check(deps, steps):
    seen = []
    for s in steps:
        need = deps.get(s, [])
        miss = [d for d in need
                if d not in seen]
        if miss:
            return (s, miss)
        seen.append(s)
    return None

bad = check(DEPS, steps)
print('order violation:', bad)

def ug_score(t_stop, k=2, t_max=20):
    if t_stop is None:
        return 0.0
    if t_stop <= k:
        return 1.0
    return max(0.0, 1 - (t_stop - k)
               / (t_max - k))

for t in [1, 2, 5, 11, 20, None]:
    print(t, round(ug_score(t), 2))

Sources & Provenance

  • MCPHallu: Benchmarking Reasoning, Execution, and Memory Hallucinations in MCP Agents (Jiang, Guo, Bai and colleagues, 5 Nov 2026) - arXiv abstract
  • Full paper (PDF), including the subtype taxonomy, claim-graph scoring and per-model results - arXiv full text
  • Benchmark code and task release - benchmark repository

All performance numbers are attributed to the preprint as published. Sources accessed 7 November 2026. This analysis is independent editorial content for GlobalTechDigest.

Comments

Kubernetes Liveness Is Not Readiness: Why a Health Check Can Amplify an Outage

KUBERNETES FAILURE RECOVERY A failed health check is an observation. Restarting a container and withholding traffic are different responses, and choosing the wrong one can amplify an outage. By Hossam The important question is what failure should trigger A web application loses access to its shared database. Its processes are still running, but requests that require database access fail. A health endpoint reports the dependency problem. Should Kubernetes restart every application container, or stop directing those requests to it? The endpoint alone cannot choose the right recovery action. Kubernetes separates startup, liveness and readiness probes because they answer different operational questions. Startup protects initialization, liveness can trigger container termination and restart handling, and readiness controls eligibility for traffic from matching Services. A green check is meaningful only within the check and response that were designed. This is a documentation-based de...

Editorial Governance & Pages