One Model, Eight Agent Architectures: Coordination Costs Up to 4.7x the Tokens and Rarely Buys Accuracy
AGENT SYSTEMS
A controlled study holds the model, tools and tasks fixed and varies only the agent architecture - and finds the SOP, not the design, decides success, while every coordinated arm pays 1.4-4.7x tokens and 2.6-5.8x latency for gains that are mostly noise.
By Ossam Hashim
KEY FINDINGS
- Amazon and UW researchers ran eight architecture arms (function-calling single agent, skill-augmented agent at two skill-quality levels, two orchestrator implementations, three swarm variants) over 13 enterprise SOPs - 2,124 tasks per arm - with one pinned model (claude-opus-4-8) and deterministic mock tools, so differences are architecture, not environment.
- No architecture wins in aggregate: per-SOP success spans under 10% to near 100% within every arm, between-SOP standard errors of 6.3-7.8 points overlap across all arms, and no arm separates from the baseline by a per-SOP sign test.
- The single exception: a redesigned orchestrator (full SOP to every worker, no placeholder outputs, one re-dispatch round, output validation) is the only arm to significantly beat the baseline - 75.6% vs 67.4% macro success, odds ratio 1.72, p=0.007 under a Knapp-Hartung random-effects test.
- Skill quality matters more than skills: high-quality skills lift macro success to 73.1% while a degraded corpus falls to 59.8% - below the plain single agent - and on one SOP silently files near-correct answers under a concocted key.
- Cost is the real separator: coordinated arms spend 1.4-4.7x the baseline's tokens and 2.6-5.8x its latency; CSE-high pays 3.0x tokens for +5.7 points, and swarm-full pays 5.8x latency for +3.1.
Same Model, Same Tools, Eight Architectures
Picking an agent architecture is usually argued from anecdotes. Nandi, Arora and colleagues instead built the controlled version (arXiv abstract page): SOP-Bench, 13 enterprise standard operating procedures with deterministic mock-tool dispatch, so tool outcomes are fixed and only the architecture varies. The eight arms cover the design space a practitioner actually faces: a function-calling single agent with the full SOP and all tools; Constrained Skill Execution, which selects reusable skills and fences the executor to each skill's declared tools, in matched high- and low-quality corpora built with their skill-definition rubric; centralized orchestration in two implementations; and a triage-guided swarm of tool-scoped specialists at three blueprint levels. Everything runs on one pinned model revision, and they verified the baseline moves at most 2.3 points across three Opus versions before fixing it.
The SOP Is the Unit
The headline is how little the architecture explains. Within every arm, per-SOP success runs from single digits (content flagging, video annotation) to near-ceiling (referral abuse, email intent); SOP identity swamps design, the between-SOP standard errors overlap across all arms, and the macro ordering cannot be asserted as a general ranking. What does separate: the baseline beats the first orchestrator implementation on 9 of 13 SOPs, and v1's failures trace to a decomposition gap - workers see only their slice, nobody resolves a missing input, and the pipeline abstains with placeholder decisions (patient intake at 13.6%). The fix is not "more multi-agent" but "make workers commit": v2 hands every worker the full SOP, forbids placeholders, adds one re-dispatch round, and recovers exactly those SOPs (patient intake 13.6 to 98.5, email intent 75.8 to 96.2). Skills tell the same story from another angle: high-quality skills beat low-quality ones by 13.3 macro points, but the gap concentrates in two long, branching procedures, and the degraded corpus drops below the no-skills baseline. As in Scout Before You Route's lesson on routing overhead, structure is a cost you pay upfront; What LLM Agents Actually Learn showed the same SOP-dependence from the benchmark side.
Cost Is What Separates Them
While accuracy differences are mostly small and insignificant, cost differences are large and robust. Every coordinated arm spends 1.4-4.7x the baseline's tokens and 2.6-5.8x its wall-clock latency. In pooled terms: the single agent averages 24.9k tokens and 24.0 seconds per task; orchestration-v1 pays 1.4x tokens for lower accuracy; CSE-low pays 2.4x for lower accuracy; swarm-full pays 2.7x tokens and 5.8x latency for +3.1 points; CSE-high pays 3.0x tokens and 2.8x latency for +5.7. The authors frame the practitioner rule as a break-even: a candidate architecture replaces the default only when its marginal success gain justifies its total architecture cost - and on most of these 13 SOPs, nothing clears that bar.
Where the Argument Bends
The authors are careful about scope, and the care is warranted. All runs use one model family, so nothing here says whether the ranking holds for other models. The single significant win comes from a redesign that changed several things at once - full-SOP context, no placeholders, re-dispatch, validation - so the study cannot say which change carried it. The skill-quality gap, though 13.3 points, fails a sign test (7 wins, 4 losses, 2 ties) and leans on two SOPs. Mock tools make the comparison clean but remove real-service friction, and 13 SOPs is a small panel for between-SOP inference, which is exactly why the standard errors overlap. Read the study as a discipline rather than a verdict: fix the model, treat the SOP as the unit, and make every architectural upgrade pay for its tokens.
Assumptions and limitations
All results are the authors' own, from their reported runs on one pinned model over 13 SOPs with deterministic mock tools; we re-ran nothing. Findings do not generalize across model families (the authors' own caveat), most accuracy differences are not statistically significant, and the single significant result comes from a multi-change redesign. The worked example uses the paper's pooled multiples with an invented task budget for arithmetic illustration only.
Worked Example: The Break-Even as Arithmetic
The paper's decision rule is arithmetic. Using their pooled token multiples and macro success rates on an invented 100-task budget, cost per solved task separates the arms more sharply than accuracy does.
from statistics import mean
# Toy break-even (paper's pooled
# numbers; invented 100-task run).
arms = [
('FC baseline', 0.674, 1.0),
('orch-v1', 0.582, 1.4),
('CSE-low', 0.598, 2.4),
('swarm-full', 0.705, 2.7),
('CSE-high', 0.731, 3.0),
]
print('arm solved cost/solve')
for name, tsr, mult in arms:
solved = round(100 * tsr)
cps = mult * 24.9 / solved
print(name, solved,
round(cps, 2))
Sources & Provenance
- Nandi, Arora et al., "Agent Architecture Cost-Success Trade-offs: A Controlled Evaluation across Enterprise-Style SOPs" - arXiv abstract page (2611.12953)
- The same paper - full text via the abstract page (setup, Tables 1-3, per-SOP results, appendices D-E)
- Related GlobalTechDigest coverage: Scout Before You Route and What LLM Agents Actually Learn
Written by Instinct for GlobalTechDigest. Analysis, framing and toy code are original; all measured results are the cited authors'.
Comments
Post a Comment