ReTrace Compiles Agent Runs into Planner-Free Workflows, Trading Runtime Planning for Rules That Still Need Testing
AGENT SYSTEMS
A workflow that no longer asks a model what to do next can be dramatically faster on simulated tasks. ReTrace's results also show the limits: unrepaired workflows can lose accuracy, tools may still call models, and enterprise completion flags can hide failed actions.
By Ossam Hashim
KEY FINDINGS
- ReTrace converts a written procedure and logged agent runs into an evidence-linked rule document, then executable code or a workflow graph. Code selects the next step at runtime; individual tools may still call models. Planner-free does not mean every deployment is model-free.
- In the paper's telecom-only, solo-mode tau2-bench evaluation, unrepaired ReTrace reaches 75.8% task success versus the agent's 80.6%, while mean runtime falls from 170 seconds to 1.3 seconds. The tools are simulated, so these timings are not a promise for real customer-service infrastructure.
- On eight selected SOP-Bench procedures, unrepaired ReTrace trails few-shot workflow generation, 80.6% versus 83.6%. Repair using labeled training failures raises ReTrace to 87.8%, but that extra feedback makes comparison with unrepaired methods unequal.
- On the proprietary enterprise dataset, reported completion is 93.8% versus the agent's 96.8%, with mean cost per task down 24%. Median time falls, but mean time rises from 153 to 163 seconds. Completion flags are not independently audited task correctness.
Compile the Branch, Not Just the Tool Call
Kaiyuan Hou and colleagues from Columbia and Amazon study a recurring automation problem in ReTrace: Distilling Tool-Using Agents into Planner-Free Workflows. A procedure may run repeatedly on different cases while an agent repeatedly spends model calls deciding which tool to invoke next. ReTrace moves that decision-making into a one-time construction pipeline.
A tool trace alone can show that an order lookup was followed by escalation without stating which condition caused it. ReTrace also reads the agent's explanation accompanying each tool call. An extraction model records a guard, the intended action, the tool, its inputs and the source of each input. Case-specific values become variables. Alternatives mentioned in the explanation can become rules even when no logged run executed them.
An aggregation stage merges these records against the written standard operating procedure into a compact intermediate rule document. Each rule cites supporting trace steps or procedure statements. Unexecuted alternatives are marked unwitnessed. A final synthesis stage generates benchmark Python or the enterprise platform's typed workflow graph. Changing the target platform affects this last stage rather than requiring the entire extraction process to be redesigned.
What Planner-Free Actually Means
The deployed workflow keeps a history of inputs and tool results. It stops when its completion check passes; otherwise it executes the first rule whose guard holds. If no guard applies and the task is incomplete, it stops incomplete rather than asking an LLM to invent the next step.
This is a stronger removal of runtime planning than giving an agent reusable written guidance or a skill library. It is also a narrower claim than eliminating models altogether. A tool used by the workflow can still classify a document with a model. The enterprise cost figures include such calls. On the public benchmarks, the simulated tools happen not to call models, so their workflows have no runtime model calls.
Construction checks establish that code loads and cannot make planning calls. Where offline re-execution is available, candidates are ranked by how many recorded steps they reproduce. Neither property establishes that the recovered guards are correct. The explanation being distilled may be mistaken, incomplete or an unreliable account of why the original agent acted.
The Fast Result Has a Defined Evaluation Boundary
The telecom evaluation uses tau2-bench's 74 training and 40 test inputs in solo mode, with no simulated user. The main unrepaired comparison uses four successful training traces per task family, selected through training-set cross-validation. ReTrace reaches 75.8% success and 1.3 seconds per task versus the original agent's 80.6% and 170 seconds. Its construction requires at least eleven model calls once, while the agent averages 12.1 planning calls on every execution.
Those figures show the advantage of removing repeated planning in a controlled setting. They do not establish identical success, production latency or compatibility with published interactive tau2-bench rankings. The paper's intervals are wide: even repaired ReTrace's 81.2% versus the agent's 80.6% is not enough to claim parity. The repaired result also starts from two-trace workflows, not the four-trace configuration in the main unrepaired row; repair lifts that two-trace starting point from 66.7% through three rounds on training failures.
SOP-Bench is a useful counterexample to an easy headline. The study admits eight of fourteen domains and applies a stricter output-format and exact-field score, so these numbers are not directly comparable to published SOP-Bench scores. Unrepaired ReTrace scores 80.6%, below few-shot generation's 83.6% and the agent's 85.5%. The repaired workflow reaches 87.8% at 0.40 seconds per task, compared with the agent's 46.6 seconds. However, repair sees failed training inputs' labels. It is additional supervision, not evidence that the unrepaired extraction pipeline alone beats the agent.
Reasoning Helps, Except Where It Does Not
Removing reasoning from the same extraction pipeline costs 6.2 points on the telecom benchmark at four traces and 8.8 points of enterprise completion. But on SOP-Bench, removing reasoning improves the score from 80.6% to 82.3%. Short, lookup-like procedures leave less for explanations to contribute. Much of the enterprise gain is concentrated in four procedures where workflows built without reasoning fail every input at a repeated code step.
More raw traces are not automatically better either. On the telecom benchmark, feeding full traces directly to a workflow generator eventually hurts build-adjusted validation success, with build rate dropping to 31% at 32 traces from 100% at sixteen. The compact rule representation addresses a construction-input problem, not a general guarantee that compression preserves every branch. Random trace selection still matters: a single SOP-Bench procedure can vary by as much as fifty points across trace samples.
Read Completion Alongside Tool Outcomes
The enterprise study covers 79 proprietary procedures, with each method running on a somewhat different usable subset. ReTrace reports 93.8% completion against the agent's 96.8%; the paired difference is -3.0 points with a 95% interval of -8.0 to 1.0. That is not proof of equal performance. Mean task cost falls from $0.127 to $0.096, approximately 24%, while median time drops from 68 to 37 seconds and mean time rises from 153 to 163 seconds.
The mean-time reversal has an important explanation in the paper. Twenty procedures use web automation, where successful calls are much slower than quickly failed ones. ReTrace's web-tool calls succeed 86.6% of the time versus 27.3% for the agent. Some agent runs with no successful tool call still receive a complete flag from the platform. A faster flagged-complete run can therefore be a failed action rather than better automation. This makes human outcome auditing a missing measurement, not a detail that the aggregate completion figure settles.
The practical lesson is to compile only after defining the decision rule and the success test. A procedure or tool change may require rebuilding the workflow; construction-time evidence can support an incorrect branch just as easily as a useful one. As One Model, Eight Agent Architectures illustrated, the procedure and execution contract can matter more than the architecture label. Planner-free code makes behavior repeatable, including its errors.
Assumptions and limitations
This article analyzes one primary paper; all reported experimental results belong to its authors. The public evaluations use new inputs to the same procedures, not transfer to unseen procedures. Enterprise completion is platform-reported, not independently audited correctness. Construction and repair add upfront model calls, runtime tools may still call models, and reasoning traces do not guarantee sound recovered conditions. The toy decision list uses invented rules and demonstrates no benchmark result.
Worked Example: Ordered Guards Need an Incomplete State
This invented decision list illustrates the runtime structure, not generated ReTrace code or a tested business rule. The missing-input case stays incomplete rather than silently inventing a decision.
from collections import namedtuple
Case = namedtuple('Case',
'verified urgent')
def route(case):
if case.verified is None:
return 'incomplete'
if not case.verified:
return 'review'
if case.urgent:
return 'escalate'
return 'complete'
for c in [Case(None,True),
Case(False,False),
Case(True,True)]:
print(c, route(c))
Sources & Provenance
- Hou and colleagues, "ReTrace: Distilling Tool-Using Agents into Planner-Free Workflows" - primary paper (arXiv 2611.14492).
- Sections 3-5, Tables 1-2 and limitations in Section 7 support the workflow mechanism, evaluation splits, repair comparisons, runtime accounting and enterprise measurement caveats.
- Related GlobalTechDigest analysis: One Model, Eight Agent Architectures.
Written by Instinct for GlobalTechDigest. Analysis and toy code are original; measured results belong to the cited authors.
Comments
Post a Comment