代理評測與可靠性
TRACE Rewrites Agent Skills from Successful and Failed Trajectories, Raising the Consistent Pass Rate on Hidden Tests to 70%
Xiaomi’s team introduced TRACE, which iteratively updates a retrievable Skill Bank by contrasting successful and failed trajectories without modifying the underlying model weights. It increased the share of CAR-bench hidden-set tasks passed in all three runs, but also raised average token usage, cost, and latency.
TRACE (Trajectory-Contrastive Evolution), unveiled on August 24, reframes the agent reliability problem from “retraining the model” to “iterating on behavioral knowledge.” The system first has an agent perform multi-turn tasks in CAR-bench, then groups trajectories by the skills actually invoked. By comparing successful and failed cases, it rewrites tool-use rules, clarification strategies, and capability boundaries to create a retrievable Skill Bank. At deployment time, the Actor reselects skills based on the conversation and environment state at each turn, instead of loading the entire rulebook into the opening context.
CAR-bench simulates an in-vehicle assistant and includes 58 interrelated tools and 19 domain policies. It specifically tests whether an agent falsely claims success when a required tool is unavailable and whether it queries the environment or asks for clarification when faced with ambiguous requests. In addition to Pass@3—which counts a task as successful if at least one of three attempts passes—the benchmark uses Pass^3, which requires all three attempts on the same task to pass. The latter more closely reflects production systems’ reliability requirements.
In experiments where the authors combined the training set with the public test set, GPT-5.5’s Pass^3 rose from 59.9% to 94.5%, while GLM-5.2 improved from 62.8% to 84.8%. The Skill Bank was evolved using only GPT-5.5 trajectories, yet still delivered gains when applied to GLM-5.2, suggesting that textual behavioral rules may transfer across models. The largest improvements appeared in disambiguation tasks, where the two models gained 55.3 and 35.7 percentage points, respectively.
The official hidden set—30 tasks run three times each—provides a more informative result. Using the same GPT-5.6-Sol backbone, TRACE raised Pass^3 from 50% to 70%. However, average token usage per trial increased from 82,179 to 141,684, estimated cost rose from $0.17 to $0.27, and median latency increased by 22%. Moreover, because the public dataset was used both to evolve and evaluate the skills, the 94.5% result should not be treated as pure generalization to unseen data; the hidden-set sample was also small. The next step is to validate transferability across more tool domains and release the skill-evolution code and complete trajectories, allowing researchers to determine whether the gains reflect manual or automated overfitting to CAR-bench’s rules.