代理評測與可觀測性
A²E Unifies 23 Benchmarks and 9 Agent Harnesses, but Each Evaluation Cell Still Contains Only 5 Tasks
The Shanghai AI Laboratory has released A²E, which uses the Agent Task Protocol to decouple benchmarks, agent harnesses, and trajectory scoring. Its 1,035 paired runs show that the harness affects tool use and token costs, but the sample size is too small to support framework rankings.
The Shanghai AI Laboratory team has released A²E (Agent Auditing Engine), an attempt to move agent evaluation away from writing a separate benchmark adapter for every framework and toward a composable protocol. Its Agent Task Protocol standardizes task inputs, tools, sandbox state, and output trajectories. The Task, Monitor, and Evaluation layers are responsible, respectively, for execution; capturing model and tool calls as OpenTelemetry/OpenInference spans; and evaluating planning, tool use, answers, cost, and safety. The project currently lists LangGraph, CrewAI, Google ADK, AutoGen, Smolagents, Agno, LlamaIndex, Claude Agent SDK, and OpenAI Agents SDK, and covers 23 benchmarks, including SWE-bench, Terminal-Bench, and τ-bench.
The paper reports 23×9 experiments—1,035 runs in total—using the same model, decoding parameters, tools, and timeout settings. The 19 non-sandboxed benchmarks also produced 855 complete trajectories. Across the eight harnesses with token usage records, average consumption ranged from 2,063 to 7,319 tokens, a 3.5-fold difference. Success rates also diverged sharply on some multi-turn tasks; on Traject-Bench, for example, they ranged from 0.20 to 1.00. This suggests that a harness’s control loop, tool transformations, and context management are not inconsequential wrapper layers.
However, each framework-by-benchmark cell sampled only five tasks, making 0.20 the smallest possible score increment, and the authors explicitly state that the results are not suitable for ranking frameworks. Listed support also does not mean that every combination has passed end-to-end validation. CrewAI spans lack token counts, while some tool-call and hallucination fields are determined by instrumentation configuration and therefore should not be interpreted as framework behavior. Engineering teams can initially treat A²E as a local, database-backed evaluation scaffold. The next questions are whether its protocol can maintain compatibility across versions, whether repeated sample sizes will be expanded, and whether the repository will add an explicit software license.