代理與程式設計評測
τ^τ-Bench Puts “Building Agents” Itself Under Evaluation; Best Coding Agent Reaches Just 30% of Expert Reference Performance
Sierra’s new benchmark no longer tests only finished agents. Instead, it requires coding agents to build deployable customer-service systems from fragmented business materials, customer interviews, and existing code. The best combination passes just 23.9% of hidden simulated traffic, exposing shortcomings in requirements discovery, architecture exploration, and validation.
Sierra Research has released τ^τ-Bench (Hyper-τ), expanding the evaluation scope from “Can an agent serve users?” to “Can a coding agent build another agent?” Each task provides materials that a real company might leave behind, including operating manuals, customer-service records, spreadsheets, screenshots, audio, and flowcharts. It also includes a simulated customer who knows undocumented requirements. In an isolated environment, the development agent must understand the materials, interview the customer, take over an empty or existing codebase, and integrate REST APIs that may contain hidden defects.
The submission is not a written answer but a complete, executable customer-service agent. The evaluator deploys it against undisclosed simulated user traffic and scores it based on the final database state, the information communicated to customers, and the average model budget per conversation. The 53 released tasks span aviation, retail, telecommunications, and banking. The aviation and retail data were additionally rewritten with different brands, figures, and policies to reduce the likelihood that models could recall existing τ-bench tasks. The codebase supports Codex, Claude Code, OpenCode, and Prime Agent, while fixing model routing, container images, and task contracts so that the “model” and the “agent execution framework” can be compared separately.
In the initial results, Claude Opus 5 paired with Claude Code scored 23.9%, while GPT-5.6-sol paired with Codex scored 22.0%. Expert-written reference agents scored 82.2%. The paper’s analysis finds that coding agents often substitute keyword searches for thorough reading, rarely ask customers follow-up questions, and tend to patch the first architecture they can get running before validating it with self-written tests that share the same blind spots. This more closely resembles real-world agent project delivery than straightforward code-repair benchmarks. However, the 82.2% figure represents expert reference implementations rather than general human capability, and both the simulated customer and model-based judges may introduce bias. The next key question is whether different teams can reproduce the rankings under identical routing, cost, and sealed-testing conditions.