AI 代理與評測
Bench2Robust Injects Nine Types of Tool Failures; Performance Regresses in 69 of 70 Agent Evaluations
An Amazon research team transformed existing tool-use benchmarks into fault environments in which agents can be instructed to retry, switch tools, or stop. After structured recovery information was added to Qwen3-4B, its pass rate on disrupted retail tasks increased by as much as 16.8 percentage points, but the system has yet to truly validate correct abstention on unsolvable tasks.
In benchmarks, tool-using agents typically interact with APIs that always succeed, return correctly formatted responses, and provide fresh data. Real-world services, however, can time out, impose rate limits, change schemas, or even return stale or incorrect data that appears normal. An Amazon team introduced [Bench2Robust](https://arxiv.org/abs/2608.11977), which inserts a fault layer between the agent and its tools and explicitly divides each episode into three solvability classes: S1 preserves the original path, allowing recovery through retries; S2 persistently blocks the primary path, requiring the use of an equivalent tool; and S3 blocks all viable paths, making the ideal behavior to stop and escalate.
The framework returns a normal result with 60% probability, with the remaining probability distributed across nine failure types, including timeouts, rate limiting, validation failures, malformed responses, schema drift, and failures without obvious error signals, such as partial data, stale values, and factual errors. The researchers integrated it with multi-turn tasks from [τ²-bench](https://github.com/sierra-research/tau2-bench) and the Berkeley Function-Calling Leaderboard. Across 70 comparisons covering seven models, four model families, and ten task slices, 69 showed performance regressions after fault injection, with the largest decline reaching 46.7 percentage points. Even Qwen3-235B dropped by 15.8 points on τ²-retail.
The team also equipped Qwen3-4B-Thinking-2507 with Bayesian Tool Memory (BTM), placing fallback-tool mappings, validation constraints, and Beta-posterior recovery rates estimated from rollouts into the inference context. Across 402 held-out retail tasks, adding BTM to the model without retraining increased the pass rate under the no-alternative-tool configuration from 20.1% to 36.9%. When combined with DAPO reinforcement learning, pass rates under the two fault configurations reached 40.8% and 45.5%. However, ablation results showed that most of the gains came from the manually configured fallback graph and rules rather than from the posterior values themselves.
The engineering takeaway is that an agent runtime should explicitly specify interchangeable tools, retryable errors, and validation requirements before irreversible operations; it cannot rely solely on a generic “retry on error” prompt. One caveat remains: the paper does not release the Bench2Robust implementation, and S3 unsolvable tasks receive no positive score. The results therefore support only the conclusion that retrying and switching tools improve performance—not that the models have learned to abstain reliably.