AI 代理與評測
Thinkingbox Validates 507 Enterprise Workflows Through Backend State; Best Agent Achieves Just 25.25% Across 20 Consecutive Perfect Runs
Microsoft has open-sourced Thinkingbox, which uses isolated MCP tool environments and executable assertions to verify whether agents actually complete their work, rather than merely checking their responses or whether tool calls are valid. The best system achieves a 65.36% single-run success rate, but that falls to just 25.25% when the same task must succeed 20 consecutive times.
Microsoft Research has released [Thinkingbox](https://github.com/microsoft/thinkingbox) and Thinkingbox-bench, shifting the focus of agent evaluation from whether an agent produces a plausible response to whether databases, files, and business systems ultimately reach the correct state. Through a persistent MCP Session Proxy, the framework launches an isolated tool server for each conversation, loads its initial state, and captures all side effects after the agent finishes. Tests can reject omitted, incorrect, or extraneous actions.
The [paper](https://arxiv.org/abs/2608.19741) introduces 507 policy-constrained workflows spanning retail, hospitality, auto insurance, digital banking internal IT, and IT and HR support at a consulting firm. The tasks require agents to ask follow-up questions, comply with rules, and coordinate dependent tools across multi-turn conversations. Of these, 477 are evaluated primarily by their terminal backend state, while another 30 use response rubrics. The authors found that many failed trajectories still ended normally and produced valid state changes, showing that monitoring only formatting correctness can miss substantive errors.
Across multiple proprietary and open-weight models, the best system achieved a pass@1 of 65.36%, but its pass^20—the probability of succeeding on the same task in all 20 runs—was only 25.25%. This gap is especially important for enterprise agents that issue refunds, modify accounts, or send emails: occasional success does not imply deployment-grade reliability. The MIT-licensed framework can also generate offline trajectories or integrate with reinforcement learning loops, and it supports OpenAI-compatible endpoints, Azure OpenAI, and Anthropic.
One limitation is that the tasks were reconstructed from undisclosed sources, so they should not be treated as representative of real-world enterprise traffic. Each task also has only one golden terminal state, excluding cases in which multiple approaches could be equally valid. The user simulator is fixed to GPT-5.4-mini, which also grades the 30 language-evaluated tasks, and the current implementation has been formally tested only on Linux. The next step is therefore to see whether independent teams can integrate their own MCP systems, add multi-party workflows or tasks with non-unique terminal states, and reproduce the observed reliability degradation.