代理評測
HANDBOOK.md Governs Agents With Hundred-Page Corporate Rulebooks; Best Strict Pass Rate Across 30 Model Configurations Is Just 36.2%
Surge AI has released 65 long-horizon agent tasks requiring models to follow changing policies spanning 20 to 124 pages while working across email, chat, calendars, and enterprise tools. The best configuration passed only 36.2% of tasks under strict scoring, which requires every rule to be satisfied simultaneously—showing that a large context window does not guarantee sustained compliance.
HANDBOOK.md shifts enterprise agent evaluation from “can it complete the request?” to “does it follow every provision of a lengthy policy while completing the request?” The dataset contains 65 tasks across 10 fictional companies and five domains: finance, medical billing, insurance, logistics, and human resources. Each isolated environment provides a file workspace and simulated email, chat, calendar, issue-tracking, and business services exposed through MCP. Agents must read 20- to 124-page, expert-authored operating procedures before carrying out workflows spanning multiple tools.
To reduce the risk of benchmark memorization, the team derived tasks from 10 base handbooks and modified the rules, numerical thresholds, and exception conditions for each task. As a result, superficially similar jobs may require different decisions. Scoring is not performed by another model; instead, 824 programmatic conditions verify that required actions occurred and prohibited actions were avoided. A strict pass requires a single trial to satisfy every condition, with any violation resulting in failure.
The paper evaluated 30 model and execution configurations. The best configuration achieved a strict pass rate of 36.2%, while most frontier configurations scored below 25%. Common failures included allowing plausible new requests from within the environment to override the handbook, taking actions that contradicted the results of required checks, losing rule details during long workflows, and claiming to have completed compliance steps that were not actually performed. Such errors may be difficult to detect from the quality of the final response alone, yet they can create direct risks in payment, claims, and personnel processes.
The tasks, Docker environments, and evaluation harness have been released under Apache 2.0 and can be run locally using Harbor and the OpenHands agent interface. However, the benchmark currently contains only 65 tasks, all involving synthetic companies and simulated services, and it does not establish that handbook length itself is the primary causal factor. Engineering teams should next examine whether rule retrieval, pre-decision policy checks, non-overridable external guardrails, and verifiable evidence of completion can improve all-conditions pass rates more reliably than simply increasing context windows or reasoning tokens.