代理工具
QuoteBench: One Extra Layer of Shell Parsing Can Slash Success Rates for Identical Agent Commands by 55.4 to 73.2 Points
QuoteBench holds model output constant and changes only how Bash commands are transported and reparsed, directly measuring failures caused by the agent interface. Its results show that near-perfect schema adherence can still conceal widespread quoting and escaping errors.
Bash commands generated by coding agents are rarely sent directly from the model to the shell. They typically pass through a JSON tool call, string interpolation, a wrapper, or another parsing layer first. QuoteBench isolates this often-overlooked “command transport path” as an evaluation variable. Its 56 single-command tasks are drawn from 14 categories of real-world incidents. Rather than comparing command text, each task verifies the resulting file and process state inside an isolated container, allowing different but correct quoting approaches to pass.
The researchers first saved the responses generated by each model under the original contract, then passed the exact same responses through an additional double-quoted parsing layer that was intentionally not escaped correctly. Success rates fell by 55.4 to 73.2 percentage points across all eight contemporaneous model configurations, demonstrating that the losses were not caused by model resampling. When the prompt disclosed this boundary in advance, six configurations recovered 30.4 to 60.7 points; Qwen3.5-27B did not recover, while Gemini 3.1 Flash-Lite regressed slightly. GPT-5.6-sol is a particularly clear example of how averages can mislead: its apparent matched-path gap was only −3.6 points, but this concealed a −64.3-point transport penalty offset by a +60.7-point generation gain.
A separate native-tool evaluation also found that schema adherence remained between 98% and 100%, while actual correctness ranged from just 86% to 100%. Valid JSON and the correct tool name do not guarantee that the command received by the shell still preserves its intended meaning. From an engineering perspective, versioned testing should cover the model, generation contract, serialization method, parsing layers, GNU/BSD userland, and final-state verifier as a single system. The authors also note that the fixes are not novel: correctly escaping interpolation points or passing commands through temporary scripts can eliminate the effect. The main limitations are that the core benchmark contains only 56 tasks, its primary causal findings are based on eight configurations, and it does not yet cover other shells, long command chains, or multi-turn self-repair.