代理評測／軟體工程
QuoteBench Replays Fixed Model Outputs Across Command Paths, Finding a Single Additional Shell Parse Can Cut Success Rates by More Than 70 Points
QuoteBench measures model generation separately from subsequent serialization, wrapping, and shell parsing, revealing that similar aggregate scores can conceal entirely different forms of transport corruption. Fixed responses from eight configurations fell by 55.4 to 73.2 percentage points after passing through an additional parser, while some models merely learned to compensate after being told about the boundary.
Bash submitted by a coding agent may appear correct yet still change meaning when a tool interface serializes the string, inserts it into a wrapper, and passes it to a second shell. Quotes, `$()`, backticks, globs, newlines, and filenames containing spaces can all expand prematurely at the wrong layer. If an evaluation checks only whether the task was ultimately completed, it cannot distinguish between a model that originally wrote an incorrect command and a harness that corrupted correct output after generation.
QuoteBench measures this boundary using 56 single-shot Bash tasks spanning 14 categories derived from real-world failure mechanisms. Each task is verified precisely through file bytes, argv, JSON, Git state, or directory state. The experiment crosses two axes: whether the model is told that another parsing layer follows, and whether the same stored response is passed directly to `bash -c` or first routed through an unescaped nested wrapper. Replaying fixed responses makes it possible to calculate separately both “transport-induced corruption” and “compensation,” in which a model rewrites its commands after the contract discloses the boundary.
When the original responses from eight contemporaneous model configurations were subjected to the additional parse, success rates fell by 55.4 to 73.2 percentage points. After the boundary was disclosed, six configurations recovered 30.4 to 60.7 points, while the other two showed no improvement. GPT-5.6-sol’s paired aggregate scores differed by only −3.6 points, but that result actually comprised −64.3 points of transport corruption and +60.7 points of compensation. The authors also replayed the responses over local SSH; for seven configurations, the magnitude of corruption matched the synthetic wrapper results exactly. Correct escaping or switching to temporary scripts eliminated every nested-path-specific failure across all 448 configuration–task combinations.
The engineering conclusion is not that models should memorize more quoting rules, but that action representations and parsing boundaries must be part of the system contract. When argv or typed operations can be transmitted, systems should not repeatedly concatenate shell strings. Evaluations should also disclose the generation format, execution path, and final-state verifier. One limitation is that the core dataset contains only 56 single-shot POSIX/Bash tasks. It does not cover PowerShell, network and authentication errors, interactive terminals, or multi-turn repair, so the results cannot yet be used to estimate failure rates across complete coding-agent workloads.