編碼代理評測
26 Testing Prompts Fail to Reliably Improve Coding Agents; More Tests Do Not Mean Higher Accuracy
A Zstandard implementation experiment compared 26 conditions, including TDD, fuzzing, property-based testing, and formal verification. None consistently outperformed the default group, which received no additional instructions. Agents often correctly identified high-risk areas, yet generated invalid inputs, proved irrelevant properties, or encoded incorrect outputs into tests.
Engineer Dan Luu published a set of coding-agent testing experiments in which agents were asked to implement Zstandard in Rust. The prompts added 26 different conditions, including TDD, QuickCheck, Proptest, fuzzing, differential testing, mutation testing, Lean 4, Kani, TLA+, and Verus. A default group received no instructions about verification methods, and medium and xhigh reasoning effort were also compared. Each major condition included many repeated runs to avoid mistaking a one-off success for an effect of the method.
The result is not that “formal methods do not work,” but that agents generally failed to use the specified tools effectively. In 63 of 160 QuickCheck runs, the agent checked only one property. Fuzzing mostly supplied completely random bytes, repeatedly exercising the same rejection path for invalid inputs. Agents generated structured random inputs in only 10 of 160 runs; half of those runs found bugs, suggesting that the bottleneck was closer to the test-data generation strategy than to the tool itself. When using methods such as Lean and Verus, agents often proved properties unrelated to the implementation’s actual risks and ultimately still relied on ordinary Rust unit tests.
TDD led agents to write roughly twice as many tests but did not improve correctness. Explicitly instructing them to use Rust’s built-in testing framework produced a similar result. Common failure modes included ignoring the most fragile behavior, using the program’s current output to construct the supposed “correct answer,” and selecting palindromic inputs that accidentally allowed bit-order reversal bugs to pass. The default group performed above average, but the differences between groups were too small to support a precise ranking.
The implication for agent workflows is that `test more`, `use TDD`, or attaching a tutorial-style skill cannot substitute for test design. A more reliable approach is for a human or an independent specification to define the oracle, valid-input generators, coverage targets, and fault injection first, then have the agent expand the test cases. The research still covers only Rust, a compression format, and a small number of other tasks, and it is not a peer-reviewed paper. The next step is to conduct preregistered statistical comparisons across models, languages, and repository-level tasks.