AI 安全與評測
Only 10 of 30 LLM Vulnerability Validation Artifacts Passed Patched-Version Falsification After Reruns
A preregistered reproducibility audit found that a vulnerability PoC that executes and triggers a crash does not necessarily reproduce the specified CVE. In 20 of the 30 evaluable cases, the same signal still appeared in the patched version, showing that agentic security benchmarks need stricter oracles than strings and exit codes.
New research divides LLM- and agent-generated vulnerability validation artifacts into four levels: publicly available, executable, producing a candidate signal, and semantically confirmed to reproduce the specified vulnerability. The researchers first built a consensus corpus from 104 papers published between 2023 and 2026, only 59 of which had publicly accessible primary artifacts. They then selected 18 papers and reran their complete workflows. Only 10 succeeded in clean environments, rising to just 11 after environment repairs such as installing existing dependencies.
More notably, the researchers conducted a case-by-case audit of a benchmark containing 102 CVE cases. In 58 cases, the CVE identifier embedded in the execution script differed from the target claimed by the directory. For cases that produced signals, the researchers looked beyond crashes, status codes, and success markers, adding three checks: whether the signal matched the CVE’s actual postcondition, whether benign input avoided triggering it, and whether the same PoC failed against a patched version.
The results showed that among the 30 cases for which patched-version testing was possible and yielded a determination, the signal disappeared in only 10; the other 20 still triggered in the same way. Of the 19 cases with matched negative controls, seven produced false positives. When the conditions were combined into the strictest E1 evidence standard, only two signal-producing cases were fully confirmed. The paper illustrates the problem with CVE-2020-1967: the test passes a null value directly to `SSL_check_chain`, causing even OpenSSL 1.1.1g—which the NVD lists as patched—to crash. It therefore does not reproduce the actual vulnerability path involving a TLS 1.3 extension field.
For developers of security-agent evaluations, the engineering priority is to encode “triggers on the vulnerable version, does not trigger on the patched version, and does not trigger on benign input” as paired tests, while freezing images, dependencies, and oracle criteria. The current figures remain exploratory: the case-level analysis focused primarily on a single benchmark; negative controls and patched-version falsification did not yet cover every signal-producing case; and Docker resource and network constraints may have affected executability. Whether the same false-positive rate can be reproduced across additional independent benchmarks will determine how widespread the problem is.