評測與資安
FuzzingBrain-Bench Does Not Specify Target Vulnerabilities; Claude Opus 4.8 Triggers Crashes in 60 of 77 Challenges
The new benchmark packages complete vulnerable programs, fuzzing interfaces, and sanitizers into offline Docker images, scoring models by distinct, reproducible crash signatures. Its design gives credit for defects that models discover unexpectedly, but the current difficulty coefficients are derived from the same models being evaluated, leaving the rankings circular.
A team at Texas A&M University has released FuzzingBrain-Bench V1, an attempt to shift LLM vulnerability evaluation from “reproducing a known CVE” toward a more open-ended bug-finding task. For each challenge, the agent receives the complete project at a vulnerable version, a fuzzing harness, and sanitizer configuration, but no patch commit, target line of code, or reference proof of concept (PoC). The agent must generate inputs that trigger as many distinct crashes as possible within a maximum of 100 rounds and 30 minutes.
The benchmark contains 77 challenges drawn from 43 open-source projects: 36 use C, 32 use C++, and nine use Java/JVM. Forty-five challenges originate from memory-safety defects; the remainder cover issues including memory exhaustion, uncaught exceptions, reachable assertions, undefined behavior, and algorithmic complexity. Each candidate input is executed three times inside an offline Docker image and scores only if all three runs trigger the same fault class at the same location. A signature consists of the error type and up to three application stack frames, allowing duplicate inputs for the same crash to be excluded. Scoring does not rely on an LLM judge.
The authors fully evaluated Claude Haiku 4.5, Sonnet 4.6, and Opus 4.8. Opus generated crashing inputs for 60 of the 77 challenges and scored 196 out of 579 points. Sonnet reached 50 challenges and 156 points, while Haiku reached 35 challenges and 58 points. Thirteen challenges remained unsolved by every model. The API cost for a complete Opus run was approximately $193—less than Sonnet’s $253 but more than Haiku’s $43. Differences in tool-use patterns and early-termination behavior mean costs cannot be inferred from model pricing alone.
The publicly available MIT-licensed repository provides a CLI, an MCP interface, Codex and Claude Code backends, and challenge images without answer keys, making it easier for researchers to integrate other models or traditional fuzzers. However, the system can confirm only that “a new reproducible crash signature appeared.” Without an answer key, it cannot prove that a signature corresponds to a distinct vulnerability, the originally targeted defect, or an exploitable security issue. More importantly, the five-level difficulty coefficients were derived from the performance of these same three Claude models and then used to score that same set of models. Until the benchmark includes cross-model evaluations, repeated runs, and external fuzzing baselines, a score of 196 out of 579 should not be treated as a general ranking of vulnerability-discovery capability.