VICBench Traces 100 Vulnerabilities to Their Initial Introducing Commits; Existing Automated Methods Reach Only 40.1% F1

Security tools often know which commit fixed a vulnerability, but they cannot necessarily determine when the vulnerability first entered the codebase. Submitted on August 12, [VICBench](https://arxiv.org/abs/2608.12246) publishes 100 verified vulnerability-inducing commits (VICs), corresponding to 100 CVEs across 88 open-source projects, three languages—Python, Java, and C++—and 48 CWE categories. The dataset is available on [Zenodo](https://zenodo.org/records/18944736).

This is more than an exercise in software archaeology. The initial introducing commit determines which versions are actually affected and enables detectors to be tested before subsequent refactoring rewrites the vulnerable code. In VICBench, vulnerability-fixing commits modify an average of 38.6 lines, whereas the actual vulnerability-introducing commits involve an average of 252.5 lines—far more than in traditional datasets that track only a small number of deleted lines. In one example from the paper, code that accepted unsigned identity tokens was moved across files in 2018. A method relying solely on `git blame` stopped at the refactoring point and missed the true origin in 2016, underestimating the affected-version range by roughly two years.

The annotation process involved independent tracing by an author with nine years of programming experience and VIC-Agent. They identified exactly the same commit in 71 cases, with a Cohen’s κ of 0.707. The remaining 29 cases were reviewed by a second human annotator, and 11 cases from the final dataset were additionally checked by a security engineer with 16 years of experience. VIC-Agent combines CVE semantics, `git blame`, `git log -S`, and commit validation to distinguish newly introduced vulnerabilities, code moves, and pure refactoring.

Baseline results show that V-SZZ achieved only 33.3% F1 on 40 Java and C++ cases, while LLM4SZZ achieved 40.1% across all 100 cases. Although VIC-Agent reached 89.3%, it participated in constructing the dataset and therefore cannot be treated as an independent evaluation result. The dataset also contains only eight C++ cases, excludes JavaScript, Go, and Rust, and assumes that each CVE has one primary introducing commit. Engineering teams can use it to reevaluate AI code review and version-range inference systems, but the next steps still require blind testing, broader language coverage, annotations for vulnerabilities with multiple causal commits, and cross-checking against affected-version data from the [NVD](https://nvd.nist.gov/).