AI 安全與代理評測
SecRespond Reconstructs Post-Intrusion Forensics, With None of 23 Models Fully Detecting and Remediating a Single Testbed
SecRespond has security agents analyze read-only disk snapshots of compromised hosts instead of merely solving attack challenges in clean environments. Although 23 models could restate alerts, they commonly missed implants not disclosed by the alerts and failed to verify remediation.
The newly released SecRespond shifts security-agent evaluation from “can it break into a system?” to post-intrusion response that more closely resembles the daily work of a SOC. The research team created 10 isolated testbeds spanning CentOS, Ubuntu, and Windows Server, four types of initial access, and 21 MITRE ATT&CK techniques. Scenarios include Log4Shell, an exposed Docker API, a malicious npm package, RDP password spraying, and Web application RCE. After receiving alerts, vulnerability scans, baseline checks, and a read-only disk tree, agents must produce investigation updates, an intrusion report, a vulnerability report, a baseline report, and a remediation plan.
The key technical feature of the evaluation is that alerts do not provide the complete answer. The testbeds contain both alert-visible and “silent” evidence, forcing agents to proactively search for services, scheduled tasks, account backdoors, Web shells, credential theft, and persistence mechanisms. All 23 models were run through OpenCode. According to the paper, the models were better at finding issues already identified by alerts, but no model achieved both complete detection and remediation in any testbed. This suggests that cybersecurity question-answering or CVE identification capabilities cannot be directly extrapolated to end-to-end incident response.
The repository includes prompts, per-testbed checklists, execution scripts, and a structured scoring format, while the disk data is distributed separately through Hugging Face. Engineering teams can use these resources to test their own models, tool permissions, and reporting pipelines. However, the scores are assigned against checkpoints by three LLMs. Although their item-level agreement ranges from 72% to 75%, they may still share biases. In addition, all hosts and evidence are controlled synthetic data, without real-world enterprise noise, cross-host lateral movement, or high-pressure incident-response collaboration. Important next steps include establishing a human-analyst baseline, measuring consistency across repeated runs, and determining whether remediation commands can undergo executable validation on isolated copies.