AI 代理評測
ORCA-bench Reconstructs Six Days of Telemetry, With Frontier Agents Achieving Just 25.3% on Medium-Difficulty Root Cause Analysis
Using runnable microservices, 50 GB of telemetry data, and 1,079 incidents, ORCA-bench tests whether coding agents can identify root causes from ambiguous incident reports. Even with full source-code access, the best agent correctly answered only one-quarter of the medium-difficulty tasks designed to resemble real-world inputs.
The newly released ORCA-bench moves agent evaluation beyond “patching an already localized software bug” toward the kind of root cause analysis performed by on-call engineers. Its test environment is a continuously running microservices system instrumented with OpenTelemetry. Through Grafana, agents can access Prometheus metrics, Jaeger traces, and OpenSearch logs, as well as inspect the complete source code. The dataset spans six days and approximately 50 GB, comprising 1,079 root cause analysis tasks that deliberately vary the specificity of incident reports, detection latency, and the number of concurrent failures.
The results show that the ability to write code does not translate into reliable incident response. Among five frontier agents, the best achieved just 25.3% RCA Accuracy in the Medium category, which represents more realistic incident reports, and fell to 10.0% in the Hard category. The weakest model fabricated implausible root causes in 40% of incident reports. Removing source-code access degraded every measured metric, indicating that telemetry search alone still cannot replace the ability to trace observed runtime behavior back to the implementation.
The researchers had SREs manually verify the ground truth for symptoms and manually rescored the LLM judge’s results, reporting a weighted Cohen’s κ of 0.90. This makes the evaluation more credible than one based solely on automated judging. However, each incident was still investigated in isolation, and the system was far smaller than a real production environment. The 25.3% result is therefore likely an optimistic upper bound rather than a deployment threshold.
For platform teams, the next step should not simply be adopting a larger model. They should test whether an agent can construct timelines, cross-check metrics, logs, traces, and version changes, and refrain from drawing conclusions when the evidence is insufficient. Any automated remediation workflow should also separate root cause hypotheses, query records, and change approvals, preventing low-accuracy diagnoses from directly triggering production operations.