CTBench Retests Agents in 234 Telecom-Failure Sandboxes; Best Root-Cause Analysis Accuracy Reaches Just 47.62%

Researchers from Huawei and Queen Mary University of London have released CTBench, shifting telecom operations evaluation from static question answering to interactive fault diagnosis. The dataset contains 126 root-cause analysis (RCA) tasks and 108 path-reconstruction tasks. Fifteen senior telecom experts abstracted and anonymized them from real-world cases, after which independent experts who had not seen the answers solved them again. For each task, agents may use only the permitted diagnostic interfaces to query device status, routes, interfaces, tunnels, and policy outputs.

Scoring goes beyond checking the final answer. RCA is broken down into fault-node/object localization, standardized cause labels, and evidence coverage. Path tasks separately measure endpoint localization, forwarding-edge-set Intersection over Union (IoU), and evidence F1. Each task is also annotated for observability, vendor and device heterogeneity, protocol complexity, fault-propagation depth, and expert-solution length, making it possible to distinguish “guessing the right result” from actually completing the diagnosis.

Among the five agent–model combinations, Codex+GPT‑5.5 achieved the best overall results: 87.96% accuracy on path tasks, 99.38% endpoint-localization accuracy, and 95.28% path IoU. However, its path-evidence score was only 47.84%. The gap was even larger for RCA: final accuracy was 47.62%, cause-identification accuracy was 66.83%, and fault-localization accuracy was 52.90%, while the evidence score was just 15.80%. This suggests that an agent may infer the fault type from symptoms without retrieving enough device-level evidence to authorize remediation safely.

Partially observable and multi-vendor environments further magnified the problem. For ClaudeCode+Qwen3.7‑Plus, for example, accuracy on highly heterogeneous path tasks fell from 29.31% to 4.00%. Although it collected more indirect evidence in partially observable RCA tasks, cause-identification accuracy still dropped from 42.92% to 16.22%. Engineering teams should now examine whether evidence planning, cross-vendor output normalization, and safe remediation can be incorporated into the agent loop. The benchmark currently covers only RCA and path reconstruction, not actual network changes. Its dataset page is public, but downloading the data requires consenting to share contact information, which also raises the barrier to independent reproduction.