代理評測／可觀測性
AFANet Diagnoses Multi-Agent Failures With 65K Trainable Parameters, Cutting In-Domain Inference From 199 Seconds to 1.16 Seconds
AFANet converts agent conversations into graphs with temporal and same-agent edges, then uses a two-layer GCN to identify faulty agents and error types. It outperforms several LLM baselines on AEGIS-Bench, but its agent–error pair F1 remains low across datasets.
After a multi-agent system fails, a common approach is to give the complete trajectory to another LLM and ask it to determine which agent made what type of error. A new study introduces AFANet, arguing that this kind of diagnosis is closer to a structured classification problem and may not require expensive generative reasoning.
AFANet represents each conversational turn as a node, adding bidirectional temporal edges between adjacent turns and long-range edges between different utterances from the same agent. Node features include statistical signals such as position and deviation, as well as sentence embeddings generated by all-MiniLM-L6-v2. After two layers of GCN message passing, the model applies mean and max pooling to each agent’s nodes and separately optimizes two losses: whether the agent is faulty and the type of error.
On AEGIS-Bench, used as the in-domain dataset, AFANet achieves an agent-level micro-F1 of 74.16% and an agent–error pair micro-F1 of 17.42%. The latter exceeds the Qwen, GPT-4.1, o3, Gemini 2.5, and Claude Sonnet 4 baselines evaluated in the paper. The model has only about 65,000 trainable parameters, takes 1.1 hours to train on a V100, and completes in-domain and out-of-domain inference in 1.16 and 0.37 seconds, respectively. By comparison, the paper’s 7B SFT baseline requires six hours of training and 199 and 108 seconds for inference.
The efficiency figures require careful interpretation: the 65,000-parameter count excludes the frozen sentence encoder, and graph-construction preprocessing takes an additional roughly 80.8 seconds. More importantly, the model is trained only on AEGIS-Bench. When transferred to Who&When, its agent-level micro-F1 falls to 37.93%, while its pair-level micro-F1 reaches just 6.90%, showing that error classification still struggles to generalize across data distributions. Ablation results also show that removing the GNN makes little difference in-domain but causes more pronounced degradation out-of-domain. The next questions are whether the authors will release a complete implementation and whether its speed and accuracy can be reproduced on naturally occurring production failure trajectories rather than curated or artificially injected failures.