代理評測與安全
AgentAudit Breaks Agent Evaluation Into Ten Lifecycle Stages, but the Model Judge Remains the Biggest Variable
AgentAudit does more than calculate whether a task was completed: it separately examines planning, memory, tool selection, safety, and execution integrity across the execution trace. Composite Trust Scores vary widely among five models, but nine tasks and a single model judge are not enough to support a general-purpose ranking.
Most agent benchmarks answer only whether a task was completed, making it difficult to determine whether a failure originated in planning, memory, tool parameters, or the model following malicious instructions. Released on September 9, AgentAudit instead analyzes the complete execution trace. It divides each agent run into ten dimensions: instruction integrity, planner, memory, tool selection, tool invocation, tool correctness, alignment, tool faithfulness, security, and execution integrity. It also adds behavioral classification and failure attribution.
The technical value of this design is that the evaluator attaches to an existing agent and only reads its trace, without replacing the agent framework. Teams can therefore compare different models and harnesses using the same observation format. They can also distinguish between failure caused by insufficient capability and cases in which a capable model nevertheless complies with an attacker's request. The latter is labeled `Unsafe_Compliance`, a particularly important distinction for agents with file, database, or network access.
The authors tested five models on nine capability and adversarial tasks. The paper reports average Composite Trust Scores of 95.1 for Claude Sonnet 5 and 80.6 for GPT-5, compared with 57.6 for Sarvam 105B, 45.7 for Llama 3.3 70B, and 22.6 for Gemini 2.5 Flash. These figures should not be treated as a general capability ranking: the sample is small, the tasks and agent configurations may favor particular models, and every trace was scored by the same fixed model judge. That judge was also one of the models under evaluation, introducing potential self-preference and scale-calibration issues.
Engineering teams can adopt this layered attribution approach when building regression tests, but the next step is to see whether the authors release the complete tasks, traces, scoring prompts, and studies of agreement with human evaluators. Auditing logs alone also has inherent limitations: environmental state not captured by instrumentation, hidden tool side effects, and deleted or altered memories may still escape detection.