程式代理評測
AgentLens Moves Coding Agent Evaluation to Production Traces, Supporting Nightly Regression Testing and Failure Diagnosis
AgentLens goes beyond checking whether an agent solves a coding task, examining the full interaction trace and actual product outcomes. The open-source benchmark can be used to compare model versions, identify behavioral regressions, and integrate evaluation into continuous integration workflows.
AgentLens proposes a “production evaluation trace review” methodology to address common blind spots in coding agent evaluation. Traditional SWE-style benchmarks typically consider only whether the final tests pass, but production agents may also repeat searches, modify unrelated files, misuse tools, consume excessive tokens, or leave behind fragile implementations when tests pass by chance. AgentLens therefore analyzes the complete trace of interactions between the agent and its environment, allowing teams to assess final correctness, action quality, and failure modes at the same time.
The authors say they have used this methodology to compare successive versions of their own agent and detect product regressions in a nightly evaluation pipeline; the benchmark and tools have been open-sourced. This approach is well suited to agent engineering: changes to the model, system prompt, tool schema, sandbox, or package versions can increase cost or risk even when the success rate remains unchanged. Saving results at the trace level also makes it possible to categorize failures as planning errors, insufficient information gathering, execution errors, inadequate validation, or premature termination, and then map them to system components that can be modified.
However, trace-based scoring also introduces new subjectivity. If a language model reviews agent behavior, the judge model may favor particular styles, and reviewing long traces can be costly. Production tasks drawn from a single product may also be unrepresentative of other libraries or team workflows. When adopting this approach, teams should use executable tests, static rules, human spot checks, and trace review as separate evaluation layers; preserve model and environment versions; and report tokens, tool calls, time, and destructive operations in addition to success rates.