IBM Identifies Unstable Agent Steps From a Single Trajectory as ALTK-Evolve Turns Diagnostics Into Reusable Guidelines

IBM Research has updated the open-source ALTK-Evolve to address the agent “consistency gap” obscured by average success rates. On AppWorld’s 168 `test_normal` tasks, the team ran a GPT-4.1-powered ReAct agent five times per task. Its per-run average success rate, Mean@5, was 77.4%, but its Pass^5—the share of tasks completed successfully in all five runs—was only 53.0%. For production systems, the former describes expected performance on a single request, while the latter better reflects whether the same financial, contractual, or API workflow can be delivered reliably across repeated executions.

The new Consistency Analyzer does not need to interact with the external environment again, nor does it require model logits or ground-truth answers. Given an existing trajectory, it replays the context at each reasoning step and asks the model to generate multiple candidate responses. It then identifies decisions prone to “flipping” by comparing structural similarities such as tool names and parameters. An offline Guideline Generator subsequently rewrites these points as natural-language rules—for example, verify that search results are unique before proceeding, or use a start-of-line regular expression to count checklist items. After the rules are stored, merged, and retrieved, they are injected into prompts at inference time for similar tasks without modifying the model’s weights.

IBM’s published GPT-4.1 results show that Pass^5 rose from 53.0% to 69.0%, while Mean@5 increased from 77.4% to 81.0%. The consistency gap narrowed from 24.4 to 12.0 percentage points. When guidelines generated from one task variant were applied to other variants in the same scenario, Pass^5 still increased by 13 percentage points. The weaker gpt-oss-120b model also improved. This suggests that, alongside `Pass@k` or average accuracy, engineering teams should also report `Pass^k`, which requires every run to succeed.

One limitation is that the guidelines are generated by the same model using its own trajectories as references, which could entrench flawed strategies. The difficult subset of similar tasks also showed a slight decline in average accuracy. The paper evaluates only ReAct, two models, and AppWorld, so it has not yet demonstrated that the results extend to browser agents, coding agents, or non-replayable real-world transactions. Practical deployments must also assess resampling costs, conflicting guidelines, memory contamination, and whether diagnostics need to be repeated after model or tool versions change.