OmniaBench Tests General-Purpose Agents With 1,431 Cross-Scenario Tasks; Frontier Models Still Score Below 60% on Pass@1

OmniaBench expands general-purpose agent evaluation beyond a single website, toolset, or conversational format to heterogeneous scenarios. The full dataset contains 1,431 tasks and includes a 644-task hard subset designed to reduce the cost of repeated evaluations and mitigate the risk of training-data contamination after the full benchmark is made public. Each environment has an explicitly defined state space, allowing evaluation to go beyond the subjective judgment of another language model and verify whether an agent’s actions actually move the system toward the target state.

The paper reports overall Pass@1 scores of just 58.54% for Claude Sonnet 5 and 57.14% for GPT-5.6 Sol. The authors further break down failures by domain and capability, finding that even when current models invoke tools correctly, they frequently make errors in multi-step planning, maintaining constraints over extended interactions, and adapting their strategies after observing failures. For production teams, this type of diagnosis is more useful than a single aggregate score: it can help determine whether to improve prompts, state representations, validators, fallback strategies, or the underlying model.

However, Pass@1 remains sensitive to the agent framework, tool descriptions, reasoning budget, sampling settings, and retry policies. Differences among vendors’ native tool interfaces may also distort comparisons. The hard subset can only delay contamination; it cannot eliminate the possibility that future model versions will memorize the tasks. Adopters should pin model versions and execution environments, retain complete trajectories and cost records, and maintain internal tasks outside the public benchmark. The next question is whether the authors will provide long-term version governance, replayable sandboxes, and standardized configurations across agent frameworks.