Model Main Effects Fall Below 3% Across Three Agent Benchmarks, as DDR Recasts Leaderboards as a Deployment Reliability Problem

Agent leaderboards compress performance across many tasks into a single success rate, but they rarely answer whether those rankings can be reproduced on an enterprise’s own workloads. Research released through Stanford’s AIMS course treats agents, tasks, trajectory steps, and error categories as four measurement facets, using generalizability theory to decompose results from TheAgentCompany, τ²-bench, and AppWorld. The combined data include 7,783 checks from TheAgentCompany, repeated runs of three frontier agents across three τ²-bench domains, and 79,650 unit-test results from AppWorld.

REML analysis shows that, across all datasets and check types, the main effect attributable to the agent itself accounts for less than 3% of total variance, while agent × task interactions account for roughly 7% to 13%. After incorporating additional analyses, the paper’s abstract reports an overall range of 7% to 23%. This does not mean that the models do not differ; rather, their advantages depend primarily on the types of tasks they encounter, making performance difficult to interpret as a single, transferable measure of “agent capability.” The three estimators—Henderson Method I, REML, and a Bayesian binomial GLMM—produce broadly consistent core variance decompositions.

Slices of the hardest tasks expose greater deployment risk: the overall generalizability coefficient `Eρ²` for τ²-bench `action_checks` is 0.752, but falls to 0 when restricted to the hardest quartile. Across 50 70/30 holdout experiments, estimates extrapolated from the training grid are negatively correlated with holdout reliability at `r=-0.90` and overestimate it by an average of 0.30. The authors therefore propose Deployment Decision Reliability (DDR), which uses variance tables and D-studies to determine how many tasks are needed, which difficulty strata should be sampled, whether rankings change under cost constraints, and whether holdout replication was actually performed.

Engineering teams can treat DDR as an evaluation-design checklist rather than another universal score. However, the study includes only a small number of shared model families, and some cross-dataset correlation coefficients are estimated from just three families. The data also come from existing public trajectories, so they cannot directly establish that the same conclusions apply to a particular company’s private workflows. The next step is to prospectively validate the sample sizes recommended by the D-study after fixing the candidate agents, real-world task distribution, and inference costs.