Same-Model Agent Pairs Fail Together on 90% of Unsuccessful Tasks, Undermining Independence-Based Reliability Estimates

Multi-agent systems often multiply the success rates of individual components to estimate the reliability of an entire workflow. This calculation implicitly assumes that agents are conditionally independent given the task. A new study, Agent Behavioral Contracts II, directly tests that assumption. Across 18,000 preregistered task runs scored by deterministic programs without an LLM judge, two agents using the same model failed together on 90.0% of tasks where at least one agent failed, with a phi correlation coefficient of 0.916. In other words, duplicating the same model does not provide two independent layers of redundancy.

When the authors replaced models one at a time, failure dependence decreased in all six comparisons. However, once the models were already different, switching to a different provider produced no significant additional improvement. This suggests that engineering teams should track failure mechanisms and model diversity rather than looking only at whether APIs come from different companies. Positive dependence makes the actual joint failure rate higher than the product implied by independence, meaning the safety benefit of having two agents cross-check each other may be systematically overstated.

The study proposes a finite-sample certificate that does not assume a particular dependence structure. It first uses Bonferroni–Clopper–Pearson intervals to bound moments measured during joint execution, then applies linear programming to find the worst-case reliability across all compatible joint distributions. Increasing the number of moment constraints from 10 to 14 narrowed the identification interval by 85.7% and raised the certified lower bound from 0.2455 to 0.4116; the optionally stopped version had a Type I error rate of 0.0471. The code, analysis scripts, and preregistration documents have been released publicly, but this remains a single preprint, and a two-agent handoff may not represent large, heterogeneous agent graphs. Deployment teams should therefore retain per-task joint-failure data instead of reporting only each agent’s marginal success rate.