AI 評測與多代理系統
170,000 Votes Across 28 Models Show That Adding More Agents Cannot Eliminate Shared Errors
A new study evaluated 28 language models on four types of binary screening tasks and found that models often fail together on the same samples, leaving majority voting with an error floor that cannot be eliminated by enlarging the committee. The research also shows that the optimal acceptance threshold depends on the error correlations for positive and negative cases, meaning a fixed simple-majority rule may not be the safest choice.
Multi-agent systems often use “majority voting” to reduce the instability of a single model, but this assumption holds only when agents’ errors are approximately independent. A new study submitted on July 27 collected 174,384 primary-protocol votes from 28 language models across four binary screening benchmarks. It also released roughly 27,000 reasoning-protocol records, covering 20 open-weight models and eight frontier API models. The researchers separately estimated hit rates and latent inter-model correlations for good and bad cases, then fitted parameters on odd-numbered items to predict committee loss on even-numbered items.
The results show that model errors do not co-occur uniformly: for some case types, models fail in highly synchronized ways, while for others they are more complementary. As a result, simply requiring agreement from more than half of the agents can amplify either false-acceptance or false-rejection risk. For a reference agent with an accuracy of 0.8, the study measured cross-model-family correlations ranging from approximately 0.24 to 0.85. Even as committee size approaches infinity, majority voting may retain 21% to 90% of a single agent’s error rate. When the costs of different error types are asymmetric, the optimal `k-of-n` threshold also shifts away from a simple majority.
For deployment teams, the key is not merely to add models or repeat sampling, but first to estimate “per-case shared failure” on representative data. Multiple samples from the same base model, similar training data, or identical prompt templates can all create false diversity. In practice, teams should select thresholds using a held-out set and calculate correlation matrices and error costs separately for each task state. However, the study focuses on screening tasks reducible to binary decisions and uses Gaussian copula modeling; its conclusions cannot yet be directly generalized to open-ended code modification, tool execution, or agent debate. The next question is whether adding genuinely heterogeneous tools, retrieval sources, and training families can meaningfully lower this correlated-error floor.