CentaurBench: The Model Best at Solving Problems Independently May Not Be the Best at Guiding Another Agent

CentaurBench separately measures two capabilities that are often conflated. In “automation” mode, the candidate model directly delivers the complete output. In “assistance” mode, it may only provide a fixed GPT-3.5-Turbo executor with a requirements check, execution plan, and self-review checklist; it cannot write the final answer on the executor’s behalf. The seven tasks cover travel and meal planning, natural gas market analysis, operations research, tax-return review, counseling responses, and instructional design.

The study tested ten models, conducting ten independent runs for each task and mode. Outputs were anonymously compared in pairs by four LLMs using task-specific rubrics. To reduce same-family bias, judges were not allowed to evaluate answers from their own model family. The correlation between the overall automation and assistance rankings was only about 0.48, while task-level correlations ranged from -0.04 for travel planning to 0.85 for tax-return review. Claude Opus 4.8 achieved an average rank of 2.05 when answering the market-analysis task directly, but fell to 8.15 when acting as a guide. The top-ranked automation model also lost on five of the seven assistance tasks.

More notably, unassisted GPT-3.5-Turbo outperformed every assisted version on three tasks and ranked second overall, behind only GPT-5-mini. This suggests that plans that are overly complex or poorly matched to the executor’s capabilities may increase cognitive and context burden instead of enabling effective division of labor. Engineering teams seeking to reduce costs by having a “strong model guide a smaller model” cannot simply rely on general capability leaderboards. They should hold the executor model, tools, and delivery format constant, then separately measure the marginal gains produced by guidance.

The results still cannot be directly generalized to human–AI collaboration. The experiment was limited to single-turn textual guidance, seven tasks, and a single GPT-3.5-Turbo executor, while its primary rankings depended on LLM judges. Agreement among multiple judges on the same winner was only 71.0%, falling further to 67.8% in assistance mode. Future work should include human workers, iterative clarification, and tool use, while testing whether richer forms of assistance change the rankings.