代理評測
FM-Bench Has Agents Run a Football Club for 20 Years; None of 15 Models Could Infer Market Prices From Rejected Bids
Using 26 tools and roughly 340 to 400 decision points, FM-Bench tests whether agents can handle delayed rewards, hidden information, and competitors’ reactions. All 15 models completed the 20-year campaign, but rankings stabilized only in the later stages, while self-managed memory exhibited two failure modes: endless accumulation or repeated overwriting.
The newly released FM-Bench places language-model agents in a replayable football club simulator rather than an environment of independent question-and-answer tasks. Over 20 in-game years, agents must use 26 tools to draft, buy, and sell players; negotiate contracts; invest in facilities and youth development; select lineups; and respond to the board. A full run contains roughly 340 to 400 decision points. All competitors draw from shared budgets, players’ true abilities are obscured by scouting ranges with fixed biases, and the market adjusts in response to agents’ actions.
The evaluation is divided into Solo and Arena modes. Solo pits one model against 15 fixed scripts, while Arena places 15 models and one scripted baseline in the same world. The engine uses deterministic simulation: runs with the same random seed and action log can be replayed bit for bit. Scores combine honors, growth in club net worth, and squad value, eliminating the need for an LLM judge. Across three seeds, the paper reports that all 15 models completed every season, whereas the scripted baselines usually went bankrupt or were dismissed early. However, the championship changed hands among 10 models, and neither price, provider, nor token expenditure reliably predicted the rankings.
The bottlenecks revealed by the trajectories are more noteworthy than the aggregate scores. Higher-scoring agents renewed contracts earlier, kept less cash idle, and stopped making investments with excessively long payback periods near the end of a season. Yet no model could infer hidden market prices from hundreds of rejected bids. Agents’ self-managed memory also tended toward two extremes: files accumulated without limit, or plans were rewritten every season, destroying continuity.
From an engineering perspective, FM-Bench provides an auditable stress test for long-horizon workflows and is well suited to research on memory compression, state estimation, and long-term credit assignment. However, the public repository provides only the complete Solo mode; the official Arena, hidden seeds, and canonical trajectories are hosted by the authors. The project is still labeled a research preview, and its scoring version and calibration may change. The current standings therefore should not be treated as a general-purpose ranking of agent capabilities.