代理評測
FINCHAL Lets Agents Compete Directly, Using Zero-Skill Simulations and Future Prices to Separate Luck from Predictive Skill
Through remote MCP tools, FINCHAL lets humans and AI agents submit long or short positions in four assets through the same interface, then scores them in real time using future market data. It first simulates 20,000 random strategies to establish a “luck ceiling,” but the full season has yet to produce comparable model results.
On August 24, the FINAL-Bench team launched FINCHAL, an attempt to shift AI forecasting evaluation away from historical backtests that allow repeated tuning and toward prospective tests whose answers do not yet exist. For each submission, participants take a position between −1 and +1 in NVIDIA, Bitcoin, gold, or crude oil. Positions remain in effect, position changes incur asset-specific transaction costs, and leverage is fixed at 1×. Settlement uses hourly price data; equities include only regular trading hours, and the system falls back to daily data only when insufficient hourly data is available. The [design notes](https://cdn-avatars.qwak.ai/blog/FINAL-Bench/financial-forecast-challenge) say the scorer also uses closed-form, independently verifiable synthetic paths to test conditions including zero positions, inverse compounding, transaction costs, and prevention of lookahead into the next price bar.
Instead of uploading CSV files, agents connect to a remote MCP server that exposes four tools: `get_rules`, `get_data`, `submit_position`, and `check_score`. This allows a model to read the rules, retrieve historical data, and change positions autonomously, while also bringing tool descriptions, error semantics, and validation scope into the evaluation. According to the public article, an early schema claimed to support hourly data but actually returned daily data; it was later changed to reject requests explicitly when that frequency was unsupported. Such errors are particularly dangerous for agents because a response that is structurally valid but semantically incorrect can directly contaminate features and decisions. The MCP specification defines tools as model-controlled executable interfaces and recommends retaining human confirmation for high-impact operations. Although FINCHAL’s orders are paper trades, its API keys and remote HTTP endpoints should still be handled under the same type of security model. [MCP tools specification](https://modelcontextprotocol.io/specification/2025-06-18/server/tools)
At the core of the evaluation, FINCHAL first simulates 20,000 zero-skill random strategies for each asset, sets the 95th percentile on the realized market path as the luck ceiling, and then maps rankings using `-log10(1-p)`. Because the tail distributions of different assets could not be calibrated reliably, the organizers abandoned a combined overall leaderboard. This is a reasonable limitation, but it also means results can be compared only within the same asset. The [public Space](https://huggingface.co/spaces/FINAL-Bench/finchal) provides the code and scoring interface, but it currently contains only baselines and backtest reference lines; there is not yet evidence that any agent can consistently exceed the luck ceiling. Engineers should watch whether data-release and submission timestamps can be independently audited, whether agents can gain an advantage through high-frequency probing or multiple accounts, and whether the organizers publish complete failure trajectories after the season ends in December instead of showcasing only the winners.