模型評測
TsuGO Reconstructs Search Trees from Reasoning Text; Gemini 3.1 Pro Scores Just 19% on Open-Ended Hard Problems
Using 600 verifiable Go life-and-death problems, TsuGO measures whether models allocate reasoning resources to the correct branches rather than merely counting correct answers and tokens. After four multiple-choice candidates are removed, both accuracy and search efficiency fall sharply across several reasoning models, leaving a clear gap with the neural-network-guided KataGo.
Long chain-of-thought reasoning is typically evaluated by answer accuracy or efficiency per token, but these metrics cannot distinguish between a model methodically validating promising candidates and one that generates large amounts of text before arriving at the right answer by chance. Released on August 13, TsuGO reframes this question as verifiable adversarial search: a model must identify the critical first move in a Go life-and-death position and read ahead to the opponent’s best response.
The dataset contains 1,500 problems. Positions are stored in SGF and converted into five representations, including coordinates, 19×19 matrices, and images. The main experiment samples 200 problems from each of three difficulty levels and evaluates both a K=4 mode that provides four candidate points and an open-search mode with no candidates. An LLM extractor then decomposes free-form reasoning into candidate exploration, variation reading, position evaluation, and backtracking to reconstruct a process search tree. In a manual review of 300 problems, agreement between the extractor and two experts on the target structures ranged from 93% to 98%.
The core metric, SearchE, combines wasted effort on incorrect branches, the timing of the first correct-candidate hit, and the order in which the correct candidate is searched. Results show that Kimi-K2.5 fell on easy problems from 52.0% accuracy and 39.9 SearchE in K=4 mode to 27.8% and 26.4, respectively, in open mode. Although Gemini 3.1 Pro reached 80% on easy four-choice problems, it scored only 19% on open-ended hard problems. With the same budget of 200 visits, KataGo-b18 still achieved 43% on open-ended hard problems, indicating that the bottleneck for language models is not merely Go ability but also candidate generation, ranking, and sustained tracking of promising branches.
TsuGO is useful for testing whether inference-time search, tree-of-thought, or a verifier genuinely improves resource allocation, but SearchE should not be treated as a direct measurement of a model’s internal reasoning. Closed models expose only compressed reasoning summaries, while KataGo reads board states directly and is therefore not an equivalent baseline in terms of tokens or interfaces. Whether the dataset and evaluation code are released in full will also be crucial for subsequent reproduction.