代理與評測研究
Strong Models Compile Task Rules Into a Harness, Raising GPT‑5.4‑mini Accuracy From 0.488 to 0.912
In a strong-to-weak test-time transfer experiment, stronger models wrote routing, format-checking, and deterministic-solving programs for a fixed smaller model without updating its weights. The best harness corrected about 83% of baseline errors, but performance depended heavily on theory-of-mind benchmarks whose patterns could be compiled into rules.
Researchers from Salesforce AI Research and other institutions introduced a “strong-to-weak scaffolding” experiment. Rather than directly answering test questions, distilling knowledge, or fine-tuning the weaker model, a strong model builds a reusable inference harness in an agentic coding environment. This outer-layer program can include task classification, prompt templates, deterministic rules, answer-format validation, retrieved examples, and symbolic solvers. Once completed, the fixed weak model processes unseen test data through the same entry point.
The experiment combines four theory-of-mind datasets: BigToM, Hi‑ToM, MMToM‑QA, and MuMA‑ToM. Builders could see only 195 questions—a labeled validation set representing 5% of the total—and were ultimately evaluated on a hidden test set of approximately 3,900 questions. The researchers conducted 72 builds across three platforms—Cursor, Claude Code, and Codex—and builder models including GPT‑5.5, Opus‑4.7, and Gemini‑3.1‑Pro. The fixed target, GPT‑5.4‑mini, achieved a macro-average accuracy of 0.488 without a harness. Every builder configuration produced a positive gain, reaching an average of 0.763. The best GPT‑5.5/Codex combination scored 0.912, even surpassing the 0.619 achieved by GPT‑5.4 without an outer-layer program.
The analysis indicates that the gains came primarily not from additional sampling or longer reasoning, but from moving brittle reasoning into code. The proportion of questions handled directly by rules or deterministic programs correlated with accuracy at `r=0.72`. Format enforcement, greedy decoding, and question-type routing provided a shared performance floor, while deeply nested belief recursion, deception, and Bayesian goal inference remained the main sources of error.
This also represents the most important limitation. Parts of BigToM’s structure can be compiled almost completely, so the results may measure a builder’s ability to discover benchmark regularities rather than whether the weak model acquired generalizable reasoning capabilities. A manually designed UserHarness still outperformed the best automated result, scoring 0.939, and the paper’s webpage currently does not provide the complete harness code or execution artifacts. Engineering teams should view the method as a high-cost, one-time form of workflow compilation. The next step is to determine whether its gains persist across open-ended tasks, data distributions, and model versions, as well as to assess the real-world latency, maintenance costs, and failure modes introduced by additional outer-layer calls.