AI 研究
CritICL Turns Small-Model Failure Modes Into a Prompt Library, Bringing Single-Generation Performance Close to Multi-Sample Test-Time Compute
CritICL first catalogs small-model errors, failure types, and critiques offline, then adds relevant examples to a large model’s prompt instead of repeatedly sampling for every question. The authors report slightly higher accuracy than self-consistency on Qwen2.5 math benchmarks, but the cost of building the library offline and the unreleased code limit reproducibility.
Researchers at Ohio State University and Princeton University have introduced CritICL, an approach that attempts to turn small-model errors into inference-time guidance for large models. The team first had Qwen2.5 1.5B, 3B, and 7B, along with corresponding small Llama models, answer each training question five times, then extracted the incorrect answers. GPT-4o-mini generated natural-language critiques and up to five failure labels for each error, which were clustered and stored in CritBank. Each entry therefore contains a question, flawed reasoning, failure modes, and corrective feedback.
There are two inference paths. CritICL-static selects up to five common failure examples based on the error distribution of smaller models from the same model family and inserts them into a single prompt. CritICL-dynamic first asks the target model to predict which errors the current question might trigger, then retrieves matching examples, requiring two model calls. The core assumption is that models of different parameter scales within the same family share similar failure structures, so pitfalls exposed by smaller models can be used to warn larger models in advance.
Across GSM8K, MATH, AMC23, AIME24, and AIME25, the static variant achieved an overall Pass@1 of 49.8% with Qwen2.5-32B, slightly above the 49.5% produced by majority voting over seven samples. With Qwen2.5-72B, it reached 59.2%, compared with 59.0% from five-sample voting. On MATH, the average token count per question was 3,768, below the test-time compute baselines’ range of 4,192 to 7,533, but still above the 3,620 tokens used by a standard five-shot prompt. This suggests that the gains come mainly from replacing repeated outputs with longer, targeted inputs, rather than reducing all in-context learning costs across the board. A larger limitation is that building CritBank requires extensive generation by small models and labeling by an external model, while the evaluation focuses on mathematics and older models. The paper states that code is available, but as of the verification date, the GitHub repository remained empty.