AI 安全與不確定性
Small Models’ Self-Reported Confidence Cannot Directly Control Risk; Only 3 of 22 Settings Meet the 20% Error Cap
A new study tests whether “model-reported confidence” can determine when to hand a task off to a human across 11 open-weight models. Although Platt scaling reduces ECE to as low as 0.02, rigorous finite-sample validation rejects every autonomous-answering setting under a 10% risk cap.
When deploying small local models, a common approach is to ask the model to output both an answer and a confidence score, then use a threshold to decide whether to hand the task off to a human. A [new study](https://arxiv.org/abs/2608.05064) tested 11 instruction-tuned models ranging from 0.5B to 14B parameters across the Qwen2.5, Llama, and Gemma families. Using ARC-Challenge and [TruthfulQA](https://github.com/sylinrl/TruthfulQA), the researchers collected 25,168 local predictions. The results show that calibration can correct the numerical meaning of confidence scores, but cannot make confidence better at identifying errors.
The reason is that strictly monotonic transformations such as temperature scaling and Platt scaling do not change the ranking of samples by confidence, so the risk–coverage frontier and error-detection AUROC remain unchanged. If all of a model’s self-reported confidence scores exceed 0.5 while its actual accuracy is below 0.5, temperature scaling even has an ECE floor that it cannot overcome. Eight of the 22 model–task combinations approached this theoretical lower bound. By comparison, Platt scaling reduced ECE to 0.02 in some settings, but this only means that a score of “0.9” is closer to 90% accuracy; it does not make correct and incorrect answers more separable.
The study then used 200 calibration questions per setting and Clopper–Pearson upper bounds to construct finite-sample risk certificates. With the error-rate cap set at 20%, only Qwen2.5-7B on ARC and Qwen2.5-14B on ARC and TruthfulQA were approved for autonomous answering. After the cap was tightened to 10%, no combination passed. When thresholds learned on ARC were transferred directly to TruthfulQA, every tested model either exceeded the risk budget or abstained entirely.
For engineering teams, confidence calibration should be performed separately for each model and task, and “no certifiable threshold” should be treated as a normal outcome. The study covers only two multiple-choice benchmarks, quantized models, and zero-temperature decoding, while its certificates depend on calibration and deployment data being independent and identically distributed. Under domain shift, free-form generation, or tool-using agents, these guarantees cannot be carried over directly.