Five Open-Weight Models Report Nearly Constant Confidence, Making Low-Confidence-First Auditing Potentially Equivalent to Random Sampling

When a supervisor manages dozens of agents at once, a common practice is to inspect the “least confident” outputs first. [This study](https://arxiv.org/abs/2607.28317) finds that when confidence is self-reported by the agents, such rankings may contain no useful information and can even perform worse than random sampling. The researchers use a two-level Gaussian copula to model correlated failures across agents and within model families. They define δ as the proportion of incorrect answers whose confidence mimics the high-confidence distribution of correct answers, then calculate the critical value δ* at which the confidence-ranking strategy reverses.

In simulations involving 20 or 100 agents, with only 2% to 20% of outputs audited per round, δ* increased as the audit rate decreased. At B/N values of 0.02, 0.05, 0.1, and 0.2, the thresholds were 0.93, 0.89, 0.84, and 0.75, respectively. The reason is that extremely constrained budgets sample only the far tail of the confidence distribution, whereas moderate contamination first disrupts the middle ranks reached by larger audit budgets. The study also proposes diversity-Bayes, which uses a single audit result to update the posterior risk of other agents with correlated failure patterns. As correlation ρ rose from 0 to 0.8, its residual-risk advantage over random sampling increased from 0.34 to 3.47 units.

For the empirical evaluation, Qwen3 0.6B, 4B, and 8B; Mistral-7B-Instruct-v0.3; Phi-4-mini-instruct; and gpt-4o-mini each answered 500 questions from GSM8K and HotpotQA using a fixed prompt and reported confidence from 0 to 100. The five open-weight models reported nearly maximum confidence regardless of whether their answers were correct, with ECE ranging from 0.43 to 0.94 and AUROC from just 0.50 to 0.52. Point estimates for four models crossed the reversal threshold under a 10% audit budget, although their confidence intervals still spanned the threshold. The tested [Qwen3-8B model card](https://huggingface.co/Qwen/Qwen3-8B) also explicitly advises against greedy decoding, while the paper’s primary evaluation used direct answers at temperature 0, highlighting a mismatch in evaluation settings.

In engineering practice, natural-language confidence should not be used as the sole key for prioritizing an audit queue. At minimum, it should be calibrated on a held-out set and combined with task risk, historical errors, and external verifiers. The study’s replayed agent population contained only 15 agents and did not evaluate chain-of-thought reasoning, logit-based confidence, or repeated sampling. Whether these alternative signals can restore effective ranking remains an important question for future validation.