模型評測與推論
64-Way Self-Consistency Voting Actually Reduces Per-Question Accuracy on Most Hard GPQA Problems
A preregistered study found that, on most GPQA Diamond questions, Qwen2.5-7B and Llama-3-8B become increasingly likely to lock onto the wrong answer as more samples are put to a vote. Overall average scores still rise slightly, showing how aggregate metrics can conceal per-question regressions.
Self-consistency typically has a model sample multiple reasoning paths and then output the majority answer. A common assumption is that increasing the number of samples may waste compute at worst, but will not reduce quality. A new study tested this assumption using [GPQA Diamond’s 198 graduate-level science questions](https://arxiv.org/abs/2608.11403), drawing 64 samples per question from Qwen2.5-7B-Instruct-Turbo and Meta-Llama-3-8B-Instruct-Lite at a temperature of 0.7.
The results show that, relative to a single sample, 64-way voting reduced expected accuracy on 56.6% of questions for Qwen and 65.7% for Llama. The worst individual questions lost 47 and 46 percentage points, respectively. Yet aggregate accuracy still increased from 0.342 to 0.369 and from 0.273 to 0.313. A small number of questions with large improvements offset a much larger number of smaller regressions, so reporting only the overall score would produce the counterintuitive conclusion that “self-consistency works.”
The researchers first conducted an exploratory analysis on 47 questions, then locked their hypotheses and thresholds into a [Git tag and complete reproduction pipeline](https://github.com/u7k4rs6/self-consistency-backfire), and finally confirmed the findings on the remaining 151 questions. With an oracle that knows the correct answer and can select the best per-question budget from 1, 2, 4, 8, 16, 32, or 64 samples, the accuracy ceiling rises to 0.482 and 0.439. This is not, however, a deployable method. The answer-agreement gate and token-entropy gate tested by the authors both came within less than 0.002 accuracy of fixed 64-sample voting.
The likely reason is that, on difficult questions, models may consistently converge on the same plausible but incorrect distractor. In Qwen’s highest-agreement bucket, the majority answer was correct only 52.5% of the time; for Llama, the figure was just 28.6%, even lower than in its low-agreement bucket. This is a warning against treating “repeated answers agree” as a direct reliability signal in inference systems. More promising approaches to test include external verifiers, executable checks, and routers calibrated by question type. One limitation is that the experiments cover only two older 7B/8B instruction-tuned models and a single multiple-choice benchmark, so the findings cannot yet be generalized to reasoning-native or frontier models.