LLM Judges Decide First, Then Search or Abstain, Using Calibrated Thresholds to Control the Error Rate

Researchers from Dalhousie University, NYU Abu Dhabi, and Emory University have reformulated reference-free LLM-as-a-judge as a three-stage decision process. The judge first makes a determination using its parametric knowledge. Only when the predictive entropy of the verdict token exceeds the first threshold does it search the web and reconsider the verdict using the top three results. If it remains uncertain after the second evaluation, it abstains and sends the case for human review. This avoids searching for every question and prevents low-confidence verdicts from being forced into evaluation results.

The key is not to set a confidence threshold manually. Instead, the framework counts errors among accepted verdicts on a calibration set with ground-truth labels, then obtains an upper bound on the false discovery rate (FDR) using a one-sided Clopper–Pearson interval. The system searches for two thresholds that maximize acceptance coverage while requiring the upper bound to remain at or below the user-specified α. For any fixed pair of thresholds, the authors prove that, under the assumption that samples are independent and identically distributed, the error indicator after two-stage routing can still be treated as a Bernoulli variable, allowing the finite-sample guarantee to carry over. To address multiple comparisons introduced by the threshold search, the paper also reports a Bonferroni-corrected version.

The experiments sampled 2,000 questions each from TriviaQA, Natural Questions, HotpotQA, and PopQA. Answers were generated using Qwen3-8B or Llama-3.1-70B and evaluated by four judge models ranging from 4B to 14B/8B parameters. Across 100 calibration/test splits and 32 candidate-model–judge–dataset configurations, the observed FDR never exceeded α. At α=0.20, when Qwen3-14B judged answers from Qwen3-8B, coverage reached 86% on HotpotQA, compared with only 14% for Qwen3-4B. On TriviaQA, Qwen3-14B could judge approximately 45% of questions directly without searching.

This design is suitable for data cleaning, offline evaluation, and automated acceptance testing, but its statistical guarantee is not a permanent safety certificate: recalibration is required after shifts in question types, search quality, or model versions. The experiments’ “ground-truth” answers also included settings in which another Qwen model determined semantic equivalence, rather than relying entirely on human annotations. Search snippets may also contain outdated information or prompt injections. Engineering teams should next focus on distribution-shift detection, filtering for trustworthy sources, and the total cost of routing abstentions to human reviewers.