MIST Tests “Selective Trust” Across Four Paired Contexts, With Misleading Information Cutting 23 Models’ Scores by an Average of 17.1 Points

Once language models are connected to search, RAG, or user feedback, the capability they truly need is not to reject external information indiscriminately, but to determine when it should be trusted. The [MIST/SCOPE study](https://arxiv.org/abs/2608.06377), released on August 6, therefore rewrites each question into four paired versions: clean, with no additional information; misleading, which suggests an incorrect answer; correct-context, which provides an accurate clue; and irrelevant-context, whose content is similar but unrelated. MIST-1000 contains 1,000 questions that underwent a two-stage human review process, producing 4,000 contextualized examples. Of these, 800 questions were adapted from existing reasoning benchmarks, while another 200 were newly written by human contributors.

The team also introduced the SC2W metric, which counts only cases in which a model originally answered correctly but switched to an incorrect answer after misleading information was added. All 23 API and open-weight models tested were affected, with misleading contexts reducing accuracy by an average of 17.1 percentage points. Even GPT-5.5, which achieved 96.0% clean accuracy, recorded an SC2W rate of 10.5%. This is more discriminating than looking only at accuracy under misleading contexts, because a model that ignores context entirely may appear “robust” while remaining unable to use genuinely helpful retrieval results.

The post-training method SCOPE does not introduce a new loss function. Instead, it constructs preference pairs from samples that were answered correctly in the clean setting but incorrectly in the misleading setting, then feeds equal numbers of examples from all four contexts into standard DPO. Qwen3-4B’s SC2W rate fell from 35.0% to 16.3%, while Llama-3.2-3B’s declined from 31.5% to 20.6%. Neither model experienced a drop in accuracy under clean, correct, or irrelevant contexts. The repository provides LoRA-DPO, vLLM evaluation, per-question bootstrap confidence intervals, and data-validation workflows to support reproducibility.

From an engineering perspective, this paired testing approach can be incorporated into regression tests for RAG systems or agents: inject correct, incorrect, and irrelevant documents for the same question instead of measuring only the average score with versus without retrieval. However, MIST is a controlled, text-only diagnostic and does not represent the frequency of errors in real-world deployments. Its training results also cover only two small-model families, and because most questions are derived from public benchmarks, data contamination cannot be ruled out entirely.