AI 安全與評測
DFOT Quantifies LLM Over-Trust in Derived Sensor Values; Cross-Modal Distillation Improves Correction Rates by Up to 6.69 Points
A new framework breaks down “model over-trust in derived values” into metrics covering conflict acceptance, contextual misdirection, error correction, and utility harm. In a PPG-based rhythm interpretation case study, a reliability model that uses ECG during training but requires only PPG at deployment improves four endpoints, although the data and weights were not released with the code.
Researchers have introduced derived-feature over-trust (DFOT) to address a problem often overlooked in RAG and agent pipelines: heart rates, risk scores, or classification labels computed by upstream models can appear indistinguishable from directly observed facts when inserted into an LLM prompt. As a result, the model may fail to consider their scope of applicability or reliability for an individual case.
The experiments use synchronized PPG and Lead-II ECG signals. D1 deliberately creates a conflict between a PPG-derived “regular rhythm” label and an offline ECG assessment to test whether the LLM accepts the derived value without verification. In D2, both signals support a regular rhythm, but the prompt includes a history of severe atrial fibrillation to test whether background information misleads the model. The framework uses COTR, CIR, CRR, ESRM, and UHR to measure over-trust, contextual error, correction effectiveness, the specificity of case-level evidence, and harm caused by unnecessary verification, respectively.
The team constructed 50,000 four-minute records from paired MIMIC-III waveforms covering 1,275 patients, with no patient overlap among the training set, validation set, and a locked test set of 187 patients. During training, the K2 model distills rhythm logits from an ECG teacher into a PPG reliability model. At deployment, neither the ECG nor the teacher outputs are provided to the LLM. Compared with B2, which is trained using reliability labels alone, K2 improves AUROC by 0.025 on the locked test set. Across D1 and D2, correction rates and evidence specificity improve by 1.82 to 6.69 percentage points, and the confidence intervals for all four endpoints exclude zero.
The value of this work extends beyond the medical case study: it provides an interface-evaluation method applicable to OCR confidence, sensor fusion, and model-generated features. However, UHR still increases by 0.67 points, and the proposed two-percentage-point engineering threshold is not a clinical validation standard. The public repository contains only code, prompts, and statistical procedures; it omits patient data, model weights, historical outputs, and some original prompts. Engineering teams should first reproduce the data-governance and calibration processes before considering the direct inclusion of reliability text in high-risk decision-making.