多模態評測
SciFigBench: GPT-5.2 Tops Chart Description Scores, Yet Still Answers 96% of the Time When Content Is Unreadable
SciFigBench uses more than 34,000 scientific-figure test configurations to evaluate visual understanding separately from the ability to acknowledge insufficient evidence. GPT-5.2 leads in description quality, while Gemini 3.1 Pro more often refuses to draw conclusions from ambiguous or misleading evidence.
Scientific-figure benchmarks typically ask only whether a model can identify trends or answer questions. Accuracy alone, however, cannot reveal whether a model will stop guessing when an image is blurry, information is missing, or a text prompt is misleading. SciFigBench therefore separates “accurate interpretation” from “reliable behavior under uncertainty.” It includes 250 scientific figures with high-quality human annotations produced over more than 600 hours, then adds image transformations, reasoning questions, anti-misleading probes, caption-bias probes, and verified selectively blurred regions to create more than 34,000 test configurations.
The benchmark uses the A-R-I framework to measure three properties: Admittance checks whether a model acknowledges insufficient evidence; Resistance tests whether it can withstand incorrect context; and Inductance examines whether it overgeneralizes from partial information. The results show that general capability scores are not equivalent to behavioral reliability. GPT-5.2 achieved an MQM description-quality score of 91.6 and reasoning accuracy of 78.4%, but still produced specific answers in 96% of cases where the content was unreadable. Gemini 3.1 Pro achieved an MQM score of 90.2 and reasoning accuracy of 81.0%, yet acknowledged uncertainty in 71% of unreadable cases and ranked first in resistance to misleading information with a score of 0.91.
This distinction is particularly important for research-paper reading agents, experimental-data extraction, and medical or engineering report pipelines. If a system selects models solely by their accuracy on normal images, it may deploy one that describes figures well but continues inventing values after the evidence disappears. Engineering teams should incorporate blurring, cropping, caption conflicts, and refusal calibration into regression testing, while requiring outputs to reference visible regions. One limitation is that the dataset contains only 250 original figures, with more than 34,000 configurations largely derived from the same image set. Rankings may also be affected by prompt templates, model versions, and updates to providers’ backends.