多模態評測／AI 安全
SHROOM-Visions Evaluates VLM Hallucinations With 20,000 Samples Across Four Languages; Best Chinese IoU Still Only 0.53
The new shared task requires systems to localize and classify visual language model hallucinations character by character, rather than merely judging whether an entire response is trustworthy. Twenty-seven teams submitted more than 600 systems, but the closed test set and unstable adjacent rankings limit the conclusions that can be drawn.
SHROOM-Visions 2026 has released the full results of its shared task, breaking visual language model (VLM) hallucination detection down into a more granular, character-level problem. Each sample includes an image, prompt, model output, hallucinated text spans, and category labels. Participating systems must estimate, for every character in the output, the probability that it belongs to a hallucination, then classify each span as a fabricated object, incorrect description, OCR misreading, counting error, or another error type.
The dataset contains approximately 20,000 samples across Chinese, English, French, and Italian. The training set comprises about 15,200 samples generated by five different VLMs, while the closed test set contains 4,800 samples—1,200 per language. Evaluation measures not only the character-level intersection over union (IoU) between predicted and human-annotated spans, but also the correlation between predicted hallucination probabilities and empirical probabilities derived from multiple annotators, as well as category-aware correlation. This design distinguishes between “knowing that a response contains an error” and “identifying exactly which characters are wrong and what type of error they represent.”
Twenty-seven teams submitted more than 600 systems in total. Averaged across the four languages, the best results achieved a character-level correlation of 0.58, a category-conditioned correlation of 0.46, and an IoU of 0.51—roughly 30 to 40 percentage points above the official baseline. For Chinese, vroom-vroom led with a correlation of about 0.61 and an IoU of about 0.53. For English, TÜRKSAT achieved a correlation of approximately 0.55 and an IoU of 0.49. Even the best systems failed to overlap with human annotations across nearly half of the hallucinated spans, indicating that these detectors are not yet suitable for direct use as reliable output gates.
The paper also used 25,000 bootstrap iterations to assess ranking stability and found that several pairs of adjacent top systems had stable win probabilities of only about 0.35 to 0.47. Small differences in rank should therefore not be treated as definitive evidence of superiority. From an engineering perspective, it is more useful to compare failure modes across languages, error categories, calibration levels, and annotation strategies. Future work should examine whether the test data will be released, whether participating methods can be reproduced, and whether detectors remain calibrated when applied to newer generations of VLMs, longer responses, and images of real-world documents.