多模態評測
PerceptionBench Breaks Visual Reasoning Back Down into Ten Basic Perceptual Skills; None of 16 Multimodal Models Scores 60%
Moonshot AI built a taxonomy of perception failures from the earliest points of failure across 42 existing benchmarks, then used 3,000 short-answer questions to isolate ten basic visual capabilities. The results show that models with similar overall scores can have entirely different perceptual gaps, but because the test set was drawn from a larger internal question pool, contamination and evaluator bias remain concerns.
Multimodal benchmarks often combine image understanding, background knowledge, and multi-step reasoning in a single question. When a model answers incorrectly, developers struggle to determine whether the problem lies in the visual encoder, coordinate localization, or language reasoning. Moonshot AI's publicly released [PerceptionBench paper](https://arxiv.org/abs/2607.24957) instead starts at the beginning of the error chain: the research team analyzed the first point at which models failed across 42 existing benchmarks, organized the failures into ten categories of atomic perceptual capabilities, and gave each question a short, unambiguous answer to minimize reliance on external knowledge and complex reasoning.
The public dataset contains 3,000 questions. Of these, 1,800 are atomized subquestions derived from existing failure cases, while another 1,200 were rewritten using supplementary images. The questions cover capabilities including text recognition, object and attribute recognition, spatial localization, counting, pose estimation, and fine-grained visual discrimination, with stratified sampling by category and difficulty. The value of this design is not another overall leaderboard, but a capability profile: two models with similar aggregate scores may fail at small-text recognition and spatial localization, respectively, requiring different data fixes and architectural changes.
The team evaluated ten closed-source and six open models using a standardized prompt and the highest reasoning budget available for each model. No model achieved 60% accuracy, and items related to perceptual hallucinations had the lowest average performance. This suggests that longer reasoning or adding chain-of-thought may not repair signals that were not reliably extracted at the input stage—and may even wrap incorrect observations in more fluent explanations.
The complete questions, statistics, and leaderboard are available in the [Hugging Face dataset](https://huggingface.co/datasets/moonshotai/PerceptionBench), and the official [technical overview and download page](https://www.kimi.com/tr/blog/perception-bench) is also available. Engineering teams can use the ten capability categories as regression-test slices instead of focusing only on average scores. However, the 3,000 public questions were sampled from an internal pool of more than 17,000 items, and the question-validation process and human baselines still require further scrutiny. Once models have been exposed to the public test set, scores from subsequent versions may also be affected by training contamination.