多模態評測
Calling a Cropping Tool Does Not Mean a Vision Model Uses the Evidence; Gains Across Six Models Concentrate in a Few Effective Trajectories
A new study uses counterfactual cropping and observation-corruption tests to determine whether visual tools truly affect answers, finding that some models merely “call without looking.” Across five fine-grained benchmarks, the overall gains of six models are driven mainly by a small number of well-calibrated trajectories.
Multimodal models have begun proactively cropping and zooming into images during reasoning, but tool-call logs alone do not prove that a model uses the returned visual content. A team including researchers from the Shanghai AI Laboratory introduced CauAudit, which decomposes the visual-tool workflow into three nodes—tool action, returned observation, and final answer—and performs causal interventions at the policy, full-trajectory, and single-step levels.
At the policy level, the study compares tool-enabled answering with direct answering. At the trajectory level, every returned image is replaced with a random crop. At the step level, a single observation is replaced while prior reasoning is held fixed, and Visual Evidence Gain is calculated to test whether the new visual content actually changes the model’s answer preference. The study evaluates DeepEyes, Pixel Reasoner, Mini-o3, Qwen3-VL 4B/8B, and Thyme on five fine-grained visual benchmarks.
The results reveal two types of mismatch. “Calling Without Looking” occurs when a model invokes a tool but the observed content has almost no causal effect on its answer. “Looking Without Planning” occurs when the visual content is genuinely useful, but the model continues cropping after it already has the answer or keeps going until it exhausts its tool-call budget. On V*, for example, 84.8% of Mini-o3 trajectories and 59.2% of Qwen3-VL-8B trajectories hit the tool limit. After dividing trajectories into four groups—no tool calls, the two failure modes, and Calibrated—the researchers found that Calibrated was the only group to deliver consistent positive gains across all four fully evaluated models. Qwen3-VL-8B achieved an overall gain of 6.9 percentage points, with 7.6 points coming from this group; the remaining groups collectively offset part of that benefit.
The implication for agent training and evaluation is that engineering teams cannot rely solely on tool-usage rates or final accuracy. They must also test whether answers are sensitive to tool-return values and track when models stop using tools. The authors hypothesize that reinforcement learning based only on final outcomes may reward both effective and ritualistic tool calls, though this has not yet been verified through controlled training experiments. Although the paper states that the project is open source, its GitHub repository currently contains only a README and license file. Independent reproduction of the data-processing and intervention pipelines must therefore wait until the complete code is released.