GUI 代理
Layout Matching Replaces Direct Coordinate Generation, Pushing GUI Grounding to 41.3% on ScreenSpot-Pro
Peking University researchers have multimodal models interpret only the user’s operational intent, then use Text/Icon candidate detectors and a frozen CLIP model to select the click region. Accuracy on ScreenSpot-Pro rises from the strongest listed baseline’s 18.9% to 41.3%. The paper’s claim of being “regression-free” applies only to the final matching stage: the candidate-box detector still uses bounding-box regression losses, and no implementation has yet been released.
The newly proposed Layout-Aware GUI Grounding Model does not ask a large multimodal model to output click coordinates directly. In the first stage, a frozen Qwen2.5-VL-72B jointly reads the screenshot and instruction, expanding an intent such as “close File Explorer” into a visual description covering text, appearance, and relative position. In the second stage, a DINO detector adapted to GUI data generates Text and Icon candidate boxes. A frozen CLIP ViT-B/32 then computes cosine similarity between the description and each cropped region, directly selecting the highest-scoring candidate box. This decoupling lets the language model handle abstract intent while specialized models perform pixel-level recognition, avoiding answers that provide plausible-looking but nonexistent coordinates. The [paper](https://arxiv.org/abs/2608.09654) also projects each candidate box’s center, width, height, aspect ratio, and area into a spatial offset that is injected into the CLIP features, helping distinguish position-sensitive instructions such as “the back button in the upper-left corner.”
The training data comes from roughly 200,000 GUI screenshots. Parsing tools first extract text and icon regions, after which Qwen3-VL generates region descriptions. DINO was trained for 50 epochs on eight RTX 3090 GPUs. In the second stage, CLIP remains frozen, and only a single-layer geometric projection is trained. The authors report an average click accuracy of 87.1% on ScreenSpot, compared with 73.3% for UGround. On ScreenSpot-Pro—which covers 23 professional applications and contains 1,581 instruction-annotation pairs—the model reaches 41.3%, versus 18.9% for the strongest listed baseline, OS-Atlas-7B. The [official benchmark repository](https://github.com/likaixin2000/screenspot-pro-gui-grounding) can be used to verify the dataset and evaluation protocol.
The term “hallucination-free” should still be interpreted cautiously. The system only avoids free-form coordinate generation in the final stage. The preceding DINO candidate generator still learns bounding-box locations using L1, GIoU, and classification losses; if the target is absent from the candidate set, CLIP cannot recover it. The evaluation also primarily measures whether the predicted point falls within the annotated box, which does not establish that a multi-step agent can complete tasks reliably. The paper provides no code, models, or newly created training data, and some baseline results in its tables still require independent reproduction. Engineering teams should next examine candidate recall, additional inference latency, and whether a smaller intent-parsing model can preserve the same gains.