AI 安全
MissClick Exploits Coordinate Token Place Values to Attack GUI Agents, Achieving Targeted Misclick Rates of Up to 62.67%
Rather than treating GUI coordinates as ordinary text, MissClick exploits small changes to hundreds-place tokens to cause large shifts in click positions. The study significantly increased attack success rates against OS-Atlas and UGround, showing that the security of computer-use agents cannot be evaluated solely by element-localization accuracy.
GUI agents typically have a vision-language model output `(x, y)` coordinates, which an executor then converts into actual mouse or touch events. MissClick argues that this seemingly simple serialization interface creates a distinct attack surface: the model predicts decimal numbers token by token, while the parser reconstructs coordinates according to place value. Changing a single hundreds-place token can shift the actual click position by 100 normalized units—far greater than the semantic impact of a typical change to a text token.
The researchers designed two types of white-box image perturbations. MissClick-U uses differentiable “soft coordinates” to maximize the distance between the predicted click point and the correct region, with the goal of making the agent click outside the intended element. MissClick-T instead weights the loss for target digits according to their decimal place values, directing the click toward an attacker-specified region. The latter means an attack can do more than merely cause a task to fail: it could turn “Confirm” into “Delete,” or redirect an ordinary link to an authorization or payment element.
The authors tested OS-Atlas and UGround across desktop, web, and mobile interfaces. MissClick-U achieved untargeted attack success rates of 75.07% and 72.93%, respectively—improvements of 16.62 and 30.72 percentage points over existing attacks. MissClick-T achieved targeted success rates of 44.86% and 62.67%, improvements of 31.73 and 47.06 percentage points. OS-Atlas’s public implementation normalizes coordinates to a 0–1000 range, directly illustrating how place-value errors are mapped back onto the original screen.
These findings have direct implications for protecting browser and desktop agents: an output passing JSON Schema validation or coordinate range checks does not mean the resulting action is safe. The execution layer should revalidate coordinates against on-screen element boundaries, add semantic confirmation for sensitive controls, and prevent a single perturbed frame from directly triggering irreversible operations. One limitation is that MissClick assumes white-box access to model gradients; it has not yet demonstrated comparable success rates through remote APIs, screenshot compression, or transfer to unknown models. The paper also does not include public attack code, so its reported results still require independent verification.