GUI 代理／世界模型
AppDeltaWorld Predicts Mobile Interfaces with Executable HTML, Raising MobileGym Success Rate from 10.2% to 14.1%
Rather than directly generating the next screenshot, AppDeltaWorld first retrieves layouts reachable through the given action, then generates HTML diffs and fills them with synthetic images. It can create closed-loop training trajectories, but its current results still depend on a custom evaluation framework and a full implementation that has not been released.
AppDeltaWorld aims to address the lack of safe, repeatedly operable training environments for mobile GUI agents. Unlike approaches that directly predict pixels or textual descriptions, it represents the next interface state as an HTML update executable by a browser. The system first maps the current screen to an interface cluster within the app, then queries an action-transition index based on the target of a tap, long press, or swipe. It retrieves a Level-1 HTML skeleton only from candidate screens that were historically reachable, after which the model generates Level-2 HTML containing the actual text, components, and positions.
For product images or video thumbnails that are difficult to express in code, the system reserves image slots, uses a diffusion model to generate the assets, and finally renders the result in a browser. This hybrid design combining text, code, and images is intended to reduce layout drift, errors in text-dense interfaces, and impossible state transitions caused by unconstrained generation. If the transition index cannot find a target cluster matching the current state and action, the system can reject the prediction or assign it lower confidence.
The authors trained the world model on 100,149 GUI transition steps and used GUI-Owl and OpenMobile seeds to build 33,133 AppDelta trajectories. AppDeltaWorld achieved an overall score of 73.51 on CMGUIBench-500, slightly higher than GPT-Image-2's 72.34. Its component and layout scores were higher, although it did not consistently lead in functional logic. After the synthetic trajectories were added to supervised fine-tuning, AppDeltaAgent's success rate on MobileGym rose from the Qwen3-VL-8B baseline of 10.2% to 14.1%, while its GUI-only success rate on MobileWorld increased from 9.4% to 14.9%.
The technical value lies not only in generating screens, but also in enabling a model to take consecutive actions within the same world model, filter out failed transitions, and use the retained trajectories for SFT or test-time reinforcement learning. However, the overall score of 73.51 combines a Gemini judge, SigLIP, DINOv2, and layout metrics, so it may not reflect causal consistency in real applications. Average output length also reaches 8,309 tokens, higher than that of most code-only baselines. Engineering teams should next watch whether the complete codebase, index-building tools, and model weights are released, and whether the system can maintain correct transitions amid interface updates, login states, and server-side side effects.