機器人世界模型
DreamX-Phi Constrains Dual-Arm Video Prediction with SE(3) Trajectories, Temporarily Ranks First on WorldArena 2.0
DreamX-Phi injects the rigid-body transformations of two robotic arms directly into attention and constrains video generation using depth, object masks, and V-JEPA features. It scored 60.65 on a WorldArena 2.0 snapshot, but its weights and inference code will not be released until the challenge concludes, and generalization to real robots remains unverified.
The challenge for robotic video world models is not merely whether the visuals look realistic. When a candidate action changes, the predicted video must actually move the correct robotic arm along the specified path while preserving the state of the small object being manipulated. Built on Wan2.2-TI2V-5B, DreamX-Phi 1.0 takes an initial frame, a language instruction, and a dual-arm action sequence containing end-effector poses and gripper states, then predicts future observations.
Instead of compressing actions into generic control tokens, the system uses PRoPE-style geometric encoding to inject each arm’s SE(3) rigid-body transformations into attention, preserving arm identity and three-dimensional motion structure. It also uses robot-only flow to indicate where actions occur in the image. During training, a lightweight depth branch constrains scene geometry, SAM3 masks increase the loss weight for small manipulated objects, and a frozen V-JEPA teacher aligns object representations across frames. Finally, distribution-matching distillation compresses the multi-step video generator into a few-step version.
In the WorldArena 2.0 snapshot fixed on August 12, DreamX-Phi ranked first among 31 Track 1 entries with an EWMScore-P of 60.65, including a trajectory accuracy score of 57.15. In Track 2, its policy achieved a 67.19% success rate on Adjust Bottle, tying for second place. It also scored 76.88 in WorldArena 1.0 offline testing, 3.24 points above the official leader in that snapshot. These results suggest that geometrically structured action conditioning may be better suited than generic tokens to video simulators requiring precise control.
For now, however, the GitHub repository contains documentation only. The model weights and inference code will not be released until the IROS challenge ends, and the leaderboard reflects only an interim snapshot. Testing is limited to WorldArena and RoboTwin, Track 2 covers just one task, and no matched ablation study isolates the contribution of each module. DreamX-Phi also predicts only the outcomes of externally supplied actions; it does not plan actions itself and therefore cannot yet be considered a closed-loop robotic controller.