機器人與世界模型
CLAP Unifies Action Spaces Across Heterogeneous Robots, Using a Video World Model as a Cross-Embodiment Simulator
CLAP first learns physical dynamics from unlabeled videos through latent actions, then maps the representation back to robot end-effector coordinates. The research team has released the code and multiple checkpoints, but physical hallucinations in generated videos still limit its direct use in safety-critical control.
The newly released CLAP aims to address the difficulty of sharing robotic video world models across hardware platforms. Franka, WidowX, dual-arm robots, and humanoid robots all differ in their degrees of freedom, joint definitions, and control interfaces, while large volumes of human video lack action annotations altogether. The team connects these data sources through three representations: a seven-dimensional end-effector pose, natural-language actions, and 32-dimensional latent actions learned from adjacent frames.
Training follows a two-stage curriculum. The model first uses latent actions to absorb spatiotemporal dynamics from human videos and videos of multiple robot types, then continues training with end-effector conditioning that can directly control robots. The video-generation backbone is built on Stable Video Diffusion and predicts post-action futures from historical frames, robot states, and multi-view imagery. This design combines the scale of unlabeled video with the precision of geometric control signals, avoiding the need to translate abstract latent vectors into executable commands at deployment time.
The authors trained on datasets including Open X-Embodiment, EgoDex, DROID, and Bridge. Each experiment used eight H100 or H200 GPUs for approximately two to three days. The report shows that the cross-embodiment model matches the single-embodiment baseline using the same backbone on DROID. Compared with a baseline that extracts actions only from human videos, some LPIPS metrics improve by at least 61%. The experiments also find that absolute end-effector coordinates are less prone to error accumulation: on DROID, LPIPS is approximately 14.6% better than with relative coordinates. Language conditioning, however, is better suited to a more compact relative representation.
The open-source package includes checkpoints for cross-embodiment models, DROID, Bridge, the dual-arm YAM robot, and the G1 humanoid robot, along with examples for replay, keyboard control, and policy-in-the-loop operation. Its engineering value lies in using video to simulate multiple candidate actions before sending one to a physical robot. However, most evaluations use author-controlled data and roughly one hundred trajectories, and the paper itself acknowledges that the model can still generate physically implausible hallucinations. The next priorities should be independent reproduction, inference latency, long-horizon error, and uncertainty calibration—not treating visually realistic predictions as reliable simulations by default.