具身 AI／評測
GAUGE Tests Simulators and Video World Models in 1,560 Physical Experiments, With No Single Engine Winning Across the Board
GAUGE puts physics engines and generative world models through the same real-world benchmark, diagnosing errors separately in trajectories, equation forms, and physical parameters. The results show that visually plausible output—or even conformity to the correct equation form—does not guarantee accurate acceleration, momentum transfer, or oscillation periods.
GAUGE places numerical simulators commonly used in embodied AI and generative video models into a shared diagnostic framework backed by physical measurements for the first time. The dataset includes 22 categories of controlled tasks and approximately 1,560 motion-capture experiments, covering rigid-body collisions and friction, ropes, cloth, and volumetric deformable materials such as foam. The researchers also calibrated coefficients of friction, Young’s modulus, and Poisson’s ratio using methods including inclined-plane tests, material compression, and digital image correlation, avoiding the use of another simulator as the sole “ground truth.”
For physics engines, the study compares Isaac Sim, Genesis, and Newton across 14 task categories, using task-specific metrics such as trajectory RMSE, Dynamic Time Warping, momentum transfer efficiency, oscillation period, and energy loss. The results reveal no universal winner: Isaac Sim is more accurate on some rigid-body contact and rotating-platform tasks; Genesis is more competitive for dynamic cloth and most deformable-body tasks; and Newton leads in certain deformation scenarios. Impact contact, rapid cloth motion, and volumetric deformation remain shared weaknesses, suggesting that robot policies may learn the errors of a particular engine rather than transferable laws of physics.
A separate evaluation uses five rigid-body tasks to assess six generative systems, including Cosmos3, Wan, Seedance 2.0, and Genie 3. GAUGE tracks objects in generated videos and separately checks whether their trajectories follow the expected equations, whether inferred parameters are close to physically measured values, and whether those parameters remain stable over time. Although some videos can be fitted well by equations with the correct form, their estimated acceleration, collision momentum, or pendulum period is still substantially wrong. Adding negative prompts can also improve performance on one task while degrading another.
From an engineering perspective, the findings caution teams against selecting training environments based solely on visual quality or a single sim-to-real success rate. The next steps are to determine whether the observations, scene-reconstruction code, and complete evaluation pipeline will be released, and whether the rankings can be reproduced across different engine versions, solver parameters, and video sampling rates. At this stage, the video models have been tested on only five categories of rigid-body scenes, so the conclusions cannot yet be directly extended to cloth, soft bodies, or real-world robot control.