多模態與世界模型
Puffin-World Brings Gravity, Depth, and Images Into a Single World Model, Directly Generating Multi-View RGB-D and Point Clouds
Puffin-World uses a nine-channel Omni-Camera condition to unify absolute physical orientation with relative camera motion, while handling understanding, generation, and 3D reconstruction within a single architecture. The code, weights, and Puffin-16M dataset have been released, but its physical state modeling is currently focused mainly on gravity and latitude in static scenes.
Researchers from Nanyang Technological University, the University of Michigan, Beijing Jiaotong University, and ACE Robotics have released the Puffin-World paper. Rather than representing the world solely as continuous RGB frames, it explicitly models three types of state jointly: physical orientation through gravity fields and latitude maps, geometry through depth maps, and appearance through images. Given a single image or text prompt and a target camera trajectory, the model can jointly generate multi-view RGB-D and integrate the results into a colored point cloud at a consistent coordinate scale, without requiring a separate offline depth-estimation or reconstruction module.
The architecture consists of a geometry-aligned vision encoder, an LLM, a diffusion model, and lightweight connectors. Its key component, Omni-Camera, is a nine-channel per-pixel conditioning representation: a 2D up vector and latitude angle form a three-channel absolute field, while ray origins and directions form a six-channel relative geometry representation. The former provides the “real-world up” direction and a horizon anchor, while the latter encodes camera intrinsics, translation, and rotation. The model then propagates the reference frame’s gravity direction to subsequent views according to their relative rotations, reducing horizon drift during long trajectories or large camera rolls.
The Puffin-16M training dataset contains 15 million vision-language-camera samples and 1 million trajectories, covering pitch, roll, full 360-degree yaw, and compound motions. The team also generated camera annotations for approximately 44.5 million images across 28 public datasets. The authors report that the model achieved the best results on all 12 median-error metrics across four monocular camera-estimation datasets, as well as a PSNR of 17.22 and an LPIPS of 0.318 on RealEstate10K. However, these comparisons were conducted primarily by the authors, and some of the new benchmarks were also created by the same team. More importantly, “physics” currently covers only gravity, latitude, and static geometry; the model does not yet handle collisions, materials, object dynamics, or long-term causal evolution. Its future value will depend on independent reproduction, evaluation on dynamic scenes, and the terms of the NTU S-Lab License 1.0, which may restrict commercial use.