電腦視覺
WorldSculpt Decomposes Scenes with Hundreds of Objects Using a Single-Object Generative Prior, but Still Relies on Masks and Camera Poses
Alaya Lab and the University of Tokyo have introduced WorldSculpt, which generates each object in multi-view images as a complete, independently editable mesh before assembling the scene through rigid transformations. Although the model is fine-tuned only on single-object data, it can handle test environments containing 93 to 701 objects; however, it still requires reliable instance masks, 3D bounding boxes, and camera poses as input.
Most neural 3D reconstruction methods represent a room or street scene as a single NeRF, 3D Gaussian Splatting model, or unified mesh. While the visual results can be realistic, directly moving, replacing, or simulating an individual object remains difficult. WorldSculpt instead handles dense scenes by “generating objects individually and assembling them afterward”: its inputs are RGB images with camera poses, per-object instance masks, and coarse 3D bounding boxes, while its output is a collection of independent meshes positioned in a shared world coordinate system.
The system is built around Pixal3D’s single-object generative prior. For each object, it first constructs a canonical cube based on the anchor view with the highest visibility. DINOv3 extracts features from cropped images across views, which are then projected onto the same voxel grid. A permutation-invariant, IBR-style aggregator assigns residual weights across different views and voxels. The fused result is injected through zero-initialized projection layers and rank-32 LoRA adapters into Pixal3D’s two-stage DiT: the first stage predicts a sparse occupancy structure, while the second generates detailed geometry. The completed canonical mesh is then placed back into world coordinates using a known rigid transformation, eliminating the need for additional inter-object fusion or per-object ICP.
Notably, training uses only single-object data from TexVerse, with 1 to 20 views sampled randomly for each example, and does not involve joint training on complete scenes. The team also created UE-MeshyScene, comprising six Unreal Engine 5.8 environments with a total of 2,299 objects and between 93 and 701 objects per scene. The paper reports that WorldSculpt outperforms the multi-view baseline ShapeR on every reported metric. Ablation experiments on UE-MeshyScene show that replacing simple averaging with learned IBR aggregation reduces Chamfer Distance L2 by 12% and raises the F-score from 0.944 to 0.951.
However, this is not a complete pipeline for directly converting arbitrary videos into editable worlds. Masks, camera poses, and coarse localization boxes must still be supplied by upstream systems, while severely misaligned boxes, incorrect masks, low light, or motion blur can all degrade the generated results. The benchmark’s ground-truth meshes also cannot be downloaded directly because of copyright restrictions; researchers must contact the team for evaluation. Key next steps include end-to-end reproduction on real handheld videos, measuring per-object inference costs, and determining whether detection, tracking, localization, and generation can be jointly optimized.