電腦視覺
Scal3R Fixes Drift in Long-Video 3D Reconstruction With Multi-Reference Pose Queries, Training Only About 1% Additional Parameters
Scal3R retains a frozen CUT3R or STream3R backbone, using lightweight query tokens to estimate relative poses against multiple historical keyframes before passing them to online pose graph optimization. The authors report a mean ATE on KITTI more than 60% lower than the strongest online competitor, but the results use Sim(3) alignment and therefore should not be interpreted as preserving true scale.
Feed-forward 3D reconstruction systems for long videos often treat the first frame as a fixed coordinate anchor and have subsequent frames regress global camera poses. The longer the trajectory, the farther the model must extrapolate beyond its training distribution, allowing small errors to eventually cause both the trajectory and point cloud to collapse. Researchers from National Yang Ming Chiao Tung University and NVIDIA introduce Scal3R and begin by analyzing where these failures occur: even as global poses continue to drift, per-frame depth typically remains stable, suggesting that the entire geometric backbone does not need to be retrained.
Scal3R therefore freezes CUT3R or STream3R and adds projection layers, pose query tokens, and a relative pose head accounting for only about 1% of the system’s total parameters. Its asymmetric attention allows the query tokens to read image features while preventing image tokens from attending back to the new tokens, avoiding disruption to the existing point-cloud representation. For each new frame, the system estimates transformations relative to multiple historical keyframes, then integrates them into a global trajectory through incremental pose graph optimization and loop-closure detection. The implementation uses GTSAM iSAM2, DINOv2-SALAD, and FAISS. The [paper](https://arxiv.org/abs/2609.04201) reports convergence in about eight hours on a single GPU. On KITTI, the CUT3R-based version achieves a mean ATE of 69.7, compared with 182.2 for TTT3R. On vKITTI, increasing the number of reference frames from 4 to 12 reduces ATE from 15.75 to 5.63. The complete CUT3R-based pipeline runs at 14.4 FPS, versus 15.9 FPS for the original backbone.
These results suggest that reformulating long-range global regression as multiple short-range constraints may be more effective than scaling up the model, with particular relevance to robotics, automotive vision, and long-duration AR scanning. However, the figures shown on the [project page](https://linjohnss.github.io/scal3r/) remain author-reported evaluations. KITTI ATE is computed after Sim(3) alignment to ground truth, which removes global scale error; in the appendix, calibrated ORB-SLAM2 also achieves a lower mean ATE than Scal3R. Key next steps include determining whether the released code and weights can fully reproduce the results, evaluating performance without scale alignment, and testing long-term stability with dynamic objects, low-texture scenes, and trajectories without loop closures.