Local depth stays accurate while global camera position drifts in long videos.

Building 3D scenes from long videos in real time is hard. Models often calculate camera positions by measuring everything back to the very first frame. Over time, tiny errors pile up, causing the 3D shapes to collapse entirely. Yet, researchers noticed that single-frame depth estimates still work fine, showing only the camera tracking breaks down. To fix this, they built Scal3R. Instead of referencing just the first frame, Scal3R compares positions against multiple past keyframes. It does this by adding small learnable tokenslearnable tokensSmall adjustable parameters that the system tweaks during training to improve performance.—about ~1% of the model parameters—into a frozen core network using asymmetric attentionasymmetric attentionA neural network mechanism that focuses differently on different parts of the input data.. An online pose graph also corrects drift over time. Scal3R cuts position errors on the KITTI dataset by over 60% compared to the online baseline.

Why it matters

This helps devices that need to map 3D environments in real time, like robots or self-driving cars. By fixing drift on long videos, it makes these systems more reliable over extended use.

Questions answered