Original title: What, Where, and How: Probing Spatiotemporal Representations in Video Foundation Models
Sharon S. Musa, Fereshteh Forghani, Harrish Thasarathan, Sonia Joseph +2 · September 2026
Computer vision models can learn from video without human labels, but we do not fully understand what they learn about the physical world. Researchers studied two models, V-JEPA 2 and VideoMAE-v2, to see what video information they store across their layers. They trained small test tools called probes to check for camera movement, intuitive physicsThe ability to understand basic physical principles like gravity and object permanence., and unusual events. Both models track camera movement well, peaking around 60-70% of their depth, and detect anomalies moderately well. However, both score near chance on intuitive physics, showing they lack deep physical reasoning. The team also discovered that video features trace smooth paths in the models. Using curved paths to steer internal features created smoother video camera motions than simple straight-line methods.
Imagine watching a video shot from a moving car, where the camera pans across a cityscape. If you try to create a middle frame by simply averaging the start and end frames, the result might look jerky because it ignores the curved path the camera took. In this paper, researchers found that the AI's internal understanding of videos moves along similar smooth paths. By steering along these paths instead of using straight-line blending, they produced more natural video transitions.
This research clarifies what video AI models actually learn, which is crucial for improving their reliability in applications like video editing and autonomous systems. It shows that while models handle camera movement well, they struggle with physical reasoning, pointing to areas needing better design.
The study only examines two specific video foundation models, so findings may not generalize to all such models. It does not explain why intuitive physics is poorly encoded, suggesting a gap in deeper physical reasoning. The geometric steering is applied only to camera motion, not to other properties like physics or anomalies.
Ask anything about this paper. Answers come from its abstract, and say so when the abstract doesn't cover your question.
What, Where, and How: Probing Spatiotemporal Representations in Video Foundation Models — Sharon S. Musa, Fereshteh Forghani, Harrish Thasarathan et al.
Abstract licence: CC0
Explanation generated by Synthesis from the abstract — it is not the original paper. This is a preprint, so it hasn't necessarily been peer reviewed.