Original title: Zero-Shot Novel Depth Synthesis Using 3D Foundation Models Scene Representations
Denis M. Akola, David F. Fouhey · September 2026
Computer models often struggle to imagine what a 3D scene looks like from angles they have not seen. The researchers wanted to fix this by checking how much 3D foundation models, such as VGGT, already understand about the world. They hypothesized that these systems store general knowledge about entire spaces to rebuild them. First, the team showed that they could pull out hidden surfaces directly from the model's internal data. Then, they built a new method named Z3D. Z3D runs latent diffusion directly on the model's inner representations to calculate pointmaps for brand new viewpoints. The researchers found that Z3D successfully predicts realistic depth maps from novel angles across multiple datasets.
Imagine you're trying to describe how a room looks from the doorway, but you can only see it from one corner. You remember similar rooms you've been in before, so you can guess the layout. Here, the AI model uses its memory of many 3D scenes to infer depth from new viewpoints without having seen them directly. It's like using past experiences to fill in the blanks for a new perspective.
This could let industries like gaming and robotics create accurate 3D models without needing extra scans. It means systems can adapt to new environments faster by using what they already know.
The abstract does not state specific limitations of the method. It mentions that Z3D predicts depth maps across multiple datasets but does not discuss any constraints or failure cases.
Ask anything about this paper. Answers come from its abstract, and say so when the abstract doesn't cover your question.
Zero-Shot Novel Depth Synthesis Using 3D Foundation Models Scene Representations — Denis M. Akola, David F. Fouhey
Abstract licence: CC0
Explanation generated by Synthesis from the abstract — it is not the original paper. This is a preprint, so it hasn't necessarily been peer reviewed.