Original title: Beyond Retrieval: Progressive Latent Memory Evolution for Streaming Video Understanding
Hongyu Qu, Guangming Yao, Ling Xing, Xiaobin Hu +6 · September 2026
Streaming video AI systems need to watch ongoing video and answer questions while using limited memory. Most existing models store past video frames in an outside bank and pull pieces back out when asked a question. This keeps past evidence outside the model instead of turning it into a working memory that can grow and guide ongoing reasoning. To solve this, researchers built LatentStream. It groups past video into short-, mid-, and long-term pools within a set memory limit. When a question arrives, special memory units look across wider scopes of video history and pull evidence into a small, fixed-size internal memory. A special training process then uses confidence scores to refine this internal memory so the system reasons more steadily. The researchers found that LatentStream sets new state-of-the-art results on online and offline video tests.
Imagine you're keeping a diary of a long trip. Each day you write notes, but after weeks, the diary is too thick to search through. Instead of flipping pages, you write a single summary page that updates itself whenever you need to remember something specific. Here, the system does something similar with video: it condenses history into a compact memory that evolves as questions come in.
This matters for real-time video analysis in applications like security or autonomous driving. It lets AI handle long videos without running out of memory, making systems more reliable and efficient.
The abstract does not discuss computational overhead, scalability to extremely long videos, or performance in noisy real-world conditions. It presents results on specific benchmarks without addressing generalizability or resource costs.
Ask anything about this paper. Answers come from its abstract, and say so when the abstract doesn't cover your question.
Beyond Retrieval: Progressive Latent Memory Evolution for Streaming Video Understanding — Hongyu Qu, Guangming Yao, Ling Xing et al.
Abstract licence: CC0
Explanation generated by Synthesis from the abstract — it is not the original paper. This is a preprint, so it hasn't necessarily been peer reviewed.