Original title: Seeing Before Synthesizing: VLM-Guided Transition Event Discovery for Weakly-Supervised Dense Video Captioning
Ye-Chan Kim, Seunghee Choi, SeungJu Cha, Si-Woo Kim +3 · September 2026
Computers struggle to find and describe every event in long videos when they only have an ordered list of captions to learn from. Past methods used large language models to guess transition descriptions for the gaps between events. However, these guesses were not tied to actual video frames. They also forced transitions into every single gap at fixed times. To solve this, researchers built Seeing Before Synthesizing, or SBS. SBS inspects the video first using a vision-language model. It creates frame-by-frame descriptions of the gaps to spot semantic shifts, discovering transitions only when they actually happen. It then tunes the gap boundaries using both timing and meaning to match words to video. Testing on ActivityNet Captions and YouCook2 showed state-of-the-art results for describing and localizing events.
Imagine you're organizing photos from a vacation, but you only have captions like 'Beach day' and 'Mountain hike' without timestamps. You need to figure out when the beach photos end and the mountain photos start. Normally, you might guess based on the order, but if you look at each photo's content—like the change from sand to trees—you can pinpoint the exact transition. In this paper, the system looks at descriptions generated for each moment in the video to find where the content shifts, allowing it to accurately mark the transition points instead of guessing fixed intervals.
This helps computers automatically break down long videos into key moments and describe them accurately, which could improve video search, editing tools, or accessibility for visually impaired users.
The abstract doesn't say, but the method is evaluated only on two datasets, so it might not generalize to all types of videos. Also, it relies on a vision-language model, which could introduce errors if the model misinterprets frames.
Ask anything about this paper. Answers come from its abstract, and say so when the abstract doesn't cover your question.
Seeing Before Synthesizing: VLM-Guided Transition Event Discovery for Weakly-Supervised Dense Video Captioning — Ye-Chan Kim, Seunghee Choi, SeungJu Cha et al.
Abstract licence: CC0
Explanation generated by Synthesis from the abstract — it is not the original paper. This is a preprint, so it hasn't necessarily been peer reviewed.