Scene graphs and logic formulas let self-driving cars find dangerous moments in video.

Self-driving cars need to handle dangerous situations safely. Finding these critical events in huge piles of unlabelled driving footage is difficult and slow. Researchers created a new system to automatically find and extract these moments. First, the system takes sensor data from the car and high-definition maps. It turns this information into a series of scene graphsscene graphsDiagrams that represent objects and their relationships in a scene at a specific time., which are diagrams that map out objects and their relationships over time. Next, users describe the exact scenario they want to find using a formal language called Linear Temporal Logic. A computer program then checks the graphs against that description to pull out every matching moment. The researchers tested it on real-world driving datasets and showed that it effectively finds these driving scenarios.

Why it matters

Developers of automated driving systems need to test their cars on rare but critical events. Automatically finding these events in large video datasets saves time and improves safety validation.

Questions answered