Original title: FLY-EVAL++: An Evidence-Driven Evaluation Protocol for Safety-Constrained Flight Prediction with Large Language Models
Yalun Wu, Junfeng Fang, Jiawei Wang, Haotian Liu +5 · September 2026
Testing large language models on airplane flight paths is tricky. An answer can look close to correct on paper but still break safety rules, defy the laws of physics, or come out in the wrong format. Current evaluation tests fail to catch these dangerous flaws. To solve this, researchers built FLY-EVAL++. This system tests flight predictions by strictly checking whether models follow formatting protocols, obey physical laws, and follow safety rules. It then scores models across clear, separate categories. Testing 66 models revealed that safety is where they differ most. Even when models scored similarly on basic accuracy, their safety scores differed by more than 28 points. Models also made dangerous safety mistakes during physically realistic paths and broke down during multi-step forecasts.
Imagine ordering a meal at a restaurant. You expect it to taste good and look presentable. Now suppose the chef focuses only on taste and ignores how the dish is plated, so it arrives messy even though it's delicious. In this paper, LLMs predicting flight paths might be accurate but violate safety rules, so the evaluation protocol checks both accuracy and safety separately to ensure the predictions are usable and safe.
When AI models assist in flight systems, inaccurate safety checks could lead to real-world accidents. This protocol helps identify hidden dangers that accuracy scores overlook, improving aviation safety.
The abstract doesn't specify limitations, but the protocol is designed for flight trajectory and attitude prediction tasks. It may not generalize to other safety-critical domains or address all real-world implementation challenges.
Ask anything about this paper. Answers come from its abstract, and say so when the abstract doesn't cover your question.
FLY-EVAL++: An Evidence-Driven Evaluation Protocol for Safety-Constrained Flight Prediction with Large Language Models — Yalun Wu, Junfeng Fang, Jiawei Wang et al.
Abstract licence: CC0
Explanation generated by Synthesis from the abstract — it is not the original paper. This is a preprint, so it hasn't necessarily been peer reviewed.