Original title: Beyond Static Guarantees: Measuring the Static-Pass Dynamic-Fail Gap in Security-Sensitive and LLM-Generated Python Code
Jessica Pourleyli, Maitreyee Das Urmi, Glaucia Melo · September 2026
People often use static analysisChecking code for bugs without running it, like a spellcheck for programs. to check software for security flaws. Static analysis means scanning code without actually running it. However, tools that only look at code miss flaws that appear when programs run, such as attacks that rely on tricky inputs or the program's environment. Many people assume code is safe if static checkers raise no red flags. To test this, researchers created a three-stage pipeline. It first runs static scanners, uses an AI model to spot potential security weaknesses, and then tests those weaknesses by running real attacks inside isolated virtual environments called Docker containers. The researchers tested Python code that passed the static scans. They discovered that static tests gave a false sense of safety: about 1 in 7 statically clean samples were actually exploitable during dynamic testing. Common flaws were completely missed by the initial scans, showing that scanning code is not enough to prove it is safe.
You are checking a house for security before vacation. You walk through every room and ensure all windows are locked, but you never try to actually break in. A locked window can sometimes be forced open by a clever intruder. In this paper, the static scan is that walkthrough—only checking rules—and dynamic verification is actually trying the window, showing that even code passing static checks can still be exploited.
Developers trust static analysis as a cheap security gate, but this work shows that passing it does not guarantee safety. One in seven samples that looked clean actually contained exploitable bugs. For AI-generated code and security-sensitive software, this means that static checks alone are insufficient; runtime verification is necessary to catch real-world exploits.
The abstract does not explicitly list limitations, but notes that the 14.53% rate is an inclusive pipeline rate (including partially confirmed exploitability) and that outcomes vary by dataset, implying findings may not generalize across all code. The study uses only Bandit and Semgrep as static tools and evaluates only Python samples from three datasets.
Ask anything about this paper. Answers come from its abstract, and say so when the abstract doesn't cover your question.
Beyond Static Guarantees: Measuring the Static-Pass Dynamic-Fail Gap in Security-Sensitive and LLM-Generated Python Code — Jessica Pourleyli, Maitreyee Das Urmi, Glaucia Melo
Abstract licence: CC0
Explanation generated by Synthesis from the abstract — it is not the original paper. This is a preprint, so it hasn't necessarily been peer reviewed.