Original title: SWE-Gate: Passing Functional Tests Is Not Enough for Software Engineering Agents
Xin He, Yanlin Wang, Mingwei Liu, Jiachi Chen +2 · September 2026
When programmers write code, passing automatic tests is not the only goal. Real teams also have code review rules that decide if a fix gets accepted. Current tests for artificial intelligence coding agents ignore these extra rules. To solve this, researchers built SWE-Gate, a benchmark that checks both working code and review rules. It pulls real feedback from pull request comments to create tasks with separate tests for basic functions and review constraints. It also gives working examples that pass or fail those extra rules. The researchers tested four different AI models. They found that out of 644 fixes that passed the basic functional tests, 221 failed the review constraints. Testing only basic function makes coding agents look better at real software repair than they actually are.
Imagine baking a cake for a contest where it must not only taste right but also follow strict decoration rules. The cake might pass the taste test but still be disqualified for using the wrong icing color. In this paper, functional tests are like the taste checks, and review constraints are like the decoration rules that code must satisfy to be fully accepted by a development team.
This matters for developers using AI coding tools because it shows that code which works functionally can still be rejected in real teams due to style or standards. Better benchmarks mean more reliable tools that save time and reduce frustration in software projects.
The benchmark is limited to Python repositories and uses specific LLM backends, so results may not generalize to other programming languages or different agent architectures. It also relies on review constraints from historical data, which might not capture all future review scenarios.
Ask anything about this paper. Answers come from its abstract, and say so when the abstract doesn't cover your question.
SWE-Gate: Passing Functional Tests Is Not Enough for Software Engineering Agents — Xin He, Yanlin Wang, Mingwei Liu et al.
Abstract licence: CC0
Explanation generated by Synthesis from the abstract — it is not the original paper. This is a preprint, so it hasn't necessarily been peer reviewed.