Original title: Efficient SWE Agent Benchmarking via Trajectory-Aware Evaluation
Kefeng Duan, Dewu Zheng, Yanlin Wang, Xiwen Wang +6 · September 2026
Testing AI software engineering agents is expensive. Each coding task takes many steps, like reading files, changing code, and running tests. To save time, researchers often test agents on a smaller set of tasks. The problem is that older methods only look at whether an agent passed or failed, ignoring how it solved the problem. The researchers built a new framework called PTA-IRT. It looks at the actual problem-solving journey. It studies the files the agent explored, the code edits it tried, and the paths it took. By combining this process with pass or fail outcomes, PTA-IRT picks better test tasks and estimates agent skills more accurately. Across four software benchmarks, it beat earlier methods even with tiny evaluation budgets.
Imagine you're judging a high school science fair by tasting only the final baked goods. You'd know which ones taste good, but you'd miss how each student measured ingredients or handled mistakes. If two cakes are equally tasty, watching the preparation might reveal one student is more skilled with techniques. In this paper, the researchers are like judges who also observe the cooking process, not just the final dish. This helps them pick which students to evaluate more carefully with limited time, leading to more accurate rankings of skill.
Software teams spend significant time and money testing AI coding assistants. This method lets them get accurate performance estimates with fewer tests, so they can develop and improve these tools more efficiently.
The abstract doesn't specify how PTA-IRT performs with high calibration budgets or on domains outside the four SWE benchmarks tested. It also relies on the availability of historical execution trajectories, which may not always be accessible for new agents or tasks.
Ask anything about this paper. Answers come from its abstract, and say so when the abstract doesn't cover your question.
Efficient SWE Agent Benchmarking via Trajectory-Aware Evaluation — Kefeng Duan, Dewu Zheng, Yanlin Wang et al.
Abstract licence: CC0
Explanation generated by Synthesis from the abstract — it is not the original paper. This is a preprint, so it hasn't necessarily been peer reviewed.