Original title: EarlyEval: Cheaper Agent Evaluation via Early Outcome Prediction
Yuling Shi, Zhensu Sun, Junsen Dong, Chengcheng Wan +2 · September 2026
Testing artificial intelligence agents costs hundreds to thousands of dollars per run, which quickly adds up during development. Past approaches cut costs by reducing the total number of test tasks, but running each remaining task still costs full price. Researchers created EarlyEval to solve this by stopping tasks early. They noticed an agent's final success or failure is often obvious from its intermediate behavior long before it finishes. EarlyEval tracks this behavior and stops the agent as soon as it feels confident about the outcome, adding almost no extra cost per step. Across tests on SWE-bench Verified, TerminalBench, and Toolathlon, it eliminated 13%-26% of steps and large shares of processed text, stayed 89%-97% accurate, and shifted average success rates by only one to two percentage points.
Imagine cooking a complex dish with expensive ingredients. You could wait until the end to taste it, but if it's ruined, you've wasted everything. Instead, tasting it halfway through lets you decide if it's worth continuing. Here, EarlyEval predicts if an AI task will succeed or fail based on early behavior, stopping evaluations early to save resources.
For AI developers, evaluating new models is so expensive it slows progress. EarlyEval reduces costs by stopping tests early when the outcome is clear, making development faster and cheaper.
The abstract does not discuss long-term effects or performance on all AI task types, focusing on three benchmarks and noting a small average perturbation in resolve rates without specifying variation across scenarios.
Ask anything about this paper. Answers come from its abstract, and say so when the abstract doesn't cover your question.
EarlyEval: Cheaper Agent Evaluation via Early Outcome Prediction — Yuling Shi, Zhensu Sun, Junsen Dong et al.
Abstract licence: CC0
Explanation generated by Synthesis from the abstract — it is not the original paper. This is a preprint, so it hasn't necessarily been peer reviewed.