Original title: HarnessDev: Can LLMs Create and Evolve Their Own Agent Harness?
Yuhao Wu, Jingyuan Zhang, Jiajun Shi, Xinping Lei +15 · September 2026
Artificial intelligence agents rely heavily on their harness, which is the external software setup that runs them and helps them solve tasks. Changing this software can change an agent's success, but usually humans build it. Researchers wanted to see if models could build this setup themselves. They introduced HarnessDev to test this. First, a model creates a complete execution harness starting from a small seed. Next, in an evolution step, it tries to improve its setup using feedback from running tasks. The team tested how well these created setups worked and how many tokens they used. They found mixed results. On coding, search, and research tasks, model-built setups lagged far behind human-made ones. However, on writing and machine-learning tests, they matched or beat human designs. Trying to improve the setups gave some gains, but the progress was shaky, struggled to help on held-out tasks, and failed to transfer well between different models.
Imagine you're setting up a workshop to build furniture. You start with a few basic tools and have to create the rest yourself, like saws and hammers. Then you use them to build different pieces, and based on how well they turn out, you try to improve the tools. But if you're not experienced, the tools might work for simple shelves but not for complex cabinets, and your improvements might not help with new projects. In this paper, the AI is like the workshop owner, the harness is like the tools, and the benchmark tests if the AI can create tools that work for various tasks, similar to building different furniture.
This matters for AI developers and users because if models can create their own infrastructure, it could streamline deployment. However, if the generated harnesses are unreliable or don't transfer well, it poses challenges for real-world use.
The abstract indicates that evolution of harnesses produces unstable gains and only partial transfer to new tasks. Performance gains depend strongly on the model executing the harness, so results may not generalize across different LLMs. The benchmark covers specific domains, so findings might not apply to all types of tasks.
Ask anything about this paper. Answers come from its abstract, and say so when the abstract doesn't cover your question.
HarnessDev: Can LLMs Create and Evolve Their Own Agent Harness? — Yuhao Wu, Jingyuan Zhang, Jiajun Shi et al.
Abstract licence: CC0
Explanation generated by Synthesis from the abstract — it is not the original paper. This is a preprint, so it hasn't necessarily been peer reviewed.