Original title: Incremental Pooled LLM Evaluation for Cost-Effective Retrieval Model Selection
Max Nelson, Hanoz Bhathena, Aviral Joshi, Saket Sharma · September 2026
When teams build search systems to find documents for artificial intelligence models, they need to test which system works best. Scoring whether each found document is helpful costs a lot of time and money, especially when new systems are added. The researchers tested a cheaper method called pooled evaluation using a large language model to judge the results. Instead of checking every document from scratch, the system combines all found documents into one list and scores them once. When a new system arrives, the model only judges the brand-new documents it pulls up. This lets teams reuse old scores. The study found this method agrees strongly with standard scoring, saves money, and cuts evaluation costs by up to 4.9x.
Imagine comparing different vacuum cleaners for your home. Each time you try a new model, you have to clean every room and check how well it picks up dirt. But if you already have notes on how well the old models cleaned each room, you only need to clean with the new model and compare its unique performance. In this paper, the system judges documents from search models in a pool, so when a new model comes in, only its new documents are judged, and old notes are reused for comparison.
Teams building AI systems that retrieve information spend a lot of time judging search quality. This method lets them reuse previous judgments, saving money and time when testing new models.
The abstract doesn't state explicit limitations, but the approach was tested on four retrieval benchmarks and one financial news QA system. Its effectiveness in other domains or with different scales is not established.
Ask anything about this paper. Answers come from its abstract, and say so when the abstract doesn't cover your question.
Incremental Pooled LLM Evaluation for Cost-Effective Retrieval Model Selection — Max Nelson, Hanoz Bhathena, Aviral Joshi et al.
Abstract licence: CC0
Explanation generated by Synthesis from the abstract — it is not the original paper. This is a preprint, so it hasn't necessarily been peer reviewed.