Original title: Beyond Endpoint Scores: Time- and Capacity-Conditioned Evaluation of Continual Knowledge Updating
Heejin Choi · September 2026
Researchers test how AI models learn new facts over time. Usually, teams judge methods by checking test scores just once at the very end using a single model capacity. This paper shows that shortcut can be misleading. The authors compared a periodic hierarchyAn update method that periodically applies a structured hierarchy to organize new knowledge. method against cumulative replayAn update method that repeatedly retrains the model on old data along with new data to prevent forgetting. across a 24-month stream of data from Wikidata. They varied the evaluation month, the replay memory capacity, and how questions were phrased. The winning method changed depending on these settings. When the replay baseline had more capacity, it overtook the hierarchy method. Looking only at the end checkpoint hid large differences over time. The researchers conclude that the hierarchy is actually a lower-cost option rather than a true winner in quality.
Imagine you're updating a recipe book with new recipes each month. You can either add a new section every few months or keep adding recipes to the existing sections. But if you check which method works better at different times or with different amounts of note-taking space, you might find the better method changes. In this paper, the researchers are comparing ways to update AI knowledge and showing that the best way depends on when you measure success and how much adaptation you allow.
If you're building AI that needs to update with new information, picking the wrong update method could make it perform worse or cost more. This study shows that how we evaluate these methods matters as much as the methods themselves.
The abstract doesn't specify limitations, but the study is limited to two specific language models and a Wikidata data stream. It only compares two update methods and varies certain parameters, so results may not apply to other methods, models, or data types.
Ask anything about this paper. Answers come from its abstract, and say so when the abstract doesn't cover your question.
Beyond Endpoint Scores: Time- and Capacity-Conditioned Evaluation of Continual Knowledge Updating — Heejin Choi
Abstract licence: CC0
Explanation generated by Synthesis from the abstract — it is not the original paper. This is a preprint, so it hasn't necessarily been peer reviewed.