Original title: Language Models Can Control Their Own Attention
Namgyu Ho, Huzama Ahmad, Woosung Koh, Se-Young Yun +2 · September 2026
Language models waste effort reading massive amounts of text. When generating each new word in a long conversation, a model scans its entire stored memory, known as the KV cacheA storage system in language models that keeps previously computed data to avoid recomputation., just to find a few relevant details. To fix this, researchers created Declarative Attention. Instead of relying on an outside tool to score what matters, they let the model choose what to read during its step-by-step thinking. The model simply declares one of three modes: look at the whole text, zoom into a specific region, or only check recent output. The system treats these choices like commands and skips reading the rest of the stored memory. Testing showed this technique cut the total words the models had to scan by 52.0% and 31.1%, while causing only minor drops in accuracy.
Imagine reading a huge book to find a single fact mentioned in the first chapter. You'd waste time flipping through every page instead of just going to the relevant section. The hard part is knowing where to look without reading everything first. In this paper, the model itself says 'I need to check chapter one' or 'just skim the summary,' so it skips most pages and only reads what's necessary.
For anyone using AI on long documents or conversations, this makes responses faster and cheaper by ignoring most irrelevant information. It helps AI systems handle massive text without slowing down, which is crucial as we ask AI to work with longer content.
The method causes modest accuracy drops, especially for smaller models. It is evaluated only on specific models and tasks in a zero-shot setting, so it may not generalize everywhere. Further improvements could come from training-based methods.
Ask anything about this paper. Answers come from its abstract, and say so when the abstract doesn't cover your question.
Language Models Can Control Their Own Attention — Namgyu Ho, Huzama Ahmad, Woosung Koh et al.
Abstract licence: CC0
Explanation generated by Synthesis from the abstract — it is not the original paper. This is a preprint, so it hasn't necessarily been peer reviewed.