Original title: Free Pause Tokens
John Langford, Nathan Godey, Giovanni Monea, Yoav Artzi +4 · September 2026
Standard language models predict the next word one step at a time, but giving them extra thinking time can help them make better guesses. Usually, this means inserting extra thinking tokens directly into the text sequence. This eats up space and slows things down. Researchers introduced the free pause token to solve this problem. Instead of adding an extra word to the sequence, it runs extra computation in a parallel prediction stream using the same underlying model weights. It gives the model extra thinking compute right at the existing word position. In tests on a 1B parameter model, it improved next-token prediction by 2-3 centinats. At inference, it adds no context length, no cache memory, and essentially no latency.
Imagine you're answering a quiz and need more time to think about each question. Normally, you'd pause and take extra time, which slows down the entire test. Here, instead of pausing, you have a way to think harder for each question without adding any delay, so you finish just as fast but with better answers. In this paper, the language model needs extra compute to predict the next word, but instead of adding thinking tokens that lengthen the sequence, it uses a parallel process that doesn't add latency or context length.
For anyone using AI language models, this means better predictions without slower responses or higher costs. It allows models to improve accuracy without the usual trade-offs in speed or memory usage.
The abstract only reports results on a 1B parameter model; it doesn't say if it scales to larger models or other tasks. It improves next-token prediction, but doesn't address reasoning or generation. The training cost is reduced but still increased compared to standard training.
Ask anything about this paper. Answers come from its abstract, and say so when the abstract doesn't cover your question.
Free Pause Tokens — John Langford, Nathan Godey, Giovanni Monea et al.
Abstract licence: CC0
Explanation generated by Synthesis from the abstract — it is not the original paper. This is a preprint, so it hasn't necessarily been peer reviewed.