Original title: When Tokenization is Secretly Output Supervision
Tanja Baeumel, Josef van Genabith, Simon Ostermann · September 2026
Language models break text into small pieces called tokens. Usually, researchers treat tokenizationBreaking text into smaller units, like words or parts of words, for the model to process. as just preparing inputs. However, these models guess the next token one piece at a time. This means token size actually changes what the model must solve in one step, acting as secretly altered training supervision. To test this, researchers separated how inputs and outputs were split into tokens during numeric reasoning tasks. They found that output tokenization causes differences in task performance, learning speed, and internal representations, while input tokenization barely matters. This finding means comparing models with different tokenizers actually compares models trained on entirely different tasks. A survey of 120 papers showed this is rarely acknowledged: only about 10% report numeric tokenization, and 69% compare models across different tokenizers without reporting it. Output supervision finally explains why tokenization impacts models.
Imagine learning to cook by watching a chef chop vegetables. If the chef cuts them into large chunks, you might learn to make a stew; into fine dice, you might learn a stir-fry. The way the chef chops changes what recipe you end up learning. In this paper, tokenization is like how the chef chops the ingredients—it determines what the model learns to predict, not just how it receives input.
When researchers compare AI models, they might think they're testing the same task, but different tokenization schemes mean the models were trained on slightly different problems. This affects the fairness of evaluations and could lead to wrong conclusions about which model is actually better.
The experiments focus on numeric reasoning tasks, so it's unclear if the findings apply to all types of language work. The survey of papers only covers numeric reasoning, which might not represent all research areas. Also, the paper provides an explanation but acknowledges that prior work already showed tokenization affects performance.
Ask anything about this paper. Answers come from its abstract, and say so when the abstract doesn't cover your question.
When Tokenization is Secretly Output Supervision — Tanja Baeumel, Josef van Genabith, Simon Ostermann
Abstract licence: CC0
Explanation generated by Synthesis from the abstract — it is not the original paper. This is a preprint, so it hasn't necessarily been peer reviewed.