Original title: Separating Syntax from Language: A Mechanistic Account of Translation in Multilingual LLMs
Mikhail Sonkin, Tanja Baeumel, Daniil Gurgurov, Josef van Genabith +1 · September 2026
Multilingual AI models are great at translating, but we do not fully understand how they change text from one language to another internally. Earlier research showed models first grasp a concept on its own, then turn it into words in the target language. This study shows the translation steps are even more split up. Researchers created controlled multilingual datasets that isolate differences in word order across languages. They used causal interventionsTechniques that modify internal model states to test how changes affect output, revealing cause-and-effect relationships. and probingA method to examine what information a model's representations contain by training simple classifiers on them. to track internal changes during translation. They found that models figure out the target word order, or syntax, before picking the actual target words. In fact, specific parts called attention headsComponents within the model that determine which words to focus on during text processing. handle sentence structure changes while ignoring which specific language is being used.
Imagine you're packing a suitcase for a trip where you need to fit items according to strict size rules. You can't just throw things in; you have to arrange them to maximize space first, then decide what each item actually is. In this paper, translation is like packing: the model first sorts out the sentence structure to fit the target language, and only then assigns the actual words.
For developers and users of translation AI, this shows that sentence structure and language choice are separate steps. Knowing this could help improve translation accuracy by targeting specific parts of the process.
The abstract doesn't state explicit limitations, but based on its scope, this work is a mechanistic analysis within mLLMs using controlled datasets. It may not generalize to all translation tasks, languages, or model architectures, and it focuses on word-order differences rather than broader linguistic features.
Ask anything about this paper. Answers come from its abstract, and say so when the abstract doesn't cover your question.
Separating Syntax from Language: A Mechanistic Account of Translation in Multilingual LLMs — Mikhail Sonkin, Tanja Baeumel, Daniil Gurgurov et al.
Abstract licence: CC0
Explanation generated by Synthesis from the abstract — it is not the original paper. This is a preprint, so it hasn't necessarily been peer reviewed.