Original title: Lost in Reordering: Structural Sensitivity of Multilingual LLMs under Semantics-Preserving Perturbations
Karthika Nhayakkat, Rajat Verma, Maharaj Brahma, Vetcha Gnana Mahesh +3 · September 2026
Many modern artificial intelligence models excel at solving math word problems across different languages. However, we do not fully understand how well they handle sentences that shuffle word order without changing the meaning. This flexibility is common in languages like Hindi and Malayalam. Researchers created a new test dataset called IndicReStruct from math problems. They rearranged sentence components and changed active voice to passive voice, keeping the exact same meaning. They tested six leading models using different prompting methods. The models struggled, showing consistent drops in math reasoning performance. Errors often happened because the models mixed up which numbers belonged to which objects. The tests showed that middle layers inside the models matter most for fixing these reasoning mistakes.
Imagine following a recipe where the steps are written in a fixed order. If someone shuffles the steps without changing any ingredients or amounts, a chef who memorized the exact sequence might add ingredients out of order or miss a step. In this paper, the AI model is like that chef; it's trained on specific sentence structures, and when the structure changes even with the same meaning, it gets confused and makes errors in solving the problem.
This affects anyone relying on multilingual AI for education or problem-solving, especially in languages like Hindi or Malayalam where sentence structure is flexible. If the AI fails when sentences are rephrased, it can give wrong answers despite understanding the content.
The abstract indicates that the study is limited to two languages (Hindi and Malayalam) and focuses specifically on mathematical reasoning tasks using the GSM8K dataset. It does not explore other languages or types of reasoning beyond the tested perturbations.
Ask anything about this paper. Answers come from its abstract, and say so when the abstract doesn't cover your question.
Lost in Reordering: Structural Sensitivity of Multilingual LLMs under Semantics-Preserving Perturbations — Karthika Nhayakkat, Rajat Verma, Maharaj Brahma et al.
Abstract licence: CC0
Explanation generated by Synthesis from the abstract — it is not the original paper. This is a preprint, so it hasn't necessarily been peer reviewed.