Original title: Retrieved but not ranked: surface-form bias in structural retrieval, from mathematics to agent trajectories
Nabira Rashid, Manolis Kellis · September 2026
Search tools use embeddings, which turn text into lists of numbers, to match queries by meaning rather than exact words. The problem is that these tools often get fooled by surface wording. Researchers tested this by asking models to find matches with the same underlying structure but different words. They used competition math problems and action paths from virtual robots. Models work by checking how close texts are in their numerical space, but they struggled here. In math, models scored 0.0% on top matches when problems were heavily disguised. The right answer was almost always in the top 10, but the models picked wrong answers that merely shared more words in 95.2 to 99.8% of misses. On robot tasks, models dropped to or below random chance when objects changed. Using large language models to re-rank results helped, but part of that gain came from memorization.
Imagine you're looking for your keys in a cluttered drawer, but someone has labeled all items with different names. You might pick up the wrong thing because the label doesn't match what you expect. Here, embedding retrieval systems act like someone searching by labels, but when the underlying structure (what the item does) is the same but the label (wording) is different, the system fails to retrieve the correct item.
This affects anyone relying on AI to find relevant information, like students using math tutors or developers building AI agents. If retrieval systems anchor on wording instead of meaning, they can miss correct answers even when they're right in the data.
The study is limited to two specific domains, so it's unclear if the findings apply broadly. The downstream experiment shows that solver accuracy might not improve with better retrieval, indicating other constraints. Reranker performance varies by domain, suggesting no universal solution.
Ask anything about this paper. Answers come from its abstract, and say so when the abstract doesn't cover your question.
Retrieved but not ranked: surface-form bias in structural retrieval, from mathematics to agent trajectories — Nabira Rashid, Manolis Kellis
Abstract licence: CC0
Explanation generated by Synthesis from the abstract — it is not the original paper. This is a preprint, so it hasn't necessarily been peer reviewed.