Original title: When Agents See Differently: Exposing UI Desynchronization Threats in Mobile Agents
Heng Li, Fulin Zhao, Zhe Geng, Zhiyuan Yao +2 · September 2026
People expect AI mobile agents to see the exact screen that human eyes see. But humans look at physical phone screens through imperfect eyes that struggle with poor lighting and blocked views. Agents look at raw digital screenshots and hidden code details called accessibility representationsExtra information in the app's code that helps screen readers describe elements to visually impaired users.. Because of this gap, an app screen can show different things to a person and an AI agent. The authors call this mismatch human-agent UI desynchronizationWhen the same screen shows different information to a human and an AI agent.. Researchers tested if altered copies of real apps could trick agents while working normally for people. They hid changes inside apps before deployment. These changes fooled agents into doing attacker-chosen actions without knowing user instructions ahead of time. The researchers created an automated system to build these attacks in real app files. In tests across varied tasks, the attacks successfully misled agents most of the time, while everyday users rarely noticed the hidden visual changes.
Imagine you and a friend are looking at the same menu board at a restaurant. You see the prices and descriptions normally, but your friend, who is colorblind, sees some items highlighted that you don't notice. The restaurant owner could change the menu in a way that only your friend sees different options, steering them to order something expensive while you see nothing unusual. In this paper, the human is you, the agent is your friend, and the menu is the phone app screen—attackers can alter it so the agent sees different buttons than the user, leading the agent to take actions the attacker wants.
Mobile agents are increasingly handling tasks like booking flights or sending money on your behalf. This research shows that attackers can make an app appear normal to you while feeding your agent different information, potentially steering it to perform unintended actions. Since the visual changes are nearly invisible to humans, you wouldn't even know your agent was compromised.
The attacks are embedded before deployment and do not adapt to runtime changes. The evaluation is limited to five mobile-agent frameworks and three backbone models. The human study, while showing perturbations are hard to notice, may not capture all real-world conditions. The framework constructs instruction-agnostic attacks, so it may not cover scenarios where user instructions are known or agent behavior is monitored.
Ask anything about this paper. Answers come from its abstract, and say so when the abstract doesn't cover your question.
When Agents See Differently: Exposing UI Desynchronization Threats in Mobile Agents — Heng Li, Fulin Zhao, Zhe Geng et al.
Abstract licence: CC0
Explanation generated by Synthesis from the abstract — it is not the original paper. This is a preprint, so it hasn't necessarily been peer reviewed.