Original title: Are We There Yet? Assessing Computer-Use Agents for Blind Users' Accessible Interaction with Desktop Applications
Satwik Ram Kodandaram, Monalika Padma Reddy, Xiaojun Bi, Jiawei Zhou +2 · September 2026
Screen readers help blind people navigate computers, but many apps remain tricky to use. New AI systems called computer-use agentsAI programs that can operate a computer like a human, moving the mouse and typing. can help by looking at desktop apps and controlling them directly. To see if these tools actually work for blind users, researchers ran a study. They created an agent called OLLA that works with screen readers, letting users give natural requests while the AI interprets visual screens and accessibility data to click and type for them. The researchers tested the system during real everyday tasks. They found that these AI helpers still struggle. The top-performing model, GPT-5, succeeded only 52.5% of the time. The agents frequently made mistakes when locating interface items, planning steps, tracking constraints, and deciding when to stop.
Imagine you are trying to cook a new recipe, but you cannot see the ingredients or the stove. You have a friend who can see, and you tell them what to do step by step. Sometimes they grab the wrong spice or leave the pot on too long because they did not notice the timer. In this paper, the AI is that friend, and the blind user is the cook. The AI sees the screen and tries to follow commands, but it often misidentifies buttons or forgets what it was doing, so the meal only turns out right about half the time.
Screen readers already help blind people use computers, but complex tasks still require sight. This study tests whether AI agents that control the mouse and keyboard can fill that gap. The findings show where these agents fail, guiding better design for millions of blind users.
The abstract does not specify the exact tasks or applications beyond noting 12 apps, nor does it detail the four additional models' performance. It also does not state whether the 52.5% success rate is statistically significant or how it varies across tasks. The study is limited to 8 participants over three weeks, so findings may not generalize broadly.
Ask anything about this paper. Answers come from its abstract, and say so when the abstract doesn't cover your question.
Are We There Yet? Assessing Computer-Use Agents for Blind Users' Accessible Interaction with Desktop Applications — Satwik Ram Kodandaram, Monalika Padma Reddy, Xiaojun Bi et al.
Abstract licence: CC0
Explanation generated by Synthesis from the abstract — it is not the original paper. This is a preprint, so it hasn't necessarily been peer reviewed.