Original title: Diagnostic Performance of a Locally Deployed Vision Language Model for Bone Tumor Diagnosis Using Smartphone-Captured Images: Exploratory Retrospective Study
Yin Shi, Zhi-Chao Tong · September 2026
Doctors often take quick smartphone photos of computer screens displaying medical scans to share with colleagues. These pictures can have glare and wavy patterns, making them hard to read. Researchers wanted to see if an artificial intelligence model that processes text and pictures could identify rare bone tumors from these messy photos. To do this, they gave the model real photos of scans taken by a handheld phone. They tested two setups: telling the model to act as a radiologist versus an orthopedic cancer specialist. They also tested whether giving the system external medical guidelines—a method meant to supply extra knowledge—would help it diagnose cases better. Surprisingly, adding outside reference guidelines did not help. In fact, it actually caused the radiologist setup to get worse, dropping its first-choice accuracy. The extra text pushed the model toward several diagnostic errors, including chasing rare diseases.
Imagine you're packing a suitcase for a trip. You know roughly what to bring, so you pack from memory. Now suppose someone hands you a printed checklist of everything you might ever need—but the checklist includes items for a safari when you're going to the beach. Instead of helping, the extra list makes you second-guess your choices, and you leave out your swimsuit. In this paper, adding expert guidelines to the AI acted like that safari checklist: it introduced information that didn't fit the actual images, causing more errors instead of fewer.
If you send a smartphone photo of an X-ray to an AI for a second opinion, adding the official medical guidelines might make it less reliable. This study found that for rare bone tumors, retrieval-augmented generation (adding expert text) actually decreased diagnostic accuracy in one of the tested scenarios. That matters because many telemedicine systems are now building in such retrieval features, assuming more information always helps—this shows it can introduce new kinds of errors.
The abstract notes this is an exploratory study with a small sample (42 patients) and a single VLM (Qwen3-VL). The smartphone-captured images were simulated in a lab, not taken during real clinical workflows. The RAG configuration used specific external guidelines, and findings may not generalize to other retrieval sources or model architectures. Error pattern analysis was post hoc and not pre-registered. Causal claims about persona effects remain speculative without larger, controlled studies.
Ask anything about this paper. Answers come from its abstract, and say so when the abstract doesn't cover your question.
Diagnostic Performance of a Locally Deployed Vision Language Model for Bone Tumor Diagnosis Using Smartphone-Captured Images: Exploratory Retrospective Study — Yin Shi, Zhi-Chao Tong — JMIR AI — doi:10.2196/99757
Abstract licence: CC BY
Explanation generated by Synthesis from the abstract — it is not the original paper.