Original title: Scalable Oversight for AI in Mental Health: Lessons from 350,000 AI Coaching Conversations between Therapy Sessions
Matthew A. Scult, John L. Havlik, Kevin Ramotar, Ethan Goh +1 · September 2026
People often suggest that doctors should check every message an artificial intelligence sends in mental healthcare. However, research on human attention shows this plan fails when systems grow large, and it can actually make things less safe. To solve this, researchers studied an AI coaching tool used across 350,000+ conversations between therapy sessions. Instead of checking every line, they built a three-layer oversight system where humans stay in the loop without doing every check themselves. This framework works by stopping problems before they happen through preventive designBuilding safety rules directly into the AI's behavior before it runs, rather than checking outputs afterward., tracking chats in real time, and having clinicians continually evaluate the tool. They found that specific clinical reviews directly led to system improvements, and they shared recommendations for experts who evaluate mental health AI.
Imagine a lifeguard who must watch every single swimmer every second. After a few hours, the lifeguard's eyes glaze over and they miss someone struggling. Worse, the swimmers might take more risks because they think the lifeguard is watching everything. In this paper, the AI coaching system in mental health faced a similar problem: expecting clinicians to review every AI response led to missed dangers, so the researchers replaced that with a layered safety approach that doesn't rely on constant attention.
Mental health professionals are often told to check every AI message to patients. But research shows this creates dangerous blind spots as attention wanes. The authors offer a smarter three-layer approach that keeps patients safe without overwhelming clinicians.
The abstract doesn't say explicitly, but the framework comes from one AI coaching tool's deployment. It may not generalize to other mental health AI systems or clinical settings. No controlled comparison with alternative oversight methods is provided.
Ask anything about this paper. Answers come from its abstract, and say so when the abstract doesn't cover your question.
Scalable Oversight for AI in Mental Health: Lessons from 350,000 AI Coaching Conversations between Therapy Sessions — Matthew A. Scult, John L. Havlik, Kevin Ramotar et al.
Abstract licence: CC0
Explanation generated by Synthesis from the abstract — it is not the original paper. This is a preprint, so it hasn't necessarily been peer reviewed.