This voice AI skips word-by-word timing and still produces natural speech.

Generating realistic speech for movie dubbing and two-way conversations is difficult because computers often struggle to match words to audio timing naturally. To fix this, researchers built Text-Audiobox, or Text-AB. Instead of relying on rigid tools that force text to match set time durations, Text-AB reads plain text directly. It uses an attention mechanism, which acts like an internal spotlight, to learn how text and speech align on its own. It also shrinks high-quality 48 kHz audio into a compact 25 Hz code, compressing sound over 10x more than earlier systems while sounding better. The team trained a massive 3B-parameter model on 480k hours of speech. They found Text-AB produced major leaps in dubbing voice similarity, rhythm, and naturalness compared to an earlier internal system. In two-way chats, it closely matched real human recordings, easily handling natural conversation flow like interruptions, emotional shifts, and brief replies.

Why it matters

Filmmakers and game developers can dub voices into other languages more naturally. It also generates realistic conversations for media, improving quality in dubbing and dialogue synthesis.

Questions answered