A system changes a voice actor's delivery based on text directions without real examples.

Voice actors can change how they say a line based on director notes, but computer voices struggle to do this. Researchers studied direction-following text-to-speech, which creates a new voice clip based on written instructions while keeping the same speaker and words. The big problem was a lack of training data. To fix this, researchers built an automated pipeline to make fake training examples. First, an impression-controllable voice model makes speech with different styles. Then, a large language model looks at the style differences and writes instructions matching the change. Testing showed that training on these fake examples alone allows stable style changes while preserving speaker identity. Adding real recorded audio to the fake data improves how well the voice follows directions.

Why it matters

Voice actors and TTS developers need systems that adapt to performance notes, but there isn't enough data of the same line spoken in different ways. This work creates a way to generate that data, making flexible speech synthesis more achievable.

Questions answered