A single AI model can both create and surgically edit songs by learning to reconstruct them from partial instructions.

Computer models usually treat making new songs and editing existing ones as two separate jobs. Editing often relies on adding random noise and regenerating, or collecting matched pairs of songs. Researchers built a system called SongCraft to do both jobs at once. It learns by reconstructing audio from different controls. Giving it broad controls, like text descriptions and lyrics, lets it generate diverse songs. Giving it detailed controls for specific musical pieces lets it recreate the exact audio. To edit, users change just one piece while holding the rest steady. The team also aligned phonemes, the distinct speech sounds, with words, added beat controls, and organized the model's inner representations. SongCraft reached the lowest word error rate among tested baselines, kept competitive audio quality, and successfully edited lyrics, vocal melodies, beats, and singer identities.

Why it matters

Song editing today is clunky: you either re-record, use tools that damage audio, or need paired training data. This shows a single model can generate songs from scratch and let you change just the lyrics, tune, beat, or singer—without losing the rest. For musicians, that means faster iteration and cleaner edits; for listeners, songs tailored without losing their feel.

Questions answered