Original title: Embodied Musical Gesture Analysis from Audio Representations: A Review of Datasets, Audio Descriptors, and Deep Learning Approaches
Ioanna Miliaresi · September 2026
When musicians play instruments, their body movements create the sound. Computer programs can now analyze complex musical recordings, but scientists want to know if these audio models actually detect the physical actions behind the music. The authors reviewed existing research by connecting music science, computer analysis, and multimodal machine learning, which combines different types of data like sound and video. They built a framework to sort movements by gesture type, how easily sounds reveal them, and how physical actions produce audio. They also surveyed relevant datasets and testing methods. They found that current audio models only capture partial, task-dependent traces of physical movements. The systems pick up physical actions best when a gesture is tightly tied to producing a sound. However, evidence remains limited for higher-level expressive and communicative body movements. Proving whether models truly grasp physical performance processes rather than simple sound patterns remains an open challenge.
Imagine trying to guess what a chef did in the kitchen just by tasting the soup. You can tell if they chopped onions (sharp, pungent flavor) or added cream (smooth, rich texture). But can you tell if they were stirring with their left hand or right? Or if they were humming while they worked? Some actions leave clear traces in the taste, others are invisible. In this paper, the soup is the audio recording, the chef's actions are the musician's gestures, and the AI is the taster trying to figure out what happened—but it can only detect the actions that strongly affect the sound.
When you listen to a recording, you hear sound—but the musician's physical gestures (how hard they pressed a key, how fast they moved a bow) shaped every note. If AI can decode those hidden movements from audio alone, it could teach instruments remotely, help dancers sync to live music, or let composers control sound with their bodies. Right now we don't know if AI actually understands the body behind the music, and this paper explains why that question is so hard to answer.
The review does not claim that any existing method successfully decodes embodied gestures from audio; it only surveys evidence that learned representations contain partial traces. It notes limited benchmarking resources, poor generalization across performers and instruments, and the unresolved ambiguity between gesture-related encoding and mere acoustic correlation. The framework itself is conceptual and not empirically validated.
Ask anything about this paper. Answers come from its abstract, and say so when the abstract doesn't cover your question.
Embodied Musical Gesture Analysis from Audio Representations: A Review of Datasets, Audio Descriptors, and Deep Learning Approaches — Ioanna Miliaresi — Arts — doi:10.3390/arts15100217
Abstract licence: CC BY
Explanation generated by Synthesis from the abstract — it is not the original paper.