Human and machine interact to make better music
Digital technologies are entering almost every aspect of our lives, becoming ever more intertwined with society. With this movement comes a desire to create more natural ways for humans to interact with computers. Researchers are hoping to incorporate human-computer interaction (HCI) into new elements of life, including culture, emotion and experience. Body movement is one way to do this. The MIM project created a new type of digital music instrument based on sensorimotor learning mechanisms — closely linked to body movements. This research was undertaken with the support of the Marie Skłodowska-Curie programme. “Music performance is often seen as one of the ultimate accomplishments of human skills,” says Mr Baptiste Caramiaux, research scientist at the National Centre for Scientific Research (CNRS), University Paris-Saclay and lead researcher in the MIM project. “Surprisingly, it is not yet understood how practice is concretely impacting the acquisition of music skills. We know that certain movements and exercises need to be repeated. But is repetition the best strategy?” In this context, the MIM project sought to better understand if it is better to repeat sequences of movement in order to acquire them, or to vary the movements during practice. Ultimately, the idea is to build interactive technology that can guide musicians during practice. Insights into human creativity There were several technological components involved in the project. The first related to devices used to capture human movement during music performance, focusing on the piano. They set up a state-of-the-art motion capture system, placing reflective markers on the performer’s fingers, which were tracked by a set of 10 infrared cameras surrounding the performer. This allowed the researchers to extract finger positions with high temporal and spatial resolution. A second aspect was movement analysis, using a set of algorithms to analyse the finger movements and gain insights. The MIM team developed two types of tools: some that analyse how the movement improves over time, and others that are able to learn the structure of the movement variability. The results showed that movement variability can be a constructive mechanism in music skill acquisition. Specifically, that varying movements when learning new music skills can be both beneficial to learning and a way to stamp an individual’s ‘signature’ on the music being created. “The impact is important. Understanding movement variability structure may help the design of movement-based interactive systems able to better understand and to foster human movement learning,” says Caramiaux. Improving human ingenuity The main goal of the project was to enhance musical performance in general, proposing ways to learn music skills more efficiently. “It may sound very ambitious, and it is,” says Caramiaux. The applications can also go beyond music. “I am collaborating with researchers at IRCAM and Sorbonne University in Paris on musical systems applied to motor rehabilitation,” says Caramiaux. The next step is to push forward with the design of interactive systems able to guide musicians as they practice. Such systems, involving machine learning methods, would be able to learn how musicians learn and adapt accordingly. “The results we obtained are very promising and shaped my current and future research agenda at CNRS: using machine learning to support and challenge human learning and creativity,” Caramiaux adds.
Keywords
MIM, machine learning, creativity, music, practice, algorithms, insights
