Audio sketching goes high tech
Creativity is changing rapidly with the advent of digital technology, enriching not only our evolving art world but also applications involving communication, gaming and computer interaction. Whereas pen and paper were once the de facto method of sketching, computers have added a myriad of ways for artists to express themselves. One relatively novel way of doing so is by integrating the sonic behaviour of objects in artwork. This was the aim of an innovative EU-funded project dubbed SKAT-VG (Sketching audio technologies using vocalizations and gestures). To explain, ‘if Bob wants to communicate to Alice what a squeaky door sounds like, he may use words or, more directly, he may imitate that sound with his voice, possibly accompanied by gestures,’ says project coordinator Davide Rocchesso from Università Iuav di Venezia. ‘People are very good at mimicking sounds with voice and gestures, especially those that are difficult to describe using words.’ he clarifies. SKAT-VG worked on exploiting these natural capabilities in the field of sound design. The main idea from a technical standpoint involves digitally sketching audio technologies. ‘Starting from a vocal sketch, a machine extracts a parametric sound model, which can be further manipulated by voice and gesture as well as by conventional knobs and sliders, to produce refined sound creations,’ explains Rocchesso. To achieve its aims, the project team included experts from a variety of complementary disciplines including cognitive psychology, gesture analysis, machine learning, audio application development, voice production and interaction design. It then analysed the way that people employ gestures and vocalisations to communicate sounds, assessing current approaches of sound designers, machine learning and sketching tools. In effect, sound sketching can be considered a new medium that can support artists in innovating and creating. ‘Sound artists and designers are involved in the development of complex products such as games, movies, or cars,’ says Rocchesso. ‘If properly assisted by SKAT-VG technologies, voice sketching becomes a creativity booster and facilitates the circulation of sound ideas.’ During the project, artists came together to collaborate and explore creative possibilities using sound sketching techniques. ‘Several sound designers actually modified their work practices to introduce vocal sketching,’ says Rocchesso. ‘They discovered how voice and gestures are terrific means to work collaboratively on sound design projects.’ Indeed, enabling designers to use their voice and hands directly for sketching the auditory aspects of an object makes it easier to exploit sound’s functional and aesthetic possibilities. ‘Sound certainly contributes to the aesthetic character of a product and is often a key ingredient for making the product usable and safe.’ confirms Rocchesso. One interesting arena where the technology plays out is electric cars, which require a large repertory of sounds to support the safety of drivers and pedestrians. Equally interesting, interactive environments such as those seen in computer games could also exploit SKAT-VG to couple sounds tightly with actions. Put simply, ‘sketches, whether in relation to sound or to any other aspect of a product, are a means to let creative ideas emerge and be communicated,’ says Rocchesso. The project has advanced our understanding on how vocalisations – and vocal imitations in particular – are produced and perceived. It has shed light on the role of gestures in complementing voice-based communication within a variety of applications. A safer, richer, more stimulating and more exciting world could emerge from this new, innovative technology.
Keywords
Sketching audio, computer game, vocal sketch, sound artist, interactive environment
