Final Report Summary - ENABLING AVUIS (Enabling Audiovisual User Interfaces for Multisensorial Interaction)
INTRODUCTION AND OBJECTIVES
Enabling Audiovisual User Interfaces is a research project that investigated how human-computer interactions can be audiovisualized – that is to say, both sonified and visualized – in order to improve user experience and usability. To address this issue, a new UI (User Interface) paradigm was conceived – AVUI (AudioVisual User Interface). AVUI links interaction, sound and image, building upon the concept of Graphical User Interface (GUI) by adding interconnected sound and image. The research hypothesis is: the introduction of AVUI, integrating interrelated sonic and visual feedback, reacting to user interactions, will lead to more usable, accessible, playful and engaging UIs, as compared to a traditional GUI – particularly in use cases where accessibility and/or engagement are determinant. AVUIs were applied to prototypes and case studies, which were developed together with users, following a User-Centered Design approach (UCD). After reaching conclusions from these, an AVUI framework was developed. The 2-year Marie Curie fellowship took place at Goldsmiths, University of London – EAVI (Embodied AudioVisual Interaction) research group, under the supervision of Prof. Atau Tanaka. The project website is: avuis.goldsmithsdigital.com
MAIN RESULTS
The outcomes of the project are: two mobile applications (one available free in the Apple AppStore, another as installable source code); software library for enabling AVUIs (ofxAVUI, for the openFramworks C++ framework); code base for these software projects and other prototypes; the theorization of interaction on the use of audiovisual elements in the UI, as presented in the five papers published; and outreach activities. The work conducted was centered on four stages: the prototyping of AVUIs (1); the development of two case study using AVUIs (2 and 3); the release of a toolkit enabling the implementation of further AVUIs (4). Transversely to this four stages, dissemination activities were conducted (5).
1) AVUI prototypes
The development of AVUI prototypes focused on tools for audiovisual expressive interaction and performance. The development followed a UCD approach. This UCD approach involved artists and developers in several stages: interviews; a brainstorming workshop; two hackathons to develop prototypes; public performances with those prototypes; and evaluation of the prototypes by other artists. The research conducted suggests design principles for systems for audiovisual performances.
The prototypes are available as open source online at: www.github.com/avuis
Three papers have been written and published on the AVUI prototypes.
2) ShapeTones - audiovisual accessible game
ShapeTones is an audiovisual memory game for iOS (iPhone/iPad). A sequence of 3 shapes and tones (ShapeTones) is played, and the player tries to reproduce it with three taps. Tapping different areas of the screen triggers different ShapeTones. Different variations of ShapeTones are available across different levels. The game aims to address the importance of using combined and congruent image and sound. Aspects such as performance and engagement were evaluated taking into account different combinations of sound and image. The research points to recommendations regarding congruence between image and visuals. The game is accessible to visually and hearing impaired users and was co-developed in partnership with another institution (Queen Mary University of London).
The game can be downloaded for free from links to the Apple AppStore.
One paper has been written and published on ShapeTones.
3) AV Zones - software for audiovisual performance
AVZones (AudioVisual Zones) is an iPad app for audiovisual performance, composed of an audio sequencer/looper with a visualizer. By default, 3 audiovisual columns or “zones” allow for the manipulation of 3 audio loops. Each zone had 3 XY pads for audio manipulation: pitch shift, delay and filter. The app explores the integration of sound and image in a touchscreen environment for the purpose of audiovisual performance. It allowed to implement some of the design principles identified in the AVUI prototypes.
The app can be downloaded and uploaded to an iOS device from http://www.avzones.org.
One paper has been written and published on AV Zones.
4) ofxAVUI - toolkit for implementing Audiovisual User Interfaces
Building upon the prototypes and case studies, ofxAVUI is a toolkit for the popular open-source openFrameworks C++ coding platform that enables the creation of user interfaces (UI) with integrated audio and visual feedback. Different UI elements (such as buttons, sliders and XY pads) are combined into a UI “zone”. Each zone has a sound and a correspondent visualization, which respond to changes in the UI. The ofxAVUI hackathon and subsequent evaluation of the framework allowed to detect what could be important use cases, and what can be done to potentiate its adoption.
The toolkit can be downloaded from https://github.com/AVUIs/ofxAVUI
One final paper is being written on ofxAVUI.
5) Outreach activities
A web site was setup for the project: avuis.goldsmithsdigital.com. It connects to several online resources, such as: software library, tutorials, reference materials and source code examples; in order to promote the usage of the library, and facilitate the implementation of AVUIs. Four workshops; three hackathons; 25 talks, demonstrations and performances took place to discuss and present audiovisual interfaces and diffuse the usage of AVUIs.
POTENTIAL IMPACT
Potential impact of the research project:
• To assist in the development of ”advanced interfaces", contributing to a "profound change of how we interact with computers" and continuing "to transform the information and entertainment industry and all services industries" (from “FP7 ICT – Information and Communication Technologies: Work Programme 2013”). In particular, the current project fits in the strategic area "multisensory interfaces”, specifically with the toolkit ofxAVUI. This research is relevant for designers and researchers in the field of Human-Computer Interaction.
• To develop "creative experience tools", which "make use of all our senses and allow for richer, more collaborative and interactive experiences” in line with the FP7 ICT report (objective ICT-2013.8.1). Case study AV Zones pointed in this direction. This could be of particular interest for musicians, motion graphic designers and audiovisual artists.
• To contribute to the development of ”accessible and intuitive solutions for personalized interfaces to smart environments and innovative services", with low levels of digital literacy/skills, and older persons”, using a multisensorial approach. Project ShapeTones is a case study in this area. Persons with sensory impairments are a particular target for this approach.
Numerous outreach activities were conducted during the study, with software developers; researchers; artists; designers; students; as well as other interested participants, such as impaired users. These have the following expected impacts:
• Bring awareness to the importance of multisensorial interfaces for user experience.
• Increase the awareness of the relevance of accessibility for ICT, and how multisensorial approaches to human-computer interaction can improve usability.
• Diffusion of the research results to industry and academia, and adoption of AVUIs.
WEBSITE AND CONTACT
avuis.goldsmithsdigital.com
Nuno N. Correia
mail@nunocorreia.com
Enabling Audiovisual User Interfaces is a research project that investigated how human-computer interactions can be audiovisualized – that is to say, both sonified and visualized – in order to improve user experience and usability. To address this issue, a new UI (User Interface) paradigm was conceived – AVUI (AudioVisual User Interface). AVUI links interaction, sound and image, building upon the concept of Graphical User Interface (GUI) by adding interconnected sound and image. The research hypothesis is: the introduction of AVUI, integrating interrelated sonic and visual feedback, reacting to user interactions, will lead to more usable, accessible, playful and engaging UIs, as compared to a traditional GUI – particularly in use cases where accessibility and/or engagement are determinant. AVUIs were applied to prototypes and case studies, which were developed together with users, following a User-Centered Design approach (UCD). After reaching conclusions from these, an AVUI framework was developed. The 2-year Marie Curie fellowship took place at Goldsmiths, University of London – EAVI (Embodied AudioVisual Interaction) research group, under the supervision of Prof. Atau Tanaka. The project website is: avuis.goldsmithsdigital.com
MAIN RESULTS
The outcomes of the project are: two mobile applications (one available free in the Apple AppStore, another as installable source code); software library for enabling AVUIs (ofxAVUI, for the openFramworks C++ framework); code base for these software projects and other prototypes; the theorization of interaction on the use of audiovisual elements in the UI, as presented in the five papers published; and outreach activities. The work conducted was centered on four stages: the prototyping of AVUIs (1); the development of two case study using AVUIs (2 and 3); the release of a toolkit enabling the implementation of further AVUIs (4). Transversely to this four stages, dissemination activities were conducted (5).
1) AVUI prototypes
The development of AVUI prototypes focused on tools for audiovisual expressive interaction and performance. The development followed a UCD approach. This UCD approach involved artists and developers in several stages: interviews; a brainstorming workshop; two hackathons to develop prototypes; public performances with those prototypes; and evaluation of the prototypes by other artists. The research conducted suggests design principles for systems for audiovisual performances.
The prototypes are available as open source online at: www.github.com/avuis
Three papers have been written and published on the AVUI prototypes.
2) ShapeTones - audiovisual accessible game
ShapeTones is an audiovisual memory game for iOS (iPhone/iPad). A sequence of 3 shapes and tones (ShapeTones) is played, and the player tries to reproduce it with three taps. Tapping different areas of the screen triggers different ShapeTones. Different variations of ShapeTones are available across different levels. The game aims to address the importance of using combined and congruent image and sound. Aspects such as performance and engagement were evaluated taking into account different combinations of sound and image. The research points to recommendations regarding congruence between image and visuals. The game is accessible to visually and hearing impaired users and was co-developed in partnership with another institution (Queen Mary University of London).
The game can be downloaded for free from links to the Apple AppStore.
One paper has been written and published on ShapeTones.
3) AV Zones - software for audiovisual performance
AVZones (AudioVisual Zones) is an iPad app for audiovisual performance, composed of an audio sequencer/looper with a visualizer. By default, 3 audiovisual columns or “zones” allow for the manipulation of 3 audio loops. Each zone had 3 XY pads for audio manipulation: pitch shift, delay and filter. The app explores the integration of sound and image in a touchscreen environment for the purpose of audiovisual performance. It allowed to implement some of the design principles identified in the AVUI prototypes.
The app can be downloaded and uploaded to an iOS device from http://www.avzones.org.
One paper has been written and published on AV Zones.
4) ofxAVUI - toolkit for implementing Audiovisual User Interfaces
Building upon the prototypes and case studies, ofxAVUI is a toolkit for the popular open-source openFrameworks C++ coding platform that enables the creation of user interfaces (UI) with integrated audio and visual feedback. Different UI elements (such as buttons, sliders and XY pads) are combined into a UI “zone”. Each zone has a sound and a correspondent visualization, which respond to changes in the UI. The ofxAVUI hackathon and subsequent evaluation of the framework allowed to detect what could be important use cases, and what can be done to potentiate its adoption.
The toolkit can be downloaded from https://github.com/AVUIs/ofxAVUI
One final paper is being written on ofxAVUI.
5) Outreach activities
A web site was setup for the project: avuis.goldsmithsdigital.com. It connects to several online resources, such as: software library, tutorials, reference materials and source code examples; in order to promote the usage of the library, and facilitate the implementation of AVUIs. Four workshops; three hackathons; 25 talks, demonstrations and performances took place to discuss and present audiovisual interfaces and diffuse the usage of AVUIs.
POTENTIAL IMPACT
Potential impact of the research project:
• To assist in the development of ”advanced interfaces", contributing to a "profound change of how we interact with computers" and continuing "to transform the information and entertainment industry and all services industries" (from “FP7 ICT – Information and Communication Technologies: Work Programme 2013”). In particular, the current project fits in the strategic area "multisensory interfaces”, specifically with the toolkit ofxAVUI. This research is relevant for designers and researchers in the field of Human-Computer Interaction.
• To develop "creative experience tools", which "make use of all our senses and allow for richer, more collaborative and interactive experiences” in line with the FP7 ICT report (objective ICT-2013.8.1). Case study AV Zones pointed in this direction. This could be of particular interest for musicians, motion graphic designers and audiovisual artists.
• To contribute to the development of ”accessible and intuitive solutions for personalized interfaces to smart environments and innovative services", with low levels of digital literacy/skills, and older persons”, using a multisensorial approach. Project ShapeTones is a case study in this area. Persons with sensory impairments are a particular target for this approach.
Numerous outreach activities were conducted during the study, with software developers; researchers; artists; designers; students; as well as other interested participants, such as impaired users. These have the following expected impacts:
• Bring awareness to the importance of multisensorial interfaces for user experience.
• Increase the awareness of the relevance of accessibility for ICT, and how multisensorial approaches to human-computer interaction can improve usability.
• Diffusion of the research results to industry and academia, and adoption of AVUIs.
WEBSITE AND CONTACT
avuis.goldsmithsdigital.com
Nuno N. Correia
mail@nunocorreia.com