Periodic Reporting for period 2 - SOLOC (Representational Mechanisms of Neural Location Encoding of Real-life Sounds in Normal and Hearing Impaired Listeners.)
Période du rapport: 2021-06-01 au 2022-05-31
Humans make use of spatial hearing continuously to make sense of the world around. For example, to rapidly localize events in the environment and to communicate in noisy situations. Specifically, spatial hearing helps you to focus on a voice of interest in the midst of background noise (e.g. a colleague’s voice in the midst of voices of other colleagues and computers whizzing). Thus, spatial hearing is crucial for humans and the inability to localize sounds hampers communication in everyday life. Yet, it is still unknown how the human brain computes the location of real-life sounds in real-world listening situations because prior research concentrated on localization of simple sounds (for example, pure tones) in strictly controlled listening situations and experiments.
Importantly, knowledge of these brain mechanisms is needed to help hearing impaired listeners. HI listeners (over 34 million EU citizens and 5% of the worldwide population) experience great difficulties with understanding speech in noise environments. These problems persist even with an assistive hearing device such as a cochlear implant. Reduced spatial hearing contributes to these problems as it makes it difficult to filter out the voice of interest based on location information. As a result of these persistent communication problems, hearing impaired listeners are more prone to social isolation, low academic achievements, and unemployment. Besides the high personal impact, this also has a high economic impact on society.
In this research project, I took a novel approach by bringing together multiple scientific disciplines to address this problem. That is, the objectives of this Marie Sklodowska-Curie Action (MSCA) were (1) to develop a neurobiological-inspired deep neural network (DNN) model of location encoding of real-life sounds in the human brain; (2) to validate deep neural networks as models of sound location encoding in the human brain using measurements of neural activity; and (3) to employ the DNNs to investigate the neural representation of sound location in cochlear implant users and to develop signal processing strategies for cochlear implants that optimize subsequent spatial processing in the brain.
One of the main outcomes of the Action is a neurobiological-inspired convolutional neural network model (Objective 1). Our results show that such a model can accurately predict sound localization in the horizontal plane and that network localization acuity resembles human localization acuity for frontal locations. Crucially, the research outcomes highlight the potential of neurobiological-inspired deep neural network models as an approach to modeling human (spatial) hearing. Future neuroscientific research and clinical research is expected to benefit from the developed models, for example to assess neuronal sound location processing and to optimize signal processing strategies for cochlear implants that maximize subsequent spatial processing in the brain.
Importantly, knowledge of these brain mechanisms is needed to help hearing impaired listeners. HI listeners (over 34 million EU citizens and 5% of the worldwide population) experience great difficulties with understanding speech in noise environments. These problems persist even with an assistive hearing device such as a cochlear implant. Reduced spatial hearing contributes to these problems as it makes it difficult to filter out the voice of interest based on location information. As a result of these persistent communication problems, hearing impaired listeners are more prone to social isolation, low academic achievements, and unemployment. Besides the high personal impact, this also has a high economic impact on society.
In this research project, I took a novel approach by bringing together multiple scientific disciplines to address this problem. That is, the objectives of this Marie Sklodowska-Curie Action (MSCA) were (1) to develop a neurobiological-inspired deep neural network (DNN) model of location encoding of real-life sounds in the human brain; (2) to validate deep neural networks as models of sound location encoding in the human brain using measurements of neural activity; and (3) to employ the DNNs to investigate the neural representation of sound location in cochlear implant users and to develop signal processing strategies for cochlear implants that optimize subsequent spatial processing in the brain.
One of the main outcomes of the Action is a neurobiological-inspired convolutional neural network model (Objective 1). Our results show that such a model can accurately predict sound localization in the horizontal plane and that network localization acuity resembles human localization acuity for frontal locations. Crucially, the research outcomes highlight the potential of neurobiological-inspired deep neural network models as an approach to modeling human (spatial) hearing. Future neuroscientific research and clinical research is expected to benefit from the developed models, for example to assess neuronal sound location processing and to optimize signal processing strategies for cochlear implants that maximize subsequent spatial processing in the brain.
Work was divided into three work packages (WPs) focused on research and an additional four WPs dedicated to management of the project (WP4), training and transfer of knowledge (WP5), dissemination and exploitation (WP6), and communication (WP7). In WP1, I created deep neural network (DNN) models of neural location processing of real-life sounds in real-world listening environments. This WP yielded a journal publication and a depository of spatialized, real-life sounds that will be released to the public in the near future. The database can be used by the wider scientific community for further research in neuroscience, audition and computational modelling. WP2 aimed to evaluate the validity of the deep neural networks as a model of sound location processing in the human brain by using measurements of neural activity. I conducted a study utilizing invasive intracranial recordings in neurosurgical patients to obtain insight into single-source sound location processing in multi-source listening scenes to better understand one of the fundamental problems of hearing impaired listeners: speech-in-noise perception. WP2 yielded a conference presentation and another conference presentation and one journal manuscript are currently underway. To promote transfer-of-knowledge, the Fellow organized three workshops for academics and assisted in the supervision of early career researchers. Executing the projects of the fellowship in a timely manner strengthened the management and administrative skills of the Fellow. Finally, during the first period, the Fellow earned a teaching qualification in the Netherlands (University Teaching Qualification) and was appointed a part-time position at Columbia University to continue her successful collaboration with the outgoing host after the end the Fellowship. The research outcomes were presented in a high-quality scientific paper in a computational neuroscience journal and at a scientific conference in the field of cognitive neuroscience and audition. Furthermore, the research materials and data sets collected over the course of the MSCA will propel numerous scientific studies forward and contribute to future publications of other research groups.
This MCSA progressed research beyond the current state-of-the-art by investigating the computational and representational mechanisms underlying the transformation from real-life sounds in ecologically valid listening settings to a neural representation of sound location. The results spark a range of new questions and open up more research avenues concentrating on the neural mechanisms of sound processing of real-life, ecologically valid listening situations. Further, this MCSA extended the frontiers of the use of deep neural network models to comprehend neurophysiological processing mechanisms by developing a novel, neurobiological-inspired DNN-model of location encoding of real-life sounds in the human auditory pathway. The outcomes of the MCSA are also expected to impact clinical developments in speech-in-noise understanding in cochlear implant users beyond the current state-of-the-art. Specifically, the outcomes can be used to investigate the brain representation of naturalistic spatial hearing in cochlear implant users and to predict the outcomes of smart signal processing algorithms that maximize the availability of spatial cues for cochlear implant users on neural processing in CI users. Thus, the uniquely multidisciplinary and intersectional approach that characterized this Marie Sklodowska-Curie Action, connected experts and expertise across cognitive neuroscience, computational modelling and clinical audiology, and brought together scientists, clinicians and industry professionals to progress research and applications beyond the current state-of-the-art.