Periodic Reporting for period 4 - SONORA (The Spatial Dynamics of Room Acoustics)
Período documentado: 2022-11-01 hasta 2023-10-31
The SONORA project aims to increase the general understanding of how complex sound scenes are impacted by the spatial dynamics of room acoustics. This knowledge is crucial in the design of signal processing algorithms for audio acquisition and reproduction problems in real-life situations, where moving sound sources and observers interact with room acoustics in a complicated manner.
A major part of the project will be devoted to the development of novel room acoustics models and to the unification of existing models. The room acoustics models developed in this project will be data-driven models with a physically motivated structure, and are expected to fill the existing gap between geometric and wave-based models. This will be achieved by formulating existing and novel models in a dictionary- based mathematical framework and introducing a new concept coined as the equivalent boundary model, aimed at relaxing the prior knowledge required on the physical room boundary.
A second part of the project will focus on the development of a protocol for measuring spatiotemporal sound fields. This protocol will be rooted in a novel sound field sampling theory which exploits the spatial sparsity of sound sources by invoking the compressed sensing paradigm.
Thirdly, novel signal processing algorithms capable of handling spatiotemporal sound fields will be designed. By employing recent advances in large-scale optimization and multidimensional scaling, fast and matrix-free algorithms will be obtained that do not require prior knowledge of the sound scene geometry.
The SONORA research results are anticipated to have a notable impact in various audio acquisition and reproduction problems, including acoustic signal enhancement, audio analysis, room inference, virtual acoustics, and spatial audio reproduction. These problems have many applications in speech, audio, and hearing technology, hence a significant benefit for industry and for technology end users is expected in the long run.
A major part of the project will be devoted to the development of novel room acoustics models and to the unification of existing models. The room acoustics models developed in this project will be data-driven models with a physically motivated structure, and are expected to fill the existing gap between geometric and wave-based models. This will be achieved by formulating existing and novel models in a dictionary- based mathematical framework and introducing a new concept coined as the equivalent boundary model, aimed at relaxing the prior knowledge required on the physical room boundary.
A second part of the project will focus on the development of a protocol for measuring spatiotemporal sound fields. This protocol will be rooted in a novel sound field sampling theory which exploits the spatial sparsity of sound sources by invoking the compressed sensing paradigm.
Thirdly, novel signal processing algorithms capable of handling spatiotemporal sound fields will be designed. By employing recent advances in large-scale optimization and multidimensional scaling, fast and matrix-free algorithms will be obtained that do not require prior knowledge of the sound scene geometry.
The SONORA research results are anticipated to have a notable impact in various audio acquisition and reproduction problems, including acoustic signal enhancement, audio analysis, room inference, virtual acoustics, and spatial audio reproduction. These problems have many applications in speech, audio, and hearing technology, hence a significant benefit for industry and for technology end users is expected in the long run.
The main results of the SONORA project are summarized here, along the three overall objectives: modeling, sensing, processing.
Modeling: we have worked to develop novel room acoustics models that are capable of capturing the spatiotemporal behavior of complex sound scenes. These models possess desirable properties for signal processing purposes by being scalable to varying room types and sizes, robust to physical uncertainties and variations, physically interpretable, perceptually relevant, invertible and cheaply auralizable.
- A novel equivalent source model for room acoustics, which associates a directional impulse responses rather than a free-field Green's function to each equivalent source:
https://doi.org/10.1109/TASLP.2017.2730284
- A novel data-driven room acoustics model based on low-rank matrix/tensor decompositions, which comes with a solid physical justification that is rooted in the modal theory of room acoustics:
https://eurasip.org/Proceedings/Eusipco/Eusipco2021/pdfs/0000111.pdf
- A novel convergence theorem for modeling acoustic wave propagation with the multipole expansion method (MEM), that allows to optimally select the MEM truncation order:
https://doi.org/10.1137/20M1370914
- A novel relation between geometric and wave-based models for acoustic wave propagation, based on the observation that the solution to the boundary integration equation can be asymptotically represented by only considering specular reflection points on the boundary of the domain:
https://appfa2023.silsystem.solutions/atti/001143.pdf
Sensing: we have aimed to develop a novel framework for sensing room acoustics, embracing novel sampling theories and practical measurement protocols for spatiotemporal sound fields.
- A classical Nyquist-Shannon sampling and interpolation theory for a number of spatiotemporal sound field sensing problems in which a rigorous foundation was previously lacking:
https://arxiv.org/abs/2012.09499
https://doi.org/10.1109/I3DA48870.2021.9610902
https://eurasip.org/Proceedings/Eusipco/Eusipco2022/pdfs/0002256.pdf
- A compressed sensing framework for acoustic source localization and power-spectral-density (PSD) estimation, both in single-sensor and multi-sensor contexts:
https://doi.org/10.1186/s13636-023-00304-8
https://arxiv.org/abs/2306.08514
https://doi.org/10.1109/WASPAA58266.2023.10248095
- Optimal microphone selection/positioning algorithms for sensing spatiotemporal sound fields in various problem settings:
https://doi.org/10.1109/IWAENC53105.2022.9914798
https://tinyurl.com/3xa5fz68
- An extensive open-source database of source/receiver signals, RIR measurements, and noise field measurements with diverse microphone arrays and room types:
https://doi.org/10.1186/s13636-023-00284-9
Processing: we have designed novel signal processing algorithms addressing basic estimation, prediction, and simulation problems related to audio acquisition and reproduction in complex sound scenes, leveraging the desirable properties of the newly developed room acoustics models.
- A collection of numerical optimization algorithms that are particularly suited for estimating equivalent source models for room acoustics:
https://arxiv.org/abs/1803.01621
- A low-rank-promoting algorithm for estimating low-rank data-driven models for room acoustics directly from source/receiver data:
https://doi.org/10.1109/TASLP.2023.3240650
- Novel single- and multi-channel convolution algorithms that can operate directly on low-rank room acoustics models, resulting in fast convolution at low latency:
https://ftp.esat.kuleuven.be/stadius/mjalmby/23-149.pdf
https://ftp.esat.kuleuven.be/stadius/mjalmby/23-150.pdf
- New algorithms targeted at solving the fundamental problem of sorting and clustering TDOA estimates corresponding to first-order and second-order room reflections:
https://eurasip.org/Proceedings/Eusipco/Eusipco2021/pdfs/0001730.pdf
https://doi.org/10.1109/ICASSP49357.2023.10096005
Modeling: we have worked to develop novel room acoustics models that are capable of capturing the spatiotemporal behavior of complex sound scenes. These models possess desirable properties for signal processing purposes by being scalable to varying room types and sizes, robust to physical uncertainties and variations, physically interpretable, perceptually relevant, invertible and cheaply auralizable.
- A novel equivalent source model for room acoustics, which associates a directional impulse responses rather than a free-field Green's function to each equivalent source:
https://doi.org/10.1109/TASLP.2017.2730284
- A novel data-driven room acoustics model based on low-rank matrix/tensor decompositions, which comes with a solid physical justification that is rooted in the modal theory of room acoustics:
https://eurasip.org/Proceedings/Eusipco/Eusipco2021/pdfs/0000111.pdf
- A novel convergence theorem for modeling acoustic wave propagation with the multipole expansion method (MEM), that allows to optimally select the MEM truncation order:
https://doi.org/10.1137/20M1370914
- A novel relation between geometric and wave-based models for acoustic wave propagation, based on the observation that the solution to the boundary integration equation can be asymptotically represented by only considering specular reflection points on the boundary of the domain:
https://appfa2023.silsystem.solutions/atti/001143.pdf
Sensing: we have aimed to develop a novel framework for sensing room acoustics, embracing novel sampling theories and practical measurement protocols for spatiotemporal sound fields.
- A classical Nyquist-Shannon sampling and interpolation theory for a number of spatiotemporal sound field sensing problems in which a rigorous foundation was previously lacking:
https://arxiv.org/abs/2012.09499
https://doi.org/10.1109/I3DA48870.2021.9610902
https://eurasip.org/Proceedings/Eusipco/Eusipco2022/pdfs/0002256.pdf
- A compressed sensing framework for acoustic source localization and power-spectral-density (PSD) estimation, both in single-sensor and multi-sensor contexts:
https://doi.org/10.1186/s13636-023-00304-8
https://arxiv.org/abs/2306.08514
https://doi.org/10.1109/WASPAA58266.2023.10248095
- Optimal microphone selection/positioning algorithms for sensing spatiotemporal sound fields in various problem settings:
https://doi.org/10.1109/IWAENC53105.2022.9914798
https://tinyurl.com/3xa5fz68
- An extensive open-source database of source/receiver signals, RIR measurements, and noise field measurements with diverse microphone arrays and room types:
https://doi.org/10.1186/s13636-023-00284-9
Processing: we have designed novel signal processing algorithms addressing basic estimation, prediction, and simulation problems related to audio acquisition and reproduction in complex sound scenes, leveraging the desirable properties of the newly developed room acoustics models.
- A collection of numerical optimization algorithms that are particularly suited for estimating equivalent source models for room acoustics:
https://arxiv.org/abs/1803.01621
- A low-rank-promoting algorithm for estimating low-rank data-driven models for room acoustics directly from source/receiver data:
https://doi.org/10.1109/TASLP.2023.3240650
- Novel single- and multi-channel convolution algorithms that can operate directly on low-rank room acoustics models, resulting in fast convolution at low latency:
https://ftp.esat.kuleuven.be/stadius/mjalmby/23-149.pdf
https://ftp.esat.kuleuven.be/stadius/mjalmby/23-150.pdf
- New algorithms targeted at solving the fundamental problem of sorting and clustering TDOA estimates corresponding to first-order and second-order room reflections:
https://eurasip.org/Proceedings/Eusipco/Eusipco2021/pdfs/0001730.pdf
https://doi.org/10.1109/ICASSP49357.2023.10096005
The SONORA project has delivered a number of research results that go beyond the state of the art in room acoustics modeling, sensing, and processing, along the following themes:
(1) Time domain processing and convolution: dynamic sound scenes involving moving sources and observers call for a time-domain approach to modeling, sensing, and processing. This requirement conflicts with the widely used frequency-domain approach to wave propagation modeling and acoustic signal processing. The project’s heavy focus on efficiently representing and computing continuous-time convolutions is the key element to facilitating time-domain processing.
(2) Sampling and interpolation: by recognizing that all room acoustics models rely on a number of spatial reference points and that microphone array measurements provide spatiotemporal samples of an acoustic wave field, various modeling and measurement approaches can be unified in a sampling framework. The establishment of this framework moreover allows to develop optimal interpolation methods and rigorously analyze the impact of undersampling.
(3) Hybrid geometric and wave-based modeling: the project has delivered new insights on how geometric models and wave-based models, that have traditionally been treated separately, are related. This has been achieved by showing under which conditions the (wave-based) solution to the boundary integral equation admits a geometric interpretation.
(1) Time domain processing and convolution: dynamic sound scenes involving moving sources and observers call for a time-domain approach to modeling, sensing, and processing. This requirement conflicts with the widely used frequency-domain approach to wave propagation modeling and acoustic signal processing. The project’s heavy focus on efficiently representing and computing continuous-time convolutions is the key element to facilitating time-domain processing.
(2) Sampling and interpolation: by recognizing that all room acoustics models rely on a number of spatial reference points and that microphone array measurements provide spatiotemporal samples of an acoustic wave field, various modeling and measurement approaches can be unified in a sampling framework. The establishment of this framework moreover allows to develop optimal interpolation methods and rigorously analyze the impact of undersampling.
(3) Hybrid geometric and wave-based modeling: the project has delivered new insights on how geometric models and wave-based models, that have traditionally been treated separately, are related. This has been achieved by showing under which conditions the (wave-based) solution to the boundary integral equation admits a geometric interpretation.