Understanding why we see things we don’t see
If a tree falls in the woods and you hear the sounds of it crashing to the floor, do you also see it? An event such as a tree fall is multisensory – there are sounds, sights and smells associated with it. But often we only experience these events through one sense – in this case, our hearing. So how do we recreate the image of it in our mind? This multisensory perception was the focus of the STYDS project, which was funded by the European Research Council (ERC). STYDS sought to bring together all empirical research on ‘multimodal mental imagery’, a perceptual phenomenon in which stimulation from one sense triggers another – in this case, audition to vision. “This project aimed to unify strands of research that have not been previously brought into contact with each other, including various psychological findings in neuroscience, philosophy, and so on, and to synthesise,” explains Bence Nanay, professor of Philosophy at the University of Antwerp and STYDS project coordinator.
Forming bonds between philosophy and science
Nanay conceived the concept of multimodal mental imagery, yet people have long pondered over the phenomenon – and not only scientists. Marcel Proust had many examples and observations in his books, Nanay notes, and film directors also think hard about the idea and use it a lot in their work. Through the project, Nanay spoke with several researchers to further develop the idea of multimodal imagery. “I am a somewhat unusual philosopher in the sense that I consider it to be part of my job to talk to and collaborate with empirical researchers who study the same questions as I do,” he notes. “The idea is not just to come up with empirically grounded philosophical theories, but also to have a bidirectional interaction between philosophy and the sciences where the two mutually influence each other,” Nanay adds.
Helping the blind to navigate the world
The most important concrete outcome was the book ‘Mental Imagery: Philosophy, Psychology, Neuroscience’, published by Oxford University Press in 2023. “I wrote this book in a way that no special training in either philosophy or the empirical sciences would be needed for the reader to follow the arguments,” Nanay says. The project also spawned another research avenue with potential real-world impacts for blind people. Contrary to popular belief, Nanay explains, most blind people have (multimodal) visual imagery and this visual imagery actually plays a crucial role in their navigation. This includes with the white cane, by echolocation, and even in Braille reading. Yet if one part of the brain is not used, it will be reallocated. If a blind person doesn’t use their visual cortices, they will be reassigned to something like language processing, which makes using visual areas of the brain on navigation tasks more difficult. This understanding came out of the project, as did a solution to train visual cortices of blind people and help them to navigate, which is now funded by an ERC Proof of Concept grant. “This could have an enormous impact on the quality of life of blind people,” says Nanay. “It’s also an example of how an interdisciplinary project led by a philosopher could have some real-life impact.”
Keywords
STYDS, multimodal, sensory, perception, philosophy, psychology, neuroscience, blind, navigate