Audio-visual integration for objects, location and low-level dynamic stimuli: novel insights from studying sensory substitution and topographical mapping
-
1
The Hebrew University of Jerusalem, Israel
-
2
Harvard Medical School, Boston, Department of Neurology, United States
-
3
High Tech Campus, Netherlands
The talk will present fMRI and behavioral experiments of auditory-visual integration in humans. It will focus on integration in sighted but also in sight restoration set-up, looking into the effects of learning and brain plasticity. New findings regarding the nature of sensory representations for dynamic stimuli ranging from pure tones to complex, natural object sounds will be presented. Firstly, I will highlight the use of sensory substitution devices (SSDs) in the context of blindness. In SSDs, visual information captured by an artificial receptor is delivered to the brain using non-visual sensory information. Using an auditory-to-visual SSD called "The vOICe" we find that blind achieve successful performance on object recognition and localization tasks, and specific recruitment of ventral and dorsal 'visual' structures. Comparable recruitment was observed also in sighted learning to use this device but not in sighted learning arbitrary associations between sounds and object identity. These results suggest "The vOICe" can be useful for blind individuals' daily activities but it also has a potential use to ‘guide’ visual cortex to interpret visual information arriving from prosthesis.
The three topographical senses (vision, audition and touch) are characterized by a topographical mapping of the sensory world onto primary and secondary brain areas. In such topographical maps, adjacent neurons represent adjacent sensory building blocks (e.g. visual field, tone frequency and body parts) in each sense. Using fMRI, we applied continuous and periodic sensory stimulation, to detect further topographically sensitive areas in the auditory modality. We used phase locking Fourier Techniques combined with spherical cortex-based alignment approach to detect such topographic maps. Using these methods, I will report in the second part of the talk the preliminary finding of multiple novel tonotopic maps in humans organized in mirror-symmetric representations, stretching from primary auditory cortex to occipito-temporal and parietal-temporal cortex. These maps are organized in parallel, mirror symmetry bands as found in the visual cortex. Our results suggest that mirror symmetry topographical mapping may be a fundamental principle of mapping in both vision and audition and might be a more common characteristic of associative cortex than previously suspected. Such maps, some of them located in the border of temporal and occipital and parietal cortex can serve as a basis for audio-visual transformations required in SSDs and reported in the first part of the talk. This work has been supported by an R21-EY0116168 grant (To APL), the International Human Frontiers Science Program Organization LTF grant and by an EU reintegration FP7 grant (to AA).
Conference:
10th International Conference on Cognitive Neuroscience, Bodrum, Türkiye, 1 Sep - 5 Sep, 2008.
Presentation Type:
Oral Presentation
Topic:
Symposium 3: Neural basis of multisensory integration
Citation:
Amedi
A,
Striem
E,
Hertz
U,
Stern
W,
Merabet
L,
Meijer
P and
Pascual-Leone
A
(2008). Audio-visual integration for objects, location and low-level dynamic stimuli: novel insights from studying sensory substitution and topographical mapping.
Conference Abstract:
10th International Conference on Cognitive Neuroscience.
doi: 10.3389/conf.neuro.09.2009.01.019
Copyright:
The abstracts in this collection have not been subject to any Frontiers peer review or checks, and are not endorsed by Frontiers.
They are made available through the Frontiers publishing platform as a service to conference organizers and presenters.
The copyright in the individual abstracts is owned by the author of each abstract or his/her employer unless otherwise stated.
Each abstract, as well as the collection of abstracts, are published under a Creative Commons CC-BY 4.0 (attribution) licence (https://creativecommons.org/licenses/by/4.0/) and may thus be reproduced, translated, adapted and be the subject of derivative works provided the authors and Frontiers are attributed.
For Frontiers’ terms and conditions please see https://www.frontiersin.org/legal/terms-and-conditions.
Received:
27 Nov 2008;
Published Online:
27 Nov 2008.
*
Correspondence:
Amir Amedi, The Hebrew University of Jerusalem, Jerusalem, Israel, Amir.Amedi@ekmd.huji.ac.il