AUTHOR=Chan Vincent Y., Jin Craig T., van Schaik André 

TITLE=Neuromorphic Audio-Visual Sensor Fusion on a Sound-Localising Robot

JOURNAL=Frontiers in Neuroscience

VOLUME=Volume 6 - 2012

YEAR=2012

URL=https://www.frontiersin.org/journals/neuroscience/articles/10.3389/fnins.2012.00021

DOI=10.3389/fnins.2012.00021

ISSN=1662-453X

ABSTRACT=This paper presents the first robotic system featuring audio-visual sensor fusion with neuromorphic sensors. We combine a pair of silicon cochleae and a silicon retina on a robotic platform to allow the robot to learn sound localisation through self-motion and visual feedback, using an adaptive ITD-based sound localisation algorithm. After training, the robot can localise sound sources (white or pink noise) in a reverberant environment with an RMS error of 4 to 5 degrees in azimuth. In the second part of the paper, we investigate the source binding problem. An experiment is conducted to test the effectiveness of matching an audio event with a corresponding visual event based on their onset time. The results show that this technique can be quite effective, despite its simplicity.