Exploring Cross-Modal Perception in a Virtual Classroom: The Effect of Visual Stimuli on Auditory Selective Attention

Carolin Breuer^1*Lukas Vollmer^1*Larissa Leist²Stephan Fremerey³Alexander Raake³Maria Klatte²Janina Fels¹

• ¹Institute for Hearing Technology and Acoustics, RWTH Aachen University, Aachen, Germany
• ²Cognitive and Developmental Psychology, RPTU Kaiserslautern-Landau, Kaiserslautern, Rhineland-Palatinate, Germany
• ³Audiovisual Technology Group, Technical University Ilmenau, Ilmenau, Germany

In virtual reality research, distinguishing between auditory and visual influences on perception has become increasingly challenging. To study auditory selective attention in more close-to-real-life settings, an auditory task was adapted to a virtual classroom. The new environment suggested evidence of increased attention, possibly introduced by the visual representation, gamification effects, and immersion. This could engage participants more effectively. To delve deeper into the impact of cross-modal effects, the paradigm was extended by visual stimuli. Participants were initially tasked with directing their auditory attention to a cued spatial position and categorizing animal names played from that position while ignoring distracting sounds. Animal pictures introduced in Experiment 1 were either congruent or incongruent with the auditory target stimuli, thus either supporting or competing with the auditory information. The concurrent presentation of animal pictures with the animal names increased response times compared to the auditory condition, and incongruent visual stimuli increased response times more than congruent ones. Fewer errors were made with congruent compared to incongruent pictures, and error rates of the auditory condition fell in between. When the visual stimulus was presented 750 ms or 500 ms before the auditory stimuli in Experiment 2, auditory and visual congruence effects interacted. In the 500 ms case, visually congruent stimuli decreased error rates in auditory incongruent trials. Conversely, visually incongruent stimuli decreased error rates on auditory incongruent trials at 750 ms. This reversal of effects suggests a positive priming effect at 500 ms and a semantic inhibition of return effect at 750 ms. Taken together, these findings indicate that cross-modal priming is at least partially different from multisensory integration.