Augmenting Art Crossmodally: Possibilities and Pitfalls

Charles Spence^1,2*Nicola Di Stefano³

• ¹University of Oxford, Oxford, United Kingdom
• ²Department of Experimental Psychology, Medical Sciences Division, University of Oxford, Oxford, England, United Kingdom
• ³Institute of Cognitive Sciences and Technologies, Department of Human and Social Sciences, Cultural Heritage, National Research Council (CNR), Rome, Lazio, Italy

In this narrative historical review, we take a closer look at the question of whether it is possible to augment works of art through crossmodal (specifically audiovisual) means. We start by highlighting an important distinction between three classes of audiovisual crossmodal correspondence: Namely those operating on individual sensory stimuli (so-called basic correspondences), those operating on dynamically-changing stimuli, or else on combinations of unisensory stimuli (so-called mid-level correspondences), and those operating on complex and often aesthetically-meaningful stimuli, such as music and paintings. We also highlight another important distinction between the literature on crossmodal matching and that dedicated to demonstrating crossmodal effects. The latter distinction aligns, in some sense, onto the distinction between crossmodal mapping and crossmodal effects. Although it may not be possible, in any meaningful sense, to translate works of art from one modality into another, that does not deny the possibility of augmenting a work of art by the deliberate addition of stimulation presented to another sensory modality. The aims and objectives of those who have attempted to augment works of art by introducing additional sensory stimulation are discussed.We also draw attention to a number of challenges and/or pitfalls (such as the distraction offered by recourse to the phenomenon of synaesthesia) for those interested in augmenting auditory/visual art crossmodally.