Rainbows in my ears—Synesthetic color perception with partial-reduced and morphed musical instrument timbres

Christoph Reuter^1*Isabella Czedik-Eysenberg¹Sarah Ambros¹Saleh Siddiq¹Solange Glasser²Jamie Ward³Jörg Jewanski¹

• ¹University of Vienna, Vienna, Austria
• ²The University of Melbourne, Melbourne, Australia
• ³University of Sussex, Brighton, United Kingdom

Abstract Background: Synesthesia is a perceptual phenomenon in which sensory inputs automatically elicit consistent additional sensations. Sound-color synesthesia, one of the most widely recognized forms, involves experiencing colors and shapes in response to auditory stimuli. Within this, timbre-color synesthesia, reported in 26% of music-color synesthetes, has rarely been studied in depth, particularly regarding how specific timbre manipulations influence color perception. Methods: This study investigated whether changes in timbre-induced color perception occur continuously or abruptly as sound parameters are altered, and whether generalizable patterns can be identified. For the first time, ΔE color similarity calculations were applied to objectively assess participants' timbre-color synesthesia. Participants listened to loudness-matched single tones (flute, oboe, French horn, violin, and piano) from the Vienna Symphonic Library, presented both in their original form and with manipulated timbres. Manipulations included partial reduction (removing partials from the first to the tenth) and morphing between instrument pairs across seven stages (from 100:0% to 0:100%). After each sound, participants described their color perception using a color selection field, generating RGB and HSL values of the chosen colors. Results: Findings revealed correlations between the individual color perceptions. Increasing partial reduction was associated with decreased saturation and increased lightness, particularly for flute, oboe, violin, and piano. Using audio features such as spectral centroid, harmonicity strength, and percussive loudness, generalizable rules for the perception of timbre-induced colors in timbre-color synesthetes could be observed. For morphed sounds, perceived colors shifted progressively from those associated with the initial instrument toward those associated with the target instrument across the morphing stages. Here, too, spectral centroid, harmonicity strength, and percussive loudness showed strong overall correlations between instrumental sounds and the colors they induce. Conclusion: This study demonstrates that manipulating musical instrument timbre through partial reduction and morphing induce clear but individual patterns of color perception in timbre-color synesthetes, which differ significantly from the responses of non-synesthetes.