Colour–Grapheme Synesthesia Affects Binocular Vision

In colour–grapheme synesthesia, non-coloured graphemes are perceived as being inherently coloured. In recent years, it is debated whether visual processing of synesthesia-inducing achromatic graphemes is similar to that of chromatic graphemes. Here, we exploit the phenomenon of binocular rivalry in which incompatible images presented dichoptically compete for conscious expression. Importantly, the competition only arises if the two images are sufficiently different; if the difference between the images is small, the images will fuse into a single mixed percept. We show that achromatic digits that induce synesthetic colour percepts increase the incidence of binocular rivalry compared to achromatic non-digits that do not evoke such percepts. That is, compared to achromatically perceived non-digits, synesthesia-inducing digits increase the predominance of binocular rivalry over binocular fusion. This finding shows that the synesthetic colour experience can provide the conditions for promoting binocular rivalry, much like stimulus features that induce rivalry in normal vision.

Introduction

In grapheme–colour synesthesia, non-coloured graphemes (i.e., letters or digits) may be perceived as inherently coloured. Evidence is somewhat mixed on the question of whether perception of synesthesia-inducing achromatic stimuli is comparable to that of chromatic stimuli. In a variety of behavioral studies, the influence of the synesthetic colour experience on performance on several tasks has been tested, both at a perceptual and a cognitive level. Examples range from experiments on visual search (e.g., Ramachandran and Hubbard, 2001) and perceptual grouping (e.g., Kim et al., 2006), to Stroop-like interference effects and priming effects (e.g., Gebuis et al., 2009a,b; Mattingley et al., 2001). Results of these studies suggest that synesthesia-inducing achromatic stimuli can affect processing in a manner comparable to chromatic stimuli. On the other hand, synesthetic hues appear to be impervious to brightness contrast, are differentially affected by simultaneous colour contrast (Nijboer et al., 2011), and fail to induce chromatic after-effects (Hong and Blake, 2008), suggesting that they are not identical to veridical colour at the earliest stages of visual cortical processing.

In this study we investigate how synesthetic colours affect binocular vision. When two identical images are projected to overlapping retinal locations, the images fuse, and a single stable object is perceived. However, when the images are sufficiently different, the intriguing phenomenon of binocular rivalry will typically occur, where both images will start alternating in perception (for a review on the phenomenon, see Blake and Logothetis, 2002). Thus, when two gratings with the same orientation are presented to overlapping locations, a stable percept of a single grating will emerge. If, however, the orientation of one of the gratings is rotated by 90°, perception will start to alternate between both gratings. Likewise, when two different digits (e.g., 2 and 5) in digital font are presented in white (Figure 1, achromatic digits), the contours of the digits will fall on the same retinal locations, leading to significant periods of fusion. However, when we add different colours to the digits (Figure 1, chromatic digits), the overlapping contours will have a difference in hue, and possibly in luminance. This difference in hue at the overlapping contours will increase the total period of rival percepts compared to that of fused percepts¹. We reasoned that if processing of synesthesia-inducing achromatic graphemes is similar to that of chromatic graphemes, significant periods of binocular rivalry should arise when synesthetes view achromatic digits. Thus, we expect that rival images that should fuse based on their physical characteristics will actually rival because of the synesthetic experience they induce.

FIGURE 1

Figure 1. The stimuli used in the binocular rivalry experiment. Four basic conditions were used: achromatic digits (top-left), chromatic digits (top-right), achromatic non-digits (bottom-left), and chromatic non-digits (bottom-right). Each stimulus pair was presented dichoptically. The colours are those set by one of the synesthetic participants.

Materials and Methods

Participants

Fifteen Synesthetes (3 males, 12 females; mean age 26.9, SD 9.0) and 15 pair-matched controls (4 males, 11 females, mean age 26.5, SD 8.2) took part in the experiment. Except for one pair, synesthetes and controls were matched on sex and age. In addition, each pair would be presented with the same colours, as acquired in a colour matching experiment by the synesthete of the pair. Colour–grapheme synesthesia was assessed using a questionnaire based on the one used by Rouw and Scholte (2007). Only synesthetes with no history of substance abuse, neurological disorders, and the presence of grapheme–colour associations were included. All participants had normal or corrected to normal visual acuity and reported no colour-blindness. Two of the authors (Chris L. E. Paffen and Maarten J. van der Smagt) served as controls, the rest of the participants were naïve as to the purpose of the study. Participants gave informed consent to participate in the study according to the Declaration of Helsinki.

Apparatus

Stimuli were presented using an Apple dual 2 GHz PowerPC G5 and a linearized LaCie Electron blue IV 22″ monitor, using MATLAB and the Psychtoolbox extensions. Dichoptic presentation was achieved using a mirror stereoscope. The length of the optical path (from the monitor via the mirrors to the observer’s eyes) was 57 cm.

Procedure and Stimuli

All experiments were conducted in a dimly lit room, while a chinrest supported the observer’s head. Synesthetes started this experiment by indicating the exact hues that were associated with digits 2, 5, 6, and 9 [dimensions (height versus width) were 3.9° by 2.0°], by matching these onto a disk (2.6° in diameter). These associations were tested again after the experiment, to verify consistency (i.e., test–retest reliability). This matching procedure was performed binocularly (without the use of mirrors). The luminance of the gray background was 27.9 cd/m². The luminance and colour settings of the chromatic images are summarized in Section “Appendix” (also see Results colour matching experiment).

Two dimensions were varied in the experiment: digit (2 versus 5, 6 versus 9) versus non-digit (two types of cocktail-glasses; see Figure 1) stimuli and chromatic versus achromatic presentations, leading to four basic stimulus sets: chromatic digits, chromatic non-digits, achromatic digits, and achromatic non-digits (dimensions of each image were 3.0° by 1.5°). Each synesthete and his or her matched control observer performed in the exact same conditions with the exact same hues in the chromatic conditions. Thus, the colours indicated by a single synesthete in a colour matching procedure were used in the chromatic rivalry condition by this synesthete and the matched control. For the chromatic non-digit conditions this leads to two combinations: upright cocktail-glass versus inverted cocktail-glass with matched colours of the 2 and the 5, and upright cocktail-glass versus inverted cocktail-glass with matched colours of the 6 and the 9. Figure 1 summarizes all the combinations used in the binocular rivalry experiment. The luminance of the gray background was again 27.9 cd/m² and the luminance of the achromatic images 55.1 cd/m².

Synesthetes and control participants were presented with pairs of chromatic and achromatic digitized digits and non-digits (see Figure 1), both presented dichoptically, such that their contours largely overlapped. In case of a chromatic pair, the contours in both eyes differed in hue and potentially in luminance. As a result, interocular conflict should arise, which should give rise to perceptual alternations between the images, where the digits were exclusively dominant in alternation. In the achromatic condition, however, the overlapping contours of the digits differed neither in luminance nor in hue. In this stimulus, there was – based on the physical characteristics of the images – only little interocular conflict, so it was expected that a mixture of the two images would be perceived for most of the time. Crucially however, if synesthetic colour percepts induced by the digits behave as typical colours, the proportion of time exclusive dominance occurred when viewing the achromatic digits should be much higher for synesthetes than for controls. The fraction of time either image was exclusively dominant (corresponding to summed exclusive dominance of both images) was therefore taken as the primary measure.

A trial in the binocular rivalry experiment lasted 30 s. Each rivalry combination was balanced between the eyes; every unique stimulus presentation was repeated twice, leading to a total of 48 trials. Each observer performed the experiment in two blocks, each lasting about 12 min. The participants were instructed to press a button only when one of the images was fully dominant. The reason for this instruction was that for the achromatic conditions, apart from exclusive dominance of one of the two images, perception of the display might include multiple mixed percepts, where different bits and pieces of the two half-images could combine (i.e., fuse) into a single percept. For example, the “2” and “5” might be combined and perceived as “8,” but also as “6.” To limit the number of response options, we instructed the observers only to press a button when one of the images was exclusively dominant. Before each trial, written text on the screen indicated which keys to use for each image. For example, when the text read “press the ← button for the 2 and the → for the 5” the observer pressed the left arrow key for as long the 2 was fully dominant in perception and the right arrow key for as long the 5 was fully dominant. Participants were further instructed to fixate the fixation cross and to refrain from making eye movements.

Data Analysis

For our analysis we collapsed the different conditions to the following four: achromatic digits, achromatic non-digits, chromatic digits, and chromatic non-digits. We used the total fraction of time that the images were exclusively dominant as the primary measure. A trial lasted for 30 s; if, for example, the 2 were the dominant percept for 5 s on average, and the 5 for 6 s, the fraction would be 0.37 [fraction dominance is (5 + 6)/30 = 0.37]. We chose to use total fraction dominance instead of fraction dominance per digit since a change in perceptual strength of one digit will change the dominance duration of that digit, but also that of its rival. Therefore, if for a synesthete the 2 would be perceived as red, but the 5 would only give rise to a very weak synesthetic colour experience, the dominance of the 5 would still be affected: the red 2 and the (almost) white 5 differ in hue, so the prerequisite for exclusive dominance of both of them is met.

Results

Results Colour Matching Procedure

The results of the colour matching experiment are summarized in Section “Appendix.” Three of the synesthetes experienced colours for the non-digit stimuli. These participants and their matched controls were removed from further analysis (shaded areas in Appendix), resulting in 12 participants per group remaining.

Figure 2 shows the test–retest reliability of the colour settings, which was quite high: Pearson’s correlation (rho) between pre- and post-test was 0.91, 0.97, and 0.95 for Luminance, CIEu, and CIEv, respectively. Interestingly, digits with a larger numerical value were set to lower luminance values, reflected in a negative correlation between the value of the digit on the one hand and the luminance of the colour match on the other (Pearson’s rho = −0.37, p < 0.005). This replicates the finding that digits with a larger numerical value are generally experienced as more dark in colour–grapheme synesthesia (Cohen Kadosh et al., 2007).

FIGURE 2

Figure 2. Results of the colour matching by the synesthetes. Test–retest reliability for set luminance (left), CIEu (middle), and CIEv (right) is high (see text for details).

Results Binocular Rivalry Experiment

The results (calculated as total fraction dominance) are displayed in Figure 3. We first performed a repeated measures ANOVA with colour (achromatic versus chromatic) and digit (digit versus non-digit) as within-subjects factors, and group (synesthetes versus controls) as between-subjects factor. There was a main effect of colour [F(1,22) = 57.2, p < 0.0001], no main effect of digit [F(1,22) = 3.4, p = 0.08], but a significant interaction between them [F(1,22) = 4.7, p = 0.04]. The main effect of colour indicates that chromatic images induced longer periods of rivalry compared to achromatic images. The absence of a main effect of digit indicates that total fraction dominance was not different for digits and non-digits. The significant interaction between these two reflects the fact that total fraction dominance for achromatic digits was larger than for achromatic non-digits, while there was no difference between chromatic digits and non-digits. Importantly, there was no main effect of group [F(1,22) = 2.9, p = 0.1], nor any other significant interaction between group and the two within subject factors. This analysis validates our assumption that adding colour increases the fraction of time binocular rivalry occurs, in both synesthetes and controls. Furthermore, we found no significant effects of group and digit. Thus, we found no general difference between synesthetes and controls: binocular rivalry is not generally different in synesthetes compared to controls. Digits and non-digits do not lead to fundamental differences in the experience of binocular rivalry either. We hypothesize that the interaction between colour and digit (the higher dominance fraction for achromatic digits versus achromatic non-digits) was mainly driven by synesthetes: our hypothesis states that rivalry in achromatic digits should be selectively increased in synesthetes and not in controls. To evaluate this, we compared total fraction dominance between synesthetes and controls. Unpaired t-tests reveal that total fraction dominance of achromatic digits was significantly larger in synesthetes than in controls [t(22) = 2.5, p = 0.04²]. For the other three conditions, no such difference was observed [t(22) < 1.2, p > 0.4].

FIGURE 3

Figure 3. Results of the binocular rivalry experiment. The bars show the fraction total perceptual dominance for synesthetes and controls for the four basic conditions (achromatic digits, achromatic non-digits, chromatic digits, chromatic non-digits). Total dominance in the achromatic digits condition is larger for synesthetes than for controls (*). For synesthetes, but not for controls, total fraction dominance for the achromatic digit condition is larger than for the achromatic non-digit condition (#). Error bars are 1 SEM.

Of importance to the current study is whether the perceptual alternations that synesthetes experienced when viewing achromatic graphemes were similar to those experienced by non-synesthetes in coloured stimuli. As discussed above (i.e., with the repeated measures ANOVA), we found no interaction between group and colour: total fraction dominance for coloured stimuli was not different between synesthetes and controls. To evaluate whether rivalry induced by achromatic stimuli in synesthetes differs from that induced by chromatic stimuli in non-synesthetes, we analyzed whether exclusive dominance for the achromatic digits for the synesthetes was different from that of chromatic digits for the controls. We found no difference in fraction dominance between synesthetes and their matched controls in this comparison [t(22) = −0.99, p = 0.33]. Although this does not mean that dominance for achromatic digits in synesthetes is the same as that for chromatic digits in controls, it does imply that they may be comparable in magnitude.

Our final analysis tests whether colour–grapheme synesthesia increased rivalry of achromatic digits compared to non-digits. Using paired t-tests, we found that for synesthetes, total fraction dominance was significantly larger for achromatic digits than for achromatic non-digits [t(11) = 2.3, p = 0.04], whereas no such difference was apparent for controls [t(11) = 0.8, p = 0.4].

Together, these results suggest that binocular rivalry for synesthetes is different from controls with achromatic digits (due to their colour experience of the digits), but comparable to controls with chromatic digits.

General Discussion

The aim of the current study was to investigate to what extent synesthetic colour experiences affect binocular vision. We assessed whether synesthesia-inducing achromatic graphemes are able to induce perceptual alternations between dichoptic images in the same manner as chromatic graphemes do.

The results showed that more rivalry was induced with the achromatic digits for synesthetes compared to controls. Importantly, only for synesthetes, achromatic digits evoked an amount of rivalry comparable to that of chromatic digits. More specifically, for synesthetes, the amount of rivalry in the achromatic condition does not appear to be different to that induced by chromatic conditions in both synesthetes and controls. In other words, binocular rivalry was of comparable magnitude for achromatic and chromatic digits for the synesthetes, which suggests that the synesthesia-inducing graphemes affected perception similarly as coloured graphemes. That this effect was due to the synesthetic colour experience, and not due to other processes influencing rivalry in synesthetes in general, was verified with the achromatic non-digit condition, in which both synesthetes and controls showed a comparably low amount of rivalry. Moreover, only for synesthetes, the amount of rivalry in the achromatic digit condition was significantly larger than that in the achromatic non-digit condition.

Our study is not the first to exploit binocular rivalry for assessing the perceptual reality of colour–grapheme synesthesia. Kim et al. (2006) presented rivalry-inducing letters at two positions, left and right of fixation. The letters were chosen such that for synesthetes letters to the left of fixation induced colour experiences (e.g., A and B) that were similar to those for different letters on the right of fixation (e.g., C and D). Their results showed that joint predominance (i.e., the time letters at the left and right of fixation were dominant in perception at the same time) of letters inducing the same colour was increased relative to controls. This result corroborates our finding that the synesthetic colour experience can interact with binocular rivalry. However, our study is the first showing that synesthetic colours can affect binocular vision: synesthetic colours increased the incidence of rivalry over fusion.

Do our results provide insight into the level of processing at which the synesthetic colour experiences emerge? To evaluate this, we need to review the discussion that has dominated the study of binocular rivalry for decades. Historically, there have been two dominant theories on the nature of binocular rivalry: the low-level (“eye”) theory on the one hand, and the high-level (“image”) theory on the other (for a review see Blake and Logothetis, 2002). According to the low-level view, binocular rivalry is resolved at an early stage of visual processing. More explicitly, it has been proposed that the perceptual alternations experienced when viewing rival images are caused by competition between pools of neurons, receiving mainly monocular inputs, at the stage of primary visual cortex (Blake, 1989). On the other hand, the high-level view has claimed that rivalry is resolved at the stage were image representations have already been formed, implying a later stage at which the perceptual alternations are triggered (Kovács et al., 1996; Logothetis et al., 1996). These different views complicate an effort to pinpoint the exact stage of processing at which a synesthetic experience occurs. In recent years, however, it has been proposed that the low- and high-level view need not be mutually exclusive, but can be combined in a multistage, hierarchal model incorporating both. For example, Wilson (2003) has argued that the level at which binocular rivalry is resolved depends on the specifics of the stimulus: perceptual alternation will be driven by monocular neurons (i.e., at a low-level) during conventional interocular rivalry (i.e., when dissimilar images are presented continuously). However, in some special cases, most notably during flicker and swap rivalry (Logothetis et al., 1996), the alternations are triggered by binocular neurons, at a higher processing stage. According to the above hybrid model of binocular rivalry, our stimulus should lead to predominance of competition between monocular image representations. How then, can synesthetic colours increase the incidence of rivalry compared to fusion? Interestingly, a study by Carlson and He (2004) showed that global image representations failed to induce binocular rivalry. That is, in order for rivalry to dominate over fusion, local differences in dichoptic images are necessary; different global images with matched local elements fail to induce rivalry. Our results resemble those of Carlson and He (2004) in that the synesthetic colour experience interacts with visual perception at the stage where monocular projections of dichoptic images either lead to fusion or binocular rivalry. We therefore speculate that the synesthetic colour is modulating the competition between dissimilar, dichoptic images at an early level, perhaps at the level of V1. It is likely that the synesthetic colour affects the competition between the incompatible images via feedback to lower visual areas, since the way by which visual attention (Paffen et al., 2006), numerosity (Paffen et al., 2011), faces (Yu and Blake, 1992), and visual context (Paffen et al., 2004) affect binocular rivalry is generally taken to indicate that these higher level cognitive phenomena modulate lower level processing via feedback. Feedback is also likely since elaborate processing of colour is expected to occur after the level of V1, perhaps at and beyond the level of V4 (Bouvier and Engel, 2006).

Although we have treated the synesthetic observers as a single group, colour–grapheme synesthetes are generally classified along two dimensions: first (and widely used), projectors versus associators; and second, higher versus lower synesthetes. It has been suggested that both dimensions might be equivalent (Dixon and Smilek, 2005). In our study, five projectors and seven associators were included. When the data are split on this dimension, it is apparent that there were no differences in the amount of rivalry in the achromatic digit condition for both types of synesthetes. Thus, without questioning the possible distinction between projectors and associators, our data provide no indication that the synesthetic colour perception affects the amount of rivalry differently in these two groups. We have to note here that a null-result for the difference between projectors and associators might be due to the relatively small sample size.

To conclude, we report evidence that synesthetic colours affect binocular vision: digits evoking synesthetic colours made binocular rivalry prevail over binocular fusion as if the digits were actually coloured. The results imply that the synesthetic colour experience can interact at the stage at which monocular visual information from two eyes leads either to a fused or an unstable rival percept.

Conflict of Interest Statement

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Footnotes

1. ^This assumption will be validated in the main experiment.
2. ^p-Values were corrected for multiple comparisons using Bonferroni correction.

References

Blake, R. (1989). A neural theory of binocular rivalry. Psychol. Rev. 96, 145–167.

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Blake, R., and Logothetis, N. K. (2002). Visual competition. Nat. Rev. Neurosci. 3, 13–21.

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Bouvier, S. E., and Engel, S. A. (2006). Behavioral deficits and cortical damage loci in cerebral achromatopsia. Cereb. Cortex 2, 183–191.

Carlson, T. A., and He, S. (2004). Competing global representations fail to initiate binocular rivalry. Neuron 43, 907–914.

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Cohen Kadosh, R., Henik, A., and Walsh, V. (2007). Small is bright and big is dark in synaesthesia. Curr. Biol. 17, R834–R835.

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Dixon, M. J., and Smilek, D. (2005). The importance of individual differences in grapheme-color synesthesia. Neuron 45, 821–823.

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Gebuis, T., Nijboer, T. C. W., and Van der Smagt, M. J. (2009a). Multiple dimensions in bi-directional synesthesia. Eur. J. Neurosci. 29, 1703–1710.

CrossRef Full Text

Gebuis, T., Nijboer, T. C. W., and van der Smagt, M. J. (2009b). Of colored numbers and numbered colors interactive processes in grapheme-color synesthesia. Exp. Psychol. 56, 180–187.

CrossRef Full Text

Hong, S. W., and Blake, R. (2008). Early visual mechanisms do not contribute to synesthetic color experience. Vision Res. 48, 1018–1026.

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Kim, C.-Y., Blake, R., and Palmeri, T. J. (2006). Perceptual interaction between real and synesthetic colors. Cortex 42, 195–203.

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Kovács, I., Papathomas, T. V., Yang, M., and Fehér, A. (1996). When the brain changes its mind: interocular grouping during binocular rivalry. Proc. Natl. Acad. Sci. U.S.A. 93, 15508–15511.

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Logothetis, N. K., Leopold, D. A., and Sheinberg, D. L. (1996). What is rivalling during binocular rivalry? Nature 380, 621–624.

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Mattingley, J. B., Rich, A. N., Yelland, G., and Bradshaw, J. L. (2001). Unconscious priming eliminates automatic binding of colour and alphanumeric form in synaesthesia. Nature 410, 580–582.

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Nijboer, T. C. W., Gebuis, T., te Pas, S. F., and van der Smagt, M. J. (2011). Interactions between colour and synesthetic colour: an effect of simultaneous colour contrast on synesthetic colours. Vision Res. 51, 43–47.

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Paffen, C. L. E., Alais, D., and Verstraten, F. A. J. (2006). Attention speeds binocular rivalry. Psychol. Sci. 17, 752–756.

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Paffen, C. L. E., Plukaard, S., and Kanai, R. (2011). Symbolic magnitude modulates perceptual strength in binocular rivalry. Cognition 119, 468–475.

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Paffen, C. L. E., te Pas, S. F., Kanai, R., van der Smagt, M. J., and Verstraten, F. A. J. (2004). Center-surround interactions in visual motion processing during binocular rivalry. Vision Res. 44, 1635–1639.

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Ramachandran, V. S., and Hubbard, E. M. (2001). Psychophysical investigations into the neural basis of synaesthesia. Proc. Biol. Sci. 268, 979–983.

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Rouw, R., and Scholte, H. S. (2007). Increased structural connectivity in grapheme-colour synaesthesia. Nat. Neurosci. 10, 792–797.

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Wilson, H. R. (2003). Computational evidence for a rivalry hierarchy in vision. Proc. Natl. Acad. Sci. U.S.A. 100, 14499–14503.

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Yu, K., and Blake, R. (1992). Do recognizable figures enjoy an advantage in binocular rivalry? J. Exp. Psychol. Hum. Percept. Perform. 18, 1158–1173.

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Appendix

TABLE A1

Table A1. Colour settings per synesthete.