Uni- and crossmodal refractory period effects of event-related potentials provide insights into the development of multisensory processing

To assess uni- and multisensory development in humans, uni- and crossmodal event-related potential (ERP) refractory period effects were investigated. Forty-one children from 4 to 12 years of age and 15 young adults performed a bimodal oddball task with frequent and rare visual and auditory stimuli presented with two different interstimulus intervals (ISIs). Amplitudes of the visual and auditory ERPs were modulated as a function of the age of the participants, the modality of the preceding stimulus (same vs. different) and the preceding ISI (1000 or 2000 ms). While unimodal refractory period effects were observed in all age groups, crossmodal refractory period effects differed among age groups. Early crossmodal interactions (<150 ms) existing in the youngest age group (4–6 years) disappeared, while later crossmodal interactions (>150 ms) emerged with a parietal topography in older children and adults. Our results are compatible with the intersensory differentiation and the multisensory perceptual narrowing approach of multisensory development. Moreover, our data suggest that uni- and multisensory development run in parallel with unimodal development leading.


INTRODUCTION
Event-related potentials (ERPs) have been used to assess the neuronal mechanisms of multisensory interactions (Stein and Meredith, 1993;Driver and Noesselt, 2008;Senkowski et al., 2008) and to investigate neurocognitive development (e.g., Courchesne, 1978;De Haan and Nelson, 1997;Molfese and Molfese, 2000;Nelson and Monk, 2001). In the past, multisensory development has mostly been evaluated with behavioral techniques (for a recent review see Bremner et al., 2012). Different views on how multisensory functions emerge have been proposed: While the hierarchical development view (=intersensory integration view, sometimes linked to the constructivism view of development) assumes that the modality systems initially develop in isolation and are linked only later in life (Piaget, 1952), the intersensory differentiation view proposes that multisensory interactions exist already at birth, and are then shaped by experience in the first months of life (Gibson, 1969(Gibson, , 1984Maurer, 1993). Moreover, Lewkowicz and Ghazanfar (2009) postulated that the development of multisensory processing emerges through regressive events, leading to a loss of multisensory abilities as a function of environmental experiences (multisensory perceptual narrowing view).
One approach to study the neural correlates of multisensory development is to record ERPs. Brett-Green et al. (2008) compared ERPs to bimodal stimuli and to the sum of ERPs elicited by the comprising unimodal events. They detected multisensory interactions between the auditory and the somatosensory system in children aged 6 to 13 years. A similar protracted developmental time course of multisensory development was suggested by the results of Brandwein et al. (2010), who reported a mature pattern of multisensory ERP effects not before the age of 14 years of age. Since the ERP approach chosen by these authors has been criticized (Gondan et al., 2005), we decided to adapt a paradigm that has fruitfully been used to study unisensory development (Cheour and Näätänen, 1998;Coch et al., 2005): When two stimuli are presented in succession, the amplitudes of ERPs elicited by the second stimulus are decreased as a function of the interstimulus interval (ISI). The time it takes the ERP amplitudes to fully recover is called the refractory period (Neville et al., 1983;Ritter et al., 1992). ERP refractory periods within a sensory system have been interpreted as an index of the excitability of the neural networks contributing to a particular ERP (Neville et al., 1983;Ritter et al., 1992). Unisensory ERP refractory periods have been used in humans to study the development of sensory systems (Cheour and Näätänen, 1998;Coch et al., 2005) and to understand the functional changes of sensory cortices following sensory loss (Neville et al., 1983). Coch et al. (2005) used auditory and visual refractory periods (with ISIs between 360 and 2000 ms) to investigate the developmental trajectory of the auditory and visual system between the age of 6 and 8 years. They reported adultlike patterns of unimodal refractory period effects in school-age children.
To date, crossmodal ERP refractory periods have not been systematically investigated and ERP studies that directly compare uni-and crossmodal refractory period effects in the same individuals and in populations aged from early childhood through adulthood are still missing. In adults, Fruhstorfer (1971) found an amplitude recovery of the vertex ERP when in a regular train of stimuli of the same modality (auditory or tactile) a stimulus of a different modality (tactile and auditory, respectively) was interposed. Furthermore, Davis et al. (1972) measured vertex ERP amplitudes to auditory, visual and tactile stimuli which were preceded by either a stimulus of the same or of a different modality with an ISI of 500 ms. They reported crossmodal refractory period effects for all modality pairings. The development of uni-vs. crossmodal refractory effects has, to our knowledge, not been investigated yet. Such studies would allow comparing the time course of both uni-and crossmodal development within the same paradigm. Moreover, due to the low demands required to perform the task, the very same paradigm is suitable for the use across a large age range. In our paradigm, healthy children (4-12 years of age) and adults were engaged in a simple bimodal oddball task with frequent and rare auditory and visual stimuli. ISIs (1000 and 2000 ms) between two consecutive stimuli of the  same or of different modalities were systematically manipulated and uni-and crossmodal refractory period effects for both auditory and visual ERPs were assessed. We expected ERP amplitudes to vary as a function of ISI both within and across modalities. It was assumed that in the course of development, the distribution of refractory period effects on the scalp becomes more focused to a reduced number of electrodes. A hierarchical developmental view would predict a sequential emergence of uni-and crossmodal refractory effects. By contrast, a differentiation developmental view would predict a parallel emergence of uni-and crossmodal refractory effects.

PARTICIPANTS
Forty one children (age group 1: 4-6 years, mean age 5.9 years, n = 15, 9 females, 6 males; age group 2: 7-9 years, mean age 8.4 years, n = 13, 8 females, 5 males; age group 3: 10-12 years, mean age 11.2 years, n = 13, 8 females, 5 males) and 15 adults (19-39 years, mean age 26 years, 7 females, 8 males) participated. The adults were all students of the University of Hamburg. The participants had no history of neurological, language, speech, or articulatory disease. All participants or the parents of the participating children reported normal hearing and normal or corrected-to-normal vision. Prior to the start of the experiment the parents and adult participants gave informed consent and children were asked to agree to participate in the study. In addition, children were tested with (i) <12 years. Colored progressive matrices (CPM, by Raven), (ii) ≥12 years. Standard progressive matrices (SPM, by Raven) and the motor-free visual perception test (MVPT-3, by Colarusso and Hammill) to confirm normal intelligence and normal visual perception. These data are not reported here, but all children included in the final analyses fell within the normal range. Parents received a reimbursement for travel costs and other costs associated with participating. Adult participants received course credit or were compensated with 7 C/h. The experiment was conducted according to the guidelines laid down in the Declaration of Helsinki and the study was approved by the ethics committee, Medical Association Hamburg, Nr. 2653. Twelve additional children (7 in age group 1, 3 in age group 2, and 2 in age group 3) and 3 additional adults were tested, but later excluded because of excessive artifacts in their EEG recordings.  Leinfelden-Echterdingen, Germany) and the visual animal stimuli by selecting color photographs from a pool of pictures of a digital photo database (Hemera Photo Objects, Vol. 1, Hemera, Hull, Canada). Auditory and visual stimuli were presented in a randomized order. Six blocks in children and 12 blocks in adults, each lasting approximately 3 min were run. Each block consisted of 112 stimuli including at least one deviant stimulus. The first two stimuli of a block were randomly selected from one modality; they were not included in the analyses. A modality switch occurred in about half of the trials. Auditory and visual stimuli were presented with equal probability with an ISI of 1000 ms or 2000 ms. No more than four of the same ISI appeared in a sequence, all deviants were followed by an ISI of 2000 ms. The design of the study resulted in eight different stimulation conditions listed in Table 1. The task of the participants was to press a response button with either the right or the left hand as quickly as possible to each deviant stimulus irrespective of its modality. Prior to the start of the EEG recording the experiment was explained and one practice block consisting of a random sequence of auditory and visual stimuli (60 standard stimuli and 6 deviants) was run. The children were comfortably seated in a dimly lit, sound-attenuated, and electrically shielded chamber. They were instructed not to move their eyes from the fixation cross at the center of the monitor during a block. Parents attended the experiment room if the children asked for. Visual and auditory stimuli were presented on a computer screen and from a central loudspeaker, respectively, both located at a distance of 91.5 cm from the participants. Gaze direction was monitored by an experimenter via an infrarot camera.

DATA ACQUISITION
EEG was recorded with 46 (children) and 73 (adults) active electrodes (ActiCap, Brain Products, Gilching, Germany) mounted in an elastic cap (Easycap, Falk Minow Services, Herrsching, Germany, see Figure 1). A lower number of electrodes was used in children because of their smaller heads. The EEG was recorded with a time constant of 10 s. Using BrainAmp amplifiers (Brain Products, Munich, Germany) the EEG signal was sampled at 5000 Hz, filtered online with a bandpass of 0.016 to 250 Hz and was then down sampled online to a sample rate of 500 Hz. These data were stored on a hard disk. Electrode positions were arranged according to the international 10-10 system. The left earlobe was used as reference electrode, offline a linked earlobe reference was calculated. The vertical EOG was recorded with two active electrodes beneath the right and the left eye against the common Abbreviations: [vsv]: e.g., ERP to a visual stimulus preceded by a visual stimulus with a short ISI; a, auditory; v, visual; s, short ISI; l, long ISI. Time windows used for the statistical analyses are marked in gray. Negativity is plotted upwards. reference. Recordings from the electrodes F9 and F10 were used for the offline calculation of the horizontal EOG (HEOG). For each participant ERPs were averaged at each electrode site over an epoch of 700 ms (200 ms pre-to 500 ms post stimulus onset). ERPs were separately analyzed for each stimulus modality (auditory vs. visual) and were averaged as a function of the modality, of the preceding stimulus (same vs. different) and the preceding ISI (1000 vs. 2000 ms). Epochs following deviant stimuli were not included. Trials containing eye movements artifacts (HEOG and VEOG) exceeding ± 100 (children) or ± 80 (adults) µV relative to the absolute difference between any two sample points within the epoch (500 ms post stimulus) and other artifacts (a voltage exceeding ± 140 (children) or ± 120 (adults) µV at any electrode relative of any two sample points) were eliminated semiautomatically. Due to noisy recordings at the caudal electrodes (PO9, Iz, PO10) in a large number of the participants, these electrodes were excluded from further analyses. Only participants with a minimum of 25 (children) or 50 (adults) artifact free trials for each condition were included in the statistical analyses. Fourteen additional participants were excluded due to this criterion (see section Participants). The EEG data were acquired in the lab of the Biological Psychology and Neuropsychology, University of Hamburg, Germany.

FIGURE 3 | Grand averaged ERPs to auditory stimuli in the unimodal (black line) vs. crossmodal (red line) condition and after short (dashed line) vs. long (solid line) interstimulus intervals.
ERPs are demonstrated at a subset of representative electrodes and for the four different age groups.
Abbreviations: [asa]: e.g., ERP to an auditory stimulus preceded by an auditory stimulus with a short ISI; a, auditory; v, visual; s, short ISI; l, long ISI. Time windows used for the statistical analyses are marked in gray. Negativity is plotted upwards.

Frontiers in Human Neuroscience
www.frontiersin.org July 2014 | Volume 8 | Article 552 | 5   ns  each single electrode separately for the uni-and crossmodal condition (paired t-tests, two-tailed). Topographic maps were created in Brain Vision Analyzer (Brain Products) using a linear interpolation algorithm. Finally, the ANOVA performed on the reaction times included the between-subject factor Age group (age group 1, age group 2, age group 3, adults) and the within-subject factors Modality of the current stimulus (visual vs. auditory), Modality Transition (MT) (unimodal vs. crossmodal) and ISI (short vs. long). Huynh/Feldt-corrected p-values, but uncorrected degrees of freedom, are reported when appropriate.

BEHAVIORAL DATA
Five children in age group 1 were excluded from the behavioral data analyses, because they failed pressing the button all together. The remaining children detected 94% (SE = 1.4) and the adults detected 96% (SE = 3.8) of the deviants. Detailed results of the detection rates in each age group and for each modality are reported in Table 2. Because EEG data of these participants did not show any obvious difference to the other participants, EEG data of the five children without behavioral data were included in the statistical analyses. The ANOVA performed on the reaction times revealed a main effect of Modality [F (1, 47) = 99.6, p < 0.001] and significant interactions of ISI × MT × Age group [F (3, 47) = 3.56, p < 0.05] and Modality × ISI × MT [F (1, 47) = 5.91, p < 0.05]. These results were due to (i) faster RT for the visual than for the auditory targets across the groups and (ii) the influence of the preceding ISI, the preceding modality and the age of the participant on reaction times. RT to both visual and auditory stimuli decreased with age, with slowest RT in the youngest age group, and fastest RT in adults. Only for the visual stimuli, RT depended on the preceding ISI and the preceding modality, but these effects varied by the age group [ISI × MT × Age group, F (3, 47) = 2.85, p < 0.05]. In the analyses conducted separately for each age group, no significant main effects and interactions were found in age group 1. Significant main effects for Modality [F (1, 12) = 25.74, p < 0.001] and for MT [F (1, 12) = 5.071, p < 0.05] in age group 2 and a significant main effect for Modality [F (1, 12) = 52.17, p < 0.001]) in age group 3 were revealed. In adults, the ANOVA yielded a significant main effect for Modality [F (1, 14) = 89.44, p < 0.001] and a marginally significant interaction of Modality, MT and ISI [F (1, 12) = 3.9, p < 0.1]. For detailed results of average reaction times in each age group and each modality see Table 3.

VISUAL ERP RESULTS
In adults, visual stimuli elicited a positive potential between 100 and 110 ms post stimulus with a maximum at occipital electrode sites. The following negative peak reached its maximum around 160 ms post stimulus over occipital brain regions. In children, ERPs showed a positive potential at occipital electrodes with a maximum peak around 130-140 ms post stimulus. The following negative peak at parieto-occipital electrodes reached its maximum around 170-190 ms post stimulus (age group 1), 180-200 ms post stimulus (age group 2) and 210 ms post stimulus (age group 3), respectively. Concurrent with the positive peak over the occipital scalp, a negative potential was elicited over frontal and central brain regions with a maximum around 130 ms post stimulus followed by a positive deflection at 210-220 ms post stimulus. Figure 2 presents the grand averaged ERPs to visual stimuli at 2.52, p < 0.1, time window 260-340 ms (children/adults): ISI × MT × H × E × Age group, F (5, 260) = 1.58, p < 0.1. Due to the interaction with the between-subject-factor Age group, the age groups were analyzed separately in the following analyses.

Unimodal refractory period effects
In adults and in age group 3 (10-12 years), ERP amplitudes in all analyzed time windows were affected by both the stimulus modality and the ISI and these effects depended on the recording sites (see significant main effect for ISI and/or for MT and/or interactions of MT × ISI × topographical factor, Tables 4, 7). In the post-hoc analyses for age group 3 and adults, significant unimodal refractory period effects (vlv-vsv) were mainly found over the posterior scalp in all analyzed time windows (Figure 4). In age group 1 (4-6 years), significant interactions of MT, ISI and topographical factors in the Five-Way-ANOVA pointed toward amplitude differences between 100 and 150 ms as a function of the preceding stimulus modality, the preceding ISI and the recording site (see interactions of MT × ISI × topographical factor in Table 5): ERP amplitudes at occipital recording sites were larger in the unimodal condition when a short ISI compared to when a long ISI preceded. In contrast, over frontal and fronto-central regions amplitude deflections were larger in the unimodal condition when a long ISI compared to when a short ISI preceded (Figure 2). In the time windows 150-220 ms and 260-340 ms, significant or marginally significant main effects of ISI and significant interactions between ISI and topographical factors in all calculated ANOVAs reflected the influence of the preceding ISI rather than the preceding modality on the ERP amplitude (see main effect of ISI and interactions of ISI × topographical factor in Table 5). Unimodal refractory effects were mainly found at frontal and fronto-central recording sites (Figures 2, 4). In age group 2 (7-9 years), significant or marginally significant main effect for ISI and significant interactions of ISI × one or more topographical factors in all analyzed time windows indicated unimodal refractory effects in this age group (see main effect of ISI and interactions of ISI × topographical factor in Table 6).

Crossmodal refractory period effects
In the youngest age group (4-6 years), ERP amplitudes in the time window 100-150 ms were modified by the preceding modality, the preceding ISI and the recording site (see interactions of MT × ISI × topographical factors in Table 5) and significant crossmodal refractory period effects (alv-asv) were observed at Frontiers in Human Neuroscience www.frontiersin.org July 2014 | Volume 8 | Article 552 | 9           fronto-central and fronto-temporal recording sites in the post-hoc analyses (Figure 4). In adults and age group 3 (10-12 years), significant interactions of MT, ISI and topographical factors in the Five-Way-and Three-Way-ANOVA indicated that amplitude differences were affected by the preceding stimulus modality and the ISI. These effects depended on the recording site (see interactions of MT × ISI × topographical factor in Tables 4 and 7). In contrast to the results in age group 1, crossmodal refractory effects (alv-asv) were confirmed at single parieto-occipital and occipital electrodes, respectively (Figures 2, 4), and after 150 ms post stimulus. In age group 2 (7-9 years), the preceding ISI and the recording site rather than the MT influenced ERP amplitudes (see main effect of ISI and interactions of ISI × topographical factor in Table 6).

AUDITORY ERP RESULTS
Auditory ERPs were characterized by a fronto-centrally distributed positive peak, a second, negative peak with a maximal deflection over the vertex and a third, positive peak with a vertex maximum as well. This ERP pattern was found in all age groups. The latencies of the first and the second peak decreased with increasing age: The first positive wave peaked around 60 ms post stimulus in adults and around 70 ms post stimulus in children. The next largest peak (with negative polarity) was between 80-90 ms post stimulus in adults and around 100 ms post stimulus in children. The third, positive peak showed a similar latency with approximately 180 ms post stimulus across the age groups. Figure 3 depicts the grand averaged ERPs to auditory stimuli at representative electrodes for (i) the four stimulus condition, i.e., unimodal vs. crossmodal condition and short ISI vs. long ISI, and (ii) the four age groups.
In the first 250 ms post stimulus, ERPs to auditory stimuli showed refractory effects which depended on the preceding modality, the recording site and the age of the participant [time window 90-150 ms (children)/70-120 ms (adults): ISI × MT × AP × E × Age group, F (5, 190) = 7.43, p < 0.001, time window 150-250 ms (children)/140-250 ms (adults): ISI × MT × E × Age group, F (5, 260) = 2.065, p < 0.05]. Due to the interaction with the between-subject-factor Age group, the age groups were analyzed separately in the following analyses.

Unimodal refractory period effects
In all age groups, significant or marginally significant main effects of MT and ISI and/or interactions between MT, ISI and topographical factors (see Five-Way-and Three-Way ANOVA in Tables 4-7) reflected the influence of the preceding stimulus modality, the preceding ISI and the recording site on ERP amplitudes. In the time window 150-250 ms (children) and 140-250 ms (adults), respectively, significant unimodal refractory effects (ala-asa) were confirmed along the midline electrodes in age group 1 (4-6 years) and age group 2 (7-9 years), and widespread over the scalp in the oldest children and in adults. In age group 1 and in adults, unimodal ISI effects were most prominent at fronto-polar recording sites in the first time window (90-150 ms (children) and 70-120 ms (adults), respectively). Beyond 260 ms post-stimulus, unimodal ERP refractory effects were obtained in all age groups (Figures 3, 5).

Crossmodal refractory period effects
In children aged 4-6 years, ERP amplitudes were significantly influenced by a modality switch, the preceding ISI and the recording site (see significant of marginally significant interaction of MT × ISI × topographical factor in the Five-Way-and Three-Way-ANOVA, Table 5). In the post-hoc analyses, crossmodal refractory effects were found over frontal, fronto-central, and central brain regions until 250 ms post stimulus (Figures 3, 5).
In older children and in adults, the Five-Way-and the Three-Way-ANOVA revealed significant main effects of MT and/or ISI and interactions of MT × ISI × one or more topographical factors (see Tables 4, 6, and 7), which again indicated the influence of preceding modality, the preceding ISI and the recording site on ERP amplitudes. In contrast to the results of the posthoc analyses in the youngest age group, crossmodal ISI effects were most pronounced over the posterior scalp after 150 ms post stimulus, and these effects were most reliable in adults (Figures 3, 5).

DISCUSSION
The present study tested whether uni-and crossmodal ERP refractory effects, indicating uni-and multisensory development, emerge sequentially (hierarchical developmental view) or parallel (differentiation developmental view). ERPs elicited in response to auditory and visual stimuli presented with two different ISIs (1000, 2000 ms) were analyzed as a function of the age of the participants, the modality of the preceding stimulus and the ISI. ERP amplitudes to both visual and auditory stimuli were modulated by the modality of the preceding stimulus and the ISI. The degree of modulation depended on (i) the recording site and (ii) the age of the participant. The most pronounced crossmodal refractory effects both for visual and auditory ERPs were observed in the youngest age group (4-6 years). Crossmodal refractory period effects emerged earlier and with a different and broader topography in the youngest children compared to adults: They had a frontal and fronto-central topography irrespectively of stimulus modality in the youngest age group but were most pronounced over the posterior scalp in the oldest children and in adults. By contrast, unimodal refractory effects were observed for both modality conditions in all age groups.
We interpret our data in favor of the multisensory differentiation (Lickliter and Bahrick, 2004) and the multisensory perceptual narrowing account (Lewkowicz and Ghazanfar, 2009). By contrast, our results seem to be incompatible with the idea of a strict hierarchical development (Piaget, 1952), with unimodal processing maturing prior to the emergence of the first crossmodal interactions.
Firstly, we found the most prominent crossmodal interactions in the youngest age group (4-6 years). These effects emerged within the first analyzed time window (<150 ms). By contrast, crossmodal refractory effects in older children and adults were observed beyond the time epoch of 150 ms. These results suggest that at earlier processing stages, both in the visual and auditory processing pathway, crossmodal interactions occur in young children but not in older children and adults. Interestingly, these early crossmodal interactions had a fronto-central topography which was clearly distinct from the parietal topography of later crossmodal interactions in older children and adults. These developmental trends were observed irrespectively of the stimulus modality. It might be speculated that the early crossmodal interactions observed in children were due to still existing exuberant connections between sensory systems (Innocenti and Clarke, 1984). As development progresses, crossmodal interactions become more specific as a result of crossmodal experience (Lewkowicz and Ghazanfar, 2009). More specific, crossmodal interactions might emerge from the setting up of multisensory areas such as those in the parietal cortex (Bolognini et al., 2010;Kamke et al., 2012). It is important to note that these results are not incompatible with previous reports in adults demonstrating earlier (<100 ms) crossmodal ERP effects in adults (Giard and Peronnet, 1999;Molholm et al., 2002). These authors compared the processing of unimodal vs. crossmodal stimuli, while we presented unimodal stimuli only and assessed the refractoriness of different stages of the auditory and visual processing pathway due to a preceding stimulus.
Although there were some differences between age groups, mainly in topography, reliable unimodal refractory effects were observed in all groups of children, irrespectively of age and in adults. These results are in accord with Coch et al. (2005), who reported unimodal refractory effects both for auditory and visual ERPs in their youngest group as well (6-8 years). Thus, though the finding of an earlier maturation of uni-compared to crossmodal processing would be compatible with a hierarchical view of sensory development, the parallel finding of pronounced crossmodal interactions in the youngest but not older age groups is incompatible with this account. Earlier behavioral studies have reported that unisensory processing reaches adult level earlier than multisensory processing (Röder et al., 2013). Thus, uniand multisensory processing seem to progress in parallel with a longer developmental time course of crossmodal (see Brett-Green et al., 2008;Brandwein et al., 2010) than unimodal development. Indeed, a number of recent studies has consistently shown a protracted developmental time course of multisensory processes into adolescents (Gori et al., 2008;Nardini et al., 2008;Barutchu et al., 2009).
Taken together, the present study provides evidence for the intersensory differentiation and multisensory perceptual narrowing approach to explain multisensory development. Some crossmodal interactions existing at earlier developmental ages seem to be eliminated and substituted by specific crossmodal interactions. Uni-and multisensory development runs in parallel with unimodal development leading.