Uni- and crossmodal refractory period effects of event-related potentials provide insights into the development of multisensory processing

Introduction

Event-related potentials (ERPs) have been used to assess the neuronal mechanisms of multisensory interactions (Stein and Meredith, 1993; Driver and Noesselt, 2008; Senkowski et al., 2008) and to investigate neurocognitive development (e.g., Courchesne, 1978; De Haan and Nelson, 1997; Molfese and Molfese, 2000; Nelson and Monk, 2001). In the past, multisensory development has mostly been evaluated with behavioral techniques (for a recent review see Bremner et al., 2012). Different views on how multisensory functions emerge have been proposed: While the hierarchical development view (=intersensory integration view, sometimes linked to the constructivism view of development) assumes that the modality systems initially develop in isolation and are linked only later in life (Piaget, 1952), the intersensory differentiation view proposes that multisensory interactions exist already at birth, and are then shaped by experience in the first months of life (Gibson, 1969, 1984; Maurer, 1993). Moreover, Lewkowicz and Ghazanfar (2009) postulated that the development of multisensory processing emerges through regressive events, leading to a loss of multisensory abilities as a function of environmental experiences (multisensory perceptual narrowing view).

One approach to study the neural correlates of multisensory development is to record ERPs. Brett-Green et al. (2008) compared ERPs to bimodal stimuli and to the sum of ERPs elicited by the comprising unimodal events. They detected multisensory interactions between the auditory and the somatosensory system in children aged 6 to 13 years. A similar protracted developmental time course of multisensory development was suggested by the results of Brandwein et al. (2010), who reported a mature pattern of multisensory ERP effects not before the age of 14 years of age. Since the ERP approach chosen by these authors has been criticized (Gondan et al., 2005), we decided to adapt a paradigm that has fruitfully been used to study unisensory development (Cheour and Näätänen, 1998; Coch et al., 2005): When two stimuli are presented in succession, the amplitudes of ERPs elicited by the second stimulus are decreased as a function of the interstimulus interval (ISI). The time it takes the ERP amplitudes to fully recover is called the refractory period (Neville et al., 1983; Ritter et al., 1992). ERP refractory periods within a sensory system have been interpreted as an index of the excitability of the neural networks contributing to a particular ERP (Neville et al., 1983; Ritter et al., 1992). Unisensory ERP refractory periods have been used in humans to study the development of sensory systems (Cheour and Näätänen, 1998; Coch et al., 2005) and to understand the functional changes of sensory cortices following sensory loss (Neville et al., 1983). Coch et al. (2005) used auditory and visual refractory periods (with ISIs between 360 and 2000 ms) to investigate the developmental trajectory of the auditory and visual system between the age of 6 and 8 years. They reported adult-like patterns of unimodal refractory period effects in school-age children.

To date, crossmodal ERP refractory periods have not been systematically investigated and ERP studies that directly compare uni- and crossmodal refractory period effects in the same individuals and in populations aged from early childhood through adulthood are still missing. In adults, Fruhstorfer (1971) found an amplitude recovery of the vertex ERP when in a regular train of stimuli of the same modality (auditory or tactile) a stimulus of a different modality (tactile and auditory, respectively) was interposed. Furthermore, Davis et al. (1972) measured vertex ERP amplitudes to auditory, visual and tactile stimuli which were preceded by either a stimulus of the same or of a different modality with an ISI of 500 ms. They reported crossmodal refractory period effects for all modality pairings. The development of uni- vs. crossmodal refractory effects has, to our knowledge, not been investigated yet. Such studies would allow comparing the time course of both uni- and crossmodal development within the same paradigm. Moreover, due to the low demands required to perform the task, the very same paradigm is suitable for the use across a large age range. In our paradigm, healthy children (4–12 years of age) and adults were engaged in a simple bimodal oddball task with frequent and rare auditory and visual stimuli. ISIs (1000 and 2000 ms) between two consecutive stimuli of the same or of different modalities were systematically manipulated and uni- and crossmodal refractory period effects for both auditory and visual ERPs were assessed. We expected ERP amplitudes to vary as a function of ISI both within and across modalities. It was assumed that in the course of development, the distribution of refractory period effects on the scalp becomes more focused to a reduced number of electrodes. A hierarchical developmental view would predict a sequential emergence of uni- and crossmodal refractory effects. By contrast, a differentiation developmental view would predict a parallel emergence of uni- and crossmodal refractory effects.

Materials and Methods

Participants

Forty one children (age group 1: 4–6 years, mean age 5.9 years, n = 15, 9 females, 6 males; age group 2: 7–9 years, mean age 8.4 years, n = 13, 8 females, 5 males; age group 3: 10–12 years, mean age 11.2 years, n = 13, 8 females, 5 males) and 15 adults (19–39 years, mean age 26 years, 7 females, 8 males) participated. The adults were all students of the University of Hamburg. The participants had no history of neurological, language, speech, or articulatory disease. All participants or the parents of the participating children reported normal hearing and normal or corrected-to-normal vision. Prior to the start of the experiment the parents and adult participants gave informed consent and children were asked to agree to participate in the study. In addition, children were tested with (i) <12 years. Colored progressive matrices (CPM, by Raven), (ii) ≥12 years. Standard progressive matrices (SPM, by Raven) and the motor-free visual perception test (MVPT-3, by Colarusso and Hammill) to confirm normal intelligence and normal visual perception. These data are not reported here, but all children included in the final analyses fell within the normal range. Parents received a reimbursement for travel costs and other costs associated with participating. Adult participants received course credit or were compensated with 7 €/h. The experiment was conducted according to the guidelines laid down in the Declaration of Helsinki and the study was approved by the ethics committee, Medical Association Hamburg, Nr. 2653. Twelve additional children (7 in age group 1, 3 in age group 2, and 2 in age group 3) and 3 additional adults were tested, but later excluded because of excessive artifacts in their EEG recordings.

Stimuli and Procedure

Stimuli consisted of frequently presented standard tones [90% of the auditory stimuli; white noise bursts, duration 50 ms, 70 dB(A)] and frequently presented visual standard stimuli (90% of the visual stimuli; 10 × 10 checkerboard square, visual angle: 5°, duration: 50 ms). Additionally, rarely presented deviant sounds (10% of the auditory stimuli; animal sounds, duration: 400 ms, 70 dB) and rarely presented deviant pictures (10% of the visual stimuli; pictures of animal, 5° of visual angle for the longest (horizontal or vertical) size, duration: 50 ms) were used as target stimuli. These target stimuli were sounds or pictures of animals and were taken from a Multimodal Stimulus Set (Schneider et al., 2008). Schneider et al. had created the animal sounds by selecting characteristic sounds of natural objects out of 12 sound effect CDs (100 Spectacular Sound FX, Mediaphon, Leinfelden-Echterdingen, Germany) and the visual animal stimuli by selecting color photographs from a pool of pictures of a digital photo database (Hemera Photo Objects, Vol. 1, Hemera, Hull, Canada). Auditory and visual stimuli were presented in a randomized order. Six blocks in children and 12 blocks in adults, each lasting approximately 3 min were run. Each block consisted of 112 stimuli including at least one deviant stimulus. The first two stimuli of a block were randomly selected from one modality; they were not included in the analyses. A modality switch occurred in about half of the trials. Auditory and visual stimuli were presented with equal probability with an ISI of 1000 ms or 2000 ms. No more than four of the same ISI appeared in a sequence, all deviants were followed by an ISI of 2000 ms. The design of the study resulted in eight different stimulation conditions listed in Table 1. The task of the participants was to press a response button with either the right or the left hand as quickly as possible to each deviant stimulus irrespective of its modality. Prior to the start of the EEG recording the experiment was explained and one practice block consisting of a random sequence of auditory and visual stimuli (60 standard stimuli and 6 deviants) was run.

TABLE 1

Table 1. Characteristics of the eight different stimulation conditions.

The children were comfortably seated in a dimly lit, sound-attenuated, and electrically shielded chamber. They were instructed not to move their eyes from the fixation cross at the center of the monitor during a block. Parents attended the experiment room if the children asked for. Visual and auditory stimuli were presented on a computer screen and from a central loudspeaker, respectively, both located at a distance of 91.5 cm from the participants. Gaze direction was monitored by an experimenter via an infrarot camera.

Data Acquisition

EEG was recorded with 46 (children) and 73 (adults) active electrodes (ActiCap, Brain Products, Gilching, Germany) mounted in an elastic cap (Easycap, Falk Minow Services, Herrsching, Germany, see Figure 1). A lower number of electrodes was used in children because of their smaller heads. The EEG was recorded with a time constant of 10 s. Using BrainAmp amplifiers (Brain Products, Munich, Germany) the EEG signal was sampled at 5000 Hz, filtered online with a bandpass of 0.016 to 250 Hz and was then down sampled online to a sample rate of 500 Hz. These data were stored on a hard disk. Electrode positions were arranged according to the international 10–10 system. The left earlobe was used as reference electrode, offline a linked earlobe reference was calculated. The vertical EOG was recorded with two active electrodes beneath the right and the left eye against the common reference. Recordings from the electrodes F9 and F10 were used for the offline calculation of the horizontal EOG (HEOG). For each participant ERPs were averaged at each electrode site over an epoch of 700 ms (200 ms pre- to 500 ms post stimulus onset). ERPs were separately analyzed for each stimulus modality (auditory vs. visual) and were averaged as a function of the modality, of the preceding stimulus (same vs. different) and the preceding ISI (1000 vs. 2000 ms). Epochs following deviant stimuli were not included. Trials containing eye movements artifacts (HEOG and VEOG) exceeding ± 100 (children) or ± 80 (adults) μV relative to the absolute difference between any two sample points within the epoch (500 ms post stimulus) and other artifacts (a voltage exceeding ± 140 (children) or ± 120 (adults) μV at any electrode relative of any two sample points) were eliminated semiautomatically. Due to noisy recordings at the caudal electrodes (PO9, Iz, PO10) in a large number of the participants, these electrodes were excluded from further analyses. Only participants with a minimum of 25 (children) or 50 (adults) artifact free trials for each condition were included in the statistical analyses. Fourteen additional participants were excluded due to this criterion (see section Participants). The EEG data were acquired in the lab of the Biological Psychology and Neuropsychology, University of Hamburg, Germany.

FIGURE 1

Figure 1. Electrode montage with (A) 46 (children) and (B) 73 (adults) active electrodes, EOG, vertical electrooculogram recorded by two active elecrodes beneath the right and left eye; GND, ground electrode [nose tip (children), Fpz (adults)]; REF, reference electrode on the left ear lobe. Electrodes shaded in gray were excluded from further analyses due to artifacts. Electrodes marked by a prominent circle were included in the statistical analyses.

Data Analyses

Based on earlier reports, and on visual inspection of the grand averages, mean amplitudes were calculated for the following time windows: Auditory ERPs: children: 90–150 ms, 150–250 ms, and 260–340 ms, adults: 70–120 ms, 140–250 ms, and 260–340 ms; visual ERPs: children: 100–150 ms, 150–220 ms, and 260–340 ms, adults: 90–140 ms, 140–190 ms, and 260–340 ms. Mean amplitudes of both visual and auditory ERPs were separately submitted to analyses of variance (ANOVA, using SPSS Software). The between-subject factor was Age group (age group 1, age group 2, age group 3, adults) and within-subject factors included Modality Transition (MT) (unimodal vs. crossmodal), ISI (short vs. long), Hemisphere (H) (right vs. left), Anterior Posterior (AP) (anterior, posterior), and Electrodes (E) (Fp1, F3, F7, FC1, FC5, FT7 vs. Fp2, F4, F8, FC2, FC6, FT8 vs. CP1, CP5, TP7, P3, P7, O1 vs. CP2, CP6, TP8, P4, P8, O2, see Figure 1). For midline electrodes an additional ANOVA was run, using three repeated measurement factors: Modality Transition (MT) (unimodal vs. crossmodal), ISI (short vs. long), and Electrodes (E) (Fpz (children), AFz, Fz, FCz, Cz, Pz, POz). In a second analysis, effects in the four different age groups were analyzed separately. Whenever, the interaction of factors MT and ISI and one of the topographic factors reached significance (p ≤ 0.05), the refractory period effect was tested at each single electrode separately for the uni- and crossmodal condition (paired t-tests, two-tailed). Topographic maps were created in Brain Vision Analyzer (Brain Products) using a linear interpolation algorithm. Finally, the ANOVA performed on the reaction times included the between-subject factor Age group (age group 1, age group 2, age group 3, adults) and the within-subject factors Modality of the current stimulus (visual vs. auditory), Modality Transition (MT) (unimodal vs. crossmodal) and ISI (short vs. long). Huynh/Feldt-corrected p-values, but uncorrected degrees of freedom, are reported when appropriate.

Results

Behavioral Data

Five children in age group 1 were excluded from the behavioral data analyses, because they failed pressing the button all together. The remaining children detected 94% (SE = 1.4) and the adults detected 96% (SE = 3.8) of the deviants. Detailed results of the detection rates in each age group and for each modality are reported in Table 2. Because EEG data of these participants did not show any obvious difference to the other participants, EEG data of the five children without behavioral data were included in the statistical analyses.

TABLE 2

Table 2. Detection rates of deviant targets and standard error (SE) in each age group and each modality.

The ANOVA performed on the reaction times revealed a main effect of Modality [F_{(1, 47)} = 99.6, p < 0.001] and significant interactions of ISI × MT × Age group [F_{(3, 47)} = 3.56, p < 0.05] and Modality × ISI × MT [F_{(1, 47)} = 5.91, p < 0.05]. These results were due to (i) faster RT for the visual than for the auditory targets across the groups and (ii) the influence of the preceding ISI, the preceding modality and the age of the participant on reaction times. RT to both visual and auditory stimuli decreased with age, with slowest RT in the youngest age group, and fastest RT in adults. Only for the visual stimuli, RT depended on the preceding ISI and the preceding modality, but these effects varied by the age group [ISI × MT × Age group, F_{(3, 47)} = 2.85, p < 0.05]. In the analyses conducted separately for each age group, no significant main effects and interactions were found in age group 1. Significant main effects for Modality [F_{(1, 12)} = 25.74, p < 0.001] and for MT [F_{(1, 12)} = 5.071, p < 0.05] in age group 2 and a significant main effect for Modality [F_{(1, 12)} = 52.17, p < 0.001]) in age group 3 were revealed. In adults, the ANOVA yielded a significant main effect for Modality [F_{(1, 14)} = 89.44, p < 0.001] and a marginally significant interaction of Modality, MT and ISI [F_{(1, 12)} = 3.9, p < 0.1]. For detailed results of average reaction times in each age group and each modality see Table 3.

TABLE 3

Table 3. Average reaction times (RT) and standard error (SE) for detecting deviant targets, separately for each age group, each modality and separately for all children and adults.

Visual ERP Results

In adults, visual stimuli elicited a positive potential between 100 and 110 ms post stimulus with a maximum at occipital electrode sites. The following negative peak reached its maximum around 160 ms post stimulus over occipital brain regions. In children, ERPs showed a positive potential at occipital electrodes with a maximum peak around 130–140 ms post stimulus. The following negative peak at parieto-occipital electrodes reached its maximum around 170–190 ms post stimulus (age group 1), 180–200 ms post stimulus (age group 2) and 210 ms post stimulus (age group 3), respectively. Concurrent with the positive peak over the occipital scalp, a negative potential was elicited over frontal and central brain regions with a maximum around 130 ms post stimulus followed by a positive deflection at 210–220 ms post stimulus. Figure 2 presents the grand averaged ERPs to visual stimuli at representative electrodes for (i) the four stimulus condition, i.e., unimodal vs. crossmodal condition and short ISI vs. long ISI, and (ii) the four age groups.

FIGURE 2

Figure 2. Grand averaged ERPs to visual stimuli in the unimodal (black line) vs. crossmodal (red line) condition and after short (dashed line) vs. long (solid line) interstimulus interval. ERPs are demonstrated at a subset of representative electrodes and separately for the four different age groups. Abbreviations: [vsv]: e.g., ERP to a visual stimulus preceded by a visual stimulus with a short ISI; a, auditory; v, visual; s, short ISI; l, long ISI. Time windows used for the statistical analyses are marked in gray. Negativity is plotted upwards.

In all time windows, significant or marginally significant interactions of MT, ISI, one or more topographical factors and age group pointed toward ERP amplitude differences as a function of the preceding stimulus modality, the preceding ISI and the age of the participants [time window 100–150 ms (children)/90–140 ms (adults): ISI × MT × H × AP × Age group, F_{(1, 52)} = 6.03, p < 0.001]: ISI × MT × H × AP × E × Age group, F_{(5, 260)} = 2.37, p < 0.01, time window 150–220 ms (children)/140–190 ms (adults): ISI × MT × H × AP × Age group, F_{(1, 52)} = 2.52, p < 0.1, time window 260–340 ms (children/adults): ISI × MT × H × E × Age group, F_{(5, 260)} = 1.58, p < 0.1. Due to the interaction with the between-subject-factor Age group, the age groups were analyzed separately in the following analyses.

Unimodal refractory period effects

In adults and in age group 3 (10–12 years), ERP amplitudes in all analyzed time windows were affected by both the stimulus modality and the ISI and these effects depended on the recording sites (see significant main effect for ISI and/or for MT and/or interactions of MT × ISI × topographical factor, Tables 4, 7). In the post-hoc analyses for age group 3 and adults, significant unimodal refractory period effects (vlv-vsv) were mainly found over the posterior scalp in all analyzed time windows (Figure 4). In age group 1 (4–6 years), significant interactions of MT, ISI and topographical factors in the Five-Way-ANOVA pointed toward amplitude differences between 100 and 150 ms as a function of the preceding stimulus modality, the preceding ISI and the recording site (see interactions of MT × ISI × topographical factor in Table 5): ERP amplitudes at occipital recording sites were larger in the unimodal condition when a short ISI compared to when a long ISI preceded. In contrast, over frontal and fronto-central regions amplitude deflections were larger in the unimodal condition when a long ISI compared to when a short ISI preceded (Figure 2). In the time windows 150–220 ms and 260–340 ms, significant or marginally significant main effects of ISI and significant interactions between ISI and topographical factors in all calculated ANOVAs reflected the influence of the preceding ISI rather than the preceding modality on the ERP amplitude (see main effect of ISI and interactions of ISI × topographical factor in Table 5). Unimodal refractory effects were mainly found at frontal and fronto-central recording sites (Figures 2, 4). In age group 2 (7–9 years), significant or marginally significant main effect for ISI and significant interactions of ISI × one or more topographical factors in all analyzed time windows indicated unimodal refractory effects in this age group (see main effect of ISI and interactions of ISI × topographical factor in Table 6).

TABLE 4A

Table 4A. F-values of the overall and subANOVAs (unimodal, crossmodal) for visual ERPs for the adult age-group, separately for each time window (90–140 ms, 140–190 ms, 260–340 ms).

TABLE 4B

Table 4B. F-values of the overall and subANOVAs (unimodal, crossmodal) for auditory ERPs for the adult age-group, separately for each time window (70–120 ms, 140–250 ms, 260–340 ms).

TABLE 4C

Table 4C. F-values of the overall and subANOVAs (unimodal, crossmodal) for visual and auditory ERPs at midline electrodes for the adult age-group, separately for each time window (visual: 90–140 ms, 140–190 ms, 260–340 ms, auditory: 70–120 ms, 140–250 ms, 260–340 ms).

TABLE 5A

Table 5A. F-values of the overall and subANOVAs (unimodal, crossmodal) for visual ERPs for age-group 1 (4–6 years), separately for each time window (100–150 ms, 150–220 ms, 260–340 ms).

TABLE 5B

Table 5B. F-values of the overall and subANOVAs (unimodal, crossmodal) for auditory ERPs for age–group 1 (4–6 years), separately for each time window (90–150 ms, 150–250 ms, 260–340 ms).

TABLE 5C

Table 5C. F-values of the overall and subANOVAs (unimodal, crossmodal) for visual and auditory ERPs at midline electrodes for age-group 1 (4–6 years), separately for each time window (visual: 100–150 ms, 150–220 ms, 260–340 ms, auditory: 90–150 ms, 150–250 ms, 260–340 ms).

TABLE 6A

Table 6A. F-values of the overall and subANOVAs (unimodal, crossmodal) for visual ERPs for age-group 2 (7–9 years), separately for each time window (100–150 ms, 150–220 ms, 260–340 ms).

TABLE 6B

Table 6B. F-values of the overall and subANOVAs (unimodal, crossmodal) for auditory ERPs for age-group 2 (7–9 years), separately for each time window (90–150 ms, 150–250 ms, 260–340 ms).

TABLE 6C

Table 6C. F-values of the overall and subANOVAs (unimodal, crossmodal) for visual and auditory ERPs at midline electrodes for age-group 2 (7–9 years), separately for each time window (visual: 100–150 ms, 150–220 ms, 260–340 ms, auditory: 90–150 ms, 150–250 ms, 260–340 ms).

Crossmodal refractory period effects

In the youngest age group (4–6 years), ERP amplitudes in the time window 100–150 ms were modified by the preceding modality, the preceding ISI and the recording site (see interactions of MT × ISI × topographical factors in Table 5) and significant crossmodal refractory period effects (alv-asv) were observed at fronto-central and fronto-temporal recording sites in the post-hoc analyses (Figure 4). In adults and age group 3 (10–12 years), significant interactions of MT, ISI and topographical factors in the Five-Way- and Three-Way-ANOVA indicated that amplitude differences were affected by the preceding stimulus modality and the ISI. These effects depended on the recording site (see interactions of MT × ISI × topographical factor in Tables 4 and 7). In contrast to the results in age group 1, crossmodal refractory effects (alv-asv) were confirmed at single parieto-occipital and occipital electrodes, respectively (Figures 2, 4), and after 150 ms post stimulus. In age group 2 (7–9 years), the preceding ISI and the recording site rather than the MT influenced ERP amplitudes (see main effect of ISI and interactions of ISI × topographical factor in Table 6).

TABLE 7A

Table 7A. F-values of the overall and subANOVAs (unimodal, crossmodal) for visual ERPs for age-group 3 (10–12 years), separately for each time window (100–150 ms, 150–220 ms, 260–340 ms).

TABLE 7B

Table 7B. F-values of the overall and subANOVAs (unimodal, crossmodal) for auditory ERPs for age-group 3 (10–12 years), separately for each time window (90–150 ms, 150–250 ms, 260–340 ms).

TABLE 7C

Table 7C. F-values of the overall and subANOVAs (unimoda, crossmodal) for visual and auditory ERPs at midline electrodes for age-group 3 (10–12 years), separately for each time window (visual: 100–150 ms, 150–220 ms, 260–340 ms, auditory: 90–150 ms, 150–250 ms, 260–340 ms).

Auditory ERP Results

Auditory ERPs were characterized by a fronto-centrally distributed positive peak, a second, negative peak with a maximal deflection over the vertex and a third, positive peak with a vertex maximum as well. This ERP pattern was found in all age groups. The latencies of the first and the second peak decreased with increasing age: The first positive wave peaked around 60 ms post stimulus in adults and around 70 ms post stimulus in children. The next largest peak (with negative polarity) was between 80–90 ms post stimulus in adults and around 100 ms post stimulus in children. The third, positive peak showed a similar latency with approximately 180 ms post stimulus across the age groups. Figure 3 depicts the grand averaged ERPs to auditory stimuli at representative electrodes for (i) the four stimulus condition, i.e., unimodal vs. crossmodal condition and short ISI vs. long ISI, and (ii) the four age groups.

FIGURE 3

Figure 3. Grand averaged ERPs to auditory stimuli in the unimodal (black line) vs. crossmodal (red line) condition and after short (dashed line) vs. long (solid line) interstimulus intervals. ERPs are demonstrated at a subset of representative electrodes and for the four different age groups. Abbreviations: [asa]: e.g., ERP to an auditory stimulus preceded by an auditory stimulus with a short ISI; a, auditory; v, visual; s, short ISI; l, long ISI. Time windows used for the statistical analyses are marked in gray. Negativity is plotted upwards.

FIGURE 4

Figure 4. (A) Visual refractory period effects (p ≤ 0.05) in the unimodal (black, left segment) and the crossmodal (red, right segment) condition, separately illustrated for each age group (a). Voltage difference maps of conditions with a long ISI minus conditions with a short ISI (unimodal vs. crossmodal) (b). Analyzed time windows: 100–150 ms (children), 90–140 ms (adults). Abbreviations: [vsv]: e.g., ERP to a visual stimulus preceded by a visual stimulus with a short ISI: a, auditory; v, visual; s, short ISI; l, long ISI. (B) Visual refractory period effects (p ≤ 0.05) in the unimodal (black, left segment) and the crossmodal (red, right segment) condition, separately illustrated for each age group (a). Voltage difference maps of conditions with a long ISI minus conditions with a short ISI (unimodal vs. crossmodal) (b). Analyzed time windows: 150–220 ms (children), 140–190 ms (adults). Abbreviations: [vsv]: e.g., ERP to a visual stimulus preceded by a visual stimulus with a short ISI: a, auditory; v, visual; s, short ISI; l, long ISI. (C) Visual refractory period effects (p ≤ 0.05) in the unimodal (black, left segment) and the crossmodal (red, right segment) condition, separately illustrated for each age group (a). Voltage difference maps of conditions with a long ISI minus conditions with a short ISI (unimodal vs. crossmodal) (b). Analyzed time window: 260–340 ms in children and adults. Abbreviations: [vsv]: e.g., ERP to a visual stimulus preceded by a visual stimulus with a short ISI: a, auditory; v, visual; s, short ISI; l, long ISI.

In the first 250 ms post stimulus, ERPs to auditory stimuli showed refractory effects which depended on the preceding modality, the recording site and the age of the participant [time window 90–150 ms (children)/70–120 ms (adults): ISI × MT × AP × E × Age group, F_{(5, 190)} = 7.43, p < 0.001, time window 150–250 ms (children)/140–250 ms (adults): ISI × MT × E × Age group, F_{(5, 260)} = 2.065, p < 0.05]. Due to the interaction with the between-subject-factor Age group, the age groups were analyzed separately in the following analyses.

Unimodal refractory period effects

In all age groups, significant or marginally significant main effects of MT and ISI and/or interactions between MT, ISI and topographical factors (see Five-Way- and Three-Way ANOVA in Tables 4–7) reflected the influence of the preceding stimulus modality, the preceding ISI and the recording site on ERP amplitudes. In the time window 150–250 ms (children) and 140–250 ms (adults), respectively, significant unimodal refractory effects (ala-asa) were confirmed along the midline electrodes in age group 1 (4–6 years) and age group 2 (7–9 years), and widespread over the scalp in the oldest children and in adults. In age group 1 and in adults, unimodal ISI effects were most prominent at fronto-polar recording sites in the first time window (90–150 ms (children) and 70–120 ms (adults), respectively). Beyond 260 ms post-stimulus, unimodal ERP refractory effects were obtained in all age groups (Figures 3, 5).

FIGURE 5

Figure 5. (A) Auditory refractory period effects (p ≤ 0.05) in the unimodal (black, left segment) and the crossmodal (red, right segment) condition, separately illustrated for each age group (a). Voltage difference maps of conditions with a long ISI minus conditions with a short ISI (unimodal vs. crossmodal) (b). Analyzed time windows: 90–150 ms (children), 70–120 ms (adults). Abbreviations: [asa]: e.g., ERP to an auditory stimulus preceded by a auditory stimulus with a short ISI: a, auditory; v, visual; s, short ISI; l, long ISI. (B) Auditory refractory period effects (p ≤ 0.05) in the unimodal (black, left segment) and the crossmodal (red, right segment) condition, separately illustrated for each age group (a). Voltage difference maps of conditions with a long ISI minus conditions with a short ISI (unimodal vs. crossmodal) (b). Analyzed time windows: 150–250 ms (children), 140–250 ms (adults). Abbreviations: [asa]: e.g., ERP to an auditory stimulus preceded by a auditory stimulus with a short ISI: a, auditory; v, visual; s, short ISI; l, long ISI. (C) Auditory refractory period effects (p ≤ 0.05) in the unimodal (black, left segment) and the crossmodal (red, right segment) condition, separately illustrated for each age group (a). Voltage difference maps of conditions with a long ISI minus conditions with a short ISI (unimodal vs. crossmodal) (b). Analyzed time window: 260–340 ms in both children and adults. Abbreviations: [asa]: e.g., ERP to an auditory stimulus preceded by a auditory stimulus with a short ISI: a, auditory; v, visual; s, short ISI; l, long ISI.

Crossmodal refractory period effects

In children aged 4–6 years, ERP amplitudes were significantly influenced by a modality switch, the preceding ISI and the recording site (see significant of marginally significant interaction of MT × ISI × topographical factor in the Five-Way- and Three-Way-ANOVA, Table 5). In the post-hoc analyses, crossmodal refractory effects were found over frontal, fronto-central, and central brain regions until 250 ms post stimulus (Figures 3, 5). In older children and in adults, the Five-Way- and the Three-Way-ANOVA revealed significant main effects of MT and/or ISI and interactions of MT × ISI × one or more topographical factors (see Tables 4, 6, and 7), which again indicated the influence of preceding modality, the preceding ISI and the recording site on ERP amplitudes. In contrast to the results of the post-hoc analyses in the youngest age group, crossmodal ISI effects were most pronounced over the posterior scalp after 150 ms post stimulus, and these effects were most reliable in adults (Figures 3, 5).

Discussion

The present study tested whether uni- and crossmodal ERP refractory effects, indicating uni- and multisensory development, emerge sequentially (hierarchical developmental view) or parallel (differentiation developmental view). ERPs elicited in response to auditory and visual stimuli presented with two different ISIs (1000, 2000 ms) were analyzed as a function of the age of the participants, the modality of the preceding stimulus and the ISI.

ERP amplitudes to both visual and auditory stimuli were modulated by the modality of the preceding stimulus and the ISI. The degree of modulation depended on (i) the recording site and (ii) the age of the participant. The most pronounced crossmodal refractory effects both for visual and auditory ERPs were observed in the youngest age group (4–6 years). Crossmodal refractory period effects emerged earlier and with a different and broader topography in the youngest children compared to adults: They had a frontal and fronto-central topography irrespectively of stimulus modality in the youngest age group but were most pronounced over the posterior scalp in the oldest children and in adults. By contrast, unimodal refractory effects were observed for both modality conditions in all age groups.

We interpret our data in favor of the multisensory differentiation (Lickliter and Bahrick, 2004) and the multisensory perceptual narrowing account (Lewkowicz and Ghazanfar, 2009). By contrast, our results seem to be incompatible with the idea of a strict hierarchical development (Piaget, 1952), with unimodal processing maturing prior to the emergence of the first crossmodal interactions.

Firstly, we found the most prominent crossmodal interactions in the youngest age group (4–6 years). These effects emerged within the first analyzed time window (<150 ms). By contrast, crossmodal refractory effects in older children and adults were observed beyond the time epoch of 150 ms. These results suggest that at earlier processing stages, both in the visual and auditory processing pathway, crossmodal interactions occur in young children but not in older children and adults. Interestingly, these early crossmodal interactions had a fronto-central topography which was clearly distinct from the parietal topography of later crossmodal interactions in older children and adults. These developmental trends were observed irrespectively of the stimulus modality. It might be speculated that the early crossmodal interactions observed in children were due to still existing exuberant connections between sensory systems (Innocenti and Clarke, 1984). As development progresses, crossmodal interactions become more specific as a result of crossmodal experience (Lewkowicz and Ghazanfar, 2009). More specific, crossmodal interactions might emerge from the setting up of multisensory areas such as those in the parietal cortex (Bolognini et al., 2010; Kamke et al., 2012). It is important to note that these results are not incompatible with previous reports in adults demonstrating earlier (<100 ms) crossmodal ERP effects in adults (Giard and Peronnet, 1999; Molholm et al., 2002). These authors compared the processing of unimodal vs. crossmodal stimuli, while we presented unimodal stimuli only and assessed the refractoriness of different stages of the auditory and visual processing pathway due to a preceding stimulus.

Although there were some differences between age groups, mainly in topography, reliable unimodal refractory effects were observed in all groups of children, irrespectively of age and in adults. These results are in accord with Coch et al. (2005), who reported unimodal refractory effects both for auditory and visual ERPs in their youngest group as well (6–8 years). Thus, though the finding of an earlier maturation of uni- compared to crossmodal processing would be compatible with a hierarchical view of sensory development, the parallel finding of pronounced crossmodal interactions in the youngest but not older age groups is incompatible with this account. Earlier behavioral studies have reported that unisensory processing reaches adult level earlier than multisensory processing (Röder et al., 2013). Thus, uni- and multisensory processing seem to progress in parallel with a longer developmental time course of crossmodal (see Brett-Green et al., 2008; Brandwein et al., 2010) than unimodal development. Indeed, a number of recent studies has consistently shown a protracted developmental time course of multisensory processes into adolescents (Gori et al., 2008; Nardini et al., 2008; Barutchu et al., 2009).

Taken together, the present study provides evidence for the intersensory differentiation and multisensory perceptual narrowing approach to explain multisensory development. Some crossmodal interactions existing at earlier developmental ages seem to be eliminated and substituted by specific crossmodal interactions. Uni- and multisensory development runs in parallel with unimodal development leading.

Conflict of Interest Statement

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Acknowledgments

We are grateful to Birthe Pagel for her contributions to the study paradigm and her help during setting up the study. We thank Julia C. Gädeke, Marlene Hense, and Johanna Möller for their help during data acquisition. The study was funded by grants of the University Medical Center Hamburg-Eppendorf to Jessika Johannsen and ERC 249425 “CriticalBrainChanges” to Brigitte Röder.

References

Barutchu, A., Crewther, D. P., and Crewther, S. G. (2009). The race that precedes coactivation: development of multisensory facilitation in children. Dev. Sci. 12, 464–473. doi: 10.1111/j.1467-7687.2008.00782.x

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Bolognini, N., Olgiati, E., Rossetti, A., and Maravita, A. (2010). Enhancing multisensory spatial orienting by brain polarization of the parietal cortex. Eur. J. Neurosci. 31, 1800–1806. doi: 10.1111/j.1460-9568.2010.07211.x

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Brandwein, A. B., Foxe, J. J., Russo, N. N., Altschuler, T. S., Gomes, H., and Molholm, S. (2010). The development of audiovisual multisensory integration across childhood and early adolescence: a high-density electrical mapping study. Cereb. Cortex 21, 1042–1055. doi: 10.1093/cercor/bhq170

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Bremner, A. J., Lewkowicz, D. J., and Spence, C. (2012). Multisensory Development. Oxford: Oxford University Press. doi: 10.1093/acprof:oso/9780199586059.001.0001

CrossRef Full Text

Brett-Green, B. A., Miller, L. J., Gavin, W. J., and Davies, P. L. (2008). Multisensory integration: a preliminary ERP study. Brain Res. 1242, 283–290. doi: 10.1016/j.brainres.2008.03.090

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Cheour, M., and Näätänen, R. (1998). Interstimulus interval and auditory event-related potentials in children: evidence for multiple generators. Electroencephalogr. Clin. Neurophysiol. 108, 345–354. doi: 10.1016/S0168-5597(97)00081-6

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Coch, D., Skendzel, W., and Neville, H. J. (2005). Auditory and visual refractory period effect in children and adults: an ERP study. Clin. Neurophysiol. 116, 2184–2203. doi: 10.1016/j.clinph.2005.06.005

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Courchesne, E. (1978). Neurophysiological correlates o cognitive development: changes in long-latency event-related potentials from childhood to adulthood. Electroencephalogr. Clin. Neurophysiol. 45, 468–482. doi: 10.1016/0013-4694(78)90291-2

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Davis, H., Osterhammel, P. A., Wier, C. C., and Gjerdingen, D. B. (1972). Slow vertex potentials: interactions among auditory, tactile, electric and visual stimuli. Electroencephalogr. Clin. Neurophysiol. 33, 537–545. doi: 10.1016/0013-4694(72)90244-1

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

De Haan, M., and Nelson, C. A. (1997). Recognition of mother's face by 6-month old infants. A neurobehaviuoral study. Child Dev. 68, 187–210. doi: 10.1111/j.1467-8624.1997.tb01935.x

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Driver, J., and Noesselt, T. (2008). Multisensory interplay reveals crossmodal influences on “sensory-specific” brain regions, neural responses, and judgements. Neuron 57, 11–23. doi: 10.1016/j.neuron.2007.12.013

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Fruhstorfer, H. (1971). Habituation and dishabituation of the human vertex response. Electroencephalogr. Clin. Neurophysiol. 27, 346–355. doi: 10.1016/0013-4694(69)91443-6

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Giard, M. H., and Peronnet, F. (1999). Auditory-visual integration during multimodal object recognition in humans: a behavioral and electrophysiological study. J. Cogn. Neurosci. 11, 473–490. doi: 10.1162/089892999563544

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Gibson, E. J. (1969). Principles of Perceptual Learning and Development. New York, NY: Appleton-Century-Crafts.

Gibson, E. J. (1984). “Perceptual development from the ecological approach,” in Advances in Developmental Psychology, ed M. E. Lamb (Hillsdale, NJ: Lawrence Erlbaum Associates), 243–286.

Gondan, M., Niederhaus, B., Rösler, F., and Röder, B. (2005). Multisensory processing in the redundant-target effect: a behavioral and event-related potential study. Percept. Psychophys. 67, 713–726. doi: 10.3758/BF03193527

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Gori, M., Del Viva, M., Sandini, G., and Burr, D. C. (2008). Young children do not integrate visual and haptic form information. Curr. Biol. 18, 694–698. doi: 10.1016/j.cub.2008.04.036

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Innocenti, G. M., and Clarke, S. (1984). Bilateral transitory projection to visual areas from auditory cortex in kittens. Dev. Brain Res. 14, 143–148. doi: 10.1016/0165-3806(84)90019-1

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Kamke, M. R., Vieth, H. E., Cottrell, D., and Mattingley, J. B. (2012). Parietal disruption alters audiovisual binding in the sound-induced flash illusion. Neuroimage 62, 1334–1341. doi: 10.1016/j.neuroimage.2012.05.063

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Lewkowicz, D. J., and Ghazanfar, A. A. (2009). The emergence of multisensory systems through perceptual narrowing. Trends Cogn. Sci. 13, 470–478. doi: 10.1016/j.tics.2009.08.004

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Lickliter, R., and Bahrick, L. E. (2004). “Perceptual development and the origins of multisensory responsiveness,” in The Handbook of Multisensory Processes, eds G. A. Calvert, C. Spence, and B. E. Stein (Cambridge, MA: MIT Press), 643–654.

Maurer, D. (1993). “Neonatal synesthesia: implications for the processing of speech and faces,” in Developmental Neurocognition: Speech and Face Processing in the First Year of Life, eds B. Boysson-Bardies, S. de Schonen, P. Jusczyk, P McNeilage, and J. Morton (Dordrecht: Kluwer), 109–124. doi: 10.1007/978-94-015-8234-6_10

CrossRef Full Text

Molfese, D. L., and Molfese, V. J. (2000). “The continuum of language development during infancy and early childhood: electrophysiological correlates,” in Progress in infancy Research, eds C. Rovee-Collier, L.P. Lipsitt, and H. Hayne (Mahwah, NJ: Lawrence Erlbaum Associates), 251–287.

Molholm, S., Ritter, W., Murray, M. M., Javitt, D. C., Schroeder, C. E., and Foxe, J. J. (2002). Multisensory auditory-visual interactions during early sensory processing in humans: a high-density electrical mapping study. Cogn. Brain Res. 14, 115–128. doi: 10.1016/S0926-6410(02)00066-6

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Nardini, M., Jones, P., Bedford, R., and Braddick, O. (2008). Development of cue integration in human navigation. Curr. Biol. 18, 689–693. doi: 10.1016/j.cub.2008.04.021

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Nelson, C. A., and Monk, C. (2001). “The use of event-related potentials in the study of cognitive development,” in Handbook of Developmental Cognitive Neuroscience, eds C. A. Nelson and M. Luciana (Cambridge, MA: MIT Press), 125–136.

Neville, H. J., Schmidt, A., and Kutas, M. (1983). Altered visual-evoked potentials in congenitally deaf adults. Brain Res. 266, 127–132. doi: 10.1016/0006-8993(83)91314-8

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Piaget, J. (1952). The Origins of Intelligence in Children. New York, NY: International Universities Press. doi: 10.1037/11494-000

CrossRef Full Text

Ritter, W., Paavilainen, P., Lavikainen, J., Reinikainen, K., Alho, K., Sams, M., et al. (1992). Event-related potentials to repetition and change of auditory stimuli, Electroencephalogr. Clin Neurophysiol. 83, 306–321. doi: 10.1016/0013-4694(92)90090-5

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Röder, B., Pagel, B., and Heed, T. (2013). The implicit use of spatial information develops later for crossmodal than for intramodal temporal processing. Cognition 126, 301–306. doi: 10.1016/j.cognition.2012.09.009

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Schneider, T. R., Engel, A. K., and Debener, S. (2008). Multisensory identification of natural objects in a two-way crossmodal priming paradigm. Exp. Psychol. 55, 121–131. doi: 10.1027/1618-3169.55.2.121

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Senkowski, D., Schneider, T. R., Foxe, J. J., and Engel, A. K. (2008). Crossmodal binding through neural coherence: implications for multisensory processing. Trends Neurosci. 31, 401–409. doi: 10.1016/j.tins.2008.05.002

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Stein, B. F., and Meredith, M. A. (1993). The Merging of the Sense. Cambridge, MA: MIT Press.