According to the effect size of a similar study [ηp2 = 0.10; Experiment 4 in Lunn et al. (2019)], a sample size estimation was done using G*power software (Faul et al., 2009). The result revealed that a sample of 26 participants was required to at least detect an interaction with an effect size of ηp2 = 0.10 (α = 0.05, 1-β = 0.80). Twenty-nine healthy college students participated in the experiment. They reported a normal or corrected-to-normal vision and normal hearing. Three participants were excluded because the accuracy rates were lower than 70%. Data of 26 participants (19 females; mean age = 19.92 years, SD = 1.74, range = 18–24 years) entered the final analysis. All participants signed informed consent and were paid 25 RMB. The study was approved by the Ethics Committee of the Department of Psychology, Sun Yat-sen University.

The experiment was controlled by E-prime 2.0 software¹. Participants sat 60 cm in front of a 23-inch monitor (1,920 × 1,080; 60 Hz) in a sound-attenuated, dimly lit room. The auditory stimuli were generated by Adobe Audition CC 2019 software, sampled at 44.1 kHz, and quantized to 16 bits. Before the experiment, the sound was tuned to a comfortable volume for all participants (range: 35–45 dB).

All consonants except “Y” were used (1° × 1.4°of visual angle) in the n-back task. Auditory distractors (1,000 Hz pure tone) were presented for 200 ms either on the left or right side equiprobably via headphones (SONY MDR-XB450). Visual distractors were white discs (diameter: 1.9° visual angle), presented at an eccentricity of 7.5° degrees (screen center to discs center). All visual distractors were presented either to the left or right side of the central letters with equal probability for 200 ms. In the audiovisual distractors condition, auditory and visual distractors were presented at the same side concurrently.

A 2 (Load: low vs. high) × 4 (Distractor type: auditory, visual, audiovisual vs. no distractor) within-participants design was adopted. The 1-back and 2-back tasks were used to manipulate the working memory load. The trial scheme is shown in Figure 1. Each letter series started with a cross fixation presented at the center of the screen for 500 ms. The fixation was then replaced by a stream of fourteen letters. Each letter was presented for 200 ms with an inter-stimulus interval (ISI) of 1,800 ms. Participants were required to memorize the stream of the letter and to report whether the current target letter was the same or not as the letter presented one or two steps back, in 1-back and 2-back tasks respectively. They were instructed to respond by pressing one of two buttons (LB and RB) with the joystick as quickly and accurately as possible. Response keys on the joystick were counterbalanced between participants.

In addition to the central visual targets, an auditory, visual, or audiovisual distractor was also presented concurrently with the letter in 75% of all trials. They were presented equiprobably at the left or right side of the letter for 200 ms. Participants were required to ignore the peripheral distractor and concentrate on the central letter. In the remaining 25% of all trials, no distractors were presented. Overall, the experiment consisted of 8 blocks with 8 letter series each. Participants had practiced before the formal experiments.

Reaction times (RTs) and accuracy rate (ACC) were calculated separately for each experimental condition. For all participants, RTs of correct responses between 100 and 1,800 ms were included in the analysis. Besides, RTs exceeding ± 3 SD of each participant’s mean reaction time in each experimental condition were removed. Analyses of variance (ANOVAs) were calculated separately for mean RTs and mean ACC with factors of Load (low vs. high) and Distractor type (auditory, visual, audiovisual vs. no distractor). The interference effect was calculated by subtracting the mean RT without distractors from the mean RT with distractors. If necessary, the Greenhouse-Geisser method was adopted to correct degrees of freedom. Besides, Bonferroni corrections were used for post-hoc pair-wise comparisons and simple effects.

Based on the effect size of a similar study [ηp2 = 0.10; Experiment 4 in Lunn et al. (2019)], a sample size estimation was done using G*power software (Faul et al., 2009). The result revealed that a sample of 26 participants was required to at least detect an interaction with an effect size of ηp2 = 0.10 (α = 0.05, 1-β = 0.80). A new group of thirty-two healthy college students participated in the experiment. They had a normal or corrected-to-normal vision and normal hearing. Three participants were excluded because of excessive (>25%) EEG artifacts. Data of 29 participants (16 females; mean age = 20.62 years, SD = 1.99, range = 18–26 years) entered the final analysis. All participants signed informed consent and were paid 75 RMB. The study was approved by the Ethics Committee of the Department of Psychology, Sun Yat-sen University.

The experimental apparatus and stimuli were the same as those in Experiment 1 except for the eccentricity of the distractors (4°), and the presenting mode of the auditory stimuli. Specifically, auditory distractors were presented at either the left or right side equiprobably via two invisible loudspeakers (Creative inspire T12), which were placed at the source location of the visual distractors behind the screen. Before the experiment, the sound was tuned to a comfortable volume for all participants (range: 65–75 dB).

A 2 (Load: low vs. high) × 4 (Distractor type: auditory, visual, audiovisual vs. no distractor) within-participants design was adopted. There were 105 trials for each experimental condition. The procedure and task were the same as those in Experiment 1 (Figure 1).

The EEG was recorded from 64 Ag-AgCl electrodes mounted in an elastic cap (Easy Cap, Germany) with a NeuroScan SynAmps2 Amplifier (Scan 4.5, Neurosoft Labs, Inc. Virginia, USA). A left earlobe electrode was used as an online reference. The ground electrode was located on the forehead. Vertical eye movements were monitored with two electrodes upper and below the right eye. Horizontal eye movements were recorded with two electrodes placed at the outer canthi of each eye. Electrode impedance was kept below 5 kΩ for all electrodes. Online recordings were bandpass filtered at 0.05–100 Hz (12 dB/oct, 40 dB/dec) and sampled at 500 Hz. During the experiment, participants were instructed to fixate on the center of the monitor and try not to make horizontal or vertical eye movements.

The offline analysis of EEG data was performed using Matlab R2016b and eeglab 14.1.2b². First, all scalp electrodes were re-referenced to the average of left and right earlobes. Then, the continuous EEG was bandpass filtered (IIR Butterworth, filter order = 2) at 0.05–30 Hz. An infomax independent component analysis (ICA) algorithm (Bell and Sejnowski, 1995) was applied for correcting eye movement artifacts. The SASICA plugin with ADJUST was used to identify the artifact component. Furthermore, the interval of 0–200 ms prior to the distractors served as the baseline; EEG signal epochs ended 800 ms after the onset of the distractor stimuli, yielding a total epoch of 1 s. Finally, trials with voltages exceeding ± 100 μV were excluded from ERP averages. The remaining epochs to eight different conditions were averaged separately for each participant with baseline corrections. In the present experiment, the average artifact rejection rate was 3.23% of all trials (SD = 5.02, range = 0–18.2%).

Behavioral data analyses were identical to Experiment 1. Analyses of variance (ANOVAs) were calculated separately for mean RTs and mean ACC with within-participants factors of Load (low vs. high) and Distractor type (auditory, visual, audiovisual vs. no distractor).

For the ERPs, to control for the overlap and generic cognitive process (such as contingent negative variation, CNV), the ERPs elicited by no distractor trials were subtracted from the ERPs elicited by auditory (A), visual (V), and audiovisual (AV) distractors, respectively. Then, to estimate the multisensory integration effect, the ERPs elicited by A distractors and V distractors were summated (A + V) and compared with the ERPs elicited by AV distractors. Specifically, the audiovisual distractors were integrated if significant differences were found between the (A + V) ERPs and the AV ERPs (Giard and Peronnet, 1999; Stevenson et al., 2014).

Time windows and electrodes were selected based on the previous studies (Talsma and Woldorff, 2005; Van der Burg et al., 2011), the grand average ERPs and the topographic map. Previous studies have found three phases of effects of integration and/or attention beginning at around 160 ms, and peaking at 190 ms (scalp positivity), 250 ms (negativity), and 300–500 ms (positivity) after stimulus onset (Talsma and Woldorff, 2005). We also did a mass-univariate statistical analysis with corrections based on previous studies. Specifically, we did ANOVAs with factors of Load and Distractor type at each electrode and each time-point across participants. To avoid the type-I error due to the large number of tests, the multisensory integration effects were thought to be significant only when the p-value was smaller than 0.05 at 10 (~20 ms) or more continuous time points on at least two nearby electrodes (Van der Burg et al., 2011; Alsius et al., 2014). Two time windows of 240–340 ms (electrode FP_Z) and 450–600 ms (electrode F_Z) were selected. To further test the hypothesis whether multisensory integration should be more pronounced in the high load condition than in the low load condition, the mean amplitudes in these time windows were analyzed by ANOVAs with factors of Load (low vs. high) and Distractor type (A + V vs. AV). If necessary, the Greenhouse-Geisser method was adopted to correct degrees of freedom. Besides, Bonferroni corrections were used for post-hoc pair-wise comparisons and simple effects.

The overall results of ANOVA are shown in Table 1. For RTs, ANOVA revealed a significant main effect of Load [F_{(1, 28)} = 52.86, p < 0.001, ηp2 = 0.65], indicating slower responses in high load condition than in low load condition (M = 790.25 vs. 636.85 ms, SE = 31.97 vs. 22.90). Neither the main effect of Distractor type [F_{(2.44, 68.37)} = 0.68, p = 0.54] nor the interaction between Load and Distractor type [F_{(3, 84)} = 0.30, p = 0.83] was significant.

For ACC, the main effect of Load was significant [F_{(1, 28)} = 17.81, p < 0.001, ηp2 = 0.39], indicating ACC in low load condition was higher than in high load condition (M = 0.94 vs. 0.90, SEs = 0.01). The main effect of Distractor type was significant [F_{(3, 84)} = 3.68, p < 0.01, ηp2 = 0.12]. However, post-hoc analyses did not reveal any significant pair-wise comparisons. The interaction between Load and Distractor [F_{(3, 84)} = 0.75, p = 0.52] type was not significant, either.

To control for the differential overlap and generic cognitive process, overlap correction was done before summating the A and V ERPs (Giard and Peronnet, 1999; Talsma and Woldorff, 2005; Van der Burg et al., 2011; Stevenson et al., 2014). To show the necessity of this overlap correction, ERPs elicited by stimuli with A, V, and AV distractors under high load conditions at the C_Z electrode were averaged, and then ERPs elicited by stimuli without distractor under high load conditions at the C_Z electrode were subtracted from the averaged ERP waveform. The effect of overlap correction was shown in Figure 3.

To further quantify the effectiveness of overlap correction, we tested whether the ERPs elicited by stimuli with (A + V) and AV distractors differed before the onset of the distractor. Theoretically, no significant differences in the mean amplitudes should be found during any time window of −200–0 ms. Besides, the averaged ERPs of distractors should not differ from 0 μV. Thus, the time window of −20–0 ms at the C_Z electrode was selected. The one-sample t-test was used to test whether the averaged ERPs elicited by stimuli with distractors differed from 0 μV. ANOVA with factors of Load (low vs. high) and Distractor type (A + V vs. AV) was used to test whether the ERPs elicited by stimuli with (A + V) and AV distractors differed from each other. When no-distractor ERPs were not subtracted from ERPs of (A + V) and AV distractors, although no significant results were found for the ANOVA (main effect of Distractor type: [F_{(1, 28)} = 1.56, p = 0.22, ηp2 =0.05]), the one-sample t-test showed that the averaged ERPs with distractors differed significantly from 0 μV [t₍₂₈₎ = −5.80, p < 0.001; M = −0.49 μV, SE = 0.08]. However, after subtracting the no-distractor ERPs from the (A + V) and AV ERPs, neither the t-test [t₍₂₈₎ = −1.77, p = 0.09] nor the main effect of Distractor type [F_{(1, 28)} = 0.06, p = 0.81, ηp2 = 0.03] was significant. These results showed that subtracting the ERPs elicited by no-distractor trials from the ERPs elicited by A, V, and AV distractor trials could effectively remove the overlap due to the generic cognitive process.

The results of the ANOVA conducted at each time epoch are reported in Table 1. The ANOVA of mean amplitudes for the time window of 240–340 ms showed a significant interaction between Load and Distractor type (F_{(1, 28)} = 4.57, p < 0.05, ηp2 = 0.14; see Figure 4). Follow-up analyses showed that the mean amplitudes of (A + V) ERPs were more positive than those in the AV condition while the working memory load was high (M = 0.85 vs. 0.08 μV, SE = 0.60 vs. 0.41; t₍₂₈₎ = 2.10, p < 0.05). However, under low load conditions, no significant differences in mean amplitudes were observed between these two conditions. These results showed that audiovisual distractors were integrated under high load conditions but not under low load conditions, suggesting that the working memory load modulated the integration of audiovisual distractors. Neither the main effect of Load [F_{(1, 28)} = 1.21, p > 0.05] nor the main effect of Distractor type [F_{(1, 28)} = 0.24, p > 0.05] was significant.

In addition, the ANOVAs of mean amplitudes for the late time window 450–600 ms showed no significant interaction between Load and Distractor type [F_{(1, 28)} = 0.29, p > 0.05]. Neither the main effect of Load [F_{(1, 28)} = 3.89, p > 0.05] nor the main effect of Distractor type [F_{(1, 28)} = 2.19, p > 0.05] was significant.

In reference to the effect size of a similar study [ηp2 = 0.10; Experiment 4 in Lunn et al. (2019)], a sample size estimation was done using G*power software (Faul et al., 2009). The result revealed that a sample of 26 participants was required to at least detect an interaction with an effect size of ηp2 = 0.10 (α = 0.05, 1-β = 0.80). Another new group of thirty-seven healthy college students participated in the experiment. They had a normal or corrected-to-normal vision and normal hearing. Two participants were excluded because of equipment problems. Another three participants were excluded because of excessive (>25%) EEG artifacts. Data of 32 participants (21 females; mean age = 20.19 years, SD = 2.15, range = 18–27 years) entered the final analysis. Participants signed informed consent and were paid 75 RMB. The study was approved by the Ethics Committee of the Department of Psychology, Sun Yat-sen University.

The experimental apparatus and stimuli were the same as those in Experiment 2 except for the type of distractors. Specifically, visual distractors consisted of pictures of a cat or a dog. They were presented for 500 ms at an eccentricity of 4° degrees (screen center to image center). Auditory distractors were the sounds that the animals made. All auditory stimuli were presented at either the left or right side equiprobably for 500 ms via two invisible loudspeakers (Creative inspire T12) placed at the source location of the visual distractors behind the screen. In the multisensory audiovisual distractors condition, both auditory and visual distractors of the same animal were presented on the same side concurrently. Before the experiment, the sound was tuned to a comfortable volume for all participants (range: 65–75 dB).

A 2 (Load: low vs. high) × 4 (Distractor type: auditory, visual, audiovisual vs. no distractor) within-participants design was adopted. There were 105 trials for each experimental condition. In order to convey semantic information clearly, central letters and peripheral distractors were both presented for 500 ms. The stimuli used as distractors are shown in Figure 1B. The procedure and task were the same as those in Experiment 1.

EEG recording and preprocessing were identical to Experiment 2. The average artifact rejection rate in the present experiment was 2.27% of all trials (SD = 4.19, range = 0–20.4%). ERPs in each experimental condition were averaged separately for each participant.

Behavioral data analyses were identical to Experiment 2. Analyses of variance (ANOVAs) were calculated separately for mean RTs and mean ACC with within-participants factors of Load (low vs. high) and Distractor type (auditory, visual, audiovisual vs. no distractor).

For the ERPs, the data analyses were identical to Experiment 2. After the overlap correction, time windows and electrodes were selected based on the previous studies (Talsma and Woldorff, 2005; Van der Burg et al., 2011), the grand average ERPs and the topographic map. We did a mass-univariate statistical analysis (ANOVAs with factors of Load and Distractor type at each electrode and each time-point across participants) with correction based on previous studies. That is, the multisensory integration effects were thought to be significant only when the p-value was smaller than 0.05 at 10 (~20 ms) or more continuous time points on at least two nearby electrodes (Van der Burg et al., 2011; Alsius et al., 2014). Two time windows of 250–330 ms (averaged across electrodes C_Z and FC_Z) and 440–600 ms (averaged across electrodes AF7, F5, and F7) were selected. To further test the hypothesis whether multisensory integration should be more pronounced in the high load condition than in the low load condition, the mean amplitudes in these two time windows were analyzed by ANOVAs with factors of Load (low vs. high) and Distractor type (A + V vs. AV), respectively. If necessary, Greenhouse-Geisser method was adopted to correct degrees of freedom. Besides, Bonferroni corrections were used for post-hoc pair-wise comparisons and simple effects.

The overall results of ANOVA are shown in Table 1. For RTs, ANOVA revealed a significant main effect of Load [F_{(1, 31)} = 115.59, p < 0.001, ηp2 = 0.79], indicating the responses were slower in the high load condition than in the low load condition (M = 700.72 vs. 562.29 ms, SE = 26.62 vs. 17.92). The main effect of Distractor type was significant [F_{(3, 93)} = 8.20, p < 0.001, ηp2 = 0.21], indicating the responses to letters with visual distractors (M = 639.35 ms, SE = 21.79) were slower than to letters with auditory, audiovisual distractors and with no distractors (auditory: M = 625.15 ms, SE = 21.76; audiovisual: M = 630.29 ms, SE = 21.80; no distractors: M = 631.22 ms, SE = 21.98). The interaction between Load and Distractor Type was not significant [F_{(3, 93)} = 0.24, p = 0.87, ηp2 = 0.01].

For ACC, there was a significant main effect of Load [F_{(1, 31)} = 18.88, p < 0.001, ηp2 = 0.38], indicating that ACC in the low load condition was higher than in the high load condition (M = 0.94 vs. 0.89, SEs = 0.01). Neither the main effect of Distractor type [F_{(3, 93)} = 0.57, p = 0.64, ηp2 = 0.02] nor the interaction [F_{(2, 93)} = 2.04, p = 0.11, ηp2 = 0.06] was significant.

Similar to Experiment 2, ERPs elicited by stimuli with A, V, and AV distractors under high load conditions at the C_Z electrode were averaged, and then ERPs elicited by stimuli without distractors in the high load condition at the C_Z electrode were subtracted from the averaged ERP waveform.

To further quantify the effectiveness of overlap correction, we tested whether the ERPs elicited by stimuli with (A + V) and AV distractors differed before the onset of the distractor. The time window of −20–0 ms at the C_Z electrode was selected. The one-sample t-test was used to test whether the averaged ERPs elicited by stimuli with distractors differed from 0 μV. The two-way ANOVA with factors of Load (low vs. high) and Distractor type (A + V vs. AV) was used to test whether the ERPs elicited by stimuli with (A + V) and AV distractors differed from each other. When no-distractor ERPs were not subtracted from ERPs of (A + V) and AV distractors, the one-sample t-test showed that the averaged ERPs with distractors differed significantly from 0 μV [t₍₃₁₎ = −5.33, p < 0.001; M = −0.51 μV, SE = 0.10]. The ANOVA also showed a significant main effect of Distractor type [F_{(1, 31)} = 7.12, p < 0.05, ηp2 = 0.19], indicating more negative ERPs elicited by stimuli with (A + V) distractors than with AV distractors (M = −0.95 vs. −0.49 μV, SE = 0.22 vs. 0.10; t₍₃₁₎ = 2.66, p < 0.05). However, after subtracting the no-distractor ERPs from the (A + V) and AV ERPs, neither the t-test [t₍₃₁₎ = 0.12, p = 0.90] nor the main effect of Distractor type [F_{(1, 31)} = 0.18, p = 0.67, ηp2 = 0.01] was significant. These results showed that subtracting the ERPs elicited by no-distractor trials from the ERPs elicited by A, V, and AV distractor trials effectively removed the overlap due to the generic cognitive process.

The results of the ANOVA conducted at each time epoch are reported in Table 1. The ANOVA of mean amplitudes of 440–600 ms showed a significant interaction between Load and Distractor type [F_{(1, 31)} = 4.69, p < 0.05, ηp2 = 0.13; see Figure 5]. Follow-up analyses showed that the mean amplitudes of (A + V) ERPs were more negative than those in the AV condition when the working memory load was high (M = −0.88 vs. −0.16 μV, SE = 0.38 vs. 0.25; t₍₃₁₎ = 2.67, p < 0.05). However, under low load conditions, no significant differences in mean amplitudes were observed between these two conditions. Similarly, as in Experiment 2, the present results showed that audiovisual distractors were integrated under high load conditions but not under low load conditions. Moreover, the working memory load modulated the integration of audiovisual distractors at the late stage when the audiovisual distractors were the meaningful complex stimuli. Neither the main effect of Load [F_{(1, 31)} = 2.13, p > 0.05] nor the main effect of Distractor type [F_{(1, 31)} = 2.99, p > 0.05] was significant.

In addition, the ANOVAs of mean amplitudes of the early time window (250–330 ms) showed no significant interaction between Load and Distractor type [F_{(1, 31)} = 0.26, p > 0.05]. Neither the main effect of Load [F_{(1, 31)} = 0.46, p > 0.05] nor the main effect of Distractor type [F_{(1, 31)} = 0.54, p > 0.05] was significant.