Edited by: Cuntai Guan, Nanyang Technological University, Singapore
Reviewed by: Quentin Noirhomme, Maastricht University, Netherlands; Andrea Kübler, University of Würzburg, Germany
*Correspondence: Akinari Onishi
Kouji Takano
This article was submitted to Neuroprosthetics, a section of the journal Frontiers in Neuroscience
This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
Gaze-independent brain computer interfaces (BCIs) are a potential communication tool for persons with paralysis. This study applies affective auditory stimuli to investigate their effects using a P300 BCI. Fifteen able-bodied participants operated the P300 BCI, with positive and negative affective sounds (PA: a meowing cat sound, NA: a screaming cat sound). Permuted stimuli of the positive and negative affective sounds (permuted-PA, permuted-NA) were also used for comparison. Electroencephalography data was collected, and offline classification accuracies were compared. We used a visual analog scale (VAS) to measure positive and negative affective feelings in the participants. The mean classification accuracies were 84.7% for PA and 67.3% for permuted-PA, while the VAS scores were 58.5 for PA and −12.1 for permuted-PA. The positive affective stimulus showed significantly higher accuracy and VAS scores than the negative affective stimulus. In contrast, mean classification accuracies were 77.3% for NA and 76.0% for permuted-NA, while the VAS scores were −50.0 for NA and −39.2 for permuted NA, which are not significantly different. We determined that a positive affective stimulus with accompanying positive affective feelings significantly improved BCI accuracy. Additionally, an ALS patient achieved 90% online classification accuracy. These results suggest that affective stimuli may be useful for preparing a practical auditory BCI system for patients with disabilities.
The brain-computer interface (BCI), also referred to as the brain-machine interface (BMI), translates brain signals into control signals for computers or machines (Wolpaw et al.,
In EEG-based non-invasive BCIs, sensory evoked signals that can be modulated by intention have been used. A popular system is the visual P300 BCI (Farwell and Donchin,
For persons who have difficulty to control the gaze-dependent BCIs, various gaze-independent BCI techniques have also been proposed. For example, a gaze-independent visual speller (Blankertz et al.,
Among the gaze-independent BCIs, the auditory P300 BCI applications were proposed and evaluated in clinical studies. Sellers and Donchin employed word-based stimuli (“Yes,” “No,” “Pass,” and “End”) presented in visual and auditory modalities (Sellers and Donchin,
Several techniques and stimulus types have been proposed to improve the auditory BCI accuracy. Klobassa et al. (
Affective stimuli may be effective at improving the performance of EEG-based auditory P300 BCIs. Although affective auditory stimuli have not been applied to auditory P300 BCIs before, affective facial images were used as stimuli for a visual P300 BCI, and did improve classification accuracy (Zhao et al.,
In this research, we used positive and negative affective sounds (PA: a meowing cat sound, NA: a screaming cat sound) as stimuli for an auditory P300 BCI. We expected that the two affective stimuli would modulate ERPs and improve the classification accuracy of the BCI. Permuted stimuli of the positive and negative affective sounds (permuted-PA, permuted-NA) were also used for comparison by keeping the features hidden in affective sounds, except those in the time series. We used a visual analog scale (VAS) to measure positive and negative affective feelings in the participants. Offline analysis was applied to investigate the effects of affective stimuli in detail. We also conducted an online experiment with an ALS patient to validate the methods proposed in this study.
This study was approved by the institutional ethics committee at the National Rehabilitation Center for Persons with Disabilities, and all participants provided written informed consent according to institutional guidelines. All experiments were performed in accordance with the approved guidelines.
Fifteen participants (aged 29 ± 7.2 y.o.; 7 women) took part in this experiment. Fourteen participants were right-handed and one participant was ambidextrous, according to the Japanese version of the Edinburgh Handedness Inventory (Oldfield,
The P300 BCI system used provides auditory stimuli. Participants were required to perform an oddball task, as shown in Figure
Task and experimental design.
Positive affective (PA) and negative affective (NA) sounds were associated with the answers “yes” and “no,” respectively. A meowing cat sound was used as the PA sound, while a screaming cat sound was used as the NA sound. The meowing cat sound is available at the Sound Effect Lab (cat-cry2.mp3 on
Permuted stimuli of the positive and negative affective (permuted-PA and permuted-NA) sounds were also prepared for comparison. Time-domain scrambling (
As shown in Figure
The BCI system consisted of a laptop computer, a digital sound interface, earphones, a display, and an EEG amplifier. The instructions and BCI stimuli were presented through earphones (Etymotic ER4 microPro; Etymotic Research, Elk Grove Village, IL). The sounds were processed with an external sound card (RME Fireface UC; Audio AG, Haimhausen, Germany). All stimuli were controlled via the laptop computer using MATLAB/Simulink (Mathworks Inc., Novi, MI) and the Psychophysics Toolbox. Using g.USBamp (Guger Technologies, Graz, Austria), EEG signals were recorded from C3, Cz, C4, P3, Pz, P4, O1, and O2, according to the “10–20” system. The sampling rate for the EEG recording was 128 Hz. All channels were referenced to the left mastoid and grounded to the right mastoid. These electrode locations are based on past P300 EEG studies (Comerchero and Polich,
Offline classification accuracies for positive and negative stimuli were computed using 10-fold cross-validation. ERPs were obtained when auditory stimuli were presented. This data was processed separately in target and non-target trials. When positive affective stimuli were used, a binary classifier was trained on the positive target (PA sounds) trials and non-target (NA and beep sounds) trials. In this case, 9 sessions of data were used for training while the rest was used for test data in the cross-validation. Thus, a classifier was trained on 90 target ERPs and 360 non-target ERPs. The test data containing positive target trials and non-target trials was then classified. Similarly, when negative affective stimuli were used, a binary classifier was trained on the negative target (NA sounds) trials and non-target (PA and beep sounds) trials, and then the test data containing negative target trials and non-target trials was classified. This analysis was applied to the data obtained from parts A and B. In this manner, the classification accuracies of PA, NA, permuted-PA, and permuted-NA were calculated.
In the offline classification, 700 ms epochs of EEG were extracted, smoothed (4 sample points), bandpass-filtered (Butterworth, 0.1–25 Hz), downsampled to 25 Hz, and vectorized. Stepwise linear discriminant analysis (SWLDA;
where
We also analyzed the offline classification accuracy for each part, meaning the accuracy for part A (PA + NA) and part B (permuted-PA + permuted-NA). In this case, the SWLDA classifier was trained on target ERPs and non-target ERPs without discriminating between PA and NA (also permuted-PA and permuted-NA). The parameters used in this analysis were the same as those used in the offline analysis for PA, permuted-PA, NA, and permuted-NA.
For data visualization, averaged waveforms were preprocessed in the same manner, except for changes in artifact removal and downsampling. Waveforms with artifacts exceeding ±100 μV were removed. In order to analyze the ERPs, the waveforms were not downsampled.
In order to clarify the differences between target and non-target ERPs at each time point and in each channel, squared point-biserial correlation coefficients (
where
In order to measure positive or negative affective feelings, all participants were asked how much they felt each stimulus was positive or negative using a VAS after the experiments. The VAS scores ranged from –100 (most negative) to +100 (most positive), where 0 indicates neutral. PA, permuted-PA, NA, and permuted-NA were played once in pseudo-random order for each participant.
Differences between classification accuracies and between VAS scores was assessed by means of a two-way repeated-measures ANOVA with factor permutation (the original sound or its permuted sound) and types of affect (positive or negative). We then performed a
A male patient with ALS aged 61 y.o. participated in this study. His ALS Functional Rating Scale-Revised (ALSFRS-R) (Cedarbaum et al.,
The online experiment consisted of five training sessions and five test sessions. The participant was asked to rest between sessions. Each session contained two runs, meaning the participant answered two questions per session. In the online experiment, PA, NA, and beep sounds were provided as well as in part A of the previous experiment. The participant was asked to silently count PA stimuli to answer “Yes” and NA stimuli to answer “No.” Each stimulus was provided for 10 sequences, meaning the participant had to count the target sound 10 times per run. A feedback sound was provided in the online experiment.
The online BCI system consisted of a laptop computer, digital sound interface, speaker, and EEG amplifier. The instructions and BCI stimuli were presented through the speaker (SoundLink Mini II, Bose Inc., Framingham MA). EEG signals were recorded from C3, Cz, C4, P3, Pz, P4, O1, and O2, according to the “10–20” system. The sampling rate for the EEG recording was 128 Hz. All channels were referenced at Fpz and grounded at AFz. Non-adhesive solid-gel electrodes were used (Toyama et al.,
We employed a SWLDA classifier using transfer learning as both online and offline classifiers. The data from 10 healthy participants obtained in experiment 1 (subjects 1–10) was employed to estimate the classifier for the new subject. The analysis window was 1,000 ms. Prior to training the classifier, ERP data that exceed ±100 μV was eliminated from the healthy participant data and the patient training data. Additionally, a Savitzky–Golay filter (3rd order, 61 samples) was applied and the EEG signal was downsampled to 26 Hz. First, the stepwise method was applied to the data obtained from the 10 healthy participants. The training data was not divided into positive and negative, so the training labels were only target and non-target ERPs. Second, the healthy participant data and patient training data were preprocessed using the stepwise method (
Binomial testing was applied to the classification accuracy, in order to verify that the achieved accuracy was significantly higher than the chance level (50%). The offline analysis was also performed to identify the required number of sequences.
An auditory P300 BCI with positive and negative affective sounds (PA: a meowing cat sound, NA: a screaming cat sound) was tested on 15 healthy participants. Permuted stimuli of the positive and negative affective sounds (permuted-PA, permuted-NA) were also used for comparison. Figure
Classification accuracies and VAS scores.
We used a VAS to measure positive and negative affective feelings for each participant. Figure
Physiological data analyses were applied to the ERPs. Figure
Averaged waveforms and biserial correlation coefficients (
Figure
Figure
Online and offline classification accuracies achieved by an ALS patient. The online and offline classification accuracies were calculated under same classifier with same data. The online classification accuracy was 90%, which is the same as the offline classification accuracy at sequence 10. Offline classification accuracy was computed by varying the number of sequences from 1 to 10. Also 70% classification accuracy is indicated by the horizontal dotted line.
We applied positive and negative affective sounds (PA: a meowing cat sound, NA: a screaming cat sound) with a P300 BCI. Permuted stimuli of the positive and negative affective sounds (permuted-PA, permuted-NA) were also used for comparison. A VAS was used to measure positive and negative affective feelings. We showed that a positive affective stimulus, with accompanying positive affective feelings, improved BCI accuracy. We also demonstrated that the proposed BCI was applicable for an ALS patient.
Our study revealed that the PA stimulus improved the offline classification accuracy of an auditory BCI. A previous visual BCI study found that affective facial images improved the classification accuracy of a BCI (Zhao et al.,
We demonstrated the significant differences between the classification accuracies of PA and permuted-PA, and between the VAS scores of PA and permuted PA. In contrast, when applying negative affective sounds, although both NA and permuted-NA showed decreased VAS scores, no significant differences were observed between them in either classification accuracy or VAS. This lack of change between NA and permuted-NA VAS scores may be caused by the scrambling. When scrambling the stimuli, the sounds were cut every 10 ms and the cut sounds were concatenated in pseudo-random order. This procedure removes continuous and frequency features below 100 Hz, but retains frequency features above 100 Hz. The meowing cat sound, used as the PA stimulus, showed decreased VAS scores in the permuted stimulus, suggesting that the features of the cat meowing sound disappeared. The screaming cat sound, used as the NA stimulus, did not show changed VAS scores in the permuted stimulus, suggesting that the features of the screaming cat sound remained. Although past psychological studies using affective stimuli have used two-dimensional evaluations of arousal and valence (Bradley and Lang,
In order to clarify which components of ERPs contributed to classification, we computed the squared point-biserial correlation coefficients (
This study revealed high biserial correlation coefficients for the late component of the P300 in response to auditory affective stimuli. Modulation of the late component of the P300 has been reported in past studies by using visual or auditory affective stimuli. Cuthbert et al. (
In the online experiment, an ALS patient achieved 90% classification accuracy. Our BCI is a totally vision-free system; all questions, stimuli, and feedback were provided only from the speaker. Thus, this system may be worth applying to patients who cannot see. The arbitrary yes/no questions can be provided orally by replacing the questions provided from the speaker. Our BCI system can only ask yes/no binary questions, but the affective sounds may be applied to auditory multiple-choice BCIs. Auditory multiple-choice BCIs have previously been proposed and evaluated. Halder et al. (
This study demonstrated that positive affective stimuli improve classification accuracy. However, further studies are required to determine if affective stimuli generally improve BCI classification accuracy. Moreover, classification accuracy for each part did not show the significant difference between part A and part B. The result may be caused by the increased variance of the two class data since responses obtained by positive and negative stimuli were combined. The effect of affect in part B may also influenced because permuted-NA was negative. Additionally, we applied transfer learning in the online system, but the effects of the algorithm should be evaluated and the parameters should be optimized in future studies. The affective stimuli evaluated in this study may also be applied to a multi-command BCI as a next step.
In conclusion, we demonstrated that a positive affective stimulus, accompanied by positive affective feelings, improved BCI accuracy. These results suggest that affective stimuli may be useful in developing a practical auditory BCI system for patients with physical disabilities.
AO, HO, and KK designed the experiment. AO and KT collected the data. AO, HO, and TK analyzed the data. AO, KT, TK, and KK wrote the manuscript.
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
This study was supported in part by a MHLW/AMED grant (BMI), a MEXT/AMED-SRPBS grant (BMI), and JSPS grants (15H03126, 15H05880, 16K13113, 16H05583, 16K16477). We thank Y. Nakajima and M. Suwa for their encouragement.
The Supplementary Material for this article can be found online at: