Abstract
Near-infrared spectroscopy (NIRS) is a promising research tool that found its way into the field of brain-computer interfacing (BCI). BCI is crucially dependent on maximized usability thus demanding lightweight, compact, and low-cost hardware. We designed, built, and validated a hybrid BCI system incorporating one optical and two electrical modalities ameliorating usability issues. The novel hardware consisted of a NIRS device integrated with an electroencephalography (EEG) system that used two different types of electrodes: Regular gelled gold disk electrodes and tri-polar concentric ring electrodes (TCRE). BCI experiments with 16 volunteers implemented a two-dimensional motor imagery paradigm in off- and online sessions. Various non-canonical signal processing methods were used to extract and classify useful features from EEG, tEEG (EEG through TCRE electrodes), and NIRS. Our analysis demonstrated evidence of improvement in classification accuracy when using the TCRE electrodes compared to disk electrodes and the NIRS system. Based on our synchronous hybrid recording system, we could show that the combination of NIRS-EEG-tEEG performed significantly better than either single modality only.
1. Introduction
Brain-computer interfacing (BCI) is a technology first mentioned in the 1970s (Vidal, 1973). It is usually thought to employ non-invasive brain monitoring techniques to recklessly collect brain activity data () and aims to provide people in need with a new communication method (). It may form a bridge between brains and the outside world bypassing the classical communication channels () with the help of advancements in technology (Rashid et al., 2020), such as software (), signal processing (; Wierzgała et al., 2018) as well as enhanced sensors ().
Both sophisticated hardware and software are needed to create a BCI system as it is supposed to capture seemingly noisy brain activity and translate it into an intended action (). The software part mandates signal preprocessing [e.g., filter motion artifacts ()], feature extraction [e.g., matching pursuit ()], feature selection [e.g., least absolute shrinkage and selection operator (Tay et al., 2021)], and classification (Spezialetti et al., 2018). Nowadays machine learning approaches may be used to that end as well ().
Utilizing more than one non-invasive brain monitoring technique on the same human scalp at the same time asks for so-called hybrid devices and is a relatively new direction to enhance BCI performance (; ; ; Shin et al., 2018). To this end, we employed a commercially available EEG amplifier and signals from a special electrode, the tri-polar concentric ring electrode (TCRE) (). We integrated the EEG system with a novel near-infrared spectroscopy (NIRS) system () to conduct 2D BCI experiments with a group of volunteers. Although NIRS systems are available commercially, we were inspired with the constant efforts to create new EEG systems and electrodes with enhanced performance and lower cost and decided to create a low cost portable system in an effort to make a portable dual modality more affordable. That way we were able to quantify the contribution of different modalities and their combinations.
2. Materials and methods
2.1. Data acquisition systems
The novel NIRS system () (see Supplementary Figure 1) feature an Atmel Microcontroller board (ATmega1280/V, USA) to control detector optodes and synchronize the NIR LED sources’ off/on period. It sent the collected data to a laptop at a 22 Hz sampling frequency and controls constant current sources to supply the optodes with a stable current. MATLAB (The MathWorks MATLAB R2022a) scripts in the connected laptop were used to achieve signal preprocessing, filtering, and analysis. The NIRS system casing (which houses the system electronics), sensor casings (which house the detectors’ preamplifier circuits), and the sensor holder rings (which were attached to the cap) were designed with Solid Edge software (Siemens Industry Software GmbH, Germany) and 3D printed with a simple 3D printer (Replicator2, MakerBot Industries, USA) using PLA filament (see Supplementary Figure 1).
Near-infrared spectroscopy optodes (sources/detectors, see Figure 1) feature eight NIR detectors (BPW34-B, OSRAM Opto Semiconductors GmbH, Germany). These photodiodes (PD) had a spectral range of sensitivity between 350–1100 nm, a radiant sensitive area of 7.45 mm2, and a 25 ns short switching time. Each detector is attached to a spring to gently press it to the scalp’s skin (see Figure 1).
FIGURE 1
Our NIRS optodes also have three NIRS dual sources, comprised of three 850 nm photodiodes (TSHG6400, Vishay Semiconductors, USA) and three 770 nm photodiodes (MTE1077N1-R, Marktech Optoelectronics, USA). Arranging NIRS sources and detectors in a commercial cap (NIRScap, NIRx Medical Technologies, LLC) forms ten NIRS channels (see Figure 2 for the montage used during experiments, more channels can be created with different alignments). Optode holder rings are arranged on the cap to separate the source from the detector by 3 cm (SDS) to allow light penetration into the cortical region of interest (see Figure 2). The complete NIRS system is versatile, small, and portable (size = 11 cm × 8.5 cm × 3.7 cm) weighing about ∼160 g (including sensors ∼ 390 g). Material price for the system sums to about €1,470.
FIGURE 2
Two-disk electrodes and eight Tri-Polar Concentric Ring Electrodes (TCRE, CREmedical, USA) connected to a commercial g.USB amplifier (g.tec GmbH, Schiedlberg, Austria) are used to collect scalp potential signals. The TCRE is thought to have a higher local sensitivity and spatial resolution as compared to other electrodes (; ). Signals measured by the TCRE outer ring provide an emulated EEG (eEEG) equivalent to a regular disk EEG electrode (; Zhu et al., 2014). Hence in this paper, the signal extracted from all three rings (which is processed to a single signal) is denominated tEEG while the signal collected from the TCRE outer ring falls in the category EEG. This peculiarity enables us to compare the performance of both data types during BCI experiments. To enhance EEG data spatial resolution, the scalp surface Laplacian is employed. The second spatial derivative of the surface Laplacian was shown to improve the high spatial frequency components of the brain activity near the electrode (). Besio et al. (; ) used a special sensor namely the TCRE electrode (; ) to measure the Laplacian. The signal measured by this sensor is calculated by: [16 * (Vm − Vd) − (Vo − Vd)] where Vm, Vd, Vo are the voltage of the middle ring, central disc, and outer ring of the TCRE electrode, respectively. The EEG signal collected by this electrode is called tEEG by Besio et al. (; ). When compared to disc electrode signals; tEEG signals’ spatial resolution, signal-to-noise ratio, and mutual information are improved by approximately three, four, and twelve times, respectively (). The main disadvantage of TCRE electrodes is their large surface area (diameter = 1.6 cm) compared to the typical commercial EEG electrodes (diameter = 1 cm) ().
The main disadvantage of TCRE electrodes is their large surface area (diameter = 1.6 cm) compared to the typical commercial EEG electrodes (diameter = 1 cm) ().
TCRE electrodes are mounted on the cap by 3D-printed holding rings following the international 10–20 system. Electrodes are located at C1, C2, C3, C4, Cz, FC1, FC2, and FCz. A disk electrode is located on the right mastoid and another on the left mastoid. The disk electrodes serve as the reference and ground contact, respectively. The sampling frequency of the measured signals is set to 256 Hz; the signals are notch filtered at 60 Hz frequency (because the experiments were done in the United States as the mains frequency is 60 Hz) and band-pass filtered between 0.1 and 70 Hz. The layout map for the EEG/tEEG electrodes and NIRS optodes covers the motor cortex area of the brain (see Figure 2).
2.2. Experimental paradigms
A motor imagery (MI) BCI paradigm was used during the experiments. This paradigm is employed by many researchers (; Wierzgała et al., 2018; ) and means that the subject will imagine moving his/her hands/feet rather than actually moving them. During the experiments, we input tEEG and EEG data into BCI2000 software (Schalk et al., 2004). The software provided the experiment’s cue (on a 23.8 inch LCD screen 1.5 m from the subject) and showed the subject’s feedback on their performance during online sessions.
The subjects performed the experiments in a dimly lit lab while comfortably seated. To minimize motion artifacts, they were asked to avoid moving. Sixteen subjects participated in the experiments (aged 20–36 years). They all signed an informed consent form before participating. The University of Rhode Island Institutional Review Board (IRB) approved the protocol of the experiments.
The experiments consisted of one offline and one online session. The offline session had five runs; each run consisted of twenty trials. The subjects were asked to imagine one of four MI tasks during each trial corresponding to the cue shown on the screen in front of them. The first task is preceded by a 10-s baseline period. Each task lasts for 10 s and is followed by 10 s of rest. The cue displayed was either an arrow pointing up (imagine moving both hands, Both-hands MI), an arrow pointing down (imagine moving both feet, Both-feet MI), an arrow pointing left (imagine moving left hand, Left-hand MI), an arrow pointing right (imagine moving right hand, Right-hand MI), or a blank screen to signal rest (see Figure 3).
FIGURE 3
The offline session resulted in 100 trials, with 25 trials of each task type. The offline session tasks have fixed and identical task periods, were repeated periodically and the subjects received no feedback about their performance during the task. The session’s purpose is to provide data to train the online classifier. This session was followed by an offline analysis during which the BCI2000 software classifiers were adjusted according to the subject’s performance and prepared for the online session.
The online session consisted of five runs with 20 trials each. During these trials, (100 trials per subject) tasks are shown in random order. The task cue given is a red square target appearing at one edge of the screen. A red ball would appear consecutively, and the subject would complete the task by hitting the target based on imagined movements as before. The color of the target and the ball will turn green if the subject accomplishes the task. The task cue starts with 2 s baseline recording followed by 1 s during which the target appears followed by the ball for a period between 2 and 15 s (see Figure 4). This ball’s presentation period depends on the subject’s performance: A successful hit terminates the task otherwise it would last for 15 s. The task is followed by a 1-s feedback displaying either success (green target) or failure (red target). A 15-s rest period (blank screen) completes the trial. The subjects are advised to employ the MI tasks exercised during the offline session. They imagined moving their hands/feet to move the ball up/down and so on.
FIGURE 4
3. Results and discussion
3.1. EEG/tEEG data
Collected data were preprocessed to improve the signal-to-noise ratio (SNR). For EEG/tEEG data, we used EEGlab software () and bandpass filtered the data between 1 and 48 Hz with a finite impulse response (FIR) filter to minimize the main power crosstalk. We also employed this software to reduce the motion artifacts as well as EOG, and EMG components. We used a MATLAB script to apply a notch FIR filter to reduce interference with our NIRS system.
Data from offline sessions for all subjects were analyzed using EEGlab software to plot event-related potential (ERP) and the power spectral density (PSD). Grand average plots are shown in Figure 5 (ERP grand average) and Figure 6 (PSD grand average). Each task’s grand average is calculated for 400 trials (16 subjects × 5 runs per offline session × 5 task trials per run).
FIGURE 5
FIGURE 6
Figure 5 shows the ERP grand averages for EEG data collected from electrodes C1, C2, C3, C4, and Cz. The plots are limited between −1 and 5 s to improve visibility (the original tasks were 10 s each) since no other peaks were seen in the remaining time. In Figures 5A, B, the Right-hand MI plot has a higher potential at around 0.3 s compared to other tasks. This is in agreement with the literature (; ) as the electrodes C1 and C3 monitor the brain region which controls the right-hand movement. In Figures 5C, D, the Left-hand MI plot has a higher potential at around 0.3 s compared to other tasks. This is in agreement with the literature (Romero et al., 2000; ) as the electrodes C2 and C4 monitor the brain region which controls the left-hand movement. In Figure 5E, the Both-feet MI plot has a potential of about 4.2 μV at around 0.3 s. This potential value although not the highest ERP potential in Figure 5E, has the highest Both-feet MI potential value in Figure 5. This is in agreement with the literature as the electrode Cz monitors the brain region which controls the feet movements. The lower potential compared to other electrodes is due to the special location of this region within the brain’s central sulcus ().
In Figure 6, which displays the plots for the PSD grand averages, the Rest plot around 9 Hz shows the highest power. Right-hand MI has the lowest power around 9 Hz in data from electrode C1. Left-hand MI has the lowest power around 9 Hz in data from electrodes C2 and C4. PSD at the specific brain region of interest is expected to be lower while performing the MI task compared to the rest period.
3.2. NIRS data
Data from our NIRS channels was first preprocessed and analyzed by MATLAB scripts. Data was first bandpass filtered between 0.01 and 0.1 Hz by an FIR bandpass to reduce the effect of Mayer waves (Yücel et al., 2016). Data were then smoothed by a 7-point moving average filter. Data from the 10 NIRS channels resulting from the combination of the NIRS sources and detectors (see Figure 2) was collected for the 16 subjects with a total of 400 trials for each task (16 subjects × 5 runs per offline session × 5 tasks per trial run). Only HbO2 data were used during NIRS data analysis.
In Figure 7A, NIRS data from one subject (subject 10) collected from channel 10 was plotted for 25 trials of Both hand MI task performed during the offline session, the mean across these trials was also plotted (the black line). Channel 10 monitors the brain area which controls the left hand (see Figure 2). The mean data showed an increase in value with the start of the Both-hands MI task. The increase reached its maximum value around 5 s and dropped to 0 at 10 s. The mean data for this subject (subject 10) and the other 15 subjects who participated in the offline sessions were plotted for the same task and the same channel in Figure 7B the grand average (GA) was also plotted (the black line, the peak value around 5 s). The grand average shows an increase during the task which drops in value before the end of the 10 s.
FIGURE 7
Grand averages for all the MI tasks from all the NIRS channels are shown in Figure 8. Channels 1–4 roughly monitor the brain region controlling right-hand movements, channels 5 and 6 feet movements, and channels 7–10 left-hand movements (see Figure 2). Plots of HbO2 concentration during the Right-hand MI show an increase in value from channels 1–3 while during Both-hands MI the values channels 1, 2, 4, 7–10 increase.
FIGURE 8
HbO2 concentration during the Left-hand MI increases in channels 7–10 while during the Both-feet MI the increase is visible in channel 5. The results from NIRS data as well as the results from the EEG data (ERP and PSD) are consistent with the literature (Shin et al., 2018). It shows that the physiological changes associated with brain activity at specific regions are successfully monitored by our NIRS device and hence it can be combined with tEEG for BCI purposes.
3.3. Classification accuracy (offline session)
Features were extracted from the offline session to compare the accuracy across the data-acquiring methods as well as the optimum combination for a BCI application. To extract features, a sliding window of 5 s width and 1-s step size was employed. A matching pursuit (MP) algorithm () was directly applied to EEG/tEEG data to extract seven features. To extract five more features fifteen FIR bandpass filters were used to narrow band filter the signals between 5 and 35 Hz (). Each filter has a window of 2 Hz and hence the first filtered from 5 to 7 Hz while the fifteenth filter covered 33–35 Hz. To extract features from these narrow band signals we used signal power, Teager-Kaiser Energy (TKE) operator (), multiclass CSP information theoretic feature extraction (ITFE) (), PSD mean, and amplitude mean.
To extract features by MP from EEG/tEEG data, we prepared a twenty-five Gabor atoms dictionary (see Supplementary Figure 2A). The data within the feature extraction window (5 s window) were compared with a Gabor atoms dictionary. When a match between Gabor atoms and data is found, we calculated the statistical features of matching atoms. Extracted features were mean atom frequency, mean atom amplitude, mean atom phase, mean atom scale, variance of the atom phase, maximum atom amplitude, and the maximum scalar product of each atom with the data within the window.
Seven features were extracted from NIRS measurements: two features by MP, the five other features by ITFE, HbO2 amplitude variance, HbO2 amplitude mean, HbO2 slope mean, HbO2 slope mean second derivative. To extract MP features, we created thirty atoms Gabor dictionary (see Supplementary Figure 2B). The extracted MP features are the mean atom amplitude and the maximum scalar product of each atom with the data within the window.
Extracted features were either fed directly to an ensemble classifier (; ) or the feature size was reduced by applying the feature selection method Least Absolute Shrinkage and Selection Operator (LASSO) (Tibshirani, 1996; Tay et al., 2021).
Figure 9A displays accuracy calculations resulting from the initial use of LASSO, while Figure 9B shows the accuracy calculated employing an ensemble classifier only. Accuracies above 85% (median around 88%) were achieved when combining all three data sets (EEG + tEEG + NIRS), while accuracies above 83% (median around 87%) were achieved when combining tEEG + NIRS. We used the Wilcoxon rank-sum test to compare the accuracy obtained by different data collection methods namely EEG, tEEG, and NIRS as well as their combinations. Combining NIRS with tEEG or EEG (dual modality) performed significantly better than any single modality as shown in Figures 9A, B. No significant difference was found between NIRS-EEG and NIRS-tEEG. In Figure 9Ap-values of 8 × 10–4 were achieved when EEG was compared with NIRS and 5.6 × 10–4 when compared with tEEG. Feature size reduction did not improve the accuracy of comparisons in Figures 9A, B.
FIGURE 9
3.4. Online session
Before commencing the online session for any subject, tEEG data was analyzed with the offline analysis package of the BCI2000 software (Schalk et al., 2004). The analysis showed the active channels during each task. We used the analysis results to determine classifiers for the subsequent online session. During the online sessions, signals were collected with the 10 NIRS channels and eight tEEG channels. Only twelve of the original sixteen subjects participated in the online sessions. Feature extraction of online signals was limited to the identified active channels specified during the offline analysis. Two classifiers were employed, right vs. left and up vs. down.
Twelve features were extracted from tEEG data and seven from the NIRS measurements (with the same methods used for the offline data, see above). Deviating from offline sessions, we used the common spatial pattern (CSP) (Sannelli et al., 2010) instead of ITFE which may be interpreted as a multi-class CSP (; ). As mentioned, we only required a two-class classification for the online sessions. A further technical difference between off- and online sessions was the lack of 8 EEG channels for the online montage while the offline one featured 8 EEG and 8 tEEG channels.
Figure 10A depicts the mean classification accuracies per run for NIRS, tEEG, and NIRS + tEEG data collected from subject 5 during the online session. Figure 10B illustrates boxplots for all five online runs of subject 5. In both graphs, we can see that the combined NIRS + tEEG gave better accuracy compared to a single modality. We can also see that the accuracy of the results with the novel NIRS system is within an acceptable range and their addition to tEEG data gave a significant improvement to the overall accuracy.
FIGURE 10
Figure 11 illustrates the mean accuracies of the 5 online runs during the online session. For all subjects (n = 12), the combined NIRS + tEEG gave overall higher accuracy compared to a single modality. The performances of subjects displayed were quite variable. tEEG data accuracies were higher for some subjects while NIRS measurements showed higher accuracies for others. This could be explained by the different characteristics of each measurement. NIRS is an optical measurement; it is more susceptible to ambient light, skin, and hair color. tEEG is more susceptible to motion artifacts, skin thickness and perspiration.
FIGURE 11
4. Conclusion
Near-infrared spectroscopy application in BCI is relatively new compared to the well-known EEG use. We developed a dual-modal BCI system with two non-invasive modalities at a moderate cost compared to other systems (Scholkmann et al., 2014).
Results from an offline BCI session with 16 subjects showed that their performance fell within a comparable range. It corroborated the novel NIRS system’s capability to efficiently monitor subjects’ brain activity and to assess movement intents. It also revealed differences in accuracy between measurement modalities and their combinations. NIRS performed significantly better when combined with tEEG (vs. EEG). Our decision to utilize the TCRE electrode tEEG to enhance BCI performance thus proved to be correct. Lastly, the offline session allowed us to adjust and individualize classifiers for each subject according to the subject’s performance for the following online session.
Results from the actual BCI experiments, the online session, revealed strong inter-subject differences in the classification accuracies of tEEG and NIRS data. This may be due to different skin/hair colors, skin thickness, as well as other possible physiological and anatomical factors. The general performance shows strong variances between subjects with some performing better due to an inherent capability or higher ability to focus during the sessions or both. However, our NIRS system compared very well with the usual electrical BCI modalities, and it showed even particularly promising results when combined into a hybrid NIRS + tEEG BCI system for all the subjects.
Although the NIRS system was used to monitor the motor areas of the brain, there is no reason why it can’t be efficiently employed to monitor other brain regions of interest as well. The flexibility of montages with optodes arbitrarily fixed on any EEG cap supports further studies. The number of channels may be increased by changing the source-detector alignment map and even add more inexpensive sources or detectors to yield higher channel counts. Our modular system is affordable, portable, lightweight, and can be employed efficiently with other modalities to monitor brain activity at different regions of interest.
Statements
Data availability statement
The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.
Ethics statement
The studies involving human participants were reviewed and approved by the Rhode Island IDeA Network for Excellence in Biomedical Research (RI-INBRE). The patients/participants provided their written informed consent to participate in this study.
Author contributions
RA: conceptualization, visualization, electronics design and assembly, methodology, software, formal analysis, investigation, writing – original draft, and editing. SM: electronics design, review, and editing. AA: review and editing. YB: investigation, review, and editing. KM: funding acquisition, project administration, review, and editing. WB: resources, review, and editing. UH: conceptualization, supervision, project administration, funding acquisition, review, and editing. All authors contributed to the article and approved the submitted version.
Funding
This research was partially funded by a DAAD stipend to RA and Rhode Island IDeA Network for Excellence in Biomedical Research (RI-INBRE).
Conflict of interest
SM was currently employed by Roche Diagnostics Automation Solutions GmbH (the employment was after finishing the research). The remaining authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
Publisher’s note
All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.
Supplementary material
The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fnhum.2023.1162712/full#supplementary-material
References
1
AggarwalS.ChughN. (2019). Signal processing techniques for motor imagery brain computer interface: A review.Array 12:100003. 10.1016/j.array.2019.100003
2
AlmajidyR. K.KirchR. D.ChristO.HofmannU. G. (2014). “Estimating the spatial resolution of fNIRS sensors for BCI purposes,” in Design and performance validation of phantoms used in conjunction with optical measurement of tissue VI, edsNordstromR. J.BouchardJ.-P.AllenD. W. (Bellingham, WA: SPIE), 894504. 10.1117/12.2037351
3
AlmajidyR. K.MankodiyaK.AbtahiM.HofmannU. G. (2020). A newcomer’s guide to functional near infrared spectroscopy experiments.IEEE Rev. Biomed. Eng.13292–308. 10.1109/RBME.2019.2944351
4
BabiloniF.BabiloniC.FattoriniL.CarducciF.OnoratiP.UrbanoA. (1995). Performances of surface Laplacian estimators: A study of simulated and real scalp potential distributions.Brain Topogr.835–45. 10.1007/BF01187668
5
BesioW. G.CaoH.ZhouP. (2008). Application of tripolar concentric electrodes and prefeature selection algorithm for brain-computer interface.IEEE Trans. Neural Syst. Rehabil. Eng.16191–194. 10.1109/TNSRE.2007.916303
6
BesioW. G.KokaK.AakulaR.DaiW. (2006). Tri-polar concentric ring electrode development for laplacian electroencephalography.IEEE Trans. Biomed. Eng.53926–933. 10.1109/TBME.2005.863887
7
BesioW. G.LiuX.WangL.MedvedevA. V.KokaK. (2011). Transcutaneous focal electrical stimulation via concentric ring electrodes reduces synchrony induced by pentylenetetrazole in beta and gamma bands in rats.Int. J. Neural Syst.21139–149. 10.1142/S0129065711002729
8
BesioW.AakulaR.KokaK.DaiW. (2006). Development of a tri-polar concentric ring electrode for acquiring accurate Laplacian body surface potentials.Ann. Biomed. Eng.34426–435. 10.1007/s10439-005-9054-8
9
ChoiI.RhiuI.LeeY.YunM. H.NamC. S. (2017). A systematic review of hybrid brain-computer interfaces: Taxonomy and usability perspectives.PLoS One12e0176674. 10.1371/journal.pone.0176674
10
DelormeA.MakeigS. (2004). EEGLAB: An open source toolbox for analysis of single-trial EEG dynamics including independent component analysis.J. Neurosci. Methods1349–21. 10.1016/j.jneumeth.2003.10.009
11
DietterichT. G. (2000). “Ensemble methods in machine learning,” in Multiple classifier systems (Berlin: Springer Berlin Heidelberg), 1–15. 10.1007/3-540-45014-9_1
12
FazliS.MehnertJ.SteinbrinkJ.CurioG.VillringerA.MüllerK.-R.et al (2012). Enhanced performance by a hybrid NIRS-EEG brain computer interface.Neuroimage59519–529. 10.1016/j.neuroimage.2011.07.084
13
Grosse-WentrupM.BussM. (2008). Multiclass common spatial patterns and information theoretic feature extraction.IEEE Trans. Biomed. Eng.551991–2000. 10.1109/TBME.2008.921154
14
HongK.-S.KhanM. J.HongM. J. (2018). Feature extraction and classification methods for hybrid fNIRS-EEG brain-computer interfaces.Front. Hum. Neurosci.12:246. 10.3389/fnhum.2018.00246
15
KaiserJ. F. (1990). “On a simple algorithm to calculate the “energy” of a signal,” in Proceedings of the International conference on acoustics, speech, and signal processing, (Albuquerque, NM: IEEE), 381–384. 10.1109/ICASSP.1990.115702
16
KayaM.BinliM. K.OzbayE.YanarH.MishchenkoY. (2018). A large electroencephalographic motor imagery dataset for electroencephalographic brain computer interfaces.Sci. Data5:180211. 10.1038/sdata.2018.211
17
KhanM. A.DasR.IversenH. K.PuthusserypadyS. (2020). Review on motor imagery based BCI systems for upper limb post-stroke neurorehabilitation: From designing to application.Comput. Biol. Med.123:103843. 10.1016/j.compbiomed.2020.103843
18
KokaK.BesioW. G. (2007). Improvement of spatial selectivity and decrease of mutual information of tri-polar concentric ring electrodes.J. Neurosci. Methods165216–222. 10.1016/j.jneumeth.2007.06.007
19
KooB.LeeH.-G.NamY.KangH.KohC. S.ShinH.-C.et al (2015). A hybrid NIRS-EEG system for self-paced brain computer interface with online motor imagery.J. Neurosci. Methods24426–32. 10.1016/j.jneumeth.2014.04.016
20
LotteF.GuanC. (2011). Regularizing common spatial patterns to improve BCI designs: Unified theory and new algorithms.IEEE Trans. Biomed. Eng.58355–362. 10.1109/TBME.2010.2082539
21
MaclinR.OpitzD. (2011). Popular ensemble methods: An empirical study.J. Artif. Intell. Res.11169–198. 10.1613/jair.614
22
MakeyevO.BoudriaY.ZhuZ.LennonT.BesioW. (2013). “Emulating conventional disc electrode with the outer ring of the tripolar concentric ring electrode in phantom and human electroencephalogram data,” in Proceedings of the IEEE signal processing in medicine and biology symposium, Brooklyn, NY. 10.1109/SPMB.2013.6736778
23
MallatS. G.ZhangZ. (1993). Matching pursuits with time-frequency dictionaries.IEEE Trans. Signal. Process.413397–3415. 10.1109/78.258082
24
MartiniM. L.OermannE. K.OpieN. L.PanovF.OxleyT.YaegerK. (2020). Sensor modalities for brain-computer interface technology: A comprehensive literature review.Neurosurgery86E108–E117. 10.1093/neuros/nyz286
25
McAvinueL. P.RobertsonI. H. (2008). Measuring motor imagery ability: A review.Eur. J. Cogn. Psychol.20232–251. 10.1080/09541440701394624
26
MinB.-K.MarzelliM. J.YooS.-S. (2010). Neuroimaging-based approaches in the brain-computer interface.Trends Biotechnol.28552–560. 10.1016/j.tibtech.2010.08.002
27
NeuperC.PfurtschellerG. (2010). “Electroencephalographic characteristics during motor imagery,” in The neurophysiological foundations of mental and motor imagery, edsGuillotA.ColletC. (Oxford: Oxford University Press), 65–82. 10.1093/acprof:oso/9780199546251.003.0005
28
OllikainenJ. O.VauhkonenM.KarjalainenP. A.KaipioJ. P. (2000). Effects of electrode properties on EEG measurements and a related inverse problem.Med. Eng. Phys.22535–545. 10.1016/S1350-4533(00)00070-9
29
PfurtschellerG.AllisonB. Z.BrunnerC.BauernfeindG.Solis-EscalanteT.SchererR.et al (2010). The hybrid BCI.Front. Neurosci.4:30. 10.3389/fnpro.2010.00003
30
RamadanR. A.VasilakosA. V. (2017). Brain computer interface: Control signals review.Neurocomputing22326–44. 10.1016/j.neucom.2016.10.024
31
RaoR. P. N.SchererR. (2013). Brain computer interfacing.J. Comp. Sci. Eng.7132–138. 10.5626/JCSE.2013.7.2.132
32
RasheedS. (2021). A review of the role of machine learning techniques towards brain–computer interface applications.MAKE3835–862. 10.3390/make3040042
33
RashidM.SulaimanN.Abdul MajeedP. P.MusaR. M.Ab NasirA. F.BariB. S.et al (2020). Current status, challenges, and possible solutions of EEG-based brain-computer interface: A comprehensive review.Front. Neurorobot.14:25. 10.3389/fnbot.2020.00025
34
RomeroD. H.LacourseM. G.LawrenceK. E.SchandlerS.CohenM. J. (2000). Event-related potentials as a function of movement parameter variations during motor imagery and isometric action.Behav. Brain Res.11783–96. 10.1016/S0166-4328(00)00297-7
35
SannelliC.VidaurreC.MullerK.-R.BlankertzB. (2010). Common spatial pattern patches - an optimized filter ensemble for adaptive brain-computer interfaces.Conf. Proc. IEEE Eng. Med. Biol. Soc.20104351–4354. 10.1109/IEMBS.2010.5626227
36
SchalkG.McFarlandD. J.HinterbergerT.BirbaumerN.WolpawJ. R. (2004). BCI2000: A general-purpose brain-computer interface (BCI) system.IEEE Trans. Biomed. Eng.511034–1043. 10.1109/TBME.2004.827072
37
ScholkmannF.KleiserS.MetzA. J.ZimmermannR.Mata PaviaJ.WolfU.et al (2014). A review on continuous wave functional near-infrared spectroscopy and imaging instrumentation and methodology. Neuroimage85, 6–27. 10.1016/j.neuroimage.2013.05.004
38
ShinJ.KwonJ.ImC.-H. (2018). A ternary hybrid EEG-NIRS brain-computer interface for the classification of brain activation patterns during mental arithmetic, motor imagery, and idle state.Front. Neuroinform.12:5. 10.3389/fninf.2018.00005
39
SpezialettiM.CinqueL.TavaresJ. M. R. S.PlacidiG. (2018). Towards EEG-based BCI driven by emotions for addressing BCI-Illiteracy: A meta-analytic review.Behav. Inf. Technol.371–17. 10.1080/0144929X.2018.1485745
40
TayJ. K.NarasimhanB.HastieT. (2021). Elastic net regularization paths for all generalized linear models.arXiv. [Preprint].
41
TibshiraniR. (1996). Regression shrinkage and selection via the lasso.J. R. Stat. Soc. Series B (Methodological)58267–288. 10.1111/j.2517-6161.1996.tb02080.x
42
VidalJ. J. (1973). Toward direct brain-computer communication.Annu. Rev. Biophys. Bioeng.2157–180. 10.1146/annurev.bb.02.060173.001105
43
WierzgałaP.ZapałaD.WojcikG. M.MasiakJ. (2018). Most popular signal processing methods in motor-imagery BCI: A review and meta-analysis.Front Neuroinform.12:78. 10.3389/fninf.2018.00078
44
YücelM. A.SelbJ.AastedC. M.LinP.-Y.BorsookD.BecerraL.et al (2016). Mayer waves reduce the accuracy of estimated hemodynamic response functions in functional near-infrared spectroscopy.Biomed. Opt. Express73078–3088. 10.1364/BOE.7.003078
45
ZhuZ.BrooksJ.MakeveyO.MemberI.KayS. M.FellowI.et al (2014). “Equivalency between emulated disc electrodes and conventional disc electrode human electroencephalography,” in Proceedings of the engineering in medicine and biology society (EMBC), 36th annual international conference of the IEEE (Chicago, IL: IEEE).
Summary
Keywords
BCI, monitoring brain activity, NIRS system design, multi-modal BCI, EEG, classification, matching pursuit, LASSO
Citation
Almajidy RK, Mottaghi S, Ajwad AA, Boudria Y, Mankodiya K, Besio W and Hofmann UG (2023) A case for hybrid BCIs: combining optical and electrical modalities improves accuracy. Front. Hum. Neurosci. 17:1162712. doi: 10.3389/fnhum.2023.1162712
Received
09 February 2023
Accepted
18 May 2023
Published
07 June 2023
Volume
17 - 2023
Edited by
Jude Hemanth, Karunya Institute of Technology and Sciences, India
Reviewed by
Kishor Lakshminarayanan, Vellore Institute of Technology (VIT), India; Qingshan She, Hangzhou Dianzi University, China
Updates
Copyright
© 2023 Almajidy, Mottaghi, Ajwad, Boudria, Mankodiya, Besio and Hofmann.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
*Correspondence: Ulrich G. Hofmann, ulrich.hofmann@coregen.uni-freiburg.de
Disclaimer
All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article or claim that may be made by its manufacturer is not guaranteed or endorsed by the publisher.