Hybrid Systems to Boost EEG-Based Real-Time Action Decoding in Car Driving Scenarios
Giovanni Vecchiato- Institute of Neuroscience, National Research Council of Italy, Parma, Italy
The complexity of concurrent cerebral processes underlying driving makes such human behavior one of the most studied real-world activities in neuroergonomics. Several attempts have been made to decode, both offline and online, cerebral activity during car driving with the ultimate goal to develop brain-based systems for assistive devices. Electroencephalography (EEG) is the cornerstone of these studies providing the highest temporal resolution to track those cerebral processes underlying overt behavior. Particularly when investigating real-world scenarios as driving, EEG is constrained by factors such as robustness, comfortability, and high data variability affecting the decoding performance. Hence, additional peripheral signals can be combined with EEG for increasing replicability and the overall performance of the brain-based action decoder. In this regard, hybrid systems have been proposed for the detection of braking and steering actions in driving scenarios to improve the predictive power of the single neurophysiological measurement. These recent results represent a proof of concept of the level of technological maturity. They may pave the way for increasing the predictive power of peripheral signals, such as electroculogram (EOG) and electromyography (EMG), collected in real-world scenarios when informed by EEG measurements, even if collected only offline in standard laboratory settings. The promising usability of such hybrid systems should be further investigated in other domains of neuroergonomics.
Introduction
Today's human at work is asked to continuously interact with objects and the environment to perform a wide variety of tasks. In this regard, the research field of neuroergonomics aims to unravel the neural bases of those neurophysiological processes involved in the interaction between the user and a technical system during everyday life activities (Parasuraman, 2003; Dehais et al., 2020; Gramann et al., 2021).
Because of its complexity, one of the main real-world activities targeted by neuroergonomics is driving (Navarro et al., 2018). Studies demonstrated that driving behavior is the final result of simultaneous mental processes such as attention, decision-making, vigilance, motor, and cognitive control (Calhoun et al., 2002; Calhoun and Pearlson, 2012). Driving activity can cause drowsiness, fatigue, and an increase in workload, and it is one of the primary causes of death worldwide (Borghini et al., 2014). In order to assure road safety, it becomes fundamental to have a deep understanding of those mental processes underlying the interactions existing among the driver, the car, and the external environment to predict human behavior resulting in steering and braking actions. In recent years, driving scenarios have been enriched by technological advancements in designing autonomous cars (Badue et al., 2021). However, even if intelligent systems can execute actions on behalf of the driver, the correctness of these choices can only be evaluated once we understand those mechanisms underlying the driver's behavior in simulated and real traffic scenarios. In this regard, expertise could be an essential factor worth considering since evidence collected among professional and non-expert drivers suggest that the two populations share basic neurophysiological mechanisms, whereas the expertise subtending exceptional driving abilities may be associated with specific morphological and functional cerebral architecture changes (Bernardi et al., 2013, 2014).
For this reason, several pieces of research have been conducted to identify the neural basis of transportation and car driving. A recent meta-analysis presents a neuroergonomic framework according to which the neural bases of driving behavior are categorized into strategical (i.e., navigation), tactical (i.e., overtaking), and operational (steering and braking) tasks (Navarro et al., 2018).
In this context, developing efficient brain-based systems for the real-time decoding of brain processes underlying driver's behavior would be highly beneficial for the design of assistive devices.
This perspective provides a succinct overview of the literature about hybrid systems used for action prediction and the related limitations. Then, it presents the results related to car driving scenarios as proof of the level of technological maturity achieved in the last years. In this context, driving actions are predicted (i) exploiting secondary tasks eliciting cerebral activity related to a higher level of motor control and (ii) by measuring neural correlates of motor preparation as a marker of braking and steering actions and. This methodological approach could benefit additional ecological scenarios in neuroergonomics, such as telerehabilitation and occupational safety.
From EEG-Based Action Decoders to Hybrid Systems
Electroencephalography (EEG) is one of the most used techniques for monitoring brain signals in operational environments. This measure provides the variation of electrical potentials on the scalp surface, generated by the summation of post-synaptic potentials within cortical layers (Biasiucci et al., 2019). Electroencephalography has the critical advantage of tracking brain dynamics with millisecond accuracy and is used in real-world scenarios for neuroimaging studies outside the lab. The evolution of technology allowed the removal of wires and produced wearable and long-lasting recording devices, enabling a wide range of experiments in real-world settings (Debener et al., 2012; Mihajlović et al., 2015; Mullen et al., 2015; Casson, 2019).
The integration of EEG-based action predictions into the control of an assistive technology device, such as a car, would have the great advantage of detecting, as early as possible, the movement preparation and execution, both in a laboratory and in more natural environmental settings. However, despite the noteworthy technological advancement of the last decade, there are still several issues that limit the utilization of the EEG for the real-time monitoring of actions in working environments. For instance, there is the need to improve the EEG hardware to obtain recordings more robust to artifacts and longer battery life and produce smaller devices to be socially accepted by everyone. Other psychophysiological and technological constraints make this prediction hard to achieve in real-life scenarios. Factors such as attention, memory load, fatigue, and competing cognitive processes (Gonçalves et al., 2006; Käthner et al., 2014; Calhoun and Adali, 2016), as well as user's individual characteristics such as lifestyle, gender, and age (Kasahara et al., 2015) influence brain dynamics producing significant intra- and inter-subject variability (Saha and Baumert, 2020; Saha et al., 2021). Common EEG artifacts generated by muscles and eye movements, impedance shifts, environmental noise are typically amplified in real-world scenarios, sensibly affecting the quality of EEG signals during real-time monitoring (Waard, 1996; Zander et al., 2017; Lohani et al., 2019). Also, wearing an EEG device for users within operational environments could be uncomfortable and lead to the corruption of the underlying brain processes. Although the technology provides researchers with high-impedance systems equipped with active shielded electrodes for mobile applications, these devices do not solve all the mentioned issues intrinsically characterizing all ecological environments. This low signal-to-noise ratio returned by raw EEG data requires the use of a range of conceptually very different and computationally expensive algorithms to extract significant temporal and frequency EEG features (Müller et al., 2004; Lotte et al., 2007, 2018; Krusienski et al., 2011; Bellotti et al., 2019). These algorithms often are demanding in terms of calibration because requiring large training sets and are not robust to real-life environmental noise affecting EEG recordings. Other issues relate to the high-dimensionality and non-stationarity of the EEG data, impacting the classification performance (Lotte et al., 2018). In addition, most of the classification methods used in the literature are applied for offline EEG analyses, thus requiring the improvement of this methodology for online applications to guarantee a computational efficiency for the real-time decoding of the brain activity. Hence, the computing hardware and software must warrant a sufficiently high performance and low latency to preserve the earliness of prediction (Wöhrle et al., 2017).
Hence, different physiological, behavioral, and technical data can be combined to improve the reliability of EEG-based predictions and their fully automated application for supporting the user in self-paced movements in critical environments. For example, the prediction of actions onset based on EEG analysis can be improved by the design of hybrid systems simultaneously monitoring additional peripheral signals, such as electroculogram (EOG) and electromyographic (EMG) data, depending on the context requirements (Kirchner et al., 2014). The hybrid concept was introduced in the field of the Brain Computer Interfaces (BCIs), exploiting advantages of different physiological signals and computational approaches to finally achieve specific goals better than a conventional EEG based system, such as improving the overall classification rate or reducing the rate of false positives (Pfurtscheller et al., 2010; Li et al., 2019). Hybrid systems should rely at least on one brain signal in the form of electrical, magnetic, or hemodynamic changes, and at the same time, they can incorporate peripheral or external signals to improve the whole system's performance. For instance, combinations of eye movement signals with neuronal signals usually are utilized for hybrid EEG–EOG BCIs (Usakli et al., 2009, 2010; Ma et al., 2015; Hong and Khan, 2017). Hence, the design of hybrid systems can improve the action prediction performance depending on the particular application.
Electroencephalography and EMG signals can be used to predict movements before the action onset reliably, showing that multimodal machine learning approaches can be potentially used to control an electronic device (Kirchner et al., 2014; Wöhrle et al., 2017). Unimodal EEG-based predictions can be achieved earlier with respect to unimodal EMG-based prediction, thus suggesting that EEG is more suitable for providing the user the feeling that a device delivers support on time without significant delay. Also, EEG analysis leads to more false positives than EMG due to the higher signal-to-noise ratio characterizing such neural data. In addition, which signals are relevant at which state of movement planning and execution have been systematically investigated with machine learning approaches to predict movement targets (Novak et al., 2013). This study reports that each sensing modality has its peculiarities. Electroencephalography is suitable for very early prediction or if the user cannot perform the movement. Electromyography and hand position are accurate after limb motion onset. Eye-tracking is accurate at motion onset, but it is not able to predict motion dynamics. Combining EEG and EOG results in higher accuracy than using a unimodal approach and is convenient since the two signals are often measured together. Augmenting EMG with eye-tracking allows predictions to be made earlier than with only EMG. However, this research field is not mature yet to make precise comparisons of performance and calibration times between machine learning approaches for unimodal and multimodal measurements.
Several challenges also characterize these hybrid systems. One of the significant issues in this research is identifying the best combinations of signals to reach the best prediction performance since the optimal combination could differ across users and experimental scenarios. Variables including system complexity, cost, user workload have to be evaluated when comparing hybrid systems with unimodal predictions. From the user's point of view, the complexity of hybrid systems is usually higher than that of conventional single modality recordings because they are required to wear multiple brain and body sensors. User acceptability is a crucial criterion that needs to be considered in designing and implementing such systems (Pfurtscheller et al., 2010; Li et al., 2019).
Hybrid Systems in Car Driving Scenarios
In the field of driving research, several studies addressed the issue of action detection and prediction based on the discrimination of different EEG features in simulated (Haufe et al., 2011; Gheorghe et al., 2013; Khaliliardali et al., 2015; Kim et al., 2015; Vecchiato et al., 2018, 2020, 2021) and real driving scenarios (Haufe et al., 2014; Zhang et al., 2015).
A few studies used secondary tasks to elicit neural features predicting steering during simulated and real car driving. In particular, the contingent negative variation (CNV) potential was generated by a go/no-go task to investigate the decoding of drive and brake events (Gheorghe et al., 2013; Khaliliardali et al., 2015). Results suggested that these actions can be discriminated around 320 ms before the movement with a classification performance of 0.77. In addition, Zhang et al. (2015) described an online event-related negativity (ERN) classifier to predict steering events, guided by a directional cue, both in a laboratory and real car driving scenarios. In both experimental conditions, they discriminated correct by error trials 480 and 700 ms after the directional cue. The classification performance is 0.70, but the computational timing cost is not reported, so the time interval between the directional cue and the classifier decision is unknown, and therefore whether it comes before or after the actual movement execution.
Other studies investigated the driver's action without using external cues with the advantage of limiting the additional driver's mental load. Haufe et al. (2011) explored pseudo-online emergency braking detection and evaluated that such a system in a simulation environment could eventually detect foot action around 130 ms before its onset. The possibility to decode self-generated actions detecting steering was also assessed, and in particular, whether the driver would perform a lane change in a simulated highway was predicted about 800 ms earlier the action onset with a true positive rate of 74.6% (Gheorghe et al., 2013). In addition, Vecchiato et al. (2018) identified an EEG independent component associated with the fronto-central electrodes exhibiting synchronization of theta EEG rhythm around 800 ms before the braking onset.
In line with the concept of hybrid systems, Kim et al. (2015) proposed a combination of EEG features in the time and frequency domains to distinguish three different kinds of stimulus-driven brake situations (i.e., sharp, soft, no brake). This study reported the highest EEG response-locked decoding performance at −480 and −420 ms distinguishing sharp and soft braking from no braking, respectively. It was harder to classify sharp and soft braking conditions with the same method (largest difference at −160 ms), which returned lower performance than a classifier based on EMG features. There were also significant results with hybrid decoding systems in a real car scenario where participants were asked to drive on a non-public test track (Haufe et al., 2014). They reported that a hybrid (EEG and behavioral features) classifier detected emergency braking even earlier than the laboratory setting (around 300 ms before the braking onset) (Haufe et al., 2011).
Moreover, in order to characterize the relative contribution of the EEG associated with the preparation of natural and self-initiated steering actions while driving to investigate its predictive power, the EEG related to continuous steering during the driving simulation was tested by means of canonical correlation analysis (CCA) and a linear lagged regression approach (LLR) to identify the relative contribution of the EEG signals in steering anticipation (Di Liberto, accepted). Results showed that the combination of CCA-LLR analysis is valuable to disentangle the relative contribution of behavioral and electrophysiological components—within the EEG signals—for steering prediction in a continuous driving simulation task. This result demonstrates that brain-related EEG signals significantly improve the overall decoding performance, showing that the significant contribution in predicting steering comes from non-brain-related signals, such as ocular and muscular components.
Brain and muscular activities underlying steering behavior were also investigated with the final aim to increase the overall ecology of the experimental setting (Vecchiato et al., 2021). In particular, EEG feature predicting steering action and direction elicited by responding to traffic signs displayed on a computer screen was extracted and later exploited to increase the predictive power of the EMG collected in a more ecological steering task, such as a driving simulation. The desynchronization of the mu rhythm during the motor preparation of non-ecological steering cued by the traffic sign discriminated the muscular activity of the deltoids, thus anticipating subject steering behavior of 1.5 s. In addition, the increase of EMG activity of the deltoids anticipated the contralateral steering in both non-ecological and ecological steering tasks of 200 and 500 ms relative to the action onset, making it possible to discriminate such a driving behavior. Although these variations of EMG activity appear before the action onset allowing for possible online predictions, EEG data were used to increase the available time to perform such a calculation. The identified non-ecological EEG feature correlated with the ecological EMG activity of the deltoids, providing an improvement of the discrimination power of the steering side during driving simulation (Figure 1). These findings show an approach to increase the ecology of the experimental setting by limiting the invasiveness of the neurophysiological measurements using surface EMG sensors in the ecological scenario and combining neural data collected in the non-ecological one. This approach provides a way to monitor the user performance online through a simpler to acquire muscular correlate when compared to neural data, which could be recorded offline to increase the decoding system's performance without impacting the complexity of the ecological setting.
Figure 1. NON-ECO frame. Time-frequency EEG patterns collected during left (A) and right (B) non-ecological steering, as well as their statistical comparison (C). The topography in the left part of the picture shows the average scalp map related to the cluster of independent components. ECO frame. ERSP for the EMG signals collected during the non-ecological steering task (A–F), and cross-correlation results between EEG and EMG data (G–L). The first (second) row (from the top) illustrates the EMG ERSP for the left (right) deltoid during left and right steering, as well as their statistical comparison. The third (fourth) row illustrates the EEG-EMG cross-correlation values for the left (right) deltoid during left and right steering, and the statistical comparison of the two conditions. White lines depict the left and right steering wheel angle profiles. Color bars indicate in blue (red) the decrease (increase) of EEG, EMG, and cross-correlation, as well as the statistical differences corresponding to the decrease (increase) of such activity during the left (right) steering. White and black masks delimit the statistically significant portion of the EEG, EMG, cross-correlation panels (adapted from Vecchiato et al., 2021).
Conclusions
The coupling between EEG, EMG, and ocular signals is a valid mechanism for utilizing hybrid systems for the detection and online prediction of driving actions, exemplifying how it might be possible to complement information from behavioral, physiological, and external sources to control the level of assistance needed by the driver in that context (Chavarriaga et al., 2018). This methodology could pave the way for the utilization of hybrid systems based on neural signals—collected in standard laboratory settings and processed offline—having the role in improving the predictive power of peripheral signals—collected in more ecological settings and possibly processed online—correlated with the upcoming action execution.
The predictive power returned by coupling the EEG with peripheral signals demonstrated in car driving scenarios could be further investigated in larger sets of actions to extend the validity of this approach to other neuroergonomic areas. For instance, this methodology could foster the spread of mobile brain and body applications (Makeig et al., 2009; Gramann et al., 2011) and BCI paradigms (Douibi et al., 2021; Saha et al., 2021) onto several other contexts of our daily life. The capability to remotely monitor in an ecological way an individual's action would have a tremendous impact in the rehabilitation field (Nuara et al., 2021), with the possibility to verify the compliance and adherence to treatment relieving the patient and caregivers from a massive burden in terms of time and costs. Applications could also extend beyond the clinical realm, virtually to any fields where action surveillance would be valuable for preventing harmful consequences. It is the case, for instance, of the occupational safety of workers dealing in their routine with unsafe practices, for whom the use of this ecological methodology could reduce the likelihood of occupational injuries during the performance of high-risk motor tasks (Rizzolatti et al., 2021).
Data Availability Statement
The original contributions presented in the study are included in the article/supplementary material, further inquiries can be directed to the corresponding author/s.
Author Contributions
The author confirms being the sole contributor of this work and has approved it for publication.
Funding
This study received funding from CAMLIN Limited. The funder was not involved in the writing of this article or the decision to submit it for publication.
Conflict of Interest
The author declares that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
Publisher's Note
All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.
Acknowledgments
The present study was supported by a research agreement between CAMLIN Limited and IN-CNR, and a research agreement between CAMLIN Limited and Toyota Motor Europe. The author would like to thank Maria Del Vecchio for the insightful comments she provided on the drafts of this manuscript.
References
Badue, C., Guidolini, R., Carneiro, R. V., Azevedo, P., Cardoso, V. B., Forechi, A., et al. (2021). Self-driving cars: a survey. Expert Syst. Appl. 165, 113816. doi: 10.1016/j.eswa.2020.113816
Bellotti, A., Antopolskiy, S., Marchenkova, A., Colucciello, A., Avanzini, P., Vecchiato, G., et al. (2019). “Brain-based control of car infotainment,” in 2019 IEEE International Conference on Systems, Man and Cybernetics (SMC) (Bari), 2166–2173. doi: 10.1109/SMC.2019.8914448
Bernardi, G., Cecchetti, L., Handjaras, G., Sani, L., Gaglianese, A., Ceccarelli, R., et al. (2014). It's not all in your car: functional and structural correlates of exceptional driving skills in professional racers. Front. Hum. Neurosci. 8, 888. doi: 10.3389/fnhum.2014.00888
Bernardi, G., Ricciardi, E., Sani, L., Gaglianese, A., Papasogli, A., Ceccarelli, R., et al. (2013). How skill expertise shapes the brain functional architecture: an fMRI study of visuo-spatial and motor processing in professional racing-car and naïve drivers. PLoS ONE 8, e77764. doi: 10.1371/journal.pone.0077764
Biasiucci, A., Franceschiello, B., and Murray, M. M. (2019). Electroencephalography. Curr. Biol. 29, R80–R85. doi: 10.1016/j.cub.2018.11.052
Borghini, G., Astolfi, L., Vecchiato, G., Mattia, D., and Babiloni, F. (2014). Measuring neurophysiological signals in aircraft pilots and car drivers for the assessment of mental workload, fatigue and drowsiness. Neurosci. Biobehav. Rev. 44, 58–75. doi: 10.1016/j.neubiorev.2012.10.003
Calhoun, V. D., and Adali, T. (2016). Time-varying brain connectivity in fMRI data: whole-brain data-driven approaches for capturing and characterizing dynamic states. IEEE Signal Process. Mag. 33, 52–66. doi: 10.1109/MSP.2015.2478915
Calhoun, V. D., and Pearlson, G. D. (2012). A selective review of simulated driving studies: combining naturalistic and hybrid paradigms, analysis approaches, and future directions. Neuroimage 59, 25–35. doi: 10.1016/j.neuroimage.2011.06.037
Calhoun, V. D., Pekar, J. J., McGinty, V. B., Adali, T., Watson, T. D., and Pearlson, G. D. (2002). Different activation dynamics in multiple neural systems during simulated driving. Hum. Brain Mapp. 17, 141–142. doi: 10.1002/hbm.10066
Casson, A. J. (2019). Wearable EEG and beyond. Biomed. Eng. Lett. 9, 53–71. doi: 10.1007/s13534-018-00093-6
Chavarriaga, R., Ušćumlić, M., Zhang, H., Khaliliardali, Z., Aydarkhanov, R., Saeedi, S., et al. (2018). Decoding neural correlates of cognitive states to enhance driving experience. IEEE Trans. Emerg. Top. Comput. Intell. 2, 288–297. doi: 10.1109/TETCI.2018.2848289
Debener, S., Minow, F., Emkes, R., Gandras, K., and de Vos, M. (2012). How about taking a low-cost, small, and wireless EEG for a walk?: EEG to go. Psychophysiology 49, 1617–1621. doi: 10.1111/j.1469-8986.2012.01471.x
Dehais, F., Karwowski, W., and Ayaz, H. (2020). Brain at work and in everyday life as the next frontier: grand field challenges for neuroergonomics. Front. Neuroergon. 1, 583733. doi: 10.3389/fnrgo.2020.583733
Di Liberto, G, Barsotti, M, Vecchiato, G, Ambeck-Madsen, J, Del Vecchio, M, and Avanzini, P. (accepted). Robust anticipation of continuous steering actions from electroencephalographic data during simulated driving. Sci. Rep.
Douibi, K., Le Bars, S., Lemontey, A., Nag, L., Balp, R., and Breda, G. (2021). Toward EEG-based BCI applications for industry 4.0: challenges and possible applications. Front. Hum. Neurosci. 15, 705064. doi: 10.3389/fnhum.2021.705064
Gheorghe, L., Chavarriaga, R., and Millán, J. D. R. (2013). “Steering timing prediction in a driving simulator task,” in 2013 35th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EBC) (Piscataway, NJ), 6913–6916. doi: 10.1109/EMBC.2013.6611147
Gonçalves, S. I., de Munck, J. C., Pouwels, P. J. W., Schoonhoven, R., Kuijer, J. P. A., Maurits, N. M., et al. (2006). Correlating the alpha rhythm to BOLD using simultaneous EEG/fMRI: inter-subject variability. Neuroimage 30, 203–213. doi: 10.1016/j.neuroimage.2005.09.062
Gramann, K., Gwin, J. T., Ferris, D. P., Oie, K., Jung, T.-P., Lin, C.-T., et al. (2011). Cognition in action: imaging brain/body dynamics in mobile humans. Rev. Neurosci. 22, 593–608. doi: 10.1515/RNS.2011.047
Gramann, K., McKendrick, R., Baldwin, C., Roy, R. N., Jeunet, C., Mehta, R. K., et al. (2021). Grand field challenges for cognitive neuroergonomics in the coming decade. Front. Neuroergon. 2, 643969. doi: 10.3389/fnrgo.2021.643969
Haufe, S., Kim, J.-W., Kim, I.-H., Sonnleitner, A., Schrauf, M., Curio, G., et al. (2014). Electrophysiology-based detection of emergency braking intention in real-world driving. J. Neural Eng. 11, 056011. doi: 10.1088/1741-2560/11/5/056011
Haufe, S., Treder, M. S., Gugler, M. F., Sagebaum, M., Curio, G., and Blankertz, B. (2011). EEG potentials predict upcoming emergency brakings during simulated driving. J. Neural Eng. 8, 056001. doi: 10.1088/1741-2560/8/5/056001
Hong, K.-S., and Khan, M. J. (2017). Hybrid brain-computer interface techniques for improved classification accuracy and increased number of commands: a review. Front. Neurorobot. 11, 35. doi: 10.3389/fnbot.2017.00035
Kasahara, K., DaSalla, C. S., Honda, M., and Hanakawa, T. (2015). Neuroanatomical correlates of brain-computer interface performance. NeuroImage 110, 95–100. doi: 10.1016/j.neuroimage.2015.01.055
Käthner, I., Wriessnegger, S. C., Müller-Putz, G. R., Kübler, A., and Halder, S. (2014). Effects of mental workload and fatigue on the P300, alpha and theta band power during operation of an ERP (P300) brain–computer interface. Biol. Psychol. 102, 118–129. doi: 10.1016/j.biopsycho.2014.07.014
Khaliliardali, Z., Chavarriaga, R., Gheorghe, L. A., Millán, J., and del, R. (2015). Action prediction based on anticipatory brain potentials during simulated driving. J. Neural Eng. 12, 066006. doi: 10.1088/1741-2560/12/6/066006
Kim, I.-H., Kim, J.-W., Haufe, S., and Lee, S.-W. (2015). Detection of braking intention in diverse situations during simulated driving based on EEG feature combination. J. Neural Eng. 12, 016001. doi: 10.1088/1741-2560/12/1/016001
Kirchner, E. A., Tabie, M., and Seeland, A. (2014). Multimodal movement prediction - towards an individual assistance of patients. PLoS ONE 9, e85060. doi: 10.1371/journal.pone.0085060
Krusienski, D. J., Grosse-Wentrup, M., Galán, F., Coyle, D., Miller, K. J., Forney, E., et al. (2011). Critical issues in state-of-the-art brain-computer interface signal processing. J. Neural Eng. 8, 025002. doi: 10.1088/1741-2560/8/2/025002
Li, Z., Zhang, S., and Pan, J. (2019). Advances in hybrid brain-computer interfaces: principles, design, and applications. Comput. Intell. Neurosci. 2019, e3807670. doi: 10.1155/2019/3807670
Lohani, M., Payne, B. R., and Strayer, D. L. (2019). A review of psychophysiological measures to assess cognitive states in real-world driving. Front. Hum. Neurosci. 13, 57. doi: 10.3389/fnhum.2019.00057
Lotte, F., Bougrain, L., Cichocki, A., Clerc, M., Congedo, M., Rakotomamonjy, A., et al. (2018). A review of classification algorithms for EEG-based brain–computer interfaces: a 10 year update. J. Neural Eng. 15, 031005. doi: 10.1088/1741-2552/aab2f2
Lotte, F., Congedo, M., Lécuyer, A., Lamarche, F., and Arnaldi, B. (2007). A review of classification algorithms for EEG-based brain-computer interfaces. J. Neural Eng. 4, R1–R13. doi: 10.1088/1741-2560/4/2/R01
Ma, J., Zhang, Y., Cichocki, A., and Matsuno, F. (2015). A novel EOG/EEG Hybrid human–machine interface adopting eye movements and ERPs: application to Robot Control. IEEE Trans. Biomed. Eng. 62, 876–889. doi: 10.1109/TBME.2014.2369483
Makeig, S., Gramann, K., Jung, T.-P., Sejnowski, T. J., and Poizner, H. (2009). Linking brain, mind and behavior. Int. J. Psychophysiol. 73, 95–100. doi: 10.1016/j.ijpsycho.2008.11.008
Mihajlović, V., Grundlehner, B., Vullers, R., and Penders, J. (2015). Wearable, wireless EEG solutions in daily life applications: what are we missing? IEEE J. Biomed. Health Inform. 19, 6–21. doi: 10.1109/JBHI.2014.2328317
Mullen, T. R., Kothe, C. A. E., Chi, Y. M., Ojeda, A., Kerth, T., Makeig, S., et al. (2015). Real-time neuroimaging and cognitive monitoring using wearable dry EEG. IEEE Trans. Biomed. Eng. 62, 2553–2567. doi: 10.1109/TBME.2015.2481482
Müller, K.-R., Krauledat, M., Dornhege, G., Curio, G., and Blankertz, B. (2004). Machine learning techniques for brain-computer interfaces. Biomed. Eng. 49, 11–22. doi: 10.13109/9783666351419.11
Navarro, J., Reynaud, E., and Osiurak, F. (2018). Neuroergonomics of car driving: a critical meta-analysis of neuroimaging data on the human brain behind the wheel. Neurosci. Biobehav. Rev. 95, 464–479. doi: 10.1016/j.neubiorev.2018.10.016
Novak, D., Omlin, X., Leins-Hess, R., and Riener, R. (2013). Predicting targets of human reaching motions using different sensing technologies. IEEE Trans. Biomed. Eng. 60, 2645–2654. doi: 10.1109/TBME.2013.2262455
Nuara, A., Fabbri-Destro, M., Scalona, E., Lenzi, S. E., Rizzolatti, G., and Avanzini, P. (2021). Telerehabilitation in response to constrained physical distance: an opportunity to rethink neurorehabilitative routines. J. Neurol. 2021, 1–12. doi: 10.1007/s00415-021-10397-w
Parasuraman, R. (2003). Neuroergonomics: research and practice. Theor. Issues Ergon. Sci. 4, 5–20. doi: 10.1080/14639220210199753
Pfurtscheller, G., Allison, B. Z., Brunner, C., Bauernfeind, G., Solis-Escalante, T., Scherer, R., et al. (2010). The hybrid BCI. Front. Neurosci. 4, 30. doi: 10.3389/fnpro.2010.00003
Rizzolatti, G., Fabbri-Destro, M., Nuara, A., Gatti, R., and Avanzini, P. (2021). The role of mirror mechanism in the recovery, maintenance, and acquisition of motor abilities. Neurosci. Biobehav. Rev. 127, 404–423. doi: 10.1016/j.neubiorev.2021.04.024
Saha, S., and Baumert, M. (2020). Intra- and inter-subject variability in EEG-based sensorimotor brain computer interface: a review. Front. Comput. Neurosci. 13, 87. doi: 10.3389/fncom.2019.00087
Saha, S., Mamun, K. A., Ahmed, K., Mostafa, R., Naik, G. R., Darvishi, S., et al. (2021). Progress in brain computer interface: challenges and opportunities. Front. Syst. Neurosci. 15, 578875. doi: 10.3389/fnsys.2021.578875
Usakli, A. B., Gurkan, S., Aloise, F., Vecchiato, G., and Babiloni, F. (2009). “A hybrid platform based on EOG and EEG signals to restore communication for patients afflicted with progressive motor neuron diseases,” in Conference proceedings. Annual International Conference of the IEEE Engineering in Medicine and Biology Society. IEEE Engineering in Medicine and Biology Society. Annual Conference 2009 (Minneapolis, MN), 543–546. doi: 10.1109/IEMBS.2009.5333742
Usakli, A. B., Gurkan, S., Aloise, F., Vecchiato, G., and Babiloni, F. (2010). On the use of electrooculogram for efficient human computer interfaces. Comput. Intell. Neurosci. 2010, 135629. doi: 10.1155/2010/135629
Vecchiato, G., Del Vecchio, M., Antopolskiy, S., Bellotti, A., Colucciello, A., Marchenkova, A., et al. (2020). “Alpha and beta EEG Desynchronizations anticipate steering actions in a driving simulation experiment,” in Intelligent Human Systems Integration 2020. Advances in Intelligent Systems and Computing, eds T. Ahram, W. Karwowski, A. Vergnano, F. Leali, and R. Taiar (Cham: Springer International Publishing), 260–265. doi: 10.1007/978-3-030-39512-4_41
Vecchiato, G., Vecchio, M. D., Ambeck-Madsen, J., Ascari, L., and Avanzini, P. (2021). Hybrid EEG-EMG system to detect steering actions in car driving settings. BioRxiv preprint. doi: 10.1101/2021.09.16.460615
Vecchiato, G., Vecchio, M. D., Ascari, L., Antopolskiy, S., Deon, F., Kubin, L., et al. (2018). Electroencephalographic time-frequency patterns of braking and acceleration movement preparation in car driving simulation. Brain Res. 1716, 16–26. doi: 10.1016/j.brainres.2018.09.004
Waard, D. D. (1996). The Measurement of Drivers' Mental Workload. Available online at: https://www.semanticscholar.org/paper/The-measurement-of-drivers%27-mental-workload-Waard/a4e683f49e357cc5b21107285013e9bd470aefe2 (accessed September 7, 2021).
Wöhrle, H., Tabie, M., Kim, S. K., Kirchner, F., and Kirchner, E. A. (2017). A hybrid FPGA-Based system for EEG- and EMG-based online movement prediction. Sensors 17, 1552. doi: 10.3390/s17071552
Zander, T. O., Andreessen, L. M., Berg, A., Bleuel, M., Pawlitzki, J., Zawallich, L., et al. (2017). Evaluation of a dry EEG system for application of passive brain-computer interfaces in autonomous driving. Front. Hum. Neurosci. 11, 78. doi: 10.3389/fnhum.2017.00078
Keywords: hybrid systems, action prediction, driving, EEG, EMG, EOG
Citation: Vecchiato G (2021) Hybrid Systems to Boost EEG-Based Real-Time Action Decoding in Car Driving Scenarios. Front. Neuroergon. 2:784827. doi: 10.3389/fnrgo.2021.784827
Received: 28 September 2021; Accepted: 08 November 2021;
Published: 29 November 2021.
Edited by:
Raphaëlle N. Roy, Institut Supérieur de l'Aéronautique et de l'Espace (ISAE-SUPAERO), FranceReviewed by:
Angela Rose Harrivel, National Aeronautics and Space Administration (NASA), United StatesCaterina Cinel, University of Essex, United Kingdom
Mathias Vukelić, Fraunhofer Institute for Industrial Engineering, Germany
Copyright © 2021 Vecchiato. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
*Correspondence: Giovanni Vecchiato, giovanni.vecchiato@in.cnr.it