Skip to main content

Brain-Computer Interfaces and Augmented/Virtual Reality

157kviews
Editorial12 May 2020
Editorial: Brain-Computer Interfaces and Augmented/Virtual Reality

The fusion of Brain-Computer Interfaces (BCI) with AR/VR can provide additional communication channels, by increasing the bandwidth of the human-AR/VR interaction. This is achieved either explicitly through active BCIs or implicitly using passive BCIs. Active BCIs notably allow users to issue commands to devices or to enter text without physical involvement of any kind, while passive BCIs monitor a user's state (e.g. workload level, attentional state) and can be used to proactively adapt the VR/AR interface. See (Lotte et al., 2012) for a general review on mixing VR and BCI technologies.BCIs, together with AR/VR, offer the possibility for immersive scenarios through induced illusions of an artificially perceived reality that can be utilized not only in basic BCI research but also therapeutic applications, Human-Computer Interaction, etc. To live up to these expectations, methodological advances are required for BCI interaction and stimulus design, synchronization, or dealing with VR/AR specific artifacts (Tremmel et al., 2019) and distractions.The papers in this research topic show both the upsides and the potential of BCI research for or with AR/VR technology, as well as the challenges that lie on the way to such an integration. Moreover, both passive and active BCIs are presented from current studies. In terms of passive BCIs, attentional state in AR is investigated by Vortmann et al. (2019a) and workload assessment in VR by (Tremmel et al., 2019). In terms of active BCIs, virtual avatar arms are controlled directly in motor-imagery BCI training for neurorehabilitation by Vourvopoulos et al. (2019c,b) and also in a more gamified training byŠkola et al. (2019).Overall, we see that BCI can be used in VR/AR-based training for workload and attention assessment, but also in rehabilitation contexts, where the VR components creates a much more immersive experience compared to traditional training paradigms.In the research topic, we find works both on passive and active BCI technology. Regarding passive BCI, research concentrates on two aspects which are often tackled for adaptive technology: attention and workload. From their motivation, these papers aim for an improvement of the AR/VR interface itself through user state adaptation. Vortmann et al. (2019a) perform a study on the classification of internal from external attention in an AR setting. The authors develop a novel AR task requiring continuous spatial alignment, mimicking typical AR interactions. They show that using frequency features, the classifier achieves an average classification accuracy of 85% using windows of 13s length. Recently, they demonstrated an real time implementation of the attention model (Vortmann et al., 2019b), enabling online adaptation of AR-based user interfaces, such as a smart home control in AR (Putze et al., 2019b) using SSVEP and eye tracking to select from virtual menus displayed in the environment. Other related work is by Si-Mohammed et al. (2018), a pioneer study on the feasibility and design of 3D User Interfaces exploiting AR for SSVEP-based BCI.Tremmel et al. (2019) perform a study to measure mental workload in VR. For this purpose, they adapted the well-studies n-back task for interactive VR and made it publicly available. They recorded EEG activity from 15 participants performing 0-, 1-and 2-back trials and could show that workload levels could be discriminated from the scalp recordings despite the large level of physical movement occuring during VR usage. Additionally, they showed the feasibility of using fNIRS as an alternative modality (Putze et al., 2019a), as both fNIRS (Herff et al., 2014) and the combination of fNIRS and EEG (Herff et al., 2015) have been shown to be successful in workload classification. This opens the door for future multimodal systems on workload classification in VR.Other papers concentrate on the integration of active BCI technology in VR settings. In contrast to the passive BCI contributions, this research is less about the potential improvement of the VR interface, but about leveraging this immersive technology to improve traditional BCI paradigms, for example in rehabilitation. Vourvopoulos et al. (2019b) performed a study by combining the principles of VR and BCI in a rehabilitation training platform called REINVENT (Vourvopoulos et al., 2019a), assessing its effects on four chronic stroke patients across different levels of motor impairment. The acquired post-stroke EEG signals that indicate an attempt to move, drive a virtual avatar arm (from the affected side), allowing patient-driven action observation BCI in VR. They show that EEG-based BCI-VR may benefit patients with more severe motor impairments, by increasing their communication bandwidth to VR, in-contrast to patients with more mild impairments that they can still harness existing sensorimotor pathways with EMG-based feedback in the same VR training (Marin-Pardo et al., 2019).By extending BCI-VR training from uni-manual (of the affected arm) into bi-manual control, Vourvopoulos et al. (2019c) performed a study by using NeuRow, a self-paced BCI paradigm in VR (Vourvopoulos et al., 2016) together with brain-imaging data (fMRI) in a chronic stroke patient. They found important improvements in upper extremity clinical scales (Fugl-Meyer) and identified increases in brain activation measured by fMRI that suggest neuroplastic changes in brain motor networks. These results suggest that BCI with ecologically-valid VR could be useful in chronic stroke patients with reduced upper limb motor function while together with brain-imaging data we move towards identifying the specific benefits of brain-controlled VR training environments for neurorehabilitation.Finally,Škola et al. (2019) presented a study with a gamified BCI-VR training including virtual avatar designed with the aim to maintain high levels of attention and motivation. This was achieved using a progressively increasing training, event-driven and not self-paced by providing participants with score points about their progress. Performance levels in terms of classifier performance show higher than chance level (65%) and the strength of ERDs during session was positively correlated to the subjective magnitude of the sense of ownership while the perceived ownership of the avatar body was not correlated to the BCI performance nor to the sense of agency.The presented articles present the large potential of combining AR/VR technology with BCIs to further increase the immersiveness of AR/VR and improve the usability of BCIs for rehabilitation and control. All presented papers use elaborate, task-specific experiment setups with custom AR/VR environments and custom solutions to set up sensors, input devices, and displays (see (Putze, 2019) for an overview of emerging methods and tools and (Si-Mohammed et al., 2017) for a general overview of potential applications and main scientific challenges related to the combination of AR and BCI). Future research can build on this pioneering works and the resulting best practices to derive more standardized, common experiment protocols for a large variety of research questions. This would lower the entry barrier and make the promising technology more accessible.Putze, F., Herff, C., Tremmel, C., Schultz, T., and Krusienski, D. J. (2019a). Decoding mental workload in virtual environments: a fnirs study using an immersive n-back task. In 2019 41st Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) (IEEE), 3103-3106 Putze, F., Weiβ, D., Vortmann, L.-M., and Schultz, T. (2019b). Augmented reality interface for smart home control using ssvep-bci and eye gaze. In 2019 IEEE International Conference on Systems, Man and Cybernetics (SMC) (IEEE), 2812-2817 Si-Mohammed, H., Argelaguet, F., Casiez, G., Roussel, N., and Lécuyer, A. (2017). Brain-computer interfaces and augmented reality: A state of the art. In Graz Brain-Computer Interface Conference Si-Mohammed, H., Petit, J., Jeunet, C., Argelaguet, F., Spindler, F., Evain, A., et al. (2018). Towards bci-based interfaces for augmented reality: Feasibility, design and evaluation. IEEE transactions on visualization and computer graphicš Skola, F., Tinková, S., and Liarokapis, F. (2019). Progressive training for motor imagery brain-computer interfaces using gamification and virtual reality embodiment. Multimodal head-mounted virtual-reality brain-computer interface for stroke rehabilitation. In Virtual, Augmented and Mixed Reality. Multimodal Interaction, eds. J. Y. Chen and G. Fragomeni (Cham: Springer International Publishing), 165-179 Vourvopoulos, A., Pardo, O. M., Lefebvre, S., Neureither, M., Saldana, D., Jahng, E., et al. (2019b). Effects of a brain-computer interface with virtual reality (vr) neurofeedback: a pilot study in chronic stroke patients. Frontiers in human neuroscience 13, 210 Vourvopoulos, A. T., Jorge, C., Abreu, R., Figueiredo, P., Fernandes, J.-C., and Bermúdez i Badia, S.(2019c). Efficacy and brain imaging correlates of an immersive motor imagery bci-driven vr system for upper limb motor rehabilitation: A clinical case report. Frontiers in human neuroscience 13, 244

25,284 views

55 citations

To maximize brain plasticity after stroke, several rehabilitation strategies have been explored, including the use of intensive motor training, motor imagery, and action observation. Growing evidence of the positive impact of virtual reality (VR) techniques on recovery following stroke has been shown. However, most VR tools are designed to exploit active movement, and hence patients with low level of motor control cannot fully benefit from them. Consequently, the idea of directly training the central nervous system has been promoted by utilizing motor-imagery (MI) based brain-computer interfaces (BCIs). To date, detailed information on which VR strategies lead to successful functional recovery is still largely missing, and very little is known about how to optimally integrate BCI and VR paradigms for stroke rehabilitation. The purpose of this study was to examine the efficacy of a BCI-VR system using a MI paradigm for post-stroke upper limb rehabilitation on functional assessments, and related changes in MI ability and brain imaging. To achieve this, a 60 years old male chronic stroke patient was recruited. The patient underwent a 3-week intervention in a clinical environment, resulting in 10 BCI-VR training sessions. The patient was assessed before and after intervention, as well as on a one-month follow-up, in terms of clinical scales and brain imaging using functional MRI (fMRI). Consistent with prior research, we found important improvements in upper extremity scores (Fugl-Meyer) and identified increases in brain activation measured by fMRI that suggest neuroplastic changes in brain motor networks. This study expands on the current body of evidence as more data are needed on the effect of this type of interventions not only on functional improvement but also through brain imaging to study the effect of the intervention on plasticity.

21,053 views

163 citations

Effects of a Brain-Computer Interface With Virtual Reality (VR) Neurofeedback: A Pilot Study in Chronic Stroke Patients
ORIGINAL RESEARCH
19 June 2019
Effects of a Brain-Computer Interface With Virtual Reality (VR) Neurofeedback: A Pilot Study in Chronic Stroke Patients
Athanasios Vourvopoulos, 5 more and Sook-Lei Liew
Rehabilitation for stroke patients with severe motor impairments is burdensome and demanding because most of the current rehabilitation options require some volitional movement to train the affected side. However, research has shown that survivors of severe stroke may receive modest benefits from action observation, virtual reality (VR), and brain-computer interfaces (BCIs). These approaches have shown some success in strengthening key motor pathways thought to support motor recovery after stroke. The purpose of this study was to combine the principles of action observation, VR, and BCI in a platform called REINVENT and assess its effects on four chronic stroke patients across different levels of motor impairment. REINVENT acquires post-stroke EEG signals that indicate an attempt to move and drives the movement of a virtual avatar arm, allowing patient-driven action observation neurofeedback in VR. In addition, synchronous electromyography (EMG) data were also captured to monitor overt muscle activity. Our results show that this EEG-based BCI can be used by stroke survivors across a wide range of motor disabilities. Finally, individual results suggest that patients with more severe motor impairments benefit the most from EEG-based neurofeedback, while patients with more mild impairments may benefit more from EMG-based feedback, harnessing existing sensorimotor pathways. Future research is needed to confirm these findings in a larger and more diverse population.

30,574 views

160 citations