The Use of Brain–Computer Interfaces in Games Is Not Ready for the General Public
Grégoire Cattan- IBM, Krakow, Poland
Introduction
The use of brain–computer interfaces (BCIs) based on electroencephalography (EEG) in video games has been widely investigated. Research in adaptive training, single-trial classification (Congedo, 2013; Barachant and Congedo, 2014) and the creation of affordable EEG acquisition devices (Vos et al., 2014; Yohanandan et al., 2018) has paved the way for the development of a ubiquitous BCI technology. For example, Congedo (2013) developed “Brain Invaders,” a BCI game inspired by the famous vintage game Space Invaders (Taito, Tokyo, Japan) and based on the so-called visual P300—an electromagnetic potential produced by the brain about 300 ms after a visual stimulation. Brain Invaders uses an adaptive algorithm that allows the player to plug the material and play without the need for calibration (Barachant and Congedo, 2014), while still achieving a high accuracy rate (Barachant et al., 2012). The game also demonstrates a good understanding of game design by naturally incorporating visual stimulations in the virtual environment. In this respect, Kaplan et al. (2013), Cattan et al. (2018b), and Rashid et al. (2020) provided a set of guidelines to adapt game implementation for BCI games, such as the use of turn-based games with a slow gameplay. Although Brain Invaders use a research-grade amplifier, the feasibility of using a low-cost EEG acquisition system for BCI has been demonstrated by Vos et al. (2014) and Yohanandan et al. (2018). These affordable headsets are comparable with research-grade amplifiers. In addition, Lotte et al. (2008) and Debener et al. (2012) demonstrated promising results when BCIs were used out of the lab for BCIs based on visual stimulation and movement imagination. The feasibility of using BCIs outside the lab has also been demonstrated in different context and at events. For instance, in the BCI game developed by Mentalista (Paris, France) for the 2016 European Football Championship, two players were asked to score against each other by moving a ball toward the opposite player's cage by concentrating1.
Although a positive step forwards, these achievements have led to the false opinion that BCIs are ready for entertainment – a belief that is supported by enthusiastic visions claiming, for example, that brains will be connected to the internet through USB2. This opinion is rather qualified in the scientific community, which reported that BCIs suffer from (1) a low transfer rate, (2) a lack of market-ready, affordable, and user-friendly research-grade EEG acquisition devices, and (3) a gap between the game design and graphics of video games available on the market vs. in laboratories. Priorities of these limitations for video game development are discussed in the literature (Nijholt et al., 2009; Ferreira et al., 2013; Marshall et al., 2013; van de Laar et al., 2013; Ahn et al., 2014; Cattan et al., 2018b; Kerous et al., 2018; Vasiljevic and Miranda, 2019; Pierce et al., 2020), and in general preponderance of quantitative over qualitative aspects is criticized (Nijholt et al., 2009; Vasiljevic and Miranda, 2019). This article supports the claim that BCIs are not ready for general public use, based on other aspects than performance. Limitations are further detailed in section Limitations of BCI Games of the present study, and the obstacles to public use are analyzed. The conclusion is presented in section Discussion and Conclusion.
Limitations of BCI Games
Low Transfer Rate
The transfer rate (in bits per second) is a computed measure for communication devices that reflects the speed and accuracy of a device. This measure is derived from information theory and was adapted for BCIs (Wolpaw and Wolpaw, 2012) to compare different implementations. Following the development of Riemannian Geometry during the last decade (e.g., Barachant et al., 2012), the transfer rate of BCIs has considerably increased to reach, for example in the case of a P300-based BCI, a 90% accuracy within a couple of seconds (Cattan et al., 2018b). For comparison, the Guinness World Records reports a record of stenotype writing of around 360 words per minutes with 97% accuracy3. In practice, this means that BCIs are unusable in traditional inputs, such as in keyboards or mice. Another concern is that the algorithm complexity [in terms of mathematics and set-up (e.g., Cattan et al., 2018a; Andreev et al., 2019)] behind BCIs might be an obstacle for game developers. This is true for VR games especially, as the VR market is mostly represented by small, independent companies with limited resources4,5.
Cost and Encumberment of Materials
Until the early 2000s, research-grade amplifiers were medical and were made of expensive materials and could mostly only be afforded by institutions or consortia (e.g., g.USBAmp, g.tec, Schiedlberg, Austria). Emotiv (San Francisco, US) was among the first manufacturers to release a commercial EEG cap for individual customers in 2009. However, low-cost versions of the Emotiv headsets omit a proper electrode location for P300-based BCIs, while research-grade versions still practice dissuasive pricing. Similar concerns apply to most customer-grade EEGs (Ahn et al., 2014; Vasiljevic and Miranda, 2019). That is, cheap materials only include a few and unmovable channels which are inappropriate for BCI based on visual stimulation, whereas medical-grade EEG caps are expensive. In 2013, OpenBCI (New York, US) finished a successful fundraising campaign, with the aim of providing a high-quality EEG acquisition system for <1,000 euros. Nevertheless, concerns were raised about the usability of the technology, such as its association with unstable wireless communication, non-standardized sampling rate and use of gel electrodes (e.g., Chabance et al., 2019). Some researchers (e.g., Yohanandan et al., 2018) demonstrated that similar performance could be obtained with an in-house, and thus cheaper, EEG headset. A Huffington Post (New York, US) publication presented an EEG headset that was ergonomic (i.e., with dry electrodes and wireless) for <500 euros6,7. However, such headsets are not available for public use, in the sense that, even though some are open-source, developers cannot build them in practice.
Another consideration is that any development is material-dependant, as all are based on different hardware, drivers and protocols (e.g., Ahn et al., 2014; Pierce et al., 2020). Despite the lack of standards, commercial brands such as Emotiv (San Francisco, the US) or NeuroSky (San Rose, the US) have developed plugins for game development. For instance, Rosca and Leba (2019) developed a pool game by integrating the Emotiv SDK with Unity 3D (San Francisco, the US), a notorious game engine. A review of existing BCI software can be found in Pierce et al. (2020). That said, the lack of middleware supported by a large and independent community creates maintenance and portability issues, as long as plugins are constructor-dependent or rely on the willingness of a research team.
Lack of Game Design and Graphics
Graphics and game design are a key concern and expectation for games on the market, from AAA to indie games. This is why video game studios hire developers as well as graphic designers, art work designers, concept artists and game designers. However, graphics and game design have been underestimated in the development of BCI games in laboratories. For instance, the number of frames per second (FPS) is a major concern for stimulation-based BCI, as they require the exact onset of the stimulation with a precision of around ±2 ms (Andreev et al., 2016). Indeed, if the tagging of such stimulation on the ongoing EEG does not happen all the time in the same frame in which the stimulation is displayed, a jitter is observable, which varies as an inverse function of the frame rate. For example, an FPS of around 50 Hz will output a frame every 20 ms and generate a jitter of a similar amplitude in the worst cases. This is particularly true for VR games where low FPS can result in a higher jitter (Cattan et al., 2018a). However, to our knowledge, the impact of graphical quality in FPS resulting in jitters is poorly understood in the context of BCI entertainment.
Ahn et al. (2014) conducted an opinion survey on the importance of BCI games elements. One aspect of this study was to outline the difference of perspective between developers and researcher communities regarding BCI development. For example, easiness of playing was one of the most important elements for 58% of the developers but only 19% of the researchers. However, the importance of graphics was minimized in the study when it should be a major concern for video games. In fact, the authors reported a developer opinion (later confirmed by the opinion survey for the two communities) that aesthetic is rather not considered as one of the most important factors for BCI games. However, we believe this only means that graphics are a basic requirement for video games. In Schell (2014), a pillar reference for game design, the author said about one of his work experiences in virtual reality:
“We had to make things look beautiful. […]. We used high-end graphics hardware and rich textures and models […].”
In general, laboratory BCI games demonstrate good design, being turn-based and having slow gameplay—in this aspect following the guidelines stated by the scientific community (e.g., Nijholt et al., 2009; Marshall et al., 2013). For example, Lotte et al. (2008) and Andreev et al. (2016) presented the BCI game Use the force!, which consisted of lifting up a vessel with motor imagery. Meanwhile, in the game Alphawow (van de Laar et al., 2013), the avatar's character changes its behavior according to the player's state of relaxation. These two games consider the low-transfer rate of the BCI by mapping it to a feature that is expected to fail from time to time (it can be agreed that using the mind to move objects is difficult to realize) and to not compete with traditional inputs (a keyboard cannot determine a person's relaxation state). Nevertheless, the use of BCI was restricted to a unique aspect of these games. On one hand, if this aspect is a side aspect (e.g., Alphawow), the use of BCI is not valuable because of its cost. On the other hand, if this aspect is the main aspect of the game, it means that the player's ability to finish the game depends on an unreliable input and thus leads to frustration. A subjective study on the use of BCI for gaming (Cattan et al., 2019) also reported a lack of feedback for error quantification from participants. This couples with the fact that around 20% of the users are not proficient using a typical BCI. In fact, BCI illiteracy is an issue which is also well-established in the existing literature (e.g., Nijholt et al., 2009; Allison and Neuper, 2010; Marshall et al., 2013; Vasiljevic and Miranda, 2019), although the idea that physiological traits are responsible for BCI illiteracy is controverted (Thompson, 2019; Riquelme-Ros et al., 2020). These limitations impact replay and the difficulty of games, as players who do not succeed in a BCI task will become stuck in the game without knowing how to improve. In fact, except for a proof of concept or a contest, such as the game created by Mentalista (Paris, France), these games are not suitable for public.
Discussion and Conclusion
This section discusses BCI limitations and explains why overcoming these limitations to develop a ubiquitous BCI game is challenging. These difficulties include a plateau in performance compared with mechanical inputs, technical and algorithmic complexity behind BCI, a lack of middleware for BCI development and an underestimation of graphics and design complexity compared with games in the market.
Transfer rate is the most common limitation discussed in the literature. For example, Rashid et al. (2020) has argued that “most BCI games demonstrate very low accuracy and speed as compared to conventional interfaces, suggesting that there are issues that must be addressed to facilitate the acceptance of BCI games.” According to Cattan et al. (2018b) and Rashid et al. (2020), this is particularly true for games requiring movements, such as VR games, as muscular artifacts interfere with the detection of brain signals. In this regard, the complexity of signal detection and classification is, to our belief, a key obstacle to creating effective BCIs for use in games.
From our perspective, recent developments in non-metric (e.g., Quemy, 2019) or quantum classification (Grant et al., 2018; Havenstein et al., 2019) might lead to significant improvements in BCI acceptance and performance. Indeed, non-metric classification reduces the need for data pre-processing and engineering, while quantum classification takes advantage of quantum physics to improve the speed and accuracy of classification. The emerging field of quantum machine learning has become increasingly mature thanks to the availability of open-source toolkits (e.g., Abraham et al., 2019) and cloud-based quantum machines (such as the IBM quantum experience by IBM, Armonk, US). Havenstein et al. (2019) showed the advantages of using a quantum vs. classical support vector machine for multi-class classification. Further interesting developments are expected in this field in the next decade.
Nevertheless, in our opinion, the impact of a low-transfer rate on BCI games is overestimated. In practice, if BCIs are considered an interesting yet dispensable device for video games, this is mostly due to design issues because BCIs are either used as an ancillary feature of games (despite requiring expensive materials) or as a means of competing with traditional inputs (e.g., keyboards and mice) to achieve the same task faster and with less concentration. In other words, BCIs in games should be limited to a set of aspects that cannot be achieved by traditional inputs, but at the same time should create sufficient value to justify the cost and encumberment of the material. However, despite some of the positive features previously enumerated (e.g., the use of BCIs for naturally imprecise behavior), a complete game concept that can be sold for concrete video game entertainment is lacking. Further, design reflection for BCI games is still in its infancy and is close to the prototypal-use cases created in the early 2000s (e.g., Bayliss and Ballard, 2000). Similar concerns were broached in recent studies, such as Vasiljevic and Miranda (2019) which reported that only a few studies focused on qualitative aspects of the interaction with BCI games.
In short, developers should above all be concerned with game design and game portability. In practice, BCI games cannot be downloaded and run independently of the EEG acquisition system, which is an impediment for both researchers and game developers. In this respect, the work achieved by platforms such as OpenVibe (Renard et al., 2010) or open-source initiatives that rely on a standard protocol, such as Lab Streaming Layer (Stenner et al., 2015), should be emphasized (see text footnote 7). Nevertheless, there is no support for developers (at any level, technologic, mathematical, usability) to integrate BCI in concrete game production. To our knowledge, practical obstacles such as compilation and integration of LSL DLL into game engines, along with the lack of command-line support or synchronization solutions between OpenVibe acquisition server and recreational applications, are rarely mentioned in the existing literature.
Author Contributions
The author confirms being the sole contributor of this work and has approved it for publication.
Conflict of Interest
GC was employed by the company IBM.
Footnotes
1. ^https://mentalista.fr/foot
2. ^https://www.thequint.com/explainers/what-is-neuralink-and-how-does-it-work-explained
3. ^https://www.guinnessworldrecords.com/world-records/fastest-realtime-court-reporter-(stenotype-writing)/
4. ^https://www.vrfocus.com/2020/05/why-now-is-the-time-for-aaa-studios-to-consider-vr/
5. ^https://labusinessjournal.com/news/2015/jun/17/independent-virtual-reality-studios-benefit-early-/
References
Abraham, H., AduOffei Agarwal, R., Akhalwaya, I. Y., Aleksandrowicz, G., Alexander, T., et al. (2019). Qiskit: An Open-source Framework for Quantum Computing. doi: 10.5281/zenodo.2562110
Ahn, M., Lee, M., Choi, J., and Jun, S. (2014). A review of brain-computer interface games and an opinion survey from researchers, developers and users. Sensors 14, 14601–14633. doi: 10.3390/s140814601
Allison, B., and Neuper, C. (2010). “Could anyone use a BCI?,” in Brain-Computer Interfaces: Human-Computer Interaction Series, eds D. Tan and A. Nijholt (London: Springer), 35–54. doi: 10.1007/978-1-84996-272-8_3
Andreev, A., Barachant, A., Lotte, F., and Congedo, M. (2016). Recreational Applications of OpenViBE: Brain Invaders and Use-the-Force. Hoboken, NJ: John Wiley; Sons. Available online at: https://hal.archives-ouvertes.fr/hal-01366873/document (accessed December 11, 2020).
Andreev, A., Cattan, G., and Congedo, M. (2019). Engineering Study on the Use of Head-Mounted Display for Brain-Computer Interface. GIPSA-Lab. Available online at: https://hal.archives-ouvertes.fr/hal-02166844 (accessed August 2, 2019).
Barachant, A., Bonnet, S., Congedo, M., and Jutten, C. (2012). Multiclass brain-computer interface classification by Riemannian geometry. IEEE Trans. Biomed. Eng. 59, 920–928. doi: 10.1109/TBME.2011.2172210
Barachant, A., and Congedo, M. (2014). A Plug&Play P300 BCI using information geometry. ArXiv14090107 Cs Stat. Available online at: http://arxiv.org/abs/1409.0107 (accessed December 11, 2020).
Bayliss, J. D., and Ballard, D. H. (2000). A virtual reality testbed for brain-computer interface research. IEEE Trans. Rehabil. Eng. Publ. IEEE Eng. Med. Biol. Soc. 8, 188–190. doi: 10.1109/86.847811
Cattan, G., Andreev, A., Maureille, B., and Congedo, M. (2018a). Analysis of Tagging Latency When Comparing Event-Related Potentials. Grenoble: Gipsa-Lab; IHMTEK. Available online at: https://hal.archives-ouvertes.fr/hal-01947551 (accessed December 11, 2020).
Cattan, G., Mendoza, C., Andreev, A., and Congedo, M. (2018b). Recommendations for integrating a P300-based brain computer interface in virtual reality environments for gaming. Computers 7:34. doi: 10.3390/computers7020034
Cattan, G. H., Andreev, A., Mendoza, C., and Congedo, M. (2019). A comparison of mobile VR display running on an ordinary smartphone with standard PC display for P300-BCI stimulus presentation. IEEE Trans. Games 1. doi: 10.1109/TG.2019.2957963. [Epub ahead of print].
Chabance, M., Cattan, G., and Maureille, B. (2019). Implementation of a Daemon for OpenBCI. IHMTEK. Available online at: https://hal.archives-ouvertes.fr/hal-02091644 (accessed August 3, 2019).
Congedo, M. (2013). EEG Source Analysis. Available online at: https://tel.archives-ouvertes.fr/tel-00880483 (accessed December 11, 2020).
Debener, S., Minow, F., Emkes, R., Gandras, K., and de Vos, M. (2012). How about taking a low-cost, small, and wireless EEG for a walk? Psychophysiology 49, 1617–1621. doi: 10.1111/j.1469-8986.2012.01471.x
Ferreira, A. L. S., de Miranda, L. C., de Miranda, E. E. C., and Sakamoto, S. G. (2013). A survey of interactive systems based on brain-computer interfaces. J. Interact. Syst. 4, 3–13. doi: 10.5753/jis.2013.623
Grant, E., Benedetti, M., Cao, S., Hallam, A., Lockhart, J., Stojevic, V., et al. (2018). Hierarchical quantum classifiers. Npj Quantum Inf. 4, 1–8. doi: 10.1038/s41534-018-0116-9
Havenstein, C., Thomas, D., and Chandrasekaran, S. (2019). Comparisons of Performance between Quantum and Classical Machine Learning. SMU Data Science Review, 1. Available online at: https://scholar.smu.edu/datasciencereview/vol1/iss4/11 (accessed December 11, 2020).
Kaplan, A. Y., Shishkin, S. L., Ganin, I. P., Basyul, I. A., and Zhigalov, A. Y. (2013). Adapting the P300-based brain-computer interface for gaming: a review. IEEE Trans. Comput. Intell. AI Games 5, 141–149. doi: 10.1109/TCIAIG.2012.2237517
Kerous, B., Škola, F., and Liarokapis, F. (2018). EEG-based BCI and video games: a progress report. Virtual Real. 22, 1–17. doi: 10.1007/s10055-017-0328-x
Lotte, F., Renard, Y., and Lécuyer, A. (2008). Self-Paced brain-computer interaction with virtual worlds: a quantitative and qualitative study “Out of the Lab,” in 4th international Brain Computer Interface Workshop and Training Course (Graz: Graz University of Technology). Available online at: https://hal.inria.fr/inria-00304340 (accessed December 11, 2020).
Marshall, D., Coyle, D., Wilson, S., and Callaghan, M. (2013). Games, gameplay, and BCI: the state of the art. IEEE Trans. Comput. Intell. AI Games 5, 82–99. doi: 10.1109/TCIAIG.2013.2263555
Nijholt, A., Plass-Oude Bos, D., and Reuderink, B. (2009). Turning shortcomings into challenges: brain-computer interfaces for games. Entertain. Comput. 1, 85–94. doi: 10.1016/j.entcom.2009.09.007
Pierce, S., S, C. C., Marvin, A., Anton, N., and E, G. J. (2020). Brain-computer interface software: a review and discussion. IEEE Trans. Hum. Mach. Syst. 50, 101–115. doi: 10.1109/THMS.2020.2968411
Quemy, A. (2019). Binary classification in unstructured space with hypergraph case-based reasoning. ArXiv180606232 Cs Stat. Available online at: http://arxiv.org/abs/1806.06232 (accessed August 29, 2020).
Rashid, M., Sulaiman, N., Abdul Majeed, A. P. P., Musa, R. M., Ab. Nasir, A. F., Bari, B. S., et al. (2020). Current status, challenges, and possible solutions of EEG-based brain-computer interface: a comprehensive review. Front. Neurorobotics 14:25. doi: 10.3389/fnbot.2020.00025
Renard, Y., Lotte, F., Gibert, G., Congedo, M., Maby, E., Delannoy, V., et al. (2010). OpenViBE: an open-source software platform to design, test, and use brain–computer interfaces in real and virtual environments. Presence Teleoperators Virtual Environ. 19, 35–53. doi: 10.1162/pres.19.1.35
Riquelme-Ros, J.-V., Rodríguez-Bermúdez, G., Rodríguez-Rodríguez, I., Rodríguez, J.-V., and Molina-García-Pardo, J.-M. (2020). On the better performance of pianists with motor imagery-based brain-computer interface systems. Sensors 20:4452. doi: 10.3390/s20164452
Rosca, S., and Leba, M. (2019). Design of a brain-controlled video game based on a BCI system. MATEC Web Conf. 290:01019. doi: 10.1051/matecconf/201929001019
Schell, J. (2014). The Art of Game Design: A Book of Lenses, Second Edition. Natick, MA: A K Peters (CRC Press).
Stenner, T., Boulay, C., and Medine, D. (2015). LabStreamingLayer. Swartz Center for Computational Neuroscience. Available online at: https://github.com/sccn/labstreaminglayer (accessed December 11, 2020).
Thompson, M. C. (2019). Critiquing the concept of BCI illiteracy. Sci. Eng. Ethics 25, 1217–1233. doi: 10.1007/s11948-018-0061-1
van de Laar, B., Gürkök, H., Bos, D. P.-O., Poel, M., and Nijholt, A. (2013). Experiencing BCI control in a popular computer game. IEEE Trans. Comput. Intell. AI Games 5, 176–184. doi: 10.1109/TCIAIG.2013.2253778
Vasiljevic, G., and Miranda, L. (2019). Brain-computer interface games based on consumer-grade EEG devices: a systematic literature review. Int. J. Hum. Comput. Interact. 36, 105–142. doi: 10.1080/10447318.2019.1612213
Vos, M. D., Kroesen, M., Emkes, R., and Debener, S. (2014). P300 speller BCI with a mobile EEG system: comparison to a traditional amplifier. J. Neural Eng. 11:036008. doi: 10.1088/1741-2560/11/3/036008
Wolpaw, J., and Wolpaw, E. W. (2012). Brain-Computer Interfaces: Principles and Practice. New York, NY: Oxford University Press.
Keywords: brain–computer interfaces, games, virtual reality, electroencephalography, perspectives
Citation: Cattan G (2021) The Use of Brain–Computer Interfaces in Games Is Not Ready for the General Public. Front. Comput. Sci. 3:628773. doi: 10.3389/fcomp.2021.628773
Received: 12 November 2020; Accepted: 02 March 2021;
Published: 24 March 2021.
Edited by:
Anton Nijholt, University of Twente, NetherlandsReviewed by:
Leonardo Cunha de Miranda, Federal University of Rio Grande do Norte, BrazilGabriel Vasiljevic, Federal University of Rio Grande do Norte Natal, Brazil, in collaboration with reviewer LM
Marvin Andujar, University of South Florida, United States
Copyright © 2021 Cattan. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
*Correspondence: Grégoire Cattan, gregoire.cattan@ibm.com