Electromyography (EMG) is a technique used to measure and record the electrical activity generated by the muscles (Athavale and Krishnan, 2017). EMG has been widely used for rehabilitation and for control of prosthetic limbs, and also as input control for gaming, primarily by developers and researchers. A popular commercial EMG device, the MYO armband (Figure 1A), has been used for research purposes and has also been a hacker’s favorite in finding novel uses for EMG in nonmedical fields.

Electroencephalography (EEG) is the measuring of electrical signals in the brain, through the use of electrodes that are placed on certain points on the skull, to assess the activity of specific regions of the brain. Brain–computer interfaces (BCIs) are EEG devices that communicate between the neural pathways and a computer. These have been developed to decode brain signals and translate them into a computer program (Abdulkader et al., 2015). Researchers are able to perform tests using EEG, to understand how the brain responds to certain stimuli, such as music, or how the brain behaves in a person experiencing sleep disturbances or epilepsy. These devices have made the process of retrieving the brain’s electrical signals into a less complex process so that even non-researchers can now use them to interact with their brainwaves.

There are several consumer-level EEG headsets on the market such as the Muse headband, ¹ the Emotiv Insight, ² the EPOC, ³ the Imec EEG ⁴ headset (Figure 1B), or the Galea by Open BCI ⁵ (to be released in 2022). These headsets tend to come with proprietary software, which allows the user to calibrate the device easily, provides artifact filtering, and data readouts for separate brainwaves, brain regions, and emotions. In counterpart, due to this proprietary nature, headsets such as Emotiv and Muse cannot be used with external software, without much revision and hacking into the company software. Devices such as the IMEC headset can be mounted within a VR headset, and use flexible, polymer dry electrodes. Although researchers prefer to use headsets that use wet electrodes (electrodes that are placed with conductive paste for better signal transmission), an analysis of the electrodes used in the IMEC headset showed no viable differences in signal when tested against conventional wet electrodes (Chen et al., 2014).

BCIs use two main methods for measuring responses in brain activity. Using the method of steady-state-evoked visual potentials (SSEVP), a visual stimulus occurs at a different constant frequency (Allison, et al., 2008). A stimulus that is presented at a frequency of 6–8 Hz is more likely to elicit an SSEVP (Müller and Hillyard, 2000). Through the use of a BCI that uses SSEVP, the EEG activity is measured at a certain frequency to measure the intent of the user. Examples of steady-state methods can be seen in the studies mentioned in this article.

An event-related response is when small electrical changes result from a stimulus change (Blackwood and Muir, 1990). The stimulus could be cognitive, sensory, or motor (Abdulkader et al., 2015). Examples of event-related response can be seen in examples in which the researchers monitor biological changes from specific changes in the stimuli, such as a change in the tension or music.

Electrooculography (EOG) and eye tracking are two methods for measuring the point of gaze or the motion of an eye relative to the head. EOG tracks ocular activity through the use of sensors that are placed around the eyes, in a similar manner as EMG electrodes. These sensors record the corneo-retinal potential difference between the positive and negative sides of the eyeball (Bulling et al., 2009). Alternatively, eye tracking uses a camera and image analysis for measuring the point of gaze or the motion of the eye. Eye tracking has become integrated into consumer devices such as the VIVE Pro Eye headset ⁶ and Alienware laptops, and companies such as Vive and Tobii ⁷ (Figure 1C) market eye tracking to companies as a tool for gathering analytical information to use for their businesses. Eye tracking can record where a potential customer may look or keep her attention during an advertisement or shopping trip, or how immersed a person may be during a task such as playing a game. Researchers, on their side, have implemented eye tracking to study many problems, including people with gambling (Murch et al., 2019) and gaming disorders.

Electrodermal activity (EDA) is a measure of the electrical activity or resistance level of the skin. It is a physiological response that is activated by the sympathetic system (Dawson et al., 2017). The skin’s electrical resistance typically changes during moments of arousal or physical activity, such as exercise, nervousness, or fear response. While we notice when our heart is beating quickly, or we may even perceive at moments of panic or nervousness an elevated amount of sweat in the palms, small changes in our sweat or skin conductance are not noticeable to us. Minute changes in skin conductance require the use of EDA sensors. One advantage of EDA is that it is less sensitive to noise in comparison with EMG and EEG, which receive a lot of noise artifacts from muscle and body movements (Kivikangas et al., 2011).

ECG is the measure of the electrical activity generated by the heart. Similar to EMG or EOG, the heart rate is measured using three electrodes that are stuck onto the skin of the chest over the heart. Heart rate changes during exercise and physical activity, during times of emotional duress, and bodily fatigue (Chęć et al., 2015). ECG can be used to measure stress and ECG changes in a player during gameplay (Porter et al., 2019). When measuring emotions and behavioral responses, ECG is often coupled with another sensor, usually EDA or EEG.

In some studies, researchers have used devices with biosensors such as EEG or EMG connected to a game, to aid persons with learning how to gain use of muscles or to test anticipation of movement for those with muscle impairment (Armiger and Vogelstein, 2008; Norman et al., 2016; Rincon et al., 2016; Prahm et al., 2017). In other uses, a researcher can, for instance, monitor in real time how players react to a game by measuring their attention levels or immersion level (Nacke and Lindley, 2008), stress (Karthikeyan et al., 2011), or panic levels (Russoniello et al., 2009; Wang et al., 2014).

Researchers have used EMG for designing experiences to train individuals with physical disabilities, to aid in their recovery (Armiger and Vogelstein, 2008; Converse et al., 2013; Hayes et al., 2013). One study explored the feasibility of using an EMG device built with commercial off-the-shelf sensors and placed into an arm sleeve, for physical rehabilitation (Visconti et al., 2018; Converse et al., 2013; Hayes et al., 2013) (Table 1, row 11). The surface sensors were placed over the forearm just below the elbow. Two simple games were created for the experiment: a tower defense game ⁸ and a Pong-style game. EMG is also being used to build and test assistive technology for those in need of prostheses. One group of researchers modified a Guitar Hero controller with EMG sensors to allow amputees to play the game (Armiger and Vogelstein, 2008) (Table 1, row 4). Another group looked to broaden the use of a prosthetic limb to have more input options to a game (Vujaklija et al., 2017) (Table 1, rows 57). For this, they used a prosthetic forearm and hand called the Ottobock Myoboy. ⁹ The aim of the study was to use gaming as a method of teaching users how to properly control a new prosthesis and stay motivated. The outcome showed that this method of training prosthesis users was efficient in training separate muscle movements and endurance. Studies such as these demonstrate methods of keeping those in rehabilitation interested and motivated while improving their physical state.

One related example of commercial use of an EMG device was produced in 2019 by the French company Perifit, ¹⁰ which released a device that aids new mothers in strengthening their pelvic muscles in postnatal recovery, as well as women who suffer from bladder control issues. To motivate these users to do the exercises and to do them correctly, the company provides several games with their applications, where the user interacts with them by using the device as a controller (Table 1, row 12). These games change exercises at each level, depending on which recovery program the user has chosen.

EMG can also act as a noninvasive and reliable signal for measuring emotions. Soares et al. (2017) employed a combination of EMG and EDA signals for reading emotions during a playing experience (Table 1, rows 6, 27). They placed EMG sensors on the facial muscles for tracking facial expressions to pinpoint positive or negative valence. For sensing EMG and EDA, these researchers used a Bitalino. The Bitalino ¹¹ (Figure 1D), from the Portuguese company Plux, is a small board that fitted with the corresponding sensors, can be used for measuring multiple simultaneous biosignals, including EMG, EDA, EEG, and ECG. It has become popular among researchers and hobbyists, due to its open-source platform, its affordability, and its reliability.

Whereas the aforementioned use of EMG for measuring emotions is still scarce, the use of EEG for the same purpose is much more widespread. Numerous studies have been done in monitoring EEG signals during gameplay, to analyze the emotional mindset of a player during game interaction. One direct benefit of measuring emotions during gameplay is to use the biological data to change the difficulty or experience of the game (Carofiglio et al., 2019).

These devices show great possibilities, not just for analysis of emotional states and adaptive gaming, but can also provide opportunities for health-related research and for severely disabled persons to have a means of interacting with video games. Researchers have studied applications of using BCIs in games to search for new interventions for people with Autism and Autism Spectrum Disorder (Friedrich et al., 2014). One study used a BCI to test the possibility of removing attention impairments from children with autism (Mercado et al., 2019). In this study, the use of a BCI with a video game improved the attention level of the children with two showing no attention issues post-study. While researchers are using BCIs to aid those with an emotional disorder or other disabilities, there still remains a gap in testing the possibilities in BCI gaming with persons with severe physical disabilities.

Relaxation and meditation has become a popular topic in research as well as at a commercial level. In a seminal experiment, Hjelm and Browall (2000) created a game using a BCI to test the alleviation of stress (Table 1, row 2). This experiment involved two gamers “competing” to become relaxed by moving a metal ball with their alpha and beta signals. More recently, Ahn et al. (2014) targeted the brain’s alpha signals for an adapted version of World of Worldcraft (Table 1, row 19). In this experiment, the user could control the game through conventional input as well as affect changes in the game through their alpha brain signals. The company Neurosky, ¹² which produces BCIs, launched a game entitled Neuroboy (Table 1, row 17), to accompany their headset in 2007. As of 2020, the company has released numerous applications and games either for free or purchase, such as Neuro Tower Defense (Table 1, row 18), a strategy game that also tests the brain to remain calm during the task of defense. Neurosky is not the only commercially available headset that provides its own applications to go along with its BCIs. Other headsets such as Muse are in fact marketed with the main purpose of reducing the stress of the user.

Widely used in commercial and marketing studies, eye gaze is still not as commonly employed in gaming as other biosignals. It is mostly used as an implementation for hands-free gaze-based interaction, which has clear applications in Virtual Reality (VR), as well as for allowing those with severe motor disabilities to interact with video games and other applications. In the case of using VR, the game VRailSurfer was developed and participants played it by controlling the direction of the character and avoiding obstacles through gaze and eye movement (Kumar and Sharma, 2016) (Table 1, row 23). The technique called dwell-time-based selection (Isokoski et al., 2009) uses visual fixation for allowing the eyes to act as a “mouse emulator.” One group tested a Guitar Hero style game, evaluating where the eyes fixated (Vickers et al., 2010) (Table 1, row 25). The Eyeharp is a software-based musical instrument that permits people with disabilities to play music with gaze or with head movements, allowing them to play melodies simply by looking at the notes on the screen (Vamvakousis and Ramirez, 2016). On a different line, Balasubramanian and Adalarasu (2007) combined EOG with EEG for measuring the onset of fatigue while users played a driving game (Table 1, row 24).

This eye-tracking technology has already been implemented commercially by companies such as Tobii and HTC Vive. The gaming giant Ubisoft worked with the Tobii-tracking software and hardware (Figure 1C) to include this capability into the PC versions of more than 130 of their games. Eye tracking is also used in a process called foveated rendering, which tracks the eye and renders images in greater quality where the eye gazes. This allows for lower machine latency as the image can maintain optimum frame rate inside the focal area while leaving the area outside of view at lower quality when not being tracked. The VR headset HTC VIVE Pro Eye, released in 2019, has eye-tracking capabilities within the headset. Vive has also released an SDK for developers to easily integrate this into their game.

Several studies have shown that as the level of challenge or speed of a game increases, the skin conductance level also augments (Parsons and Reinebold, 2012) (Table 1, row 13). EDA correlates with the levels of nervousness or fear, the integration of this biosignal into gaming brings possibilities for adaptive games, such as introducing more challenges in a level depending on the user’s state of calm Chanel et al. (2008) used EDA (along with other biosignals) to measure the level of excitement, boredom, and anxiety in participants during gameplay of Tetris. The company Valve used skin response to measure players’ sweat levels during testing phases of the game Left 4 Dead 2 (Polygon, 2020). ¹³ This allowed them to measure stress levels and study and adjust the time that the objective should take to finish. EDA has also shown possibilities in therapeutic games, for instance in the treatment of phobias Kritikos et al. (2021) use EDA with users with arachnophobia in a Virtual Reality environment. The users were exposed to an environment with spiders and their skin response was measured and the intensity of the experience was controlled in real time. Researchers are also using physiological signals to solve issues in experiences, such as Virtual Reality sickness (Martin et al., 2018). In this study, EDA was used along with a measurement of cardiac activity to test a detection method for users who become motion sick within VR.

ECG is another biosignal often used in gaming for measuring emotional states during gameplay. Vachiratamporn et al. (2015) used EEG and ECG to monitor players’ emotional states during a horror game (Table 1, row 20). They reported that ECG was an effective means of predicting the user’s affect during pre-scary and post-scary moments. ECG and skin conductance were also used during an experiment in which players played the game Half Life 2 (Dekker and Champion, 2007) (Table 1, row 21). In this experiment, the researchers modified the game in several ways to react and change, depending on the readings, such as sound volume, shader effects (color changes), and rewards. Nonplayable characters would try to scare the player if they were reading too calmly. In 2016, the game company Flying Mollusk released a commercial game entitled Nevermind (Table 1, row 26), which is usable with consumer level heart rate monitors. The developers created the horror genre game with the purpose of creating a more frightening experience for the player, based upon their biological signals.

ECG monitoring during gameplay can also note the varying engagement levels of players during a game (Giakoumis et al., 2011). By monitoring the engagement level of players, game developers can adjust game difficulty (Dynamic Difficulty Adjustment) and algorithms to keep players interested (Kivikangas et al., 2011; Xue et al., 2019). The company Valve has long been researching gameplay using biosignals (Ambinder, 2011), and has even patented a method for capturing player biosignals in real time to affect gameplay (Bond and Ambinder, 2009).

Using ECG during gameplay can also be used for training of breath and staying calm during stressful situations. An experiment was done using members of the Canadian Forces with this purpose (Bouchard et al., 2012) (Table 1, row 22). The participants were given refresher training on Stress Management, which included breathing practices. The participants played an adapted version of a game while ECG and skin conductance were monitored.

Biosignal data can help researchers understand how a person interacts with a game—a technique that major companies have also employed to test game design and experience. Researchers can harness the biosignals of a player to provide real-time interaction and feedback in response to the player’s emotional state. This implementation in which the users’ behavior or emotional state directly affects gameplay is known as affective gaming (Kotsia et al., 2013). Affective gaming is a branch of affective computing, a type of computing that influences, arises from, or relates to human emotion (Picard, 1995). An affective program can receive input from gestures and biological signals. This is particularly helpful in gathering information in regard to learning in games (Cowley et al., 2013; Cowley and Ravaja, 2014), as well as understanding how emotions correlate to stimuli such as music (Bo et al., 2017; Suto and Oniga, 2018) or visuals (Suhaimi et al., 2018). By designing a game to be affective, a developer or an artist can tailor the experience to the person playing the game (Liu et al., 2009; Nacke et al., 2011).

The largest challenge for using biosignals on a massive scale in a game environment, be it for control, monitoring, affective gaming, or for providing some sort of biofeedback, is indeed the hardware equipment. In order for biosignals to be employed on a mainstream level, the leading companies such as Sony, Valve, or Microsoft would need to develop controllers with biosensors unobtrusive and easy to use, and game developers would need to create games that would integrate these sensors and make the games affective or provide some type of biofeedback.

As we have reviewed, the present level of market penetration varies considerably among the different biosignals, depending on several factors such as the potential applications and benefits that these biosignals can bring, and the cost, reliability, or unobtrusiveness of the related sensors. Eye-tracking technologies, for example, which offer interesting possibilities for gaze control, with clear applications in VR and for providing control mechanisms to people with severe motor disabilities, have already been implemented commercially by companies such as Tobii and HTC Vive.

Concerning EEG, although consumer level and noninvasive headsets are being produced by companies such as Muse or Emotiv, this still constitutes a niche market, mostly targeting mindfulness and relaxation. If EEG sensing and interpretation will probably have to wait some more years for offering more reliable and useful possibilities at the consumer level, EMG, EDA, and ECG on their hand, which provide physiological information much simpler to measure and interpret, would seem to be ready for affective gaming.

In 2020, Sony released a patent for the Dualshock 5 controller (Andall and Hogarth, 2018), which incorporates the use of physiological signal technology. The controller design includes EDA sensors and an input for possible integration of other biosignals such as heart rate. Will this convey the massive use of biosignals in commercial controllers, and bring affective gameplay to the mainstream? Even in the case of not making gaming affective, game companies would likely find it useful to have the possibility of collecting users’ biodata during gameplay to adapt gameplay in updates or future releases. Perhaps the proposed commercial use of biofeedback from skin conductance from Sony will integrate this feature of bio-adaptability into Playstation 5 games in the future. If implemented, this would be an advance for biofeedback as well as the first popular commercial release of this technology.