Abstract
A cluster of research in Affective Computing suggests that it is possible to infer some characteristics of users’ affective states by analyzing their electrophysiological activity in real-time. However, it is not clear how to use the information extracted from electrophysiological signals to create visual representations of the affective states of Virtual Reality (VR) users. Visualization of users’ affective states in VR can lead to biofeedback therapies for mental health care. Understanding how to visualize affective states in VR requires an interdisciplinary approach that integrates psychology, electrophysiology, and audio-visual design. Therefore, this review aims to integrate previous studies from these fields to understand how to develop virtual environments that can automatically create visual representations of users’ affective states. The manuscript addresses this challenge in four sections: First, theories related to emotion and affect are summarized. Second, evidence suggesting that visual and sound cues tend to be associated with affective states are discussed. Third, some of the available methods for assessing affect are described. The fourth and final section contains five practical considerations for the development of virtual reality environments for affect visualization.
Introduction
Virtual Reality (VR) systems offer endless possibilities for the development of interactive experiences. They are used for the development of tools in diverse areas such as rehabilitation therapy (), exergames (; ), and robotics (). Their potential is particularly promising when combined with technological advances in Affective Computing, allowing to interpret users’ affective states as computer commands (; ; ; ) and adapt the content of a virtual environment accordingly ().
Traditional psychological tasks for the treatment and diagnosis of mental disorders can be replaced by VR systems (; ; , ). These new tools are less time-consuming and provide more realistic environments, hence higher ecological validity. Furthermore, VR might be helpful for at least two types of therapy: exposure therapy and biofeedback therapy. Exposure therapy is commonly used to treat anxiety disorders caused by phobias. Patients are systematically exposed to the stimuli that trigger the phobia in a controlled environment and with a therapist’s guidance. VR is useful for exposure therapy because it allows delivering realistic experiences while providing control over the stimuli. Previous research suggests that exposure therapy in VR might be effective for the treatment of a least three types of phobias: social phobia (), claustrophobia (), and spider-phobia ().
Biofeedback therapy is used to provide real-time feedback to the patient about their physiological activity while they perform a task (J. ). The characteristics of the task depend on the purpose of the therapy. For example, developed a biofeedback game for training attention control in children with Attention Deficit Hyperactivity Disorder (ADHD). The player performed tasks in a virtual farm, such as collecting fruits and repairing a pathway. Participants were challenged to increase their concentration, impulsivity control, and sustained attention to do these tasks. Simultaneously, electroencephalography (EEG) signals were processed in real-time to identify EEG activity associated with attention state. The player obtained a game score if attention state was detected in the EEG signals.
Similarly, developed an interactive experience to enhance empathy using neurofeedback. The participant interacted with a fictional doctor that was going through a difficult situation. Simultaneously, the participant’s EEG signals were analyzed to estimate the affective response towards the doctor. If the system detected a positive affective response in the player, the storyline would change positively (the doctor would struggle less). It was expected that those changes in the story line would reinforce the supportive, empathetic behavior of the player.
Patients who lack affective self-regulation could benefit from VR biofeedback therapy to train affective self-control, fostering mood regulation (). conducted an experiment where twenty-three participants’ brain activity was analyzed in real-time using functional Magnetic Brain Imaging (fMRI). They asked participants to evoke a happy or sad memory and provided feedback about their affective state. The feedback was provided with a bar on a screen. The bar’s level increased when the fMRI data indicated that the participant successfully evoked the happy or sad memory. Results suggest that providing visual feedback allowed participants to learn how to modulate their neural activity. But it is not clear how to implement this finding in a therapeutic application that can be accessible for a large population because 1) fMRI is an expensive technology that is not accessible for most people, 2) participants must remain motionless for long periods during fMRI recordings; otherwise, the data is corrupted, and 3) an ideal therapeutic application should consist of engaging content that motivates users to use the system. These challenges could be solved by measuring brain activity with a less expensive and more portable method than fMRI, such as electroencephalography (EEG). The visual feedback could be provided using game-like elements.
The development of an affective visualization tool in VR would require at least two components: 1) A set of VR stimuli with affective content whose properties can be adjusted in real-time, and 2) a technique to continuously assess affective states in an online fashion without interrupting the VR experience. This literature review was elaborated to understand how to develop those components. Both requirements are addressed in three subsections. Firstly, theories related to emotions and affect are presented. Secondly, findings related to visual and sound cues that are associated with affective responses are analyzed. And thirdly, some of the most common methods for detecting affective states are summarized.
Theoretical Models of Emotion and Affect
The terms emotion and affect are often used interchangeably in the literature, but they are not exactly the same. There is not a general agreement about how to define these concepts. In this manuscript, emotions are defined as mental states that coordinate the operation of cognitive processes. This definition is based on the assumption that the human mind is designed as a computational system that consists of a series of information-processing programs (). Emotions are a particular type of program that coordinate other programs’ operation (). Affect is defined as the cognitive representation of the bodily changes that come with emotions (; ). Neither emotion nor affect can be directly observed or measured. However, the definition of affect is directly associated with the physiological states, while the definition of emotion is not. Therefore, it is reasonable to use electrophysiological signals to infer affective states, which might allow to infer some characteristics of emotional states.
Emotion Theories
The Ortony, Clore and Collins (OCC) theory of emotions () has been widely used in the field of computer science to model users’ emotional responses (e.g., ; ). This theory describes emotions in terms of twenty-two categories and assumes a clear distinction between each category. This approach is compatible with existing emotion recognition algorithms because these are usually based on categorizing emotions (e.g., ; ). According to the OCC theory (), the first step in an emotional response is the perception of the situation. Then the situation is evaluated (appraisal), and finally, the emotional response emerges. However, this theory does not consider the physiological changes associated with emotions.
Similarly, Robert Plutchnik proposed a structural model of emotions (), commonly known as Plutchnik’s wheel of emotions. This model consists of eight primary states: ecstasy, adoration, terror, amazement, grief, loathing, rage, and vigilance. According to Pluthnik’s theory, any emotion can be described as a combination of a subset of those basic states. Here emotions are defined as a sequence of reactions towards a stimulus. This sequence includes a cognitive evaluation of the stimuli (appraisal), feelings (subjective experience of the emotion), autonomic neural activity, and behavioral responses.
There are at least three other major emotion theories: the James-Lang Theory (
), the Cannon-Bard Theory (
;
), and the Schachter-Singer Theory (
). According to
, these theories have in common the assumption that emotional responses have four components but differ in the order those components take place during an emotional response. The components are:
• Appraisal: The cognitive, rationalized evaluation of the context where the emotional response is produced.
• Feeling: The subjective, momentary experience of the emotion.
• Physiological change: The bodily changes produced by the emotional response.
• Behavior: The observable conduct that comes with the emotion.
According to the , the first step in an emotional response is the cognitive evaluation of the situation. Then, physiological changes are produced in the body, at the same time that a behavioral response is produced. Lastly, feelings take place.
The Cannon-Bard Theory (; ) proposes that all the elements of an emotional response are independent of each other, and there is no particular order in which they occur. This theory is not compatible with the convincing amount of evidence indicating that emotional stimuli tend to trigger automatic changes in the body (e.g., ; ; ). Overall, these previous studies suggest interdependence between physiological changes and the other components of emotion.
According to the , physiological changes occur first. Then the user tries to find an explanation in the environment for those physiological changes. Depending on the explanation found, a cognitive label is assigned to the bodily changes perceived. Therefore, the physiological changes indicate the intensity of the emotional experience, but cognitive factors determine the emotion’s valence (pleasant vs unpleasant).
Theoretical Models of Affect
Theoretical models of affect can be classified into two major groups: discrete and dimensional models. Discrete models are based on a categorical division of affective responses, while dimensional models represent affect as an array of continuous variables. Both types of models are commonly used in Affective Computing to build affect recognition models (e.g., ; ; ).
In broad terms, discrete models propose the existence of a few primary states, such as happiness, sadness, and anger. Affective responses are a combination of a subset of those fundamental states. A prominent example of the influence of discrete models in psychology research can be found in an experiment conducted by in New Guinea. In this experiment, stories with emotional content were told to 153 participants. One hundred thirty of them (84.97%) had no previous contact with the western culture. After each story had been told, participants saw a series of pictures of facial expressions and were asked to choose the more coherent face with the story. Interestingly, participants associated similar facial expressions to the same stories, regardless of their cultural background. Based on this evidence, it was proposed that there are at least six facial expressions that are universal (i.e., they are not affected by culture): happiness, anger, sadness, disgust, surprise, and fear. These results are consistent with earlier contributions from Charles Darwin, who pointed out the existence of activation patterns in facial muscles which are associated with affective states (; ).
Dimensional models have their roots in the early contributions of Wilhelm Wundt, who proposed that affective responses have three dimensions: valence (pleasant–unpleasant), arousal (arousing–subduing), and intensity (strain–relaxation) (). On this basis, the Circumplex Model of Affect () was developed, representing affect in a two-dimensional space, where valence and arousal are equivalent to the x-axis and y-axis, respectively.
Other authors have proposed the Evaluative Space Model (ESM) (), which has three dimensions: Negativity in the x-axis, Positivity in the y-axis, and Net predisposition (to withdraw or approach a stimulus) in the z-axis. Unlike the Circumplex Model of Affect (), the ESM () contemplates the existence of affective responses with simultaneous negative and positive activation (“bitter-sweet” affective states). For example, while playing a terror video game, the user might feel fear, and at the same time, might feel excited because is aware that there is not a real danger. An analysis about dimensional models of affect can be found in .
The ESM proposes the existence of the negativity bias and the positivity offset. The negativity bias implies that negative activation produces more changes in the motivation to withdraw or approach stimuli than positive activation. Evidence supporting the existence of the negativity bias indicates that negative stimuli tend to produce more salient behaviors than positive stimuli (), and negative stimuli tend to be associated with higher arousal than positive stimuli (). The negativity bias suggests that terror video games should trigger higher arousal than video games associated with positive affective states. However, a recent study indicates that the arousal level triggered by terror video games is slightly lower than the arousal triggered by video games associated with positive affective states ().
The positivity offset implies a slight positive motivation to approach unknown stimuli in a neutral environment. This mechanism has been associated with humans’ natural tendency to explore new, unthreatening environments, even when that behavior is not associated with a reward (, p. 12). Further research about the positivity offset could help understand how to motivate VR users to explore virtual environments. For example, to stimulate engagement of players with VR games.
Visual and Sound Cues
Building a virtual environment for affective visualization requires content that any user can associate with a wide range of affective states, regardless of cultural differences or personal preferences. Therefore, this section presents recent studies suggesting an association between features of audio-visual elements and affective states. We aim to provide general guidelines for representing affect with audio-visual elements, rather than defining a set of rigid rules.
Visual Cues
Rounded objects are associated with higher valence and lower arousal than sharp objects (). And rounded lines are perceived as more attractive than straight or angular lines (; ). Given that attractiveness is associated with positive affective states, rounded lines are likely associated with positive valence. Additional studies suggest that visual complexity plays a role in the likability of objects. People tend to prefer extremely simple or extremely complex objects (). Given that likability tends to trigger positive valence (), an intermediate level of complexity is more likely associated with negative valence.
A cross-cultural study showed that the most critical factors in the affective meaning of color are brightness and saturation, while hue has a secondary role (). These results are consistent with evidence reported in but contrast with recent studies indicating that hue has a significant role in the affective state associated with a color palette (). Additional evidence suggests that blue, green, and purple are among the most pleasant hues, while yellow is among the most unpleasant. Green-yellow, blue-green, and green are the most arousing, while purple-blue and yellow-red are among the least arousing (). Similarly, it has been found that the most pleasant colors are those with higher saturation and brightness (; ). However, other studies suggest that there are not universal associations between colors and affective states. People tend to like colors associated with objects they like and dislike colors associated with objects they dislike (). Additional evidence indicates that color associations change according to the context where colors are used (), supporting the hypothesis that there are not universal associations between colors and affective states. Yet, it is possible to establish color palettes that allow to communicate affective states. For example, bright, unsaturated colors are more suitable to communicate calm, while dark, red colors are more suitable to communicate disturbance ().
Textures may influence the affective meaning of color (; ). This has been demonstrated by pairing colors with computer-generated textures and asking participants to rate the color-texture pairs using four scales: Warm-Cool, Masculine-Feminine, Hard-Soft, and Heavy-Light. Results suggest that texture significantly influences the evaluation on the Hard-Soft scale and has a minor impact on the other scales. However, this evidence does not allow to identify associations between particular texture patterns and affective responses.
Non-static visual elements have other visual properties besides color, shape, and texture. Some of these additional properties are speed, motion shape, direction, and path curvature. Fast-moving objects are associated with higher arousal than slow-moving objects (; ). But there are contradictory findings regarding the type of valence associated with speed. One study suggest that fast movements are related to positive affective states (), while other study indicates the opposite ().
Linear motion with straight paths is associated with low arousal and positive valence (). Jerky paths are associated with higher arousal than straight paths in linear motion (; ). But the curvature of the path has no incidence in affective associations when applied to spherical or radial motion (). Inward movements are related to more positive affective states than outward movements (). Downwards-right motion tends to be linked to positive states, while upwards-left motion tends to be associated with negative states (). In general, angular paths are related to more negative affective states than linear paths (). And spherical motion patterns tend to be associated with higher arousal than linear motion patterns.
Sound Cues
Previous research indicates that the location of a sound source influences the affective states associated with that sound. When the user cannot see where the object is (outside of the field of view), it is often associated with more arousing affective states than when the user can see it (inside the field of view) (; ). Similarly, sounds located further away in the space are related to less arousing responses (). The perception of an approaching sound is associated with more arousing responses than the perception of it moving away (). These phenomena are likely to be linked to mechanisms enforced by evolution (). Our primitive ancestors had more chances to survive if they were aware of the most potentially dangerous objects, such as those they could not see, were closer to them, or were approaching them.
The reverberation of the sound, which is associated with space’s size, can influence affective associations (). Lower reverberation (smaller rooms) is linked to more pleasant states than higher reverberation (larger rooms). Perhaps, because the primitive human being was better protected from predators in closed spaces, leading to an evolutionary process that favors the activation of attentional resources when we are in open areas.
Other studies indicate that asking people to rate pictures with affective content while listening to the sound of a heartbeat can influence their affective evaluations, as well as their heart rate (). Here, the sound of a heart rate faster than the listener’s one tends to increase their heart rate, while a slower sound seems to relax the listener’s heart rate. Therefore, playing a fast heartbeat in the background might be an effective way of representing an increase in arousal.
On the other hand, music is pivotal for affective visualization because it can contribute to create more immersive experiences. However, it is a vast topic that will not be fully covered in this manuscript. Yet, it is important to mention that tempo influences music’s affective perception (). Faster tempo tends to be associated with higher arousal ratings, while slower tempo tends to be associated with lower arousal ratings. To the extent of our knowledge, there is no evidence suggesting that tempo influences valence ratings.
Major and minor chords are associated with positive and negative affective states, respectively (). Similarly, dissonant harmonies tend to be strongly associated with anger, and to a lesser extent, with fear (). And it is possible to compose music based on people’s affective states (). However, it remains an open question whether it is feasible to do it in real-time, based on the user’s electrophysiological signals.
Personalized Affective Visualizations
There might be individual differences in the affective states that each user associates with the same audio-visual stimuli. These individual differences could be amplified as a consequence of personal experiences. An ideal system for affective visualization should account for those individual differences, delivering personalized visual representations of affective states, similar to .
developed an Augmented Reality (AR) system that automatically creates visual representations of the user’s affective states. The visualizations consisted of fractals generated using Procedural Content Generation (PCG). The visual properties of the fractals varied according to the affective state detected in the user. However, the evidence reported by does not allow to establish whether participants perceived that the fractals’ graphical properties represented their affective states.
Additional studies indicate that it is possible to use PCG to create content dynamically, adjusting it to the preferences of the user. This approach is known as experience-driven procedural content generation (EDPCG) (; ). In broad terms, EDPCG consists of an iterative process where the content is constantly modified based on the user’s feedback.
The general functioning of EDPCG is the same as an evolutionary algorithm (EA), which is an optimization process inspired by natural evolution. In a natural environment, the organisms that are better adapted to their habitat tend to have more reproductive success, hence more likely to pass their genes to the next generation. Similarly, objects can be created programmatically in a virtual environment and tested to identify the most successful ones. The criteria to identify which objects are more successful is based on a previously defined goal. This goal is defined by the developer based on the purpose of the application. During each iteration, the objects that are more successful at reaching the goal are identified. In the following iterations, new sets of objects are created, and the characteristics of the most successful objects tend to remain, whereas the characteristics of the least successful tend to disappear. It is assumed that repeating this process several times allows to reach the optimal parameters required to achieve the goal. For example, if the goal is to create personalized visual representations of positive affective states, and the EA detects that the user tends to associate red, rounded objects with positive affective states, the game would produce objects that would tend to be more red and more angular. An introduction to EA can be found in Eiben and Smith ().
Additional research indicates that it is possible to create automatically visual compositions in VR using Deep Convolutional Neural Networks (DCNN) (). Overall, the process consists of merging features from two images to create a third image. This approach could be combined with EDPCG (; ) to create personalized affective visualizations. The process would involve at least three steps: 1) Create a set of VR content that all users will observe and used that content as a baseline. This initial set of content could be developed following the guidelines described in Table 1; 2) Capture user feedback about the visual stimuli to establish the affective state that each user associates to each piece of VR content; And 3) use DCNNs to merge features of the initial content onto new, personalized VR content.
TABLE 1
| High arousal | Low arousal | High (positive) valence | Low (negative) valence | ||
| Static visual cues | Shape | Angular | Rounded | Rounded | Angular |
| Lines | Rounded | Straight | |||
| Hue | Green-yellow, blue-green, and green | Purple-blue and yellow-red | Blue, green, and purple | Yellow | |
| Saturation | N/A | N/A | Saturated | Unsaturated | |
| Brightness | N/A | N/A | Bright | Dark | |
| Visual complexity | N/A | N/A | Extremely complex or extremely simple | Neither complex nor simple | |
| Non-static visual cues | Speed | Fast | Slow | Some studies suggest that fast motion is associated with positive valence, while others suggest the opposite | |
| Motion shape | Spherical | Linear | N/A | N/A | |
| Direction | N/A | N/A | Downwards-right, inward | Upwards-left, outward | |
| Path curvature | Jerky | Straight | Jerky | Straight | |
| Sound cues | Source location | Outside field view | Inside field view | N/A | N/A |
| Distance to the sound source | Near | Far | N/A | N/A | |
| Sound source movement | Approaching | Receding | N/A | N/A | |
| Heartbeat sound | Fast (above 100 bpm) | Slow (below 60 bpm) | N/A | N/A | |
| Music | Tempo | Fast | Slow | N/A | N/A |
| Harmony | N/A | N/A | Major scale | Minor scale | |
Summary of audio-visual cues associated with affective states, according to previous studies.
Assessment of Affective States
Users’ feedback should be captured using methods that do not interrupt the VR experience, such as body movements (see Section Behavioral Measures) or electrophysiological signals, similar to . Methods for assessing affective states can be grouped into three categories: self-report questionnaires, behavioral measures, and electrophysiological signals. Each method has advantages and disadvantages that will be discussed below.
Self-Reports
Self-reports allow participants to evaluate their affective state by answering a series of questions. They can be used to verify the accuracy of the acquired information through other methods, such as behavioral and electrophysiological signals. Data collected through self-reports are often used as a ground-truth in the field of HCI.
In general, self-report measures are relatively easy to implement because they only require to display a series of questions on a paper sheet or a screen. Unlike behavioral and electrophysiological methods, self-reports are considered a direct measure because they allow asking participants directly about their mental states (). However, they are susceptible to be biased by rational processes. For example, participants who believe that it is expected from them to respond in a certain way might adjust their responses to fulfill that expectation, causing a phenomenon known as experimenter bias (). Some available tools for the assessment of affective responses are the Positive and Negative Affect Schedule (PANAS) (), Self-Assessment Manikin (SAM) (), and Pick a Mood (PAM) (). The PANAS (see Figure 1) consists of 20 words related to negative and positive feelings (ten negatives and ten positives). Participants use those words to report their affective state. Each word can receive a rating from one to 5.
FIGURE 1
The SAM () is an instrument that uses three scales: valence (pleasant/unpleasant), arousal (tension/relaxation) and dominance (inhibition/uninhibition). Each scale has five pictograms. Participants can select the blank spaces between each pictogram to indicate intermediate states. Therefore, answers to each scale can take values between one and 9 (see Figure 2). Given that this instrument is based on dimensions, it is compatible with dimensional models of affect. The SAM () is one of the most established instruments for assessing affect (over 7.000 citations) and has been used for the development of batteries of stimuli with emotional content, such as the International Affective Pictures System (IAPS) () and the DEAP dataset ().
FIGURE 2
On the other hand, the PAM () is based on discrete states. Therefore, it is compatible with discrete models of affect. This instrument also uses pictorial cues to assess participant’s states. There are eight mood types plus a neutral one: excited, cheerful, relaxed, calm, bored, sad, irritated, and tense. There are three characters for each of these states: a man, a woman, and a robot (gender-neutral character). In comparison to the SAM (), PAM’s characters () are more similar to a real human being (see Figure 1), which might be an advantage because it could be easier for participants to feel identified with the characters of the PAM ().
The PAM has been used to understand how to design objects and experiences that could stimulate mood regulation (), analyze the effect of immersive virtual environments on gaming Quality of Experience (QoE) (), and analyze whether the effect of color on affective states varies across different VR rooms (). Using paper questionnaires to analyze experiences in virtual environments might require to interrupt the VR experience. This limitation can potentially be counterbalanced by using subjective rating scales inside the virtual environment ().
Behavioral Measures
Behavioral measures allow inferring affective states from observable conducts, such as body movements (; ), voice patterns (; ), and facial expressions (). During an experiment conducted by , participants listened to a series of audio recordings with emotional content while their body movements were videotaped. Results suggested that sadness is associated with dropping the head while boredom is related to leaning the face in one hand. Building on that, recent research indicates that it is possible to infer arousal from body movements in virtual reality users (; ). In general, faster body movements are associated with higher arousal.
It is possible to automatically analyze users’ affective states based on their voice patterns (). Usually, a set of features are defined and used to build a classification model. Some of the features used for automatic speech emotion recognition are pitch, loudness, and tempo (; ). This approach is coherent with evidence suggesting that changes in vocalization patterns have an effect on the affective evaluation of speech, e.g. (; ).
Eye-tracking has been an essential measure of various individual states or even personality traits (). demonstrated recently how this measure could be easily obtained from modern smartphones using built-in system libraries (). The accuracy of this approach is comparable to other webcam or selfie-cam-based systems. However, having eye-tracking systems easily accessible in millions of devices opens up opportunities for remote or in-the-field studies with a much higher ecological validity than studies relying on heavy equipment traditionally used in laboratory investigations.
As mentioned in section Introduction. Theoretical models of affect, facial expressions are associated with affective states (). These expressions can be analyzed visually and described in terms of the Facial Action Coding System (FACS) (). The FACS is an instrument that describes all the possible movements of the facial muscles. Each movement is defined as an Action Unit (AU). Facial expressions can be described as a combination of a subset of all the Action Units defined in the FACS (). In a study conducted by , AUs were used for real-time analysis of the facial expressions of video streaming users. Additional studies suggest that human facial expressions can be collected using crowdsourcing techniques (), and its analysis can be optimized using statistical models that adapt automatically to the characteristics of the data (). However, facial recognition with camera sensors might be challenging to implement in VR because the Head-Mounted Display (HMD) covers the user’s face. Therefore, facial electromyography (fEMG), a technique introduced in the following section, might be more suitable for capturing VR users’ facial expressions ().
Electrophysiology
Electrophysiological methods allow measuring changes in the electrical potentials of the body. Usually, facial electromyography (fEMG), electrocardiography (ECG), and electroencephalography (EEG) are used to record facial muscle, heart, and brain activity, respectively. This section focuses on methods to infer emotions in terms of the Circumplex Model of Affect () (see Section Theoretical Models of Affect). Therefore, the focus is on techniques that can be used to infer valence and arousal. There are many approaches for affect detection using electrophysiological signals that are not based on the Circumplex Model of Affect () and are not included in this manuscript.
Arousal can be inferred from features extracted from ECG signals. The beat-to-beat intervals of the ECG signal (often referred to as RR-Intervals, RRI) are extracted, detecting its peaks and calculating the time lapse between each peak. These RRIs are used to analyze the heart rate variability (HRV). Prominent examples of time-domain features used to analyze HRV are the root mean square of successive differences (RMSSD) and the standard deviation of NN intervals (SDNN). It has been found that higher HRV is associated with higher emotional arousal (). It is possible to extract features from the ECG signal in the frequency domain by calculating the LF/HF ratio. The low-frequency component (LF) (0.04–0.15 Hz) is associated with parasympathetic activity, while the high-frequency component (HF) (0.15–0.4 Hz) is associated with sympathetic activity (). The activation of the parasympathetic system is associated with relaxation, and activation of the sympathetic system is associated with arousal. Therefore, increased activity in the HF component indicates higher arousal (). Further research has shown that it is possible to infer arousal from EEG signals in VR users employing long short-term memory (LSTM) recurrent neural networks (RNN) ().
A recent study compared the benefits of implementing HRV biofeedback in virtual reality with a traditional HRV biofeedback therapy (), suggesting that the VR implementation produces more benefits for users in terms of relaxation self-efficacy, reduced mind wandering, and control of attentional resources. A similar approach was proposed in , introducing a breathing biofeedback algorithm. This algorithm combines features extracted from electrocardiography activity with data inferred from diaphragm movements. The experiment was conducted using a chest band (Polar H10), which is a reliable, relatively inexpensive sensor. Results suggest that this approach can help to foster more regular and slower breathing in VR users.
Valence can be inferred from EMG and EEG signals. Previous evidence suggests that the Corrugator Supercilii muscle activity (located above the eyebrows) is associated with negative affective states. In contrast, the Zygomaticus Major muscle activity (located in the cheeks) is related to positive affective states (). Changes in facial muscle activity can occur without conscious awareness of the participant (; ). However, it might be challenging to implement EMG in a VR system because the pressure of the Head-Mounted Display (HDM) on the electrodes can create artifacts on the recorded signal.
Asymmetry in the cortical activity of the frontal cortex is also associated with valence. It has been found that positive and negative emotions are processed in the left and right frontal cortex, respectively (; ; ). Additionally, it has been found that cortical activity decreases as the alpha power (frequencies between 8 and 13 Hz) increases (). Therefore, increased processing of positive stimuli is associated with decreased alpha power in the left frontal cortex (higher activity in the left side of the brain). Similarly, increased processing of negative stimuli is associated with decreased alpha power in the right frontal cortex (higher activity in the right side of the brain) (; ; ).
These findings are coherent with results obtained by in a study where the brain activity of video game players was recorded using EEG. Participants were asked to report their affective state using the SAM () after the game session ended. Results indicated a positive correlation between self-reported valence and alpha asymmetry. Likewise, analyzed the brain activity of 32 participants who watched forty musical videos and rated their emotional reactions to each video using the SAM (). A positive correlation was found between self-reported valence and alpha power in the right occipital region of the brain.
Eye-movements and eye-blinks cause artifacts in the EEG signals and are usually reflected in the activity of the frontal region of the brain. In non-stationary VR applications, it is particularly challenging to remove artifacts caused by muscle activity, head movements, or electrical activity from the VR headset (). It is possible to remove these artifacts using Independent Component Analysis (ICA). This technique allows to identify the components of an EEG signal that are not produced by brain activity (). The maximum number of independent components (ICs) that can be identified using ICA depends on the number of electrodes used. For example, a recording with 32 electrodes will allow to identify up to 32 ICs. Therefore, increasing the number of electrodes might help identify the artifacts in the signal with more precision. For a complete analysis about using ICA in non-stationary and stationary settings, see .
An additional challenge is to process the EEG signals in real-time. ICA can be used in real-time (), but it was not designed for that purpose. An alternative is Artifact Subspace Reconstruction (ASR) (; ), a technique designed for online artifact removal. ASR uses data recorded from the user as a baseline. Then, principal component analysis (PCP) is applied to identify the EEG channels that contain artifacts. The data of the corrupted channels are reconstructed using the baseline data as a reference. There is software available that can facilitate the implementation of ASR, such as BCILAB (), OpenBiVE () and Neuropype (Intheon Labs, California).
Brain-Computer Interfaces
The implementation of electrophysiological signals in VR systems leads to the development of interfaces that allow interpreting users’ brain activity as computer commands (). One of the basic assumptions underlying the development of Brain-Computer Interfaces (BCIs) is that mental processes originate in the brain. But there are BCIs that measure electrophysiological responses in other places of the body (e.g., ), such as the heart and face, because processes that originate in the brain can produce changes in the activity of other body parts.
There are different techniques for measuring brain activity that can be used for the development of BCIs. For example, electrocorticography (ECoG), Positron Emission Tomography (PET), and functional Magnetic Resonance Imaging (fMRI), among others. However, electroencephalography (EEG) is the method most frequently used in BCIs because 1) it provides high temporal resolution (i.e., relatively large amount of data points recorded per second); 2) does not create health risks for the user because the electrodes can be easily placed and removed from the scalp; 3) can be portable, which is important for applications where the user is moving; and 4) is less expensive than most of the other methods ().
According to , there are three types of BCIs: active, passive, and reactive. Active BCIs require the active participation of the user to generate an action. For example, patients who lack motor control can use mental commands to move a wheelchair (Voznenko et al., 2018). Passive BCIs do not require the conscious involvement of the user. They can be used, for example, to analyze the cognitive load of car drivers automatically (Almahasneh et al., 2014). Reactive BCIs use mental activity that occurs as a response to external stimuli. An example is a neurofeedback video game where threatening stimuli are presented, and players have to control their anxiety to obtain game score (Schoneveld et al., 2016). A VR application for affective visualization, would usually involve either a passive or a reactive BCI.
The typical workflow in a BCI involves at least four steps (
;
):
1) Preprocessing pipeline: Filter out the signal’s noise and keep only the components that reflect brain activity. This process involves (but is not limited to) filtering frequency bands and removing artifacts caused by eye-movements or muscle activity. An introduction to signal processing can be found in .
2) Feature extraction: Isolate the information related to the psychological construct of interest based on previous neuroscience studies (see Section Electrophysiology).
3) Classifier definition: A classification model is created using prerecorded data. The classifier is tested offline, and an estimate of the accuracy of the classification is calculated. In general, classifiers are trained using data that has been previously labeled by humans. Machine Learning algorithms are used to identify patterns in the data that tend to be associated with each label.
4) Classification application: The classification is implemented in the BCI to perform online analysis of the brain activity. The outputs of the classification are used as computer commands.
Practical Considerations
This section contains five practical considerations that might help during the development of a VR system for affective visualization.
1) Which are the initial steps for designing a virtual environment? First, define who will use the virtual environment (target group) and what the user will do inside that environment. This will help to have a more clear idea about the interaction events that will occur during the experience. Look for other interactive experiences, such as games and art installations, that can serve as inspiration. This will trigger ideas and will help to understand how to implement them. Then, define the graphical layout of your environment (color palette, typographies, and textures).
2) Which software should I use for VR development? Unity is probably one of the best options. There are alternatives, such as Vizard, a virtual reality software for research. However, to the extent of our knowledge, Unity is the only game engine compatible with open-source solutions, such as LSL and Excite-O-Meter. Therefore, it is relatively easy and inexpensive to develop virtual environments that rely on the user’s physiological data using Unity.
3) How to integrate Unity with electrophysiological equipment? One possibility is to use LabStreamingLayer (LSL), a tool for collecting time-series data in experimental settings. Essentially, LSL allows to collect and synchronize the data and stream it into Unity. At the same time, it allows to send data from Unity (e.g., markers) to the signal processing software. Another option is to setup a UDP Broadcast to send information through your network.
4) Is there a ready-to-use solution for integrating Unity with electrophysiological equipment? Yes. Excite-O-Meter () is a Unity plugin for visualizing cardiovascular activity, which is built on top of LSL. It can be used to visualize Heart Rate Variability (HRV) (see SectionElectrophysiology). By default, the Excite-O-Meter provides a time-series graph of the data. But you can customize it to build other types of visualizations.
5) How to define the sampling rate for recording electrophysiological signals? According the Nyquist-Shannon sampling theorem, the sampling rate should be twice the maximum frequency of interest. For example, if you are interested in frequencies of up to 128 Hz, you should use a sampling rate of at least 256 Hz. A sampling rate of 256 Hz means that you are collecting 256 data points per second.
6) The usable information for each type of signal is located in a different frequency range. Therefore, the maximum frequency of interest for each signal is different. For example, the usable information in an ECG signal is up to 100 Hz. Therefore, the sampling frequency for ECG signals should be at least 200 Hz. However, previous studies indicate that ECG recordings at 200 Hz contain noise in the high-frequency components (). This noise can be reduced by recording at a higher sampling rate. Therefore, it is considered a good practice to record ECG signals at a sampling rate between 256 and 512 Hz, EMG signals at a sampling rate between 512 and 1024 Hz, and EEG signals at a sampling rate between 256 and 512 Hz.
Discussion
This manuscript aims to understand how to develop VR systems for affective visualization. These systems would involve the development of at least two components: a virtual environment and an affect detection technique. The development of both components requires the understanding of theories related to emotion and affect. Therefore, the manuscript analyses previous research related to 1) theories of emotion and affect, 2) audio-visual cues associated with affective states, and 3) methods for assessment of affective states.
Studies discussed in Section Visual and Sound Cues suggest that specific visual and sound cues can represent users’ emotions. However, most of these studies were conducted in experimental settings where the stimuli were carefully controlled. It is unclear whether the same psychological responses would occur if a combination of these cues were used simultaneously. For example, a particular combination of colors associated with positive states may result in an unbalanced visual composition that produces negative affective states. Or there might be motion patterns that are more prone to produce motion sickness in VR users, triggering negative states. Moreover, the novelty of a VR system in new users might bias the emotions they associate with the audio-visual stimuli.
Other studies mentioned in Section Visual and Sound Cues suggest that leftwards linear motion tends to be associated with negative valence (; ). This finding was obtained during experiments conducted in a western society, where time is represented as a progression to the right (). Therefore, it is likely that western users associate leftward motion with negative affective states because that type of motion is culturally associated with regressing in time. However, in other cultures, such as the Hebrew culture, people represent time as a progression to the left (). Therefore, it is possible that Hebrew users would associate leftward linear motion with positive valence.
Recent studies have demonstrated that affective states can be elicited by triggering psychogenic shivering (PS) (; ), using a device that controls the temperature in the upper back of the participants. Additional research indicates that the ability to be empathetic with others’ emotions can be influenced by delivering electrical stimulation in the vagus nerve (), and by inducing affective states in the observer through videos (). It remains an open question how to use those findings to develop Mixed Reality (MR) technologies for empathy enhancement, as proposed by .
Most of the existing techniques for inferring affective states from electrophysiological signals are based on a small number of discrete states e.g., (; ). But the amount of distinct affective states that can be detected using this approach is limited. Therefore, it might be convenient to formulate affect detection problems in terms of statistical regression. This approach would allow creating a model capable of describing affective states in terms of a continuum containing an infinite amount of distinct affective states. Previous studies suggest that it is possible to infer arousal from EEG signals () as a continuous variable. Future studies could investigate whether it is possible to use a similar approach to express valence in terms of a continuous variable.
Finally, it is possible to use a programmatic approach to create virtual reality content in real-time, using procedural content generation (PCG) (; ; ; ). PCG allows to create content dynamically that adjusts to user feedback. Electrophysiological signals could be used to capture user feedback without interrupting the VR experience. This approach would allow to create personalized virtual environments for emotion visualization, similar to or .
Statements
Author contributions
All authors contributed to the study conception and analysis. JG and WR contributed with analysis of data related to gaming and Virtual Reality. J-NV-A contributed to the analysis of data related to psychology and electrophysiology. RS contributed with the redaction of the manuscript and with data related to electrocardiography and eye-tracking. SM contributed to the analysis of data related to sound design and Machine Learning. AP performed the literature search, data analysis and wrote the first draft. All authors commented on previous versions of the manuscript.
Funding
This work was supported by the strategic partnership between the Technische Universität Berlin, Germany and the University of Technology Sydney, Australia.
Acknowledgments
We acknowledge the support of the German Research Foundation and the Open Access Publication Fund of TU Berlin. We appreciate the generosity of Nick Busietta in giving us access to the Psydocs of LiminalVR (liminalvr.com). That documentation was crucial for writing Section 2. Visual and Sound Cues of this manuscript.
Conflict of interest
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
Publisher’s note
All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.
Abbreviations
HCI, Human Computer Interaction; VR, Virtual Reality; fMRI, functional Magnetic Brain Imaging; EEG, electroencephalography; OCC theory of emotions, Ortony, Clore and Collins; ESM, Evaluative Space Model; PANAS, Positive and Negative Affect Schedule; SAM, Self-Assessment Manikin; PAM, Pick a Mood; IAPS; International Affective Pictures System; FACS, Facial Action Coding System; AU, Action Unit; HMD, Head–Mounted Display; EMG, Electromyography; ECG, electrocardiography; EEG, electroencephalography; RRI, RR-Intervals; HRV, heart rate variability; RMSSD, root mean square of successive differences; SDNN, standard deviation of NN intervals; LF/HF ratio, Low Frequency/High Frequency ratio; LSTMRNN, long short-term memory recurrent neural networks; ICA, Independent Component Analysis; ICs, independent components; ASR, Artifact Subspace Reconstruction; PCP, principal component analysis; BCIs, Brain–Computer Interfaces; ECoG, electrocorticography; PET, Positron Emission Tomography.
References
1
AntonsJ.-N.ArndtS.SchleicherR.MöllerS. (2014). “Brain Activity Correlates of Quality of Experience,” in Quality of Experience. Editors MöllerS.RaakeA. (Springer International Publishing), 109–119. 10.1007/978-3-319-02681-7_8
2
ArndtS.PerkisA.Voigt-AntonsJ.-N. (2018). Using Virtual Reality and Head-Mounted Displays to Increase Performance in Rowing Workouts. Proc. 1st Int. Workshop Multimedia Content Anal. Sports - MMSports’18, 45–50. 10.1145/3265845.3265848
3
AronoffJ. (2006). How We Recognize Angry and Happy Emotion in People, Places, and Things. Cross-Cultural Res.40 (1), 83–105. 10.1177/1069397105282597
4
AronoffJ.WoikeB. A.HymanL. M. (1992). Which Are the Stimuli in Facial Displays of Anger and Happiness? Configurational Bases of Emotion Recognition. J. Personal. Soc. Psychol.62 (6), 1050–1066. 10.1037/0022-3514.62.6.1050
5
BanseR.SchererK. R. (1996). Acoustic Profiles in Vocal Emotion Expression. J. Personal. Soc. Psychol.70 (3), 614–636. 10.1037/0022-3514.70.3.614
6
BarM.NetaM. (2006). Humans Prefer Curved Visual Objects. Psychol. Sci.17 (8), 645–648. 10.1111/j.1467-9280.2006.01759.x
7
BardP. (1934). On Emotional Expression after Decortication with Some Remarks on Certain Theoretical Views: Part I. Psychol. Rev.41 (4), 309–329. 10.1037/h0070765
8
BarrettL. F.Bliss‐MoreauE. (2009). “Chapter 4 Affect as a Psychological Primitive,” in Advances in Experimental Social Psychology (Elsevier), 41, 167–218. 10.1016/S0065-2601(08)00404-8
9
BartramL.PatraA.StoneM. (2017). Affective Color in Visualization. Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, 1364–1374. 10.1145/3025453.3026041
10
BelgerJ.KrohnS.FinkeC.TrompJ.KlotzscheF.VillringerA.et al (2019). “Immersive Virtual Reality for the Assessment and Training of Spatial Memory: Feasibility in Individuals with Brain Injury,” in 2019 International Conference on Virtual Rehabilitation. Tel Aviv, Israel: ICVR, 1–2. 10.1109/ICVR46560.2019.8994342
11
Bermudez i BadiaS.QuinteroL. V.CameiraoM. S.ChiricoA.TribertiS.CipressoP.et al (2019). Toward Emotionally Adaptive Virtual Reality for Mental Health Applications. IEEE J. Biomed. Health Inform.23 (5), 1877–1887. 10.1109/JBHI.2018.2878846
12
BlandónD. Z.MuñozJ. E.LopezD. S.GalloO. H. (2016). “Influence of a BCI Neurofeedback Videogame in Children with ADHD. Quantifying the Brain Activity through an EEG Signal Processing Dedicated Toolbox,” in 2016 IEEE 11th Colombian Computing Conference. Popayań, Colombia: CCC, 1–8. 10.1109/ColumbianCC.2016.7750788
13
BlumJ.RockstrohC.GöritzA. S. (2020). Development and Pilot Test of a Virtual Reality Respiratory Biofeedback Approach. Appl. Psychophysiol Biofeedback45 (3), 153–163. 10.1007/s10484-020-09468-x
14
BlumJ.RockstrohC.GöritzA. S. (2019). Heart Rate Variability Biofeedback Based on Slow-Paced Breathing with Immersive Virtual Reality Nature Scenery. Front. Psychol.10, 2172. 10.3389/fpsyg.2019.02172
15
BlumS.JacobsenN. S. J.BleichnerM. G.DebenerS. (2019). A Riemannian Modification of Artifact Subspace Reconstruction for EEG Artifact Handling. Front. Hum. Neurosci.13. 10.3389/fnhum.2019.00141
16
BradleyM. M.LangP. J. (1994). Measuring Emotion: The Self-Assessment Manikin and the Semantic Differential. J. Behav. Ther. Exp. Psychiatry25 (1), 49–59. 10.1016/0005-7916(94)90063-9
17
BullP. (1978). The Interpretation of Posture through an Alternative Methodology to Role Play. Br. J. Soc. Clin. Psychol.17 (1), 1–6. 10.1111/j.2044-8260.1978.tb00888.x
18
BurdeaG. C.CioiD.KaleA.JanesW. E.RossS. A.EngsbergJ. R. (2013). Robotics and Gaming to Improve Ankle Strength, Motor Control, and Function in Children with Cerebral Palsy-A Case Study Series. IEEE Trans. Neural Syst. Rehabil. Eng.21 (2), 165–173. 10.1109/TNSRE.2012.2206055
19
CacioppoJ. T.GardnerW. L.BerntsonG. G. (1997). Beyond Bipolar Conceptualizations and Measures: The Case of Attitudes and Evaluative Space. Pers Soc. Psychol. Rev.1 (1), 3–25. 10.1207/s15327957pspr0101_2
20
CamgözN.YenerC.GüvençD. (2002). Effects of Hue, Saturation, and Brightness on Preference. Color Res. Appl.27 (3), 199–207. 10.1002/col.10051
21
CannonW. B. (1927). The James-Lange Theory of Emotions: A Critical Examination and an Alternative Theory. Am. J. Psychol.39 (1/4), 106. 10.2307/1415404
22
CassaniR.MoinnereauM.-A.FalkT. H. (2018). “A Neurophysiological Sensor-Equipped Head-Mounted Display for Instrumental QoE Assessment of Immersive Multimedia,” in 2018 Tenth International Conference on Quality of Multimedia Experience. Sardinia, Italy: QoMEX, 1–6. 10.1109/QoMEX.2018.8463422
23
CavazzaM.AranyiG.CharlesF.PorteousJ.GilroyS.KlovatchI.et al (2014). Towards Empathic Neurofeedback For Interactive Storytelling Application/pdf, 19. 10.4230/OASICS.CMN.2014.42
24
LangeC. G.JamesW. (1922). The Emotions. (Baltimore, Maryland: Williams & Wilkins), Vol. 1. 10.1037/10735-000
25
ColzatoL. S.SellaroR.BesteC. (2017). Darwin Revisited: The Vagus Nerve Is a Causal Element in Controlling Recognition of Other's Emotions. Cortex92, 95–102. 10.1016/j.cortex.2017.03.017
26
ConatiC.ZhouX. (2002). “Modeling Students' Emotions from Cognitive Appraisal in Educational Games,” in Intelligent Tutoring Systems. Editors CerriS. A.GouardèresG.ParaguaçuF. (Springer Berlin Heidelberg), 944–954. 10.1007/3-540-47987-2_94
27
CordaroD. T.KeltnerD.TsheringS.WangchukD.FlynnL. M. (2016). The Voice Conveys Emotion in Ten Globalized Cultures and One Remote Village in Bhutan. Emotion16 (1), 117–128. 10.1037/emo0000100
28
CosmidesL.ToobyJ. (1994). “Origins of Domain Specificity: The Evolution of Functional Organization,” in Mapping the Mind. Editors HirschfeldL. A.GelmanS. A.. 1st ed. (Cambridge University Press), 85–116. 10.1017/CBO9780511752902.005
29
DarwinC. (1872). The Expression of the Emotions in Man and Animals. John Murray. 10.1037/10001-000
30
DavidsonR. J. (1992). Emotion and Affective Style: Hemispheric Substrates. Psychol. Sci.3 (1), 39–43. 10.1111/j.1467-9280.1992.tb00254.x
31
DesmetP. M. A. (2015). Design for Mood: Twenty Activity-Based Opportunities to Design for Mood Regulation. Int. J. Des.9 (2), 2015
32
DesmetP. M. A.RomeroN.VastenburgM. H. (2016). Mood Measurement with Pick-A-Mood: Review of Current Methods and Design of a Pictorial Self-Report Scale. Jdr14 (3), 241. 10.1504/JDR.2016.07975110.1504/jdr.2016.10000563Vastenburg
33
DimbergU. (1982). Facial Reactions to Facial Expressions. Psychophysiology19 (6), 643–647. 10.1111/j.1469-8986.1982.tb02516.x
34
DimbergU.ThunbergM.ElmehedK. (2000). Unconscious Facial Reactions to Emotional Facial Expressions. Psychol. Sci.11 (1), 86–89. 10.1111/1467-9280.00221
35
DimbergU.ThunbergM. (2012). Empathy, Emotional Contagion, and Rapid Facial Reactions to Angry and Happy Facial Expressions. PsyCh J.1 (2), 118–127. 10.1002/pchj.4
36
DrossosK.FlorosA.GiannakoulopoulosA.KanellopoulosN. (2015). Investigating the Impact of Sound Angular Position on the Listener Affective State. IEEE Trans. Affective Comput.6 (1), 27–42. 10.1109/TAFFC.2015.2392768
37
EbeY.UmemuroH. (2015). “Emotion Evoked by Texture and Application to Emotional Communication,” in Proceedings of the 33rd Annual ACM Conference Extended Abstracts on Human Factors in Computing Systems, 1995–2000. 10.1145/2702613.2732768
38
EibenA. E.SmithJ. E. (2015). Introduction to Evolutionary Computing. Springer, 2.
39
EkmanP. (2006). Darwin and Facial Expression: A century of Research in Review. Malor Books. Cambridge, MA: Ishk.
40
Editor EkmanP.FriesenW. V. (1971). Constants across Cultures in the Face and Emotion. J. Personal. Soc. Psychol.17 (2), 124–129. 10.1037/h0030377
41
EkmanP.FriesenW. V. (1976). Measuring Facial Movement. Environ. Psychol. Nonverbal Behav.1 (1), 56–75. 10.1007/BF01115465
42
FefferM.RudovicO.Oggi)PicardR. W. (2018). “A Mixture of Personalized Experts for Human Affect Estimation,” in Machine Learning and Data Mining in Pattern Recognition. Editor PernerP. (Springer International Publishing), 316–330.
43
FengC.BartramL.RieckeB. E. (2014). “Evaluating Affective Features of 3D Motionscapes,” in Proceedings of the ACM Symposium on Applied Perception, ’14. Vancouver, Canada: SAP, 23–30. 10.1145/2628257.2628264
44
Fernández-SotosA.Fernández-CaballeroA.LatorreJ. M. (2016). Influence of Tempo and Rhythmic Unit in Musical Emotion Regulation. Front. Comput. Neurosci.10. 80. 10.3389/fncom.2016.00080
45
FisherR. J. (1993). Social Desirability Bias and the Validity of Indirect Questioning. J. Consum Res.20 (2), 303. 10.1086/209351
46
FuhrmanO.BoroditskyL. (2010). Cross-Cultural Differences in Mental Representations of Time: Evidence from an Implicit Nonlinguistic Task. Cogn. Sci.34 (8), 1430–1451. 10.1111/j.1551-6709.2010.01105.x
47
GaeblerM.MuñozJ. E.de MoojiJ.QuinteroL. E.TrompJ.KlotzscheF.et al (2021). Excite-O-Meter. Available at: https://www.exciteometer.eu/. (Accessed July 28, 2021).
48
GaoX.-P.XinJ. H.SatoT.HansuebsaiA.ScalzoM.KajiwaraK.et al (2007). Analysis of Cross-Cultural Color Emotion. Color Res. Appl.32 (3), 223–229. 10.1002/col.20321
49
GarciaJ. A.NavarroK. F. (2014). “The Mobile RehAppTM: An AR-based mobile Game for Ankle Sprain Rehabilitation,” in 2014 IEEE 3nd International Conference on Serious Games and Applications for Health. Rio de Janeiro, Brazil: SeGAH, 1–6. 10.1109/SeGAH.2014.7067087
50
GeorgiouT.DemirisY. (2017). Adaptive User Modelling in Car Racing Games Using Behavioural and Physiological Data. User Model. User-adap Inter.27 (2), 267–311. 10.1007/s11257-017-9192-3
51
GerardiG. M.GerkenL. (1995). The Development of Affective Responses to Modality and Melodic Contour. Music Perception: Interdiscip. J.12 (3), 279–290. 10.2307/40286184
52
GreinacherR.KojicT.MeierL.ParameshappaR. G.MollerS.Voigt-AntonsJ.-N. (2020). “Impact of Tactile and Visual Feedback on Breathing Rhythm and User Experience in VR Exergaming,” in 2020 Twelfth International Conference on Quality of Multimedia Experience. Athlone, Ireland: QoMEX, 1–6. 10.1109/QoMEX48832.2020.9123141
53
GreinacherR.Voigt-AntonsJ.-N. (2020). “Accuracy Assessment of ARKit 2 Based Gaze Estimation,” in Human-Computer Interaction. Design and User Experience. Editor KurosuM. (Springer International Publishing), Vol. 12181, 439–449. 10.1007/978-3-030-49059-1_32
54
HaarA. J. H.JainA.SchoellerF.MaesP. (2020). Augmenting Aesthetic Chills Using a Wearable Prosthesis Improves Their Downstream Effects on Reward and Social Cognition. Sci. Rep.10 (1), 21603. 10.1038/s41598-020-77951-w
55
HarischandraJ.PereraM. U. S. (2012). “Intelligent Emotion Recognition System Using Brain Signals (EEG),” in 2012 IEEE-EMBS Conference on Biomedical Engineering and Sciences, 454–459. 10.1109/IECBES.2012.6498050
56
HernandezJ.LiY.RehgJ. M.PicardR. W. (2014). “BioGlass: Physiological Parameter Estimation Using a Head-Mounted Wearable Device,” in 2014 4th International Conference on Wireless Mobile Communication and Healthcare - Transforming Healthcare Through Innovations in Mobile and Wireless Technologies. Athens, Greece: MOBIHEALTH), 55–58. 10.1109/MOBIHEALTH.2014.701590810.4108/icst.mobihealth.2014.257219
57
HofmannS. M.KlotzscheF.MariolaA.NikulinV. V.VillringerA.GaeblerM. (2018). “Decoding Subjective Emotional Arousal during a Naturalistic VR Experience from EEG Using LSTMs,” in 2018 IEEE International Conference on Artificial Intelligence and Virtual Reality. (Taichung, Taiwan: AIVR), 128–131. 10.1109/AIVR.2018.00026
58
HoppeS.LoetscherT.MoreyS. A.BullingA. (2018). Eye Movements during Everyday Behavior Predict Personality Traits. Front. Hum. Neurosci.12, 105. 10.3389/fnhum.2018.00105
59
HupontI.GraciaJ.SanagustinL.GraciaM. A. (2015). “How Do New Visual Immersive Systems Influence Gaming QoE? A Use Case of Serious Gaming with Oculus Rift,” in 2015 Seventh International Workshop on Quality of Multimedia Experience. Costa Navarino, Greece: QoMEX, 1–6. 10.1109/QoMEX.2015.7148110
60
HusterR. J.StevensS.GerlachA. L.RistF. (2009). A Spectralanalytic Approach to Emotional Responses Evoked through Picture Presentation. Int. J. Psychophysiology72 (2), 212–216. 10.1016/j.ijpsycho.2008.12.009
61
JaquesP. A.VicariR. M. (2007). A BDI Approach to Infer Student's Emotions in an Intelligent Learning Environment. Comput. Edu.49 (2), 360–384. 10.1016/j.compedu.2005.09.002
62
KapurA.KapurA.Virji-BabulN.TzanetakisG.DriessenP. F. (2005). “Gesture-Based Affective Computing on Motion Capture Data,” in Affective Computing and Intelligent Interaction. Editors TaoJ.TanT.PicardR. W. (Springer Berlin Heidelberg), 1–7. 10.1007/11573548_1
63
KitsonA.DiPaolaS.RieckeB. E. (2019). “Lucid Loop,” in Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems, 1–6. 10.1145/3290607.3312952
64
KlugM.GramannK. (2020). Identifying Key Factors for Improving ICA‐based Decomposition of EEG Data in mobile and Stationary Experiments. Eur. J. Neurosci.. 10.1111/ejn.14992
65
KoelstraS.MuhlC.SoleymaniM.Jong-Seok LeeLee.YazdaniA.EbrahimiT.et al (2012). DEAP: A Database for Emotion Analysis ; Using Physiological Signals. IEEE Trans. Affective Comput.3 (1), 18–31. 10.1109/T-AFFC.2011.15
66
KoenigS. T.CrucianG. P.DuenserA.BartneckC.Dalrymple-AlfordJ. C. (2011). Validity Evaluation of a Spatial Memory Task in Virtual Environments
67
KotheC. A.MakeigS. (2013). BCILAB: a Platform for Brain-Computer Interface Development. J. Neural Eng.10 (5), 056014. 10.1088/1741-2560/10/5/056014
68
LangP. J.BradleyM. M.CuthbertB. N. (2008). International Affective Picture System (IAPS): Affective Ratings of Pictures and Instruction Manual. Gainesville, FL: University of Florida. Technical Report A-8
69
LeslieG.PicardR.LuiS. (2015). An EEG and Motion Capture Based Expressive Music Interface for Affective Neurofeedback. 10.13140/RG.2.1.4378.6081
70
LiZ.TongL.WangL.LiY.HeW.GuanM.et al (2016). Self-regulating Positive Emotion Networks by Feedback of Multiple Emotional Brain States Using Real-Time fMRI. Exp. Brain Res.234 (12), 3575–3586. 10.1007/s00221-016-4744-z
71
Lipson-SmithR.BernhardtJ.ZamunerE.ChurilovL.BusiettaN.MorattiD. (2020). Exploring Colour in Context Using Virtual Reality: Does a Room Change How You Feel?. Virtual Reality. 10.1007/s10055-020-00479-x
72
LockyerM.BartramL.RieckeB. E. (2011). Simple Motion Textures for Ambient Affect. Comput. Aesthetics Graphics, Visualization, 8. 10.2312/COMPAESTH/COMPAESTH11/089-09610.1145/2030441.2030461
73
LucassenM. P.GeversT.GijsenijA. (2011). Texture Affects Color Emotion. Color Res. Appl.36 (6), 426–436. 10.1002/col.20647
74
MakeigS.JungT.-P.BellA. J.GhahremaniD.SejnowskiT. J. (1997). Blind Separation of Auditory Event-Related Brain Responses into Independent Components. Proc. Natl. Acad. Sci.94 (20), 10979–10984. 10.1073/pnas.94.20.10979
75
MalikM.BiggerJ. T.CammA. J.KleigerR. E.MallianiA.MossA. J.et al (1996). Heart Rate Variability: Standards of Measurement, Physiological Interpretation, and Clinical Use. Eur. Heart J.17 (3), 354–381. 10.1093/oxfordjournals.eurheartj.a014868
76
Martínez-TejadaL.Puertas-GonzálezA.YoshimuraN.KoikeY. (2021). Exploring EEG Characteristics to Identify Emotional Reactions under Videogame Scenarios. Brain Sci.11 (3), 378. 10.3390/brainsci11030378
77
MattekA. M.WolfordG. L.WhalenP. J. (2017). A Mathematical Model Captures the Structure of Subjective Affect. Perspect. Psychol. Sci.12 (3), 508–526. 10.1177/1745691616685863
78
MavridouI.McGheeJ. T.HamediM.FatoorechiM.ClealA.Ballaguer-BalesterE.et al (2017). “FACETEQ Interface Demo for Emotion Expression in VR,” in 2017 IEEE Virtual Reality (Los Angeles, California: VR), 441–442. 10.1109/VR.2017.7892369
79
McDuffD.KalioubyR. E.PicardR. W. (2012). Crowdsourcing Facial Responses to Online Videos. IEEE Trans. Affective Comput.3 (4), 456–468. 10.1109/T-AFFC.2012.19
80
MullenT. R.KotheC. A. E.ChiY. M.OjedaA.KerthT.MakeigS.et al (2015). Real-time Neuroimaging and Cognitive Monitoring Using Wearable Dry EEG. IEEE Trans. Biomed. Eng.62 (11), 2553–2567. 10.1109/TBME.2015.2481482
81
NormanJ. F.BeersA.PhillipsF. (2010). Fechner's Aesthetics Revisited. Seeing and Perceiving23 (3), 263–271. 10.1163/187847510X516412
82
OrtonyA.CloreG. L.CollinsA. (1988). The Cognitive Structure of Emotions. Cambridge University Press.
83
PaganiM.LombardiF.GuzzettiS.SandroneG.RimoldiO.MalfattoG.et al (1984). Power Spectral Density of Heart Rate Variability as an index of Sympatho-Vagal Interaction in normal and Hypertensive Subjects. J. Hypertens. Suppl.2 (3), S383–S385.
84
PalmerS. E.SchlossK. B. (2010). An Ecological Valence Theory of Human Color Preference. Proc. Natl. Acad. Sci.107 (19), 8877–8882. 10.1073/pnas.0906172107
85
PeperkornH. M.AlpersG. W.MühlbergerA. (2014). Triggers of Fear: Perceptual Cues versus Conceptual Information in Spider Phobia. J. Clin. Psychol.70 (7), 704–714. 10.1002/jclp.22057
86
PerkisA.TimmererC.BarakovićS.HusićJ. B.BechS.BosseS.et al (2020). QUALINET White Paper on Definitions of Immersive Media Experience (IMEx).
87
PetriT. (2009). Exploring Relationships between Audio Features and Emotion in Music. Front. Hum. Neurosci.3. 10.3389/conf.neuro.09.2009.02.033
88
PfurtschellerG.Lopes da SilvaF. H. (1999). Event-related EEG/MEG Synchronization and Desynchronization: Basic Principles. Clin. Neurophysiol.110 (11), 1842–1857. 10.1016/S1388-2457(99)00141-8
89
PicardR. W.VyzasE.HealeyJ. (2001). Toward Machine Emotional Intelligence: Analysis of Affective Physiological State. IEEE Trans. Pattern Anal. Machine Intell.23 (10), 1175–1191. 10.1109/34.954607
90
PinillaA.TamayoR. M.NeiraJ. (2020). How Do Induced Affective States Bias Emotional Contagion to Faces? A Three-Dimensional Model. Front. Psychol.11. 10.3389/fpsyg.2020.00097
91
Pion-TonachiniL.Sheng-H. H.MakeigS.Tzyy-Ping JungT. P.CauwenberghsG. (2015). “Real-time EEG Source-Mapping Toolbox (REST): Online ICA and Source Localization,” in 2015 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society. Milano, Italy: EMBC, 4114–4117. 10.1109/EMBC.2015.7319299
92
PiwekL.PollickF.PetriniK. (2015). Audiovisual Integration of Emotional Signals from Others' Social Interactions. Front. Psychol.9. 10.3389/fpsyg.2015.00611
93
PlutchikR. (1982). A Psychoevolutionary Theory of Emotions. Soc. Sci. Inf.21 (4–5), 529–553. 10.1177/053901882021004003
94
PolzehlT.SchmittA.MetzeF.WagnerM. (2011). Anger Recognition in Speech Using Acoustic and Linguistic Cues. Speech Commun.53 (9–10), 1198–1209. 10.1016/j.specom.2011.05.002
95
PorcuS.FlorisA.Voigt-AntonsJ.-N.AtzoriL.MollerS. (2020). Estimation of the Quality of Experience during Video Streaming from Facial Expression and Gaze Direction. IEEE Trans. Netw. Serv. Manage.17, 2702–2716. 10.1109/TNSM.2020.3018303
96
PutnamH. (1967). “The Nature of Mental States,” in Art, Mind, and Religion. Editors CapitanW. H.MerrillD. D. (Pittsburgh, PA: Pittsburgh University Press), 1–223.
97
RaffeW. L.ZambettaF.LiX.StanleyK. O. (2015). Integrated Approach to Personalized Procedural Map Generation Using Evolutionary Algorithms. IEEE Trans. Comput. Intell. AI Games7 (2), 139–155. 10.1109/TCIAIG.2014.2341665
98
RayW.ColeH. (1985). EEG Alpha Activity Reflects Attentional Demands, and Beta Activity Reflects Emotional and Cognitive Processes. Science228 (4700), 750–752. 10.1126/science.3992243
99
RenardY.LotteF.GibertG.CongedoM.MabyE.DelannoyV.et al (2010). OpenViBE: An Open-Source Software Platform to Design, Test, and Use Brain-Computer Interfaces in Real and Virtual Environments. Presence: Teleoperators and Virtual Environments19 (1), 35–53. 10.1162/pres.19.1.35
100
ReuderinkB.MühlC.PoelM. (2013). Valence, Arousal and Dominance in the EEG during Game Play. Ijaacs6 (1), 45. 10.1504/IJAACS.2013.050691
101
RobitailleP.McGuffinM. J. (2019). Increased Affect-Arousal in VR Can Be Detected from Faster Body Motion with Increased Heart Rate. Proc. ACM SIGGRAPH Symp. Interactive 3D Graphics Games, 1–6. 10.1145/3306131.3317022
102
RussellJ. A. (1980). A Circumplex Model of Affect. J. Personal. Soc. Psychol.39 (6), 1161–1178. 10.1037/h0077714
103
RyaliC. K.GoffinS.WinkielmanP.YuA. J. (2020). From Likely to Likable: The Role of Statistical Typicality in Human Social Assessment of Faces. Proc. Natl. Acad. Sci. USA117 (47), 29371–29380. 10.1073/pnas.1912343117
104
SchachterS.SingerJ. (1962). Cognitive, Social, and Physiological Determinants of Emotional State. Psychol. Rev.69 (5), 379–399. 10.1037/h0046234
105
SchererK. R.OshinskyJ. S. (1977). Cue Utilization in Emotion Attribution from Auditory Stimuli. Motiv. Emot.1 (4), 331–346. 10.1007/BF00992539
106
SchoellerF.BertrandP.GerryL. J.JainA.HorowitzA. H.ZenasniF. (2019b). Combining Virtual Reality and Biofeedback to Foster Empathic Abilities in Humans. Front. Psychol.9, 2741. 10.3389/fpsyg.2018.02741
107
SchoellerF.HaarA. J. H.JainA.MaesP. (2019a). Enhancing Human Emotions with Interoceptive Technologies. Phys. Life Rev.31, 310–319. 10.1016/j.plrev.2019.10.008
108
SemertzidisN.ScaryM.AndresJ.DwivediB.KulweY. C.ZambettaF.et al (2020). Neo-Noumena. Proc. 2020 CHI Conf. Hum. Factors Comput. Syst., 1–13. 10.1145/3313831.3376599
109
ShibanY.PeperkornH.AlpersG. W.PauliP.MühlbergerA. (2016). Influence of Perceptual Cues and Conceptual Information on the Activation and Reduction of Claustrophobic Fear. J. Behav. Ther. Exp. Psychiatry51, 19–26. 10.1016/j.jbtep.2015.11.002
110
ShibanY.ReichenbergerJ.NeumannI. D.Mã¼hlbergerA. (2015). Social Conditioning and Extinction Paradigm: A Translational Study in Virtual Reality. Front. Psychol.6. 10.3389/fpsyg.2015.00400
111
ShiotaM. N.KalatJ. W. (2012). Emotion. Second Edition. Belmont, CA: Linda Schreiber-Ganster
112
SitaramR.LeeS.RuizS.RanaM.VeitR.BirbaumerN. (2011). Real-time Support Vector Classification and Feedback of Multiple Emotional Brain States. NeuroImage56 (2), 753–765. 10.1016/j.neuroimage.2010.08.007
113
SutherlandM. R.MatherM. (2012). Negative Arousal Amplifies the Effects of Saliency in Short-Term Memory. Emotion12 (6), 1367–1372. 10.1037/a0027860
114
Tajadura-JiménezA.LarssonP.VäljamäeA.VästfjällD.KleinerM. (2010b). When Room Size Matters: Acoustic Influences on Emotional Responses to Sounds. Emotion10 (3), 416–422. 10.1037/a0018423
115
Tajadura-JiménezA.VäljamäeA.AsutayE.VästfjällD. (2010a). Embodied Auditory Perception: The Emotional Impact of Approaching and Receding Sound Sources. Emotion10 (2), 216–229. 10.1037/a0018422
116
Tajadura-JiménezA.VäljamäeA.VästfjällD. (2008). Self-Representation in Mediated Environments: The Experience of Emotions Modulated by Auditory-Vibrotactile Heartbeat. CyberPsychology Behav.11 (1), 33–38. 10.1089/cpb.2007.0002
117
ThayerJ. F.HansenA. L.Saus-RoseE.JohnsenB. H. (2009). Heart Rate Variability, Prefrontal Neural Function, and Cognitive Performance: The Neurovisceral Integration Perspective on Self-Regulation, Adaptation, and Health. Ann. Behav. Med.37 (2), 141–153. 10.1007/s12160-009-9101-z
118
UnpingcoJ. (2014). Python for Signal Processing: Featuring IPython Notebooks. Springer
119
ValdezP.MehrabianA. (1994). Effects of Color on Emotions. J. Exp. Psychol. Gen.123 (4), 394–409. 10.1037/0096-3445.123.4.394
120
VogtT.AndréE.BeeN. (2008). “EmoVoice—A Framework for Online Recognition of Emotions from Voice,” in Perception in Multimodal Dialogue Systems. Editors AndréE.DybkjærL.MinkerW.NeumannH.PieracciniR.WeberM. (Springer Berlin Heidelberg), 188–199.
121
Voigt-AntonsJ.-N.LehtonenE.PalaciosA. P.AliD.KojicT.MollerS. (2020). “Comparing Emotional States Induced by 360° Videos via Head-Mounted Display and Computer Screen,” in 2020 Twelfth International Conference on Quality of Multimedia Experience. Athlone, Ireland: QoMEX, 1–6. 10.1109/QoMEX48832.2020.9123125
122
WatsonD.ClarkL. A.TellegenA. (1988). Development and Validation of Brief Measures of Positive and Negative Affect: The PANAS Scales. J. Personal. Soc. Psychol.54 (6), 1063–1070. 10.1037/0022-3514.54.6.1063
123
WilliamsD.KirkeA.MirandaE.DalyI.HwangF.WeaverJ.et al (2017). Affective Calibration of Musical Feature Sets in an Emotionally Intelligent Music Composition System. ACM Trans. Appl. Percept.14 (3), 1–13. 10.1145/3059005
124
WilmsL.OberfeldD. (2018). Color and Emotion: Effects of Hue, Saturation, and Brightness. Psychol. Res.82 (5), 896–914. 10.1007/s00426-017-0880-8
125
WolpawJ. R.BirbaumerN.McFarlandD. J.PfurtschellerG.VaughanT. M. (2002). Brain-computer Interfaces for Communication and Control. Clin. Neurophysiol.113 (6), 767–791. 10.1016/S1388-2457(02)00057-3
126
WundtW. (1897). Outlines of Psychology. London, England: Williams and Norgate. C. H. Judd, Trans.10.1037/12908-000
127
YannakakisG. N.TogeliusJ. (2011). Experience-Driven Procedural Content Generation. IEEE Trans. Affective Comput.2 (3), 147–161. 10.1109/T-AFFC.2011.6
128
ZanderT. O.KotheC. (2011). Towards Passive Brain-Computer Interfaces: Applying Brain-Computer Interface Technology to Human-Machine Systems in General. J. Neural Eng.8 (2), 025005. 10.1088/1741-2560/8/2/025005
Summary
Keywords
virtual reality, affect, emotion, electrophysiology, visual design, visualization
Citation
Pinilla A, Garcia J, Raffe W, Voigt-Antons J-N, Spang RP and Möller S (2021) Affective Visualization in Virtual Reality: An Integrative Review. Front. Virtual Real. 2:630731. doi: 10.3389/frvir.2021.630731
Received
18 November 2020
Accepted
23 July 2021
Published
06 August 2021
Volume
2 - 2021
Edited by
Doron Friedman, Interdisciplinary Center Herzliya, Israel
Reviewed by
Louis Nisiotis, University of Central Lancashire, Cyprus
Anja S. Göritz, University of Freiburg, Germany
Updates
Copyright
© 2021 Pinilla, Garcia, Raffe, Voigt-Antons, Spang and Möller.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
*Correspondence: Andres Pinilla, andres.pinilla@qu.tu-berlin.de
† ORCID: Andres Pinilla orcid.org/0000-0002-0812-7896 Jaime Garcia orcid.org/0000-0001-5718-1605 William Raffe orcid.org/0000-0001-5310-0943 Jan-Niklas Voigt-Antons orcid.org/0000-0002-2786-9262 Robert P. Spang orcid.org/0000-0001-6580-9060 Sebastian Möller orcid.org/0000-0003-3057-0760
This article was submitted to Virtual Reality and Human Behaviour, a section of the journal Frontiers in Virtual Reality
Disclaimer
All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article or claim that may be made by its manufacturer is not guaranteed or endorsed by the publisher.