Performance of game sessions in VR vs standard 2D monitor environment. An EEG study

Urszula Malinowska^1,2Jakub Wojciechowski³Marek Waligóra²Jacek Rogala^4*

• ¹Institute of Experimental Physics, Faculty of Physics, University of Warsaw, Warsaw, Masovian, Poland
• ²Nencki Institute of Experimental Biology, Polish Academy of Sciences, Warszawa, Masovian, Poland
• ³Bioimaging Research Center, Institute of Physiology and Pathology of Hearing (IFPS), Kajetany, Poland
• ⁴Neurophysiology, University of Warsaw, Warsaw, Poland

Improving the effectiveness of treatments for mental disorders has been and continues to be a key issue in medicine, especially with the significant increase in the prevalence of nervous system dysfunction, not only due to dementia, trauma, or neurodegenerative diseases but also disorders such as Attention deficit hyperactivity disorder (ADHD), which is increasingly prevalent in younger people (Oehrlein et al., 2016).The use of new technological advances may allow for better outcomes while reducing the duration or number of cumbersome therapy sessions. With the rapid development of Virtual Reality (VR) technology, 1 I n r e v i e w there are more and more reports on research and clinical applications (Wright, 2014;Parsons et al., 2013).Researchers have demonstrated the effectiveness of VR in therapeutic applications, and VR systems have been used in the treatment of many psychiatric disorders, pain management, and neurorehabilitation for several years (Bohil et al., 2011). Studies on brain function in VR environments mostly related to passive watching of 3D videos indicate that the fully immersive 3D-enriched environment requires the allocation of more brain and sensory resources for cognitive/motor control than 2D presentations (Slobounov et al., 2015). In another EEG-VR study (Malik et al., 2015), Malik et al. reported higher EEG absolute power across various brain regions during 3D video watching, especially including the occipital region in the theta band and the frontal and parietal regions in the alpha band, while in fMRI, Chen et al. (Chen et al., 2015) observed increased visual fatigue caused by watching 3D TV compared to 2D. Convergent results on increased load in VR settings are also shown by studies using interactive game or simulator environments.For example, Kakkos and colleagues' findings (Kakkos et al., 2019) demonstrated significant alterations of alpha and theta band power, indicating increased workload (Hogervorst et al., 2014;Mühl et al., 2014;Brouwer et al., 2012) in a VR environment. Therefore, although these studies are valuable, they do not allow for the evaluation of the effectiveness of multisession training protocols. Another issue related to evaluating the effectiveness of cognitive training based on video games is their complexity due to the involvement of multiple cognitive and motor tasks, making it difficult to isolate the impact of the environment on a single task. To address the above issues, we designed an exploratory investigation aiming at a progress comparison of a gamified working memory task in a 2D computer screen and in head-mounted display virtual reality environments during three consecutive training sessions spread over one week. We monitored the course of neural, behavioral, and neuronal processing using EEG across the three sessions, including classical spectral and time-frequency analyses, spectral entropy, and neural connectivity estimated by Phase Locking Value. Additionally, we used theta to beta ratio to monitor changes of cognitive load over the sessions.Finally, we monitored behavioral performance using reaction times and accuracy of test performance.Experiments for both groups were conducted in the same dimly lit room, at room temperature. For the 2D display group we used a standard 17" LCD monitor with participants sitting in comfortable chairs at a distance of approximately 50 cm from the screen. In the VR group, instead of monitors, participants were using Oculus Rift CV1 goggles. The field of view (FOV) for the computer screen was estimated as 40 • and for goggles 90 • , the VR version of the game included a detailed 270 • view of the spaceship cabin and space beyond the spaceship, including stars and meteors. To ensure a similar difficulty level for both groups and mitigate potential artifacts caused by excessive head movements in the VR environment the shooting target appeared in the center of FOV (approximately 20 • ) thus minimizing head movements in the VR environment and keeping a similar level of difficulty in the 2D one. The VR headset was worn over the EEG cap making sure that the harness of the headset did not press any electrode. To keep participants motivated and engaged throughout the test, the experimental task was adapted to an animated spacecraft shooting video game using the Unity programming environment for monitor screen and VR goggles. Otherwise both game variants were identical. Screenshots of the game are presented in Fig. 1. Initial tests, performed before the main experiment, showed no effect of VR headset on EEG signal quality.Frontiers 2Figure 1. Delayed Match-To-Sample task: 1.5 second encoding phase, next 5 second to maintain the stimulus in working memory, and 2 seconds of a retrieval phase when participants pressed a button corresponding to the stimulus (left/right ctrl) that matched stimulus presented during encoding. In red are marked 1 second windows selected for EEG analysis.For the cognitive procedure, we used the Delay Match-To-Sample (DMTS) task (Blough, 1959), which allows for studying working memory and attention in human subjects. We chose DMTS because it involves multiple memory functions, i.e. encoding, maintenance, and retrieval of stimulus representations in sequential order which allow for their individual analyses unlike the n-back task which requires only constant information updating (Ribeiro et al., 2019). In its basic version, the DMTS task comprised three phases: sample, delay, and choice. Each trial starts with the presentation of a sample to be memorized; next a delay was introduced. During the delay, participants were asked to maintain a representation of the sample. When the delay was over another stimulus or multiple stimuli were simultaneously presented to the participant. Participants were then asked to make a decision about does this stimulus match the sample presented in the first phase or does not. Our version of DMTS ( Żygierewicz et al., 2022) started with a 2 second familiarization period, where no action from participants was required, followed by the sample encoding phase lasting 1.5 seconds, and then the animated silhouette of the spaceship was presented to participants. The delay phase -maintenance of the sample in memory, where no stimulus was presented lasted 5 seconds. During the last, 2 seconds long retrieval -choice phase, participants were presented with the spacecraft silhouette matching or not the sample. For analyses we chose 1 second window in each of the task phases. In the encoding and retrieval phases the window began at the time of stimulus appearance.In the maintenance phase window began 0.5 second after the end of the stimulus presentation to allow for fade-out of visual processing, which could interfere with memory retention processing readings. Window length was set to 1 second to cover the whole period of stimulus presentation. The duration of each phase was established experimentally during pilot trials to achieve the average performance of 70% accuracy. The Twenty eight participants (13 women), aged 26-40 years who were not experienced game players were recruited through advertisements at local universities and employment agencies. The participants were randomized into VR and 2D display groups and performed three sessions of the adapted DMTS task during EEG recording. The EEG signal was recorded using Digitrack software (Elmiko l.t.d.) with 21 electrodesFrontiers 3arranged in a 10-20 system, referenced to the right ear. All subjects provided informed written consent in accordance with the Helsinki Declaration as well as Institutional Review Board requirements.EEG signals were recorded at 500 Hz sampling rate. Artifact removal procedure was described in detail in ( Żygierewicz et al., 2022). Preprocessing steps contained semi-automated methods, which included 0.5 Hz high-pass and 45 Hz low-pass filtering, baseline correction, exclusion of trials containing muscle artifacts, and independent component analysis (ICA). The identified eye movement and muscle artifacts components were removed. Next, for each channel and trial, we extracted three 1 second long windows (marked on Fig.1 as W1, W2, and W3) in order to investigate EEG signals properties in the encoding, maintenance, and retrieval phases of each task trial. If we set the beginning of the windows in each trial at time 0, the time spans of the three windows will encompass correspondingly: W1 [0, 1000ms], W2 [2000ms, 3000ms], W3[6500ms, 7500ms]. In the last step, we extracted only epochs with correct responses, which were used in all subsequent analyses. Analysis was performed in the following classical EEG bands: delta (δ; 0.5-4Hz); theta (θ; 4-7 Hz); alpha (α; 8-12 Hz); beta-1 (β1; 13-20 Hz); beta-2 (β2; 21-30 Hz) and gamma (γ; 31-40 Hz). Spectral analyses included a comparison between groups of subjects of average power from the 1 second windows in the frequency bands. All power estimates were computed using MATLAB 'pwelch' function (with settings: 250 ms segments with 50% overlap, windowed with a Hamming window).The obtained spectra were averaged in each window for a given channel and subject. For more detailed time-frequency analyses we used the 'newtimef' function implemented in EEGLAB (Delorme et al., 2011) in the described above windows and frequency bands. We applied Morlet wavelets decomposition using 3 cycles and 1 second long window with 200 time points.Signal complexity: Complexity was evaluated by means of Spectral Entropy (SpEn) -theoretical information measure that provided an estimation of EEG regularity. SpEn definition was based on the formulation of Shannon's entropy (Shannon, 1948), where the probability was replaced by the estimated power of spectral density PSD:SpEn = - 1 log(L) 40Hz f =1Hz P SD(f ) * log[P SD(f )](1)where L was the number of spectral components.Connectivity was assessed by Phase Locking Value (PLV) -a measure of phase dependence (Lachaux et al., 1999), here estimated between pairs of sites measured in separate frequency bands (defined above δ, θ, α, β1, β2, and γ);P LV = 1 N N n=1 exp(i(ϕ 1 [n] -ϕ 2 [n]))(2)where ϕ 1 and ϕ 2 were phases of signal in a given frequency range for a pair of electrodes (1) and (2).Attention and processing capacity: As the measure of cognitive processing capacity we used the θ/β ratio (TBR) Clarke et al. (2019). The θ/β ratio was calculated by dividing absolute theta power by absolute beta power (β1 or β2) at each time window at the given trial and electrode site. To test differences between EEG signals in 2D monitor display and VR groups, the results of each measure were first averaged across all trials for each subject and channel in the three selected windows (W1, W2, W3). Next, a non-parametricFrontiers 4unpaired Wilcoxon test with the significance threshold of p<0.05 or p<0.01 was used for each measure, and frequency band, and compared separately between the group of players in each game session. All data were analyzed in MATLAB (8.5.0, Math-Works, USA) using in-house scripts and EEGLAB toolbox (Delorme and Makeig, 2004).The VR group scored higher in the task compared to the 2D monitor display group, but only in the first game session Fig. 2 Most prominent group differences in spectral analysis were found in the first session, in line with most behavioral differences. In the first session during the sample encoding phase, we observed significantly (p<0.05) stronger spectral activity for the VR than the monitor display group, mostly in the δ band on frontal and right central EEG channels and in the β2 and γ bands in the right posterior electrodes. During the maintenance and retrieval phases, the VR group revealed stronger activity in the δ band in one frontal and left parietal electrode and also more powerful activity in the β2 and γ band, especially in the posterior and parietal areas on the right side. In the following sessions differences between groups gradually decreased, finally showing significant differences only in the posterior electrodes in β2 and γ bands in the last session.The sites with significant spectral differences between groups of players are presented in Fig. 3 in red.More detailed time-frequency analyses revealed group differences in all EEG bands except for α band. A most striking difference between analyzed groups was the higher spectral power of the VR group in session 1 in all EEG bands except for δ one where no differences were found (Fig. 4). The number of electrodes showing group differences decreased in subsequent sessions. Most numerous differences showing higher spectral power of the VR group were found in the θ and γ bands in the encoding phase (Fig. 4Frontiers I n r e v i e w First, we observed smaller EEG signal complexity in the VR group in the first session. Differences between monitor display and VR groups in average spectral entropy from all EEG channels were significant in encoding phase (p=0.021), but not in maintenance (p=0.16), and sample matching (p=0.18) phases during the first session of the DMTS task (Fig. 6(A)). This effect disappeared in sessions 2 (p>0. Red color denotes higher spectral power in the VR group, blue in the monitor group. Spectral bands or task phases where significant differences were not found are not shown or left empty. All differences significant at p<0.01.In session 1 the θ/β2 ratio was significantly smaller in VR than in the monitor group. The differences were found on posterior channels in all three phases of the task. Channels with significant differences in different task phases are shown in Fig. 7. That was consistent with an indication of power spectra differences, where θ band did not show significant differences between the groups of analysis, and β2showed higher power for the VR group (Fig. 3). In session 2 VR players showed lower θ/β2 ratio in Frontiers 7I n r e v i e w posterior channels in comparison to monitor display group (Fig. 7(B)), mostly in encoding and retrieval phases. In session 3 we observed further reduction of this effect (Fig. 7).EEG signals interrelations, measured by PLV, revealed in session 1 stronger connectivity in the VR group of subjects compared to the monitor group. That was observed especially in δ, θ, and α frequencies in left centro-parietal sites during all three analyzed phases of the DMTS task. For higher frequencies, we noted significantly stronger connections between left frontal and centro-parietal regions for VR subjects (Fig. 8). This effect, similar to spectral differences, was not observed in sessions 2 and 3 for which the Frontiers 8Figure 7. Average θ/β2 ratios presented as bars for monitor display (white/light blue) and for VR (gray/dark blue) shown for 3 consecutive sessions S1-S3 (rows) and for windows W1-W3 (columns). For subjects in the 2D group electrodes with significantly higher values of average θ/β2 ratio (on sig. level 0.05) are marked in blue, and those with the ratio higher on a significance level 0.01 are underlined.only differences between VR and monitor display groups were observed for a few connections in higher frequencies stronger for VR group.Results of our exploratory study comparing 2D monitor display and VR groups in three game sessions showed group differences only in the first session. The VR group members outperformed the monitor display group in the first session and maintained their performance level throughout the remaining two, while the monitor display group increased performance in each session, eventually leveling up with the VR group. The studies comparing behavioral and neurophysiological effects of the VR and computer screenbased tests over multiple training sessions are very rare. One of the very few such investigations is the study performed by Berger and colleagues (Berger et al., 2022), who investigated the effect of neurofeedback training in VR and 2D environments on SMR power. Although their results pointed to a linear increase of the SMR power only in the VR groups, the graphical implementation of the task in both environments was different, therefore one can not exclude that the effect was due to differences in visual stimulus. The study also did not investigate putative differences in cognitive performance. From that perspective more interesting seems to be the experiment conducted by Barrett et al. (Barrett et al., 2022). In this experiment three groups of participants performed identical categorization tasks also in terms of the visual stimulus implemented in the three environments: VR its computer screen version and the flat version also on a computer screen. The first and second task implementations exactly match the settings of our experiment, however the cognitive task was different. Although this study included only one session, the performance differences in our study were found only in the first session. Interestingly, Barrett et al. (Barrett et al., 2022) found no behavioral differences between the groups, but the number of fixations in the VR environment was significantly higher than in the 2D one. Also other, previous experiments comparing monitor displayFigure 8. Phase locking value connectivity differences between monitor and VR groups in 3 consecutive sessions. From left for session 1, session 2, and session 3. Red colors mark pairs of electrodes with significantly higher values for subjects in the VR group (on significance level 0.01), and blue for the monitor display group.and VR environments in a single session showed no benefit of greater immersion experience elicited by VR environment (Pallavicini and Pepe, 2019;Buttussi and Chittaro, 2017;Lugrin et al., 2013). Our seemingly contradictory behavioral performance results may stem from specific game design solely based on a simplified Delay Match-to-Sample task. Games utilizing more selective cognitive functions could be more sensitive to the environmental effects than more complex games, which require more variability and demand on cognitive functions than one function. Another explanation of these contradictions could be possible differences in the interactions between the task at hand and the environment.To further investigate the effect of the game environment on memory performance we analyzed EEG data. The pattern of behavioral group differences was followed by EEG activity. Analyses of the spectral power differences across sessions revealed two prominent observations: first, higher power of the δ band in the VR group (in the relation to monitor display group) in the encoding phase of the first session, which gradually diminished in subsequent sessions; and second, differences in the higher frequency bands mostly in the second session in posterior locations. Although δ band activity is primarily associated with deep sleep, some studies also reported relationships between δ band activity and cognitive task performance. The work of Mathewson et al. (Mathewson et al., 2012), found a positive correlation between δ band activity in the frontal and parietal electrodes and learning rate in video games. Increased δ band activity was also found to be correlated with higher attention to internal processes (Harmony et al., 1996). These findings may suggest that observed game performance differences could arise from faster memory learning rate elicited by increased attention to internal processes in the VR group. In subsequent sessions, participants Frontiers I n r e v i e w from the monitor group would gradually adapt to the game environment, improving their learning rate through increased attention to internal processes leveraging their learning rate as indicated by diminished differences in δ band activity. Indeed, more detailed time-frequency analyses revealed increased δ power in the second session in the monitor display group in frontal electrodes, while there were no differences in the VR group. The notion of the VR effect on the attention state in the first session can be further confirmed by the results of spectral entropy analyses. Spectral entropy is often considered in the context of attention (Ke et al., 2014;Lesenfants and Francart, 2020;Lesenfants et al., 2018) pointing out that the state of the attention can be reliably detected using this method. Our results showed lower spectral entropy in the VR group than in the monitor display group in the encoding phase in session 1. In the remaining two sessions spectral entropy of the monitor display group gradually decreased, reaching a significant difference against the first session in session 3 and leveling with the VR group, in which entropy level was stable over all three sessions. Interestingly, all spectral differences between the VR and monitor display groups were found in the encoding phase, suggesting a positive effect of the VR environment in this particular phase of working memory processing. This observation can be further confirmed by stronger, left hemisphere theta band connectivity most pronounced in the encoding phase as revealed by PLV analyses. Summerfield & Mangels (Summerfield and Mangels, 2005) found that item-context binding during the encoding phase is mediated by fronto-posterior EEG phase locking within and between hemispheres in the θ band, also Weiss et al. (Weiss et al., 2000) observed increased θ band synchronizations during successful encoding of concrete and abstract nouns. Strong fronto-parietal reciprocal coupling during Working Memory tasks (visual and auditory), in different frequencies, especially θ, γ was also confirmed in the work of Blinowska et al. (Blinowska et al., 2013). Finally, Sato & Yamaguchi (Sato and Yamaguchi, 2007) found that θ coherence increased during successful encoding of the object-place associations. Our second observation concerned the activity in higher EEG frequencies (β and γ). Spectral analyses of the VR and monitor display groups revealed higher activity in the VR participants mostly in the maintenance phase in session 1 and in all task phases in session 2. Several studies indicated the role of higher frequency EEG bands in the maintenance phase of working memory. Notably, Fernández et al. (Fernández et al., 2021) found an increase of β activity in occipital and occipito-temporal regions during the delay period of DMS tasks. Also, Palva et al. (Pavlov and Kotchoubey, 2022) posited that β activity is related to the maintenance of object representations in working memory. More detailed time-frequency analyses showed higher spectral power in the γ band in the monitor display group in session 2 in the maintenance and retrieval phases, which may indicate that participants of this group adapted to the game environment. The better performance of the VR group in session 1 may also be explained by the higher spectral power in that group in the θ and γ bands in the encoding phase, as revealed by time-frequency analysis. θ and γ bands were found to constitute a working memory mechanism (Alekseichuk et al., 2016;Chaieb et al., 2015;Roux and Uhlhaas, 2014;Kamiński et al., 2011) and higher spectral power of those bands in the encoding phase of VR group may indicate higher performance of their short term memory processes resulting from attention to internal processes (Harmony et al., 1996). Finally, our study revealed that subjects in the monitor display group exhibited significantly higher average θ/β2 (21-30 Hz) ratios than those in the VR group. The θ/β ratio was initially thought to reflect the arousal mechanism, but subsequent research has suggested that it represents cognitive processing capacity (Clarke et al., 2019) and serves as an objective indicator of executive cognitive control, particularly attention control, in healthy adults. The θ/β ratio, which is the ratio of fronto-central θ (4-7 Hz) to β oscillations (13-30 Hz), has been found to be negatively correlated with attentional control, reinforcement learning, executive function, and age (Finley et al., 2022). Therefore, the higher θ/β2 (21-30Hz) ratio in the monitor display group suggests that the group requires higher mobilization of attention, especially during the maintenance phase. However,observed in our study differences in EEG activity may indicate faster learning rate in the VR environment than in the 2D one; the differences did not survive multiple comparison correction which indicate rather low effect. An important limitation of our study pertains to the insufficient control over the benefits derived from the stereoscopy effect. Introducing an additional experimental group could elucidate whether the effects noted in the initial session were a result of the experienced immersion in the VR due to stereoscopy or a large visual display spanning most of the field of view. Future research should also consider expanding the number of sessions to approximately 10-15 to better understand the long-term impacts of VR training.However, given the trend of diminishing differences between the tested groups observed in our study, we propose that any long-term effects of VR training could, at most, be moderate.Our findings show that the impact of the VR environment may be differentiated for various cognitive functions and the benefits are primarily limited to the learning rate. Therefore, further studies comparing behavioral results and learning rates for different cognitive functions are needed before any clinical applications. Another conclusion that could be drawn from this study is that observed differences in brain activity induced by different task environments do not necessarily result in desired behavioral improvements.I n r e v i e w I n r e v i e w