Psychophysiology in Studying VR-Mediated Interactions: Panacea or a Trick? Valuable Applications, Limitations, and Future Directions

Over the last 10 years, we have seen a growing interest in the usage of psychophysiology–i.e., examining physiological signals, such as electrodermal activity, heart rate, and electroencephalography, to study psychological phenomena–in Virtual Reality (VR) psychological research. The change is reflected in a rise in Google Scholar query results—from 811 in 2009 to 2,500 in 2019. This increase reveals a significant shift in the methodology of this field where both direct (e.g., questionnaires) and indirect measures of users’ experience (e.g., psychophysiology) are used. At the same time, research interest in the social applications of VR technology (i.e., using VR to communicate, interact, and stay in presence of agents/avatars) has been consistently growing (discussion: Churchill et al., 2012). The opinions on the usage of psychophysiology in VR research vary from enthusiastic (e.g., Blascovich et al., 2002; Meehan et al., 2002; Wiederhold and Rizzo, 2005; Bombari et al., 2015; Kisker et al., 2019) to reserved (e.g., Slater, 2004; Friedman et al., 2005; Llobera et al., 2010, p. 11). In this paper, we reflect on this polarization, discussing the possible advantages, limitations, and future directions of this methodology in the context of building impactful VR communication platforms.


INTRODUCTION
Over the last 10 years, we have seen a growing interest in the usage of psychophysiology-i.e., examining physiological signals, such as electrodermal activity, heart rate, and electroencephalography, to study psychological phenomena-in Virtual Reality (VR) psychological research. The change is reflected in a rise in Google Scholar query results-from 811 in 2009 to 2,500 in 2019. This increase reveals a significant shift in the methodology of this field where both direct (e.g., questionnaires) and indirect measures of users' experience (e.g., psychophysiology) are used. At the same time, research interest in the social applications of VR technology (i.e., using VR to communicate, interact, and stay in presence of agents/avatars) has been consistently growing (discussion: Churchill et al., 2012).
The opinions on the usage of psychophysiology in VR research vary from enthusiastic (e.g., Blascovich et al., 2002;Meehan et al., 2002;Wiederhold and Rizzo, 2005;Bombari et al., 2015;Kisker et al., 2019) to reserved (e.g., Slater, 2004;Friedman et al., 2005;Llobera et al., 2010, p. 11). In this paper, we reflect on this polarization, discussing the possible advantages, limitations, and future directions of this methodology in the context of building impactful VR communication platforms.

VALUABLE APPLICATIONS
First, we will discuss some valuable applications of psychophysiology that in our opinion can be beneficial for social VR, namely, (1) making VR research more objective, (2) enabling control over virtual character development, and (3) increasing engagement.

Making Research More Objective
From the early years of studying VR-mediated interactions, with both other users and bots, researchers aimed to quantify the VR experience. After the initial spread of the paper-pencil methods (e.g., Barfield and Hendrix, 1995;Witmer and Singer, 1998), more objective measurement (e.g., proximity measurement; Bailenson et al., 2004) was introduced. This trend was supported by Slater's (2004) heavy critique of the subjective measurement of presence, based on the inaccuracy of the declarative measurement on the one hand and the first attempts to use psychophysiological measurement in Virtual Environments (VE) on the other (Meehan et al., 2002;Hoyt et al., 2003). Recent studies show that psychophysiology may indeed be one of the ways of making VR-mediated interaction studies more objective by looking for physiological correlates of, e.g., engagement (Czarnek et al., 2020), social presence (Syrjämäki et al., 2020), or anxiety levels (Mühlberger et al., 2007;Crescentini et al., 2016).

Virtual Characters' Creation
Another promising direction is employment of the psychophysiological measures to test the virtual characters' creation, that is, a process of making decisions about its exact graphical form. In a virtual social situation, the interaction is mediated primarily by the spectator's visual perception of a virtual character. Therefore, we believe that it is essential to put more effort into the testing phase of virtual characters' creation, which would allow to standardize social virtual experience. For instance, in a recent study, Syrjämäki et al. (2020) investigated the effects of eye contact on social presence. We think that this is a particularly well-designed experiment as it combines precisely defined single experimental manipulation (eye-gaze) with subjective-free psychophysiological measurement (EDA). We find it promising to take this step-by-step, bottom-up approach in virtual character development that aims to break down complex stimuli into their basic elements. Ultimately, it can provide us with reliable knowledge of how to manipulate the virtual characters' features for making VR social interactions more engaging.

Engagement Increase
Usually, VR developers aim to create environments that engage their users in the virtual world and consequently facilitate meaningful interactions and authentic communication in VE. Some of the researchers try to make use of physiology for creating engaging training simulators (e.g., Czarnek et al., 2020;Muñoz et al., 2020) and games (Dynamic Difficulty Adjustment; e.g., Liu et al., 2009;Chanel and Lopes, 2020).
Although not yet tested enough (Barreda-Ángeles et al., 2018), in future studies, the psychophysiological methodology might become a useful tool for studying engagement in VR social situations by being an indirect, unbiased indicator of the characteristics of interaction experience. A new line of research may be focused on transferring findings based on psychophysiology to creating engaging environments in the gaming or simulator training contexts or to creating meeting spaces where interactants could feel "as if " in real social situations.

LIMITATIONS
The social VR research is a relatively new field, at the crossroads between social and technical sciences suffering from common methodological problems characteristic for interdisciplinary research (Schmälzle and Meshi, 2020). Not all researchers share enthusiasm toward the use of indirect measurement in VR.
Here, we discuss some of the limitations of this approach. In particular, we focus on (1) the danger of assuming isomorphic relationship between the signal and the psychological construct and the usage of (2) complex experimental design and (3) unstandardized stimuli.

Assuming an Isomorphic Relationship
First, a recurring problem is assuming an isomorphic (one-toone) relationship between the physiological signal (e.g., heart rate) and the psychological construct (e.g., presence) (Cacioppo et al., 2007, p. 804). The limitations of causal relationship inference between physiology and psychological construct were raised by several researchers in the field (Llobera et al., 2010, p. 11;Kivikangas et al., 2011;general discussion: Cacioppo and Tassinary, 1990). As Cacioppo and Tassinary (1990, p. 24) underline "simply knowing that manipulating a particular psychological element [..] leads to a particular physiological response [..] does not logically enable inferences about the former based on observations of the latter." Isomorphic assumption is made in many papers, both relatively old (Meehan et al., 2002) and recent (Hartanto et al., 2014, p. 2-10;Athif et al., 2020;Gill, 2020). This practice might create an illusion of having solid support for research findings when there is none. The VR research can be especially susceptible to that problem due to close ties between the research and industry. It might be tempting for VR researchers working under the market pressure to take this assumption to meet the expectations of the developers wanting their devices to be "objectively" validated.

Usage of Complex Design
The second limitation is the usage of complex designintroducing too many experimental manipulations to a single experiment. That, along with unstandardized stimuli employment (to be discussed below), may be a serious threat to the internal validity of the study (Stemmler, 2003). The more complex design, the higher the chance that change in physiological signal is due to confounders and not independent variables. Stemmler (2003, p. 241) calls physiological response "the slave of many masters, " meaning that the change in the physiological signal can be due to: the stimuli, the psychological processes, as well as the action performed by the participant. Therefore, it is important to limit the impact of the confounders by using rather simple designs preferably based on Mill (1862) method of difference. Unfortunately, in VR-based research, it is common to violate this method by creating conditions greatly differing in terms of the scenario, sensory input, virtual character characteristics, user's possible actions, etc. (e.g., Vinayagamoorthy et al., 2004, p. 149-152;Slater et al., 2006;Llobera et al., 2010, p. 4-7;Hartanto et al., 2014, p. 3-7). Consequently, it becomes impossible to causally connect the resulting physiological signal change with any particular part of experimental manipulation.

Lack of Pre-testing
Third, a specific example of the inability to control for confounders is the lack of pretesting of the stimuli and procedures used in the VR-based social experiments. As Emmerich and Masuch (2016) point out, lack of pretesting is one of the problems connected to the research in VR conducted with virtual characters (for discussion: Sterna et al., 2019). Besides virtual characters, experimental procedures in VR experiments are usually also not pretested, made ad hoc, without the usage of standardized materials or performing pilot tests (with few exceptions, e.g., Garau et al., 2005, p. 107;Zimmer et al., 2019, p. 7-12;Harjunen, 2019, p. 34-48;Niu et al., 2020, p. 060413-3). When not pretested, the efficacy of the experimental manipulation cannot be ensured. As a result, the studies become incomparable, and accumulation of knowledge impossible.

FUTURE DIRECTIONS
In the last part of the paper, we wish to reflect on the possible methodological directions of VR-mediated interactions research inspired by classical psychophysiology (Cacioppo et al., 2007). The solutions we propose aim to optimize the experimental design and results interpretability (Cacioppo et al., 2007, p. 848). We think that plenty of errors can be reduced by taking measures presented in the following sections. For analogous discussion regarding the methodological issues on the junction between communication science and neuroscience, see Schmälzle and Meshi (2020).

Higher Precision in Experimental Design
Firstly, we encourage the researchers to strive for higher precision in experimental design to reduce the impact of possible confounding factors. As aforementioned, one of the limitations of the current VR-based psychophysiological research is the usage of complex designs with many factors manipulated between the conditions. Following the current recommendations on psychophysiological research (Cacioppo et al., 2007), we suggest simplifying the manipulations made in the experiments by limiting them to one characteristic of environment or virtual character or other single feature at a time (e.g., Pan and Slater, 2007, p. 102-104;Harjunen, 2019, p. 34-38). Additionally, detailed descriptions of virtual character actions (as in, e.g., Pan et al., 2012) should be included. This approach might better warrant the precise identification of the cause of change observed in the physiological signal as a dependent variable.

Standardization and Pretesting
Secondly, we encourage the creation of standardized databases of experimental stimuli or at least pretesting the stimuli before the study (see current guidelines: Kourtesis et al., 2020). It is a common practice in psychophysiological experiments to make use of databases of experimental stimuli (e.g., Nencki Affective Picture System, NAPS; Marchewka et al., 2014), and it seems possible to create similar databases of virtual characters or its components (example of VR-video database: Li et al., 2017). VR allows for systematic changes in the environments and virtual characters (e.g., Hale et al., 2015) with precise control over all aspects of the presented stimuli (Wiederhold and Rizzo, 2005). Adopting this approach in combination with standardization increases the interpretability of research findings by linking them clearly to single experimental manipulation. It is especially relevant when making use of high-level constructs, such as presence, copresence, realism, and immersion, as these are already non-unitary and multifaceted in nature.

Joint Analyses
Thirdly, we recommend the wider use of joint analysis of two types of temporal data: VE events and physiological signal (Friedman et al., 2005). Automatic extraction of time-logged VE events can be combined with continuous measurement of physiological data to analyze the phasic component of the signal. The event-related analysis is a powerful and commonly used method in psychophysiological studies (Cacioppo et al., 2007). Nevertheless, it is regrettably rarely used solution in VR research (Liebold et al., 2017). In this type of analysis, we extract only those parts of the signal that correspond to a defined repetitive event (e.g., an action of a virtual character), average those signal fragments over repetitions of a given experimental condition and statistically test it in comparison with the means obtained for other experimental conditions (Kivikangas et al., 2011). This approach is suitable for both passive experimental designs, e.g., free-viewing task (e.g., Syrjämäki et al., 2020), and active ones, requiring participants to perform repetitive events, e.g., reacting to specifically designated stimuli (e.g., Meehan et al., 2002). As every single measurement has a random error involved, by averaging the measurements over repetitive trials, we are getting closer to the true value of the signal, thereby increasing signal-to-noise ratio. Therefore, this design might lead to significant noise reduction and ensure better confounders control.

DISCUSSION
In summary, in this paper, we reflected on the current voices raised regarding the usage of psychophysiology in social VR research. We critically discussed the methodological issues and proposed future directions in research on virtual social spaces. Although in the field of social VR research presently it is impossible to infer high-level psychological constructs, such as presence, engagement, or satisfaction, based solely on physiology (Barreda-Ángeles et al., 2018), we believe that measuring physiological changes in well-controlled conditions will bring us closer to designing VE fitted to human communication needs.

AUTHOR CONTRIBUTIONS
RS wrote the major part of the paper, contributed to the conception of the paper, and performed the article search. AS wrote the minor part of the paper and performed the article search. JP and MK wrote the minor part of the paper. All authors listed have made substantial intellectual contribution to the work, revised the manuscript, and read and approved the submitted version.