Feasibility of the virtual reality-based assessments in patients with panic disorder

Introduction: Recurrences and diagnostic instability of panic disorder (PD) are common and have a negative effect on its long-term course. Developing a novel assessment tool for anxiety that can be used in a multimodal approach may improve these problems in panic disorder patients. This study assessed the feasibility of virtual reality-based assessment in panic disorder (VRA-PD).

Methods: Twenty-five patients with PD (ANX group) and 28 healthy adults (CON group) participated in the study. VRA-PD consisted of four modules based on the key components of cognitive behavior therapy for an anxiety disorder: “Baseline evaluation module” (M0), “Daily environment exposure module” (M1), “Relaxation module” (M2), and “Interoceptive exposure module” (M3). Multiple evaluations, including self-rating anxiety scores (AS) and physiological responses [heart rate variability (HRV) index], were performed in three steps at M1, M2, and M3, and once at M0. Comparisons between patients with PD and healthy controls, factor analysis of variables in VRA-PD, changes in responses within modules, and correlation analysis between variables in VRA-PD and anxiety symptoms assessed by psychological scales were performed.

Results: All participants completed the VRA-PD without discontinuation. The ANX group reported significantly higher AS for all steps and a smaller HRV index in M1 (steps 1 and 2) and M2 (step 1). Repeated-measures analysis of covariance (ANCOVA) revealed significant interaction effects for AS in M1 (F = 4.09, p = 0.02) and M2 (F = 4.20, p = 0.02), and HRV index in M2 (F = 16.22, p < 0.001) and M3 (F = 21.22, p = 0.02). The HRV index only indicated a good model fit for the three-factor model, reflecting the construct of the VRA-PD. Both AS and HRV indexes were significantly correlated with anxiety and depression symptoms.

Discussion: The current study provides preliminary evidence that the VRA-PD could be a valid anxiety behavior assessment tool.

1. Introduction

Panic disorder (PD) is an anxiety disorder characterized by the occurrence of unexpected panic attacks, in which an overwhelming anxiety accompanied by a sequence of physiological and/or cognitive symptoms emerges suddenly and without an obvious external reason (1). According to epidemiological research, the lifetime prevalence of PD in the general adult population is between 1.4 and 4.1% (2–4). Moreover, recurrences of anxiety disorders are common, with reports of 56 and 58% for PD without agoraphobia and PD with agoraphobia, respectively (5). Recurrences of anxiety disorders have a negative effect on the long-term course of these diseases (6). Therefore, modifying treatment strategies by identifying recurrence-related predictors might contribute to the decrease in recurrence.

Over the last decades, a growing body of research has discovered diagnostic changes within anxiety disorders (2, 6), as well as between depressive and anxiety disorders (7, 8). In addition, it has been reported that such diagnostic instability is related to recurrence (9). These results on diagnostic instability in anxiety disorders are consistent with twin and family research on comorbidity findings, indicating that overlapping genetic etiological components are likely represented as personality characteristic of neuroticism (10). The predictive problem of recurrence and diagnostic instability shows the lack of an evaluation system that relies on expert interviews and self-report in the current anxiety disorder evaluation (11). Moreover, concerning the recent need for precision medicine, categorical diagnostic systems are considered a need for significant improvement in psychiatry related to these issues (12).

In this context, the National Institute of Mental Health has initiated the Research Domain Criteria (RDoC) initiative, which stresses a multimodal approach to identify relationships between neurobiological markers, clinical behaviors, and trait characteristics that span traditional diagnostic categories (13). However, few investigations of anxiety disorders using the RDoC have been described. It may be because behavior-related variables have not yet been well established in the negative valence systems and arousal and regulatory systems; domains associated with anxiety disorder in the RDoC matrix (14). Even in the previous research, although anxiety was not explicitly investigated, the “Emotion” task or “threat conditioning and extinction” task was used as a measure (15, 16). Therefore, developing a tool capable of assessing anxiety-related behavior will play a crucial part in our ability to comprehend and design treatments for those suffering from anxiety.

Virtual reality (VR) may be an advantageous tool for assessing anxiety behaviors because it immerses people in a virtual world that mirrors daily living requirements (17–20). Previous research has shown that VR offers an enormous opportunity to evaluate whether therapies can be applied in real life and can allow scientists to observe an individual’s real-time interactions with virtual entities (21, 22). We also reported that psychological factors, including communication style and life satisfaction, are related to behavior in VR (23, 24). VR can also induce intended anxiety behavior in patients with anxiety disorders, based on the findings that the use of VR is effective for cognitive behavioral therapy for anxiety disorders and the exploring the mechanism of action underlying exposure therapy (25, 26). In addition, providing and observing stimuli in real time is advantageous for applying a multimodal approach, including the simultaneous acquisition of physiological data. Moreover, the use of VR reduces human resources and costs and thus has an advantage in terms of sustainability (27). Therefore, it can be said that VR possesses the necessary qualities for assessing anxiety-related behavior.

In response to this need, we developed virtual reality-based assessments in patient with panic disorder (VRA-PD), an anxiety behavior evaluation system, to evaluate the characteristics of an individual suffering from anxiety. The VRA-PD uses the virtual environment data of the VR-based relaxation self-training program for PD, the feasibility of which has been confirmed in other studies (25). In the VRA-PD, we acquired both subjective anxiety and physiological responses in virtual environments that were developed to induce anxiety-related behaviors. The VRA-PD scenario comprises modules that represent different parts of cognitive behavioral therapy for anxiety disorders. These modules included a claustrophobic and social environment (getting on an elevator), interoceptive sensations (hyperventilation and head down), and relaxation training (28, 29).

The current study aimed to summarize the implementation method of this VRA-PD and evaluate its feasibility in assessing anxiety disorders by applying it to patients with PD and healthy adults without anxiety. By introducing the system configuration of VRA-PD and the content and purpose of its modules, we attempted to demonstrate how virtual reality technology may be used as a tool for assessing mental health. To assess it feasibility, this study explored the discriminant, construct, and convergent validity of VAR-PD in patients with PD. We hypothesized that variables within VRA-PD differ significantly between patients with PD and healthy controls, represent a three-factor model reflecting the construct of VRA-PD, and significantly correlate with anxiety symptoms.

2. Materials and methods

2.1. Participants

Patients with PD and healthy controls, matched for sex, age, and marital status, were recruited through outpatient clinics and public advertisements. All examinations and procedures were conducted at the psychiatric outpatient clinic of Korea University Guro Hospital. All patients were interviewed by a psychiatrist using the Mini-International Neuropsychiatric Interview (MINI) to screen for histories of psychiatric illness and drug use (30).

The inclusion criteria for patients with PD were as follows: (1) symptoms complied with the diagnostic criteria for PD according to the Structured Clinical Interview of the MINI; (2) age between 19 and 50 years; and (3) voluntary participation with signed informed consent. The exclusion criteria for patients with PD were as follows: (1) nervous system diseases or other mental illnesses, major physical illnesses, or serious infectious diseases; (2) alcohol or substance abuse; and (3) major depressive episodes, bipolar I disorder, or psychotic disorders. The inclusion criteria for healthy controls were as follows: (1) age between 19 and 50 years and (2) voluntary participation with signed informed consent. The exclusion criteria for healthy controls were as follows: (1) suffering from nervous system diseases or other mental illness, major physical illness, or serious infectious diseases; and (2) history of mental illness. The study was approved by the local ethics committee of Korea University Guro Hospitals (2021-GR0057), and each participant signed a written informed consent form after being notified of the objectives, methods, and possible risks of the study. All aspects of the study were conducted in accordance with the 1964 Declaration of Helsinki.

Thirty patients and 30 healthy controls met the eligibility criteria and participated in this study. Seven participants were excluded due to incomplete physiological assessment data. The final sample consisted of 53 adults, representing two groups: 25 adults with PD (ANX group) and 28 healthy adults (CON group). Table 1 shows the demographic characteristics and means assessment scores of the sample.

TABLE 1

Table 1. Demographic and psychological characteristics of each group.

2.2. System configuration and operation

This section presents the design of the proposed VRA-PD, as outlined in Figure 1. The VRA-PD scenario was developed by a psychiatrist with experience in developing VR-based programs for psychological interventions and treating patients with anxiety disorders using a VR-based relaxation self-training program. In addition, the scenario and virtual environment were modified through a review by two other psychiatrists who were experts in treating anxiety disorders.

FIGURE 1

Figure 1. Configuration diagram of the virtual reality-based assessment in panic disorder (VRA-PD). A box with rounded corners inside an angled rectangle representing a module describes the virtual environment. An empty arrow means a process that checks whether to proceed after the virtual environment. Items marked with dotted lines indicate recorded variables. M, module; DRR, diaphragmatic respiratory relaxation; PMR, progressive muscle relaxation; HV, hyperventilation; HD, head down.

After the login process, voice guidance on the purpose of the VRA-PD and how to operate the controller and head-mounted display (HMD) were provided to the users. Subsequently, users progressed through four consecutive modules; “Baseline evaluation,” “Daily environment exposure,” “Relaxation,” and “Interoceptive exposure.” Except for the baseline evaluation module, the other three modules consisted of three steps as follows: (1) pre-evaluation environment, (2) virtual environment 1, and (3) virtual environment 2. At the end of each module and virtual environment, the decision to proceed to the following module or the following virtual environment was selected.

The virtual environments of the VRA-PD were produced by modifying the virtual environments of the VR-based relaxation self-training program developed in previous research and confirmed to be suitable for treatment (25). Except for the virtual environment of the daily environment exposure module produced with animated graphics, the remaining three virtual environments were created with a 3D video filmed in a real scene using a 360-degree 3D camera (Insta360 Pro, Insta360 Inc., Irvine, CA, USA).

The virtual environments were exhibited using an HMD comprising an Oculus Quest 2 and two Oculus Touch Controllers to provide a 360-degree view with an 89-degree field of vision. Users may independently execute VRA-PD based on the built-in instructions provided as text on the screen or audibly through the audio system. Furthermore, Quest 2 and the physiological data acquisition system were connected to the same laptop to interlock the physiological data and VRA-PD. The researcher checked the progress through a laptop screen to monitor the appropriate process. Users can proceed with all VRA-PD processes by clicking icons on the screen with a spear-shaped pointer using the controller. When it was no longer feasible to continue experiencing the virtual environment due to cybersickness or rising anxiety, the user was notified that the evaluation could be terminated by removing the HMD.

2.3. In-app measurements

Subjective VRA-PD anxiety experiences and physiological data were collected in the VRA-PD. Evaluation of subjective anxiety experience (anxiety scores, AS) was based on the question, “How anxious are you right now?” The participants’ responses were rated on a visual analog scale (VAS), which presented “not at all” (0 points) at the left end of a horizontal line and “very much” (100 points) at the right end (Figure 2A-2). Subjective anxiety experience was recorded as AS, and the time at which ASs were received for each module and the variable names are shown in Figure 1.

FIGURE 2

Figure 2. Screenshots of the virtual reality-based assessment in panic disorder (VRA-PD). In the program, Modules 1, 2, and 3 consisted of three steps (pre-evaluation, step 1, and step 2). Module 0 (baseline evaluation module) with a comfortable virtual environment composed of beach scenes (A-1). A comfortable virtual environment (A-1) was employed in step 1 of Module 1. In addition, it was also used in Module 3 at 1 min and 30 s in the second half of the interoceptive exposure of Step 2 and Step 3. Users rate their self-rating anxiety scores on the visual analog scale constructed in the virtual environment (A-2). Users selected whether to proceed at each step and each module (A-3). In Module 1, Steps 2 (B-1) and 3 (C-1) each consist of 2 min process in an elevator with two and six passengers, respectively. Users performed diaphragmatic respiratory relaxation (DRR) in step 2 (B-2) of Module 2 and progressive muscle relaxation (PMR) in step 3 (C-2) for 2 min each according to human motion, guidance, and time displayed in the virtual environment. Panel (B-3) shows the hyperventilation virtual environment of module 3, and panel (C-3) shows the head down virtual environment. In Module 3, the guidance in the virtual environment was used in the same manner as in Module 2 for 30 s to provide stimulation. Panel (A-1) shows the comfortable scene for the remaining 1 min and 30 s. The program was conducted in Korean; however, the examples of the help balloon in this figure are presented in English to aid the understanding of readers.

Physiological data, composed of photoplethysmogram (PPG) data, were measured for 2 min in each virtual environment. As shown in the overall system configuration (Figure 1), data were collected once in the baseline evaluation module, thrice in the daily environment exposure module, and twice in the other two modules. PPG data were measured using a PPG sensor [Model: ubpulse T1 (Pulse Analyzer, MFDS, Certification No. 11-1296), LAXTHA Inc., Daejeon, South Korea], approved as a medical device by the Ministry of Food and Drug Safety of South Korea. Ubpulse T1 has a sampling rate of 250 Hz and a bandpass frequency of 0.3–10.6 Hz.

A PPG sensor was placed on the index finger of the left hand, and the hands connected to the sensors were placed on the armrest of a chair and relaxed. Manicured fingernails or foreign substances were removed from the fingernails attached to the pulsing electrode. In addition, tight-fitting sleeves, disposable bands, and rubber bands that exerted pressure on the arms or fingers were removed.

The 2-min PPG measurement period may not be adequate to ensure repeatability (31). In addition, artifacts induced by user movement due to the nature of the VR environment may impact data quality. The PPG parameter employed in this research was the heart rate variability (HRV) triangular index (HRV index). The HRV index is based on geometric approaches insensitive to data quality and demonstrates robustness to outliers and artifacts (32, 33).

Therefore, the heart rate variability triangular index (HRV index) was employed among the parameters obtained by analyzing PPG because, the HRV index has been examined across anxiety disorder patient groups, including those with PD (34–36), and its predictive value in heart-related disorders is also established (37). The analysis for the HRV index was conducted based on standard methods (38).

2.4. Contents of each module

2.4.1. Baseline evaluation module (module 0)

The baseline evaluation module was designed to assess the baseline state before exposure to the virtual environment of the module for participant evaluation and to introduce the evaluation method repeatedly used in each module. Users were instructed to rest for 2 min in a comfortable virtual environment composed of a beach scene with a clear sky (Figure 2A-1). In this module, physiological data were measured while experiencing a comfortable virtual environment, and the participants’ subjective anxiety experience was evaluated afterward.

2.4.2. Daily environment exposure module (module 1)

The daily environment exposure module was designed to assess anticipatory anxiety in daily environments and changes in anxiety according to changes in stress intensity. This virtual environment was designed with an elevator boarding environment, one of the ordinary daily life situations precipitating anxiety reported in a previous study using VR in patients with PD (39). The daily environment exposure module consisted of three virtual environments: (1) evaluation of anticipatory anxiety standing in front of the elevator; (2) elevators in which two passengers rode together; (3) elevators in which six passengers rode together. When the users selected the Module “Daily environment exposure” among the four icons shown after Module 0, the screen was converted into a virtual environment standing in front of the elevator entrance in the building (Figure 2A-2). In the evaluation of anticipatory anxiety, users were instructed to imagine the scene of the boarding elevator with two and six passengers for 2 min. Subsequently, two-stage virtual environments were conducted in which two and six passengers boarded the elevator together in order for 2 min each (Figures 2B-1, 2C-1). Physiological data acquisition during each virtual environment for 2 min and subjective AS ratings after each environment were performed three times for each virtual environment (Figures 2A-3).

2.4.3. Relaxation module (module 2)

The relaxation module was designed to evaluate the relaxation ability of patients with anxiety disorders. This module was designed to sequentially experience two virtual environments associated with diaphragmatic breathing relaxation and progressive muscle relaxation, which are essential elements of cognitive behavioral therapy (40). When the user finished the daily environment exposure module and pressed the “Next” icon, the screen was converted into the beach virtual environment screen used in the baseline evaluation module. They listened to the voice guidance about the module’s objectives and content, and the subjective AS was rated. Notably, physiological data were not acquired in this scene. In a virtual environment related to diaphragmatic breathing relaxation and progressive muscle relaxation, users were instructed to relieve tension by diaphragmatic breathing or forcefully contracting the muscles for 10 s and then slowly relaxing them for 20 s. Users underwent relaxation according to the help video of the assistant’s motion played in the center of the screen for 2 min (Figures 2B-2, 2C-2). Physiological data acquisition and subjective AS ratings were performed twice for each virtual environment.

2.4.4. Interoceptive exposure module (module 3)

The interoceptive exposure module was designed to evaluate the sensitivity to interoceptive stimuli related to anxiety disorders. Users who chose to proceed from the relaxation module to the interoceptive exposure module were instructed by the voice guidance for the interoceptive exposure module. They proceeded with the pre-state self-rating anxiety evaluation in the same way as the relaxation module. Afterward, users experienced two virtual environments sequentially as the video of the assistant’s motion played in the center of the screen for hyperventilation (HV) and head down (HD), respectively (Figures 2B-3, 2C-3). Because of the difficulty of performing interoceptive stimuli and the risk of inducing panic attacks, both virtual environments performed HV and head down for the first 30 s out of 2 min in the production process. The remaining 1 min and 30 s were designed to look at the comfortable environment of the beach. Physiological data acquisition and subjective AS ratings were performed twice for each virtual environment.

2.5. Psychological assessments

2.5.1. Interview measures

The panic disorder severity scale (PDSS) is a seven-item, clinician-administered standard assessment of PD and agoraphobia symptom severity (41). This study used the Korean version of the PDSS, translated and standardized in 2001 (42). Five PDSS questions were used to evaluate the basic DSM-IV symptoms of PD, while two additional measures were for assessing impairment in social and occupational functioning. Each item was graded on a scale from 0 to 4 for a total of 28 points. The PDSS has demonstrated sound psychometric properties with good internal consistency (Cronbach’s α = 0.88) and test-retest reliability (r = 0.71) (43). The Cronbach’s α for the ANX group in this study was 0.86.

2.5.2. Liebowitz social anxiety scale: Self-report version

The Liebowitz social anxiety scale: self-report version (LSAS-SR) is a self-reported version of the LSAS (44), a 24-item semi-structured interview assessment of fear and avoidance in various social and performance circumstances. Participants were given a copy of the questionnaire and were asked to score each item twice, based on how much they dreaded the circumstance and then on how often they avoided it. The psychometric qualities of the LSAS-SR were presented with good internal consistency (Cronbach’s α = 0.95) and test-retest reliability (r = 0.70) (45). Each of the 24 events was assessed on a 4-point scale (0–3), first for fear and then for avoidance. Zero represents no fear, one represents mild fear, two represents considerable fear, and three represents extreme terror. In addition, zero indicates no avoidance, one indicates occasional avoidance (1–33 percent of the time), two indicates often avoiding the circumstance (34–67 percent of the time), and three indicates always avoiding the situation (68–100 percent of the time). The Korean version of the LSAS-SR, adapted and validated by Kang et al. (46) in 2013, was used. In the current study, Cronbach’s alpha of the LSAS-SR-fear and LSAS-SR-avoidance was 0.94 and 0.95, respectively, indicating an acceptable level of internal consistency.

2.5.3. Generalized anxiety disorder scale

The generalized anxiety disorder scale (GAD-7) is a 7-item self-reported instrument designed to screen for GAD symptoms (47). Participants were asked how often they had been disturbed by anxiety symptoms (such as difficulty relaxing or excessive concern) over the previous 2 weeks. On a 4-point Likert-type scale, items are ranked from 0 (not at all) to 3 (almost every day). Higher scores indicate more severe symptoms of GAD. In primary care settings and with the general population, the GAD-7 has been proven to have sufficient psychometric qualities, including strong internal consistency (Cronbach’s α = 0.92), clinical value, and construct validity (48, 49). This study used the Korean version of the GAD-7 with confirmed psychometric qualities (50). The GAD-7 also showed good reliability in the current study (Cronbach’s α = 0.93).

2.5.4. Hospital anxiety and depression scale

The hospital anxiety and depression scale (HADS) consists of two subscales: the HADS-A, designed to detect anxious states, and the HADS-D, designed to detect depressive states (51). Each subscale consisted of seven items with a 4-point ordinal response format. Scores ranged from 0 to 21 for each subscale, with higher scores indicating higher levels of anxiety or depression. Participants answered each item by thinking about how they felt and/or behaved during the past week. In this study, the Korean versions of the HADS with good internal consistency [Cronbach’s α = 0.89 (HADS-A) and 0.86 (HADS-D)] were used (52). Cronbach’s α of the HADS-A and HADS-D was 0.94 and 0.84, respectively, indicating high levels of internal consistency in the current study.

2.5.5. Simulator sickness questionnaire

The simulator sickness questionnaire (SSQ), a 16-item questionnaire, was used to examine the occurrence and severity of cybersickness symptoms when immersed in virtual environments (53). The SSQ has been extensively used to assess and minimize simulator sickness and to investigate any significant impacts (e.g., age, gender, and equipment characteristics) on simulator sickness (54). Cronbach’s α for the Korean version of the SSQ was 0.89, indicating good internal consistency (55). The SSQ also showed good reliability in the current study (Cronbach’s α = 0.93).

2.6. Statistical analyses

All statistical analyses were performed using a commercial software package (IBM SPSS Statistics v28.0, AMOS v.28.0, for Windows, IBM Corporation, Armonk, NY, USA). Demographic variables, and psychological assessments were compared between the ANX and CON groups using an independent t-test for continuous data and the chi-square test for categorical data. Distributions of the subjective AS and physiological measures of the VRA-PD were presented with corresponding item skewness and kurtosis values and then compared between the two groups using the Mann–Whitney test. A repeated-measures analysis of covariance (ANCOVA) was performed to evaluate content validity of the VRA-PD, controlling for age, sex, smoking usage, alcohol usage, and HADS-D scores. Since the difference in the subjective AS between the two groups was significant, both the group factor and the repetition effects were included. For physiological data, repeated-measures ANCOVA for only the repetition effect was executed only in the ANX group, a target group. Next, the VRA-PD construct validity for each acquired variable was examined to show whether the variables in the VRA-PD fit well by module. To validate the factor structure of the VRA-PD, we employed confirmatory factor analyses with maximum likelihood implemented in AMOS v.28.0 concerning each variable in VRA-PD to account for probable non-normality in the data distribution. We tried a three-factor model in which the variables from each module were added to the module factor because each module provides a different stimulus. Model fit was assessed using the chi-squared statistic, comparative fit index (CFI), Tucker–Lewis index (TLI), root mean square error of approximation (RMSEA), and standardized root mean squared residual (SRMR). Although a consensus on acceptable levels of fit indices is still missing (56), a value of 0.95 or more for CFI and TLI, 0.08 or less for RMSEA, and 0.08 or less for the SRMR is considered an acceptable fit (57). Concerning convergent validity, we analyzed correlations among psychophysiological measures in the VRA-PD with psychological assessments concerning anxiety PDSS, LSAS-SR, GAD-7, HADS-D, and SSQ using Spearman’s rank correlation analysis. Statistical significance was set at p < 0.05.

3. Results

3.1. Demographic and self-assessment scales

Statistical tests did not show significant differences in age, years of education, sex, and marital status distribution between the two groups. The ANX group presented more nicotine and alcohol use than the CON group (Table 1). A significant difference was found in the psychological characteristics associated with anxiety, except for the LSAS-SR-fear and LSAS-SR-avoidance subscale scores.

3.2. Acceptability of VRA-PD

Participants in the ANX and CON groups did not discontinue the VRA-PD because of anxiety symptoms, including omitted cases due to inadequate data quality. Cybersickness measured by the SSQ after VRA-PD use was more significant in the ANX group (mean = 34.68, SD = 12.46) than in the CON group (mean = 21.93, SD = 8.21; t₅₁ = −4.44, p < 0.001).

3.3. Distribution of variables in VRA-PD and discriminant validity

Table 2 presents the distribution and descriptive statistics of AS and HRV index of each module with means, standard deviations, median, skewness, and kurtosis concerning ANX and CON groups. The median of all ASs in the CON group was 0, and most of the scores had skewness of more than two, indicating that it was not a normal distribution (58). The skewness and kurtosis values of ASs from the ANX group were within the acceptable range, considering they are a normal distribution. The ANX group showed greater self-rating AS than the CON group for all ratings concerning each module except for baseline AS.

TABLE 2

Table 2. Distribution and descriptive statistics of the psychophysiological data in each module.

The HRV index was significantly lower in the ANX group elevators 2 and 6 of module 1 (M1) and in the diaphragmatic respiratory relaxation (DRR) environment of module 2 (M2) than in the CON group (M1 elevator 2: Mann–Whitney U = −3.86, p < 0.001; M1 elevator 6: Mann–Whitney U = −2.43, p = 0.015; M2 DRR: Mann–Whitney U = −2.82, p = 0.005).

3.4. Content validity

3.4.1. Self-rating anxiety scores

For the repeated measured ANCOVA results of AS for each module (Table 3), a significant interaction effect between group (ANX vs. CON) and time (pre vs. during vs. post) was found for M1 and M2, rather than for the AS of Module 3 (M3). For all modules, a significant main effect of the group was shown, and the post hoc test showed higher AS in the ANX group than in the CON group. A significant main effect of time was revealed for MI.

TABLE 3

Table 3. Results of repeated-measured ANOVA for self-rating anxiety scores in the two groups (ANX and CON) and different time concerning each module.

FIGURE 3

Figure 3. Illustration of the interaction effect of group (ANX and CON) x time (pre, step 1, and step 2) obtained from repeated-measured analysis of covariance (ANCOVA). The adjusted value for sex, age, smoking usage, alcohol usage, and HADS-D are presented for each module. Error bars mean standard errors. The post hoc comparison of the significant ANCOVA results is also described within the plot. The symbols in the graph mean significantly different self-rating anxiety scores between pre and during (#), between pre and post (y), and between during and post (※) from the post hoc comparisons with Bonferroni-corrected p < 0.05. AS, anxiety scores; DRR, diaphragmatic respiratory relaxation; PMR, progressive muscle relaxation.

Figure 3 shows the significant results among the paired t-tests for each group, conducted as a post hoc test for the interaction effect shown in M1 and M2. In the CON group, there was no significant change in AS at each measurement time point (pre vs. E6: t₂₇ = 1.28, Bonferroni-corrected p = 0.424; pre vs. E2: t₂₇ = 1.00, Bonferroni-corrected p = 0.652; E2 vs. E6: t₂₇ = 0.70, Bonferroni-corrected p = 0.980) in M1, and AS M2 progressive muscle relaxation (PMR) was significantly decreased compared to AS M2 pre- and AS M2 DRR (pre vs. PMR: t₂₇ = −2.83, Bonferroni-corrected p = 0.018; DRR vs. PMR: t₂₇ = −2.42, Bonferroni-corrected p = 0.044) in M2.

In the ANX group, AS M1 E6 was significantly increased compared to AS M1 pre and AS M1 E2 (pre vs. E6: t₂₄ = 3.251, Bonferroni-corrected p = 0.006; E2 vs. E6: t₂₄ = 3.258, Bonferroni-corrected p = 0.006). In M2, AS decreased significantly from pre-to DRR and PMR (pre vs. DRR: t₂₄ = −2.954, Bonferroni-corrected p = 0.014; pre vs. PMR: t₂₄ = −4.899, Bonferroni-corrected p < 0.001; DRR vs. PMR: t₂₄ = −3.024, Bonferroni-corrected p = 0.012).

3.4.2. Physiological data

Regarding the HRV index, a significant repetition effect was found for M2 (F₂,₂₂ = 16.21, p < 0.001) and M3 (F₂,₂₂ = 21.22, p < 0.001), rather than for M1 (F₂,₂₂ = 1.15, p = 0.34). The HRV-index significantly increased compared to E6 in both DRR and PMR in M2 (E6 vs. DRR: t₂₄ = 6.07, Bonferroni-corrected p < 0.001; E6 vs. PMR: t₂₄ = 3.39, Bonferroni-corrected p = 0.001). In M3, both HV and head down (HD) decreased significantly compared with PMR (PMR vs. HV: t₂₄ = −5.35, Bonferroni-corrected p < 0.006; PMR vs. HD: t₂₄ = −4.42, Bonferroni-corrected p < 0.006).

3.5. Factor structure, confirmatory factor analysis

Table 4 presents the CFA results without correlating the error terms for each variable in the total sample. Regarding the fit statistics, the results for the HRV index only indicated a good model fit, with chi-square statistics (χ2 = 391.92, χ2/df = 48.99), RMSEA = 0.026, CFI = 0.949, TLI = 0.903, and SRMR = 0.063. Figure 4 shows the standardized estimates of the confirmatory model and the correlations among the modules.

TABLE 4

Table 4. Confirmatory factory analysis of self-rating anxiety score and heart rate variability (HRV) index in virtual reality-based assessment in panic disorder (VRA-PD).

FIGURE 4

Figure 4. The standardized model of the three-factor heart rate variability (HRV) index of virtual reality-based assessment in panic disorder (VRA-PD). M, module; DRR, diaphragmatic respiratory relaxation; PMR, progressive muscle relaxation; HV, hyperventilation; HD, head down.

3.6. Convergent validity

Table 5 presents the results of the Spearman correlation analysis. Several HRV indexes exhibited substantial relationships with anxiety-related PDSS, LSAS-SR, and GAD-7. The HRV index was also significantly correlated with depression-related HADS-D rather than with the SSQ. AS exhibited a strong association with anxiety-related measures. There was also a strong association between AS, HADS-D, and SSQ.

TABLE 5

Table 5. Correlations among variables in virtual reality-based assessment in panic disorder (VRA-PD), psychological characteristics and simulator sickness questionnaire (SSQ) in the ANX group.

4. Discussion

The current study explored the feasibility of a VR program for evaluating anxiety behavior in patients with PD. The VRA-PD execution was conducted with patients with PD and healthy adults to identify the reliability and validity of the VRA-PD on the anxiety-related behaviors of PD. The results suggested that the VRA-PD can potentially be a reliable and valid research tool.

Participants were recruited from among volunteers with PD and those without any psychiatric, nervous, or medical illnesses to clarify the differences in anxiety symptoms. There was no significant difference in the demographic characteristics between the two groups; however, there was a significant difference in the usage of alcohol and smoking. Alcohol consumption and smoking have been linked to increased anxiety disorders, including PD (59, 60). As previous studies have shown that the use of substances can affect physiological data (61), these differences support the inclusion of smoking and alcohol use as covariates in the analysis.

Concerning the psychological characteristics of the recruited subjects, in the CON group, the mean PDSS scores were 0 because they had not experienced any panic attacks; however, the mean PDSS scores of the ANX group were 14.08, which is interpreted as markedly ill (62). In the case of LSAS, there was no difference between the two groups in both subscales of fear and avoidance, whereas, in the GAD-7, HADS-A, and HADS-D, it was confirmed that the ANX group was significantly higher than the CON group. Unlike PDSS and LSAS, which are disease-specific, GAD-7 and HADS-A are mainly used as screening tools. This difference seems to have been confirmed, as it is known that specificity is poor for a detailed diagnosis of anxiety disorders (48, 63). People with PD are known to have a relatively high level of depression (64); therefore, it is reasonable to include the HADS-D as a covariate when looking at how depression affects anxiety and other physiological data.

Regarding acceptability, our sample of 53 participants, including 25 patients with PD and 28 healthy adults, completed all the modules in the VRA-PD without discontinuing. Given that the weighted mean SSQ score of the projection type of the head-mounted display used in the previous study was 29.9 (65), the mean SSQ score of 21.93 of the CON group suggests that VRA-PD causes an acceptable level of adverse effects of cybersickness. Previous studies have suggested that anxiety symptoms can mediate cybersickness in a VR environment (66), and the head down also increases cybersickness in highly stressed people (67). Therefore, the significantly high mean SSQ scores of the ANX group could be caused by interoceptive stimuli in the interoceptive exposure module and anxiety symptoms of PD. Additionally, compared to the 36.27 SSQ mean scores reported in a previous VR exposure study in patients with phobia (68), the 34.68 SSQ mean score in the ANX group was considered acceptable.

The median self-rating AS obtained from the VRA-PD was 0 for most items in the CON group and was significantly higher in the ANX group than in the CON group. Inducing subjective anxiety was significantly different in patients with the PD compared with the CON group while causing an acceptable level of anxiety in the ANX group. This result shows that using VRA-PD in patients experiencing anxiety could differentiate the two groups. These results support both its validity and acceptability. In addition, the distribution, skewness, and kurtosis values in the ANX group, unlike the CON group, were within the range of the standard for the normal distribution (58), which also supports the feasibility of applying VRA-PD to patients with PD. Moreover, our results support the use of subjective discomfort in the virtual environment utilization evaluation of PD patients. The subjective unit of discomfort score is included in the self-report unit of analysis of the “acute threat” sub-construct of the “Negative valence system” domain in the RDoC framework. Therefore, it will be possible to use the subjective anxiety scores in the virtual environment we used in future RDoC framework studies.

Regarding physiological data, the HRV index in the ANX group was lower in the two elevator virtual environments of M1 than in the CON group. Even in measures with no significant difference, the HRV index generally tended to be low in the ANX group. Even though there has not been a study of physiological data analysis using a virtual environment in PD, the results of this study are consistent with the results of previous studies— the reduced HRV index in patients with anxiety disorders (69).

Regarding the factorial construct evidence of validity, our data, obtained via confirmatory factor analyses, showed that the hypothesized model was only suitable for the HRV index. This result seems to support the idea that the modules of VRA-PD induce different physiological responses, as intended for the HRV index. It seems that the self-rating AS did not show validity because the current VRA-PD measured self-rating anxiety as a VAS score for overall anxiety, which is insufficient to show the differences between the modules. Previous research on dysfunctional communication using VR has shown that the differences between modules are well reflected when specific subjective responses according to the contents of the virtual environment are acquired (23). Therefore, it would be necessary to prepare questionnaires and ratings that can reflect the characteristics of the module to increase the feasibility of the self-rating behavior parameter.

In the repeated-measures ANCOVA, the self-rating AS results were intended to evaluate the content validity of each VRA-PD module. The interaction effect of group and time was confirmed in M1 and M2. In the post hoc test, subjective anxiety in the ANX group increased with time in M1 and decreased in M2. Because M1 was designed to make the user feel anxiety gradually and M2 was meant to relax the user repeatedly, the change in the subjective AS shows that each part of the VRA-PD worked as intended. The insignificant result of this change in M3 indicated that the difference between the two types of interoceptive exposure might not be significant. Moreover, a short relaxation period can affect anxiety (70), and there is also the possibility that the comfortable environment shown in each of the two interoceptive exposure virtual environments affects their response.

Among the physiological responses of the VRA-PD modules, significant changes were observed in M2 and M3 concerning the HRV index. In detail, the HRV index increased compared to pre in the relaxed environment of M2 and decreased compared to pre in the exposure environment of M3. Previous research has revealed that anxious individuals have a lower HRV index than healthy individuals (69); hence, this finding can be interpreted as indicating that the VRA-PD relaxation environment induces a relaxed state in the user, while M3 induces a tense state. Therefore, it can be considered that VRA-PD also induces some valid changes in the physiological domain.

The physiological data showed significant changes in the HRV index in M3, unlike in the self-rating AS. These results suggest that the virtual environment of M3 is insufficient to affect subjective anxiety but is sufficient to induce physical changes. This interpretation is consistent with the results of the current study that no change was observed in the physiological data in M1, even though the self-rating AS changed significantly. In anxiety disorders, both the cognitive domain and physiological responses are important (28). When developing an RDoC study, assessing and integrating a variety of both biological and behavioral measures simultaneously are recommended (14). Therefore, our findings are interesting because they show the possibility that behavior and biological signals are acquired simultaneously through a virtual environment and classify users’ cognitive and physical thresholds concerning anxiety. Follow-up studies introducing the RDoC framework are required to establish this possibility, including developing virtual environments related to the paradigms included in RDoC.

Significant correlations were confirmed between VRA-PD variables and anxiety-related measures in the self-rating ASs and physiological variables. This result suggests that the users’ subjective and physical responses in the VRA-PD environment reflect the psychological aspects of their anxiety and show convergent validity related to anxiety. The HADS-D and SSQ also demonstrated a significant correlation between the HRV index measured in VRA-PD and a part of AS, which shows that the measurement in VRA-PD is not specific to anxiety, contrary to expectations. However, this result may also reflect that depression is a significant factor affecting anxiety, and cybersickness is a significant element in the stress experienced in a VR environment (7, 66). Therefore, to obtain a more precise conclusion on convergence validity, it will be necessary to analyze the correlation of factors unrelated to anxiety and cybersickness in future studies. Moreover, for VRA-PD to be clinically valid, a study that directly compares different types of anxiety disorder diagnoses is required.

This study has some limitations. First, our study evaluated behavioral parameters, physiological data, and anxiety-related data in patients with PD and in healthy controls. Results based solely on panic disorder patients may not generalize to a different diagnosis in anxiety disorders. The second limitation of this research is the extent of the validation evidence (i.e., concept, content, and concurrent validity) and the absence of reliability estimations. Using such restricted metrics does not provide a comprehensive scope of evidence for the VRA-PD. The third limitation in our study was the relatively small sample size for each group. Therefore, we need to carefully interpret the results and secure a larger sample size for future studies. A fourth limitation is that we did not include psychoactive drugs usage in the analysis. Because psychoactive drug use can affect the patient’s physiological response, the patient’s response to anxiety disorders may be masked. Our fifth limitation was the lack of assessments concerning participants’ movement and measurement time while acquiring physiological data. Even a tiny movement can change the quality of the physiological data (71), so it is essential to reduce measurement errors by controlling motion-related parameters. Finally, the 2 min measurement time of this study can have affected the quality of physiological data. Previous studies have shown that a 5-min measurement time is needed to enhance the repeatability and reduce uncertainty of measurements (31). However, previous studies also used 2 min for physiological data acquisition (72), and the need to secure acceptability for actual clinical application of VRA-PD supports a measurement time of 2 min. Therefore, additional considerations regarding optimal measurement time and motion parameter correction are required in future studies.

This study provides preliminary evidence and considerations for the future use of VR in anxiety behavior assessments. It also showed that VRA-PD could induce intentional subjective anxiety and physiological responses in patients with PDs. In addition, the variables obtained from the VRA-PD could reflect the differences in both the cognitive and physiological domains of anxiety behavior between PD and healthy controls. Additionally, our results showed the validity of the modular construction of the VRA-PD and its concurrent validity with other anxiety assessment tools. For VRA-PD to be used as an evaluation tool for anxiety behavior, a future study using a multimodal approach, including other biosignals and brain images, is required for patients with more diverse anxiety disorders. However, given the potential for practical advantages of VR formats and the importance of a dimensional approach, we believe that this study should excite scholars and practitioners.

Data availability statement

The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.

Ethics statement

The studies involving human participants were reviewed and approved by the Ethics Committee of Korea University Guro Hospitals. The patients/participants provided their written informed consent to participate in this study.

Author contributions

B-HK, J-JK, and JK: conceptualization. S-HK, CH, H-GJ, and M-SL: participant evaluation and data acquisition. B-HK and JK: formal analysis and writing—original draft preparation. JK: writing—review, editing, and supervision. All authors contributed to the manuscript and approved the submitted version.

Funding

This research was supported by the Bio and Medical Technology Development Program of the National Research Foundation (NRF) and funded by the Korean government (MSIT) (NRF-2021R1I1A1A01046025 and NRF-2021M3E5D1A01015191) and the National Research Foundation of Korea (NRF) grant funded by the Korean government (MSIT) (No. NRF-2021M3E5D9025019).

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher’s note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Abbreviations

VRA-PD, virtual reality-based anxiety behavior evaluation systems; VR, virtual reality; AS, anxiety scores; DRR, diaphragmatic respiratory relaxation; PMR, progressive muscle relaxation; PPG, photoplethysmogram; HMD, head-mounted display; HRV, heart rate variability; PDSS, panic disorder severity scale; LSAS-SR, Liebowitz social anxiety scale: Self-report version; GAD-7, generalized anxiety disorder scale; HADS, hospital anxiety and depression scale; SSQ, simulator sickness questionnaire; CFI, comparative fit index; TLI, Tucker–Lewis index; RMSEA, root mean square error of approximation; SRMR, standardized root mean squared residual.

References

1. American Psychiatric Association. Diagnostic and statistical manual of mental disorders. 5th ed. Arlington, VA: American Psychiatric Association (2013).

Google Scholar

2. Eaton W, Kessler RC, Wittchen HU, Magee WJ. Panic and panic disorder in the United States. Am J Psychiatry. (1994) 151:413–20. doi: 10.1176/ajp.151.3.413

PubMed Abstract | CrossRef Full Text | Google Scholar

3. Weissman M, Bland RC, Canino GJ, Faravelli C, Greenwald S, Hwu HG, et al. The cross-national epidemiology of panic disorder. Arch Gen Psychiatry. (1997) 54:305–9. doi: 10.1001/archpsyc.1997.01830160021003

PubMed Abstract | CrossRef Full Text | Google Scholar

4. de Jonge P, Roest A, Lim C, Florescu S, Bromet E, Stein D, et al. Cross-national epidemiology of panic disorder and panic attacks in the world mental health surveys. Depress Anxiety. (2016) 33:1155–77. doi: 10.1002/da.22572

PubMed Abstract | CrossRef Full Text | Google Scholar

5. Bruce S, Yonkers K, Otto M, Eisen J, Weisberg R, Pagano M, et al. Influence of psychiatric comorbidity on recovery and recurrence in generalized anxiety disorder, social phobia, and panic disorder: a 12-year prospective study. Am J of Psychiatry. (2005) 162:1179–87. doi: 10.1176/appi.ajp.162.6.1179

PubMed Abstract | CrossRef Full Text | Google Scholar

6. Scholten W, Batelaan N, van BA, Wjh Penninx B, Smit J, van Oppen P. Recurrence of anxiety disorders and its predictors. J Affect Disord. (2013) 147:180–5. doi: 10.1016/j.jad.2012.10.031

PubMed Abstract | CrossRef Full Text | Google Scholar

7. Rhebergen D, Batelaan N, de Graaf R, Nolen W, Spijker J, Beekman A, et al. The 7-year course of depression and anxiety in the general population. Acta Psychiatr Scand. (2011) 123:297–306. doi: 10.1111/J.1600-0447.2011.01677.X

PubMed Abstract | CrossRef Full Text | Google Scholar

8. Fichter M, Quadflieg N, Fischer U, Kohlboeck G. Twenty-five-year course and outcome in anxiety and depression in the upper bavarian longitudinal community study. Acta Psychiatr Scand. (2010) 122:75–85. doi: 10.1111/J.1600-0447.2009.01512.X

PubMed Abstract | CrossRef Full Text | Google Scholar

9. Scholten W, Batelaan N, Penninx B, van Balkom A, Smit J, Schoevers R, et al. Diagnostic instability of recurrence and the impact on recurrence rates in depressive and anxiety disorders. J Affect Disord. (2016) 195:185–90. doi: 10.1016/j.jad.2016.02.025

PubMed Abstract | CrossRef Full Text | Google Scholar

10. Middeldorp C, Cath D, van Dyck R, Boomsma D. The co-morbidity of anxiety and depression in the perspective of genetic epidemiology. a review of twin and family studies. Psychol Med. (2005) 35:611–24. doi: 10.1017/S003329170400412X

PubMed Abstract | CrossRef Full Text | Google Scholar

11. Insel T, Cuthbert B, Garvey M, Heinssen R, Pine D, Quinn K, et al. Research domain criteria (RDoC): toward a new classification framework for research on mental disorders. Am J of Psychiatry. (2010) 167:748–51. doi: 10.1176/APPI.AJP.2010.09091379

PubMed Abstract | CrossRef Full Text | Google Scholar

12. Insel T. The NIMH research domain criteria (RDoC) project: precision medicine for psychiatry. Am J of Psychiatry. (2014) 171:395–7. doi: 10.1176/appi.ajp.2014.14020138

PubMed Abstract | CrossRef Full Text | Google Scholar

13. Carcone D, Ruocco A. Six years of research on the National Institute of Mental Health’s Research domain criteria (RDoC) initiative: a systematic review. Front Cell Neurosci. (2017) 11:46. doi: 10.3389/fncel.2017.00046

PubMed Abstract | CrossRef Full Text | Google Scholar

14. NIMH. RDoC Matrix. (2022). Available online at: https://www.nimh.nih.gov/research/research-funded-by-nimh/rdoc/constructs/rdoc-matrix (accessed Oct 19, 2022).

Google Scholar

15. Marin M, Hammoud M, Klumpp H, Simon N, Milad M. Multimodal categorical and dimensional approaches to understanding threat conditioning and its extinction in individuals with anxiety disorders. JAMA Psychiatry. (2020) 77:618–27. doi: 10.1001/JAMAPSYCHIATRY.2019.4833

PubMed Abstract | CrossRef Full Text | Google Scholar

16. Tozzi L, Staveland B, Holt-Gosselin B, Chesnut M, Chang S, Choi D, et al. The human connectome project for disordered emotional states: protocol and rationale for a research domain criteria study of brain connectivity in young adult anxiety and depression. Neuroimage. (2020) 214:116715. doi: 10.1016/J.NEUROIMAGE.2020.116715

PubMed Abstract | CrossRef Full Text | Google Scholar

17. Corriveau Lecavalier N, Ouellet É, Boller B, Belleville S. Use of immersive virtual reality to assess episodic memory: a validation study in older adults. Neuropsychol Rehabil. (2018) 30:462–80. doi: 10.1080/09602011.2018.1477684

PubMed Abstract | CrossRef Full Text | Google Scholar

18. Han K, Ku J, Kim K, Jang H, Park J, Kim J, et al. Virtual reality prototype for measurement of expression characteristics in emotional situations. Comput Biol Med. (2009) 39:173–9. doi: 10.1016/j.compbiomed.2008.12.002

PubMed Abstract | CrossRef Full Text | Google Scholar

19. Ku J, Jang H, Kim K, Park S, Kim J, Kim C, et al. Pilot study for assessing the behaviors of patients with schizophrenia towards a virtual avatar. Cyberpsychol Behav. (2006) 9:531–9. doi: 10.1089/cpb.2006.9.531

PubMed Abstract | CrossRef Full Text | Google Scholar

20. Park K, Ku J, Choi S, Jang H, Park J, Kim S, et al. A virtual reality application in role-plays of social skills training for schizophrenia: a randomized, controlled trial. Psychiatry Res. (2011) 189:166–72. doi: 10.1016/j.psychres.2011.04.003

PubMed Abstract | CrossRef Full Text | Google Scholar

21. Sanchez D, Weiner E, van Zelderen A. Virtual reality assessments (VRAs): exploring the reliability and validity of evaluations in VR. Int J Select Assess. (2022) 30:103–25. doi: 10.1111/IJSA.12369

CrossRef Full Text | Google Scholar

22. Gutiérrez-Maldonado J, Rus-Calafell M, González-Conde J. Creation of a new set of dynamic virtual reality faces for the assessment and training of facial emotion recognition ability. Virtual Real. (2014) 18:61–71. doi: 10.1007/s10055-013-0236-7

CrossRef Full Text | Google Scholar

23. Kim J, Jung Y, Shin Y, Kim M, Eom H, Kim E, et al. Feasibility of a virtual reality-based interactive feedback program for modifying dysfunctional communication: a preliminary study. BMC Psychol. (2020) 8:50. doi: 10.1186/s40359-020-00418-0

PubMed Abstract | CrossRef Full Text | Google Scholar

24. Kim J, Jung Y, Shin Y, Kim M, Eom H, Kim E, et al. Development and validation of a virtual reality-based training program for promoting subjective well-being. Psychiatry Investig. (2020) 17:1207–15. doi: 10.30773/pi.2020.0311

PubMed Abstract | CrossRef Full Text | Google Scholar

25. Jeong H, Oh J, Paik M, Kim H, Jang S, Kim B, et al. Development and feasibility assessment of virtual reality-based relaxation self-training program. Front Virtual Real. (2022) 2:722558. doi: 10.3389/frvir.2021.722558

CrossRef Full Text | Google Scholar

26. van Dis E, van Veen S, Hagenaars M, Batelaan N, Bockting C, van den Heuvel R, et al. Long-term outcomes of cognitive behavioral therapy for anxiety-related disorders: a systematic review and meta-analysis. JAMA Psychiatry. (2019) 77:265–73. doi: 10.1001/jamapsychiatry.2019.3986

PubMed Abstract | CrossRef Full Text | Google Scholar

27. Brahnam S, Jain L. Advanced computational intelligence paradigms in healthcare 6: virtual reality in psychotherapy, rehabilitation, and assessment. Berlin: Springer (2011).

Google Scholar

28. Otte C. Cognitive behavioral therapy in anxiety disorders: current state of the evidence. Dialogues Clin Neurosci. (2022) 13:413–21. doi: 10.31887/DCNS.2011.13.4/COTTE

PubMed Abstract | CrossRef Full Text | Google Scholar

29. Carpenter J, Andrews L, Witcraft S, Powers M, Smits J, Hofmann S. Cognitive behavioral therapy for anxiety and related disorders: a meta-analysis of randomized placebo-controlled trials. Depress Anxiety. (2018) 35:502–14. doi: 10.1002/DA.22728

PubMed Abstract | CrossRef Full Text | Google Scholar

30. Lecrubier Y, Sheehan DV, Weiller E, Amorim P, Bonora I, Sheehan K, et al. The mini international neuropsychiatric interview (MINI). A short diagnostic structured interview: reliability and validity according to the CIDI. Eur Psychiatry. (1997) 12:224–31. doi: 10.1016/S0924-9338(97)83296-8

CrossRef Full Text | Google Scholar

31. Schroeder E, Whitsel E, Evans G, Prineas R, Chambless L, Heiss G. Repeatability of heart rate variability measures. J Electrocardiol. (2004) 37:163–72. doi: 10.1016/J.JELECTROCARD.2004.04.004

PubMed Abstract | CrossRef Full Text | Google Scholar

32. Vollmer M. A robust, simple and reliable measure of heart rate variability using relative RR intervals. Comput Cardiol. (2015) 42:609–12. doi: 10.1109/CIC.2015.7410984

CrossRef Full Text | Google Scholar

33. Malik M, Xia R, Odemuyiwa O, Staunton A, Poloniecki J, Camm A. Influence of the recognition artefact in automatic analysis of long-term electrocardiograms on time-domain measurement of heart rate variability. Med Biol Eng Comput. (1993) 31:539–44. doi: 10.1007/BF02441992

PubMed Abstract | CrossRef Full Text | Google Scholar

34. Chang H, Chang C, Tzeng N, Kuo T, Lu R, Huang S. Generalized anxiety disorder, comorbid major depression and heart rate variability: a case-control study in Taiwan. Psychiatry Investig. (2013) 10:326–35. doi: 10.4306/PI.2013.10.4.326

PubMed Abstract | CrossRef Full Text | Google Scholar

35. Pittig A, Arch J, Lam C, Craske M. Heart rate and heart rate variability in panic, social anxiety, obsessive–compulsive, and generalized anxiety disorders at baseline and in response to relaxation and hyperventilation. Int J of Psychophysiol. (2013) 87:19–27. doi: 10.1016/J.IJPSYCHO.2012.10.012

PubMed Abstract | CrossRef Full Text | Google Scholar

36. Choi K, Jang E, Kim A, Fava M, Mischoulon D, Papakostas G, et al. Heart rate variability for treatment response between patients with major depressive disorder versus panic disorder: a 12-week follow-up study. J Affect Disord. (2019) 246:157–65. doi: 10.1016/J.JAD.2018.12.048

PubMed Abstract | CrossRef Full Text | Google Scholar

37. Shaikh al arab A, Guédon-Moreau L, Ducrocq F, Molenda S, Duhem S, Salleron J, et al. Temporal analysis of heart rate variability as a predictor of post traumatic stress disorder in road traffic accidents survivors. J Psychiatr Res. (2012) 46:790–6. doi: 10.1016/J.JPSYCHIRES.2012.02.006

PubMed Abstract | CrossRef Full Text | Google Scholar

38. Khandoker A, Jelinek H, Palaniswami M. Heart rate variability and complexity in people with diabetes associated cardiac autonomic neuropathy. Proceedings of the 30th annual international conference of the IEEE engineering in medicine and biology society, EMBS’08 - “personalized healthcare through technology”. Vancouver, BC: IEEE (2008). p. 4696–9. doi: 10.1109/iembs.2008.4650261

PubMed Abstract | CrossRef Full Text | Google Scholar

39. Botella C, Villa H, García-Palacios A, Baños R, Perpiñá C, Alcañiz M. Clinically significant virtual environments for the treatment of panic disorder and agoraphobia. Cyberpsychol Behav. (2004) 7:527–35. doi: 10.1089/CPB.2004.7.527

PubMed Abstract | CrossRef Full Text | Google Scholar

40. Pompoli A, Furukawa T, Efthimiou O, Imai H, Tajika A, Salanti G. Dismantling cognitive-behaviour therapy for panic disorder: a systematic review and component network meta-analysis. Psychol Med. (2018) 48:1945–53. doi: 10.1017/S0033291717003919

PubMed Abstract | CrossRef Full Text | Google Scholar

41. Shear M, Brown T, Barlow D, Money R, Sholomskas D, Woods S, et al. Multicenter collaborative panic disorder severity scale. Am J Psychiatry. (1997) 154:1571–5. doi: 10.1176/AJP.154.11.1571

PubMed Abstract | CrossRef Full Text | Google Scholar

42. Kim J. Development of the Korean version of the panic disorder severity scale. Korean J Psychopathol. (2001) 10:140–51.

Google Scholar

43. Shear M, Rucci P, Williams J, Frank E, Grochocinski V, vander Bilt J, et al. Reliability and validity of the panic disorder severity scale: replication and extension. J Psychiatr Res. (2001) 35:293–6. doi: 10.1016/S0022-3956(01)00028-0

CrossRef Full Text | Google Scholar

44. Heimberg R, Horner KJ, Juster H, Safren S, Brown E, Schneier F, et al. Psychometric properties of the liebowitz social anxiety scale. Psychol Med. (1999) 29:199–212. doi: 10.1017/S0033291798007879

PubMed Abstract | CrossRef Full Text | Google Scholar

45. Fresco D, Coles M, Heimberg R, Liebowitz M, Hami S, Stein M. The liebowitz social anxiety Scale: a comparison of the psychometric properties of self-report and clinician-administered formats. Psychol Med. (2001) 31:1025–35. doi: 10.1017/S0033291701004056

PubMed Abstract | CrossRef Full Text | Google Scholar

46. Kang JH, Lee JA, Oh KS, Lim SW. Validation and clinical efficacy of the Korean liebowitz social anxiety scale: clinician administered. Korean J Clin Psychol. (2013) 32:291–312.

Google Scholar

47. Spitzer R, Kroenke K, Williams J, Löwe B. A brief measure for assessing generalized anxiety disorder: the GAD-7. Arch Intern Med. (2006) 166:1092–7. doi: 10.1001/archinte.166.10.1092

PubMed Abstract | CrossRef Full Text | Google Scholar

48. Kertz S, Bigda-Peyton J, Bjorgvinsson T. Validity of the generalized anxiety disorder-7 Scale in an acute psychiatric sample. Clin Psychol Psychother. (2013) 20:456–64. doi: 10.1002/CPP.1802

PubMed Abstract | CrossRef Full Text | Google Scholar

49. Tiirikainen K, Haravuori H, Ranta K, Kaltiala-Heino R, Marttunen M. Psychometric properties of the 7-item generalized anxiety disorder scale (GAD-7) in a large representative sample of Finnish adolescents. Psychiatry Res. (2019) 272:30–5. doi: 10.1016/J.PSYCHRES.2018.12.004

PubMed Abstract | CrossRef Full Text | Google Scholar

50. Lee B, Kim Y. The psychometric properties of the generalized anxiety disorder scale (GAD-7) among Korean university students. Psychiatry Clin Psychopharmacol. (2019) 29:864–71. doi: 10.1080/24750573.2019.1691320

CrossRef Full Text | Google Scholar

51. Zigmond A, Snaith R. The hospital anxiety and depression scale. Acta Psychiatr Scand. (1983) 67:361–70.

Google Scholar

52. Oh S, Min K, Park D-B. A study on the standardization of the hospital anxiety and depression scale for Koreans: a comparison of normal, depressed and anxious groups. J Korean Neuropsychiatr Assoc. (1999) 38:289–96.

Google Scholar

53. Kennedy R, Lane N, Berbaum K, Lilienthal M. Simulator sickness questionnaire: an enhanced method for quantifying simulator sickness. Int J Aviat Psychol. (1993) 3:203–20. doi: 10.1207/s15327108ijap0303_3

CrossRef Full Text | Google Scholar

54. Sevinc V, Berkman M. Psychometric evaluation of Simulator Sickness Questionnaire and its variants as a measure of cybersickness in consumer virtual environments. Appl Ergon. (2020) 82:102958. doi: 10.1016/J.APERGO.2019.102958

PubMed Abstract | CrossRef Full Text | Google Scholar

55. Kim H, Park J, Choi Y, Choe M. Virtual reality sickness questionnaire (VRSQ): motion sickness measurement index in a virtual reality environment. Appl Ergon. (2018) 69:66–73. doi: 10.1016/J.APERGO.2017.12.016

PubMed Abstract | CrossRef Full Text | Google Scholar

56. Jackson D, Gillaspy J, Purc-Stephenson R. Reporting practices in confirmatory factor analysis: an overview and some recommendations. Psychol Methods. (2009) 14:6–23. doi: 10.1037/A0014694

PubMed Abstract | CrossRef Full Text | Google Scholar

57. Hu L, Bentler P. Cutoff criteria for fit indexes in covariance structure analysis: conventional criteria versus new alternatives. Struct Equ Model. (2009) 6:1–55. doi: 10.1080/10705519909540118

CrossRef Full Text | Google Scholar

58. Kline T. Psychological testing: a practical approach to design and evaluation. Thousand Oaks, CA: Sage Publications, Inc (2014). doi: 10.4135/9781483385693

CrossRef Full Text | Google Scholar

59. Cosci F, Schruers K, Abrams K, Griez E. Alcohol use disorders and panic disorder: a review of the evidence of a direct relationship. J Clin Psychiatry. (2007) 68:685. doi: 10.4088/JCP.v68n0608

PubMed Abstract | CrossRef Full Text | Google Scholar

60. Zvolensky M, Feldner M, Leen-Feldner E, McLeish A. Smoking and panic attacks, panic disorder, and agoraphobia: a review of the empirical literature. Clin Psychol Rev. (2005) 25:761–89. doi: 10.1016/J.CPR.2005.05.001

PubMed Abstract | CrossRef Full Text | Google Scholar

61. Kageyama T, Imai H, Kabuto M, Nishikido N, Honda Y, Kurokawa Y, et al. Effects of obesity, current smoking status, and alcohol consumption on heart rate variability in male white-collar workers. Int Arch Occup Environ Health. (1997) 69:447–54. doi: 10.1007/S004200050173

PubMed Abstract | CrossRef Full Text | Google Scholar

62. Furukawa T, Shear M, Barlow D, Gorman J, Woods S, Money R, et al. Evidence-based guidelines for interpretation of the panic disorder severity scale. Depress Anxiety. (2009) 26:922–9. doi: 10.1002/da.20532

PubMed Abstract | CrossRef Full Text | Google Scholar

63. Spinhoven P, Ormel J, Sloekers P, Kempen G, Speckens A, van Hemert AM. A validation study of the hospital anxiety and depression scale (HADS) in different groups of dutch subjects. Psychol Med. (1997) 27:363–70. doi: 10.1017/S0033291796004382

PubMed Abstract | CrossRef Full Text | Google Scholar

64. Uhde T, Roy-Byrne P, Vittone B, Boulenger J, Post R. Phenomenology and Neurobiology of Panic Disorder. In: AH Tuma, J Maser editors. Anxiety and the anxiety disorders. New York, NY: Routledge (2019). p. 557–76. doi: 10.4324/9780203728215-39

PubMed Abstract | CrossRef Full Text | Google Scholar

65. Sharples S, Cobb S, Moody A, Wilson J. Virtual reality induced symptoms and effects (VRISE): comparison of head mounted display (HMD), desktop and projection display systems. Displays. (2008) 29:58–69. doi: 10.1016/j.displa.2007.09.005

CrossRef Full Text | Google Scholar

66. Pot-Kolder R, Veling W, Counotte J, van der Gaag M. Anxiety partially mediates cybersickness symptoms in immersive virtual reality environments. Cyberpsychol Behav Soc Netw. (2018) 21:187–93. doi: 10.1089/CYBER.2017.0082/ASSET/IMAGES/LARGE/FIGURE3.JPEG

CrossRef Full Text | Google Scholar

67. Kim H, Kim D, Chung W, Park K, Kim J, Kim D, et al. Clinical predictors of cybersickness in virtual reality (VR) among highly stressed people. Sci Rep. (2021) 11:12139. doi: 10.1038/s41598-021-91573-w

PubMed Abstract | CrossRef Full Text | Google Scholar

68. Bouchard S, St-Jacques J, Renaud P, Wiederhold B. Side effects of immersions in virtual reality for people suffering from anxiety disorders. JCR. (2009) 2:127. doi: 10.3233/SHTI210961

PubMed Abstract | CrossRef Full Text | Google Scholar

69. Chalmers J, Quintana D, Abbott M, Kemp A. Anxiety disorders are associated with reduced heart rate variability: a meta-analysis. Front Psychiatry. (2014) 5:80. doi: 10.3389/FPSYT.2014.00080

PubMed Abstract | CrossRef Full Text | Google Scholar

70. Tolin D, Davies C, Moskow D, Hofmann S. Biofeedback and neurofeedback for anxiety disorders: a quantitative and qualitative systematic review. Adv Exp Med Biol. (2020) 1191:265–89. doi: 10.1007/978-981-32-9705-0_16

CrossRef Full Text | Google Scholar

71. Berwal D, Vandana C, Dewan S, Jiji CV, Baghini M. Motion artifact removal in ambulatory ECG signal for heart rate variability analysis. IEEE Sens J. (2019) 19:12432–42. doi: 10.1109/JSEN.2019.2939391

CrossRef Full Text | Google Scholar

72. Li K, Rüdiger H, Ziemssen T. Spectral analysis of heart rate variability: time window matters. Front Neurol. (2019) 10:545. doi: 10.3389/FNEUR.2019.00545/BIBTEX

CrossRef Full Text | Google Scholar