MIND-VR: Design and Evaluation Protocol of a Virtual Reality Psychoeducational Experience on Stress and Anxiety for the Psychological Support of Healthcare Workers Involved in the COVID-19 Pandemic
Federica Pallavicini1* 
Eleonora Orena2 
Simona di Santo3,4 
Luca Greci5 
Chiara Caragnano6 Paolo Ranieri7 Costanza Vuolato2 
Alessandro Pepe1 Guido Veronese1 
Antonios Dakanalis8 Angelo Rossini3 Carlo Caltagirone3 
Massimo Clerici8 
Fabrizia Mantovani1- 1Department of Human Sciences for Education “Riccardo Massa”, University of Milano Bicocca, Milan, Italy
- 2IRCCS Neurological Institute Carlo Besta, Milan, Italy
- 3IRCCS Fondazione Santa Lucia, Roma, Italy
- 4Department of Systems Medicine, Università degli Studi di Roma Tor Vergata, Roma, Italy
- 5Institute of Intelligent Industrial Technologies and Systems for Advanced Manufacturing (STIIMA) National Research Council of Italy (CNR), Lecco, Italy
- 6Department of Psychology, University of Milano Bicocca, Milan, Italy
- 7University of Milano Bicocca, Specialization School in Psychology, Milan, Italy
- 8Department of Medicine and Surgery, University of Milano Bicocca, Monza, Italy
To ensure the continuity of healthcare and to counter the spread of the COVID-19 pandemic, doctors and nursing staff at hospitals must face an insidious, invisible danger that is stretching the healthcare system far past its capacity. Excessive workload, inadequate protection from contamination, the need to manage patients experiencing extreme suffering and being kept apart from their families put medical personnel at high risk to experience stress and anxiety. Numerous scientific studies have shown that, among various therapeutic programs, virtual reality represents a highly specialized and effective tool for the prevention and treatment of stress and anxiety. However, the solutions developed using this technology for the management of stress and anxiety induced by the COVID-19 pandemic are still very limited, and none of these have been developed specifically for use with healthcare professionals. Therefore, this paper will detail the design and evaluation protocol of MIND-VR, a virtual reality-based psychoeducational experience on stress and anxiety developed following a user-centered design approach. The virtual experience will be tested on a sample of Italian hospital healthcare personnel involved in the COVID-19 pandemic emergency. MIND-VR is available free of charge, both in Italian and English, on the project website (https://mind-vr.com/).
Introduction
The novel coronavirus disease (COVID-19) pandemic is a global health emergency that is dramatically affecting the daily lives of millions of people worldwide (Johns Hopkins Coronavirus Resource Center, 2020), with serious consequences not only on a socioeconomic level but also on individuals’ mental health (Cai et al., 2020a; Ma et al., 2020; Rogers et al., 2020; Vindegaard and Eriksen Benros, 2020; Zhao et al., 2020; Zheng, 2020). The adverse psychological consequences concern all those people forced into isolation and/or quarantine (Brooks et al., 2020), but especially healthcare workers involved directly in handling COVID-19 patients (Chen et al., 2020; Kang et al., 2020; Liu et al., 2020).
Excessive workload, inadequate protection from contamination, and the need to manage patients experiencing extreme suffering and being kept apart from their families put medical personnel at high risk of experiencing stress (Bohlken et al., 2020; Greenberg et al., 2020; Qiu et al., 2020; Spoorthy, 2020), and anxiety (Huang et al., 2020; Liu et al., 2020; Zhu et al., 2020). Stress classically is defined as “the bodily processes that result from circumstances that place physical or psychologic demands on an individual”, and it is usually a short-lived response (Selye, 1973). Anxiety, instead, according to the American Psychiatric Association (APA), refers to “an emotion characterized by feelings of tension, worried thoughts and physical changes like increased blood pressure”, and it generally is a long-term condition (Kazdin, 2000).
Past studies published after the 2003 outbreak of severe acute respiratory syndrome (SARS) have revealed that healthcare workers experienced high levels of stress and anxiety (Tam et al., 2004; Grace et al., 2005; Wu et al., 2005; McAlonan et al., 2007). Similar results have been reported by recent studies conducted during the COVID-19 pandemic on healthcare workers from different countries around the world, including China (Cai et al., 2020b; Chen et al., 2020; Du et al., 2020; Kang et al., 2020; Lai et al., 2020; Liu et al., 2020), United States (Shechter et al., 2020), India (Wilson et al., 2020), and Italy (Babore et al., 2020; Di Tella et al., 2020; Vagni et al., 2020).
Excessive stress is a critical factor that could compromise healthcare workers’ performance, (Dellve et al., 2011; Gandi et al., 2011), particularly during an emergency (Müller et al., 2009). Besides, high levels of stress and anxiety may have long-term physical and psychological consequences (Conway et al., 2008; Vinstrup et al., 2020). Therefore, urgent actions are needed to protect the mental health of those on the front line who deal with the management of the COVID-19 pandemic health emergency daily (Carmassi et al., 2020; Chirico et al., 2020; Krystal and McNeil, 2020). As underlined by a recent review (Weerasekara and Smedberg, 2019) and by the Italian Workers Compensation Authority and the Italian National Institute of Health, psychological support services should be based on enhancing the coping strategies for managing stress and anxiety of healthcare workers (Chirico et al., 2020; Italian National Institute of Health, 2020). However, little attention has been paid so far to the practical implementation of psychological interventions to help hospital staff (Chen et al., 2020; Cheng et al., 2020; Weiner et al., 2020).
Within this context, new approaches to mental health integrating advanced technologies such as virtual reality should be considered (Pallavicini and Bouchard, 2019; Imperatori et al., 2020; Riva et al., 2020; Riva and Wiederhold, 2020). Nevertheless, the solutions developed using this technology for the management of stress and anxiety induced by the COVID-19 pandemic are still very limited (Riva and Wiederhold, 2020), and none of these have been developed specifically for use with healthcare professionals. Therefore, this paper will detail the design and evaluation protocol of MIND-VR, a virtual reality-based psychoeducational experience on stress and anxiety developed following a user-centered design (UCD) approach. This virtual experience will be tested on a sample of Italian hospital healthcare personnel involved in the COVID-19 pandemic emergency.
Background
Virtual Reality for the Management of Stress and Anxiety and Its Relevance During the COVID-19 Pandemic
Several studies have shown that virtual reality represents an advanced and effective tool for the prevention and treatment of stress and anxiety (Parsons and Rizzo, 2008; Maples-Keller et al., 2017; Oing and Prescott, 2018; Wechsler et al., 2019). The use of this immersive technology can help patients to learn useful resources to better manage anxiety (Gaggioli et al., 2014; Anderson et al., 2017; Lindner et al., 2019; Pizzoli et al., 2019), with therapeutic benefits similar to or even greater than those of traditional programs such as cognitive-behavioral therapy (CBT) interventions, (e.g. in vivo exposure and guided imagery) (Gerardi et al., 2008; Wechsler et al., 2019).
As underlined in a recent article, several characteristics make virtual reality particularly interesting in the prevention and treatment of stress and anxiety, especially during this historical moment linked to the COVID-19 pandemic (Imperatori et al., 2020). Among these characteristics, one of the most relevant is that virtual reality can help people acquire coping skills useful for managing stress and anxiety, including relaxation techniques (Stetz et al., 2011; Serrano et al., 2016; Anderson et al., 2017; Pizzoli et al., 2019), biofeedback (Pallavicini et al., 2009; Gradl et al., 2018), and mindfulness training (Kosunen et al., 2016; Navarro-Haro et al., 2017; Seabrook et al., 2020). Indeed, this technology has proven to be effective at inducing relaxation, eliciting positive emotions in users (Anderson et al., 2017; Lindner et al., 2019; Pizzoli et al., 2019; Pallavicini et al., 2020; Pallavicini and Pepe, 2020). In particular, the visual presentation of a relaxing virtual scenario can facilitate the practice of patients and the consequent mastery of relaxation techniques, making the experience more vivid and real than both photos/videos (Repetto et al., 2009; Gorini et al., 2011; Pallavicini et al., 2013) and using one’s imagination and memory (Villani and Riva, 2008; Villani et al., 2009). For example, in April 2020, a virtual reality experience called “The Secret Garden” was developed to help people manage stress generated by the COVID-19 pandemic (Riva and Wiederhold, 2020).
In addition to the ability of virtual reality to reduce levels of stress and anxiety through immersion in relaxing virtual environments, a particularly significant feature is linked to the usefulness of this technology within psychoeducational programs. This is particularly relevant since, as will be described in the following paragraph, mental health literacy–defined as knowledge and beliefs about mental disorders, which aid their recognition, management, or prevention (Jorm, 2000)–represents a key facilitator of formal help-seeking behavior (Rickwood et al., 2005; Gulliver et al., 2010).
The Potential of Virtual Reality for Psychoeducation on Stress and Anxiety
The term psychoeducation refers to interventions aimed at providing information on a psychological disorder in a clear and articulated way to make the patient more competent and aware of his problem, with a view to individual empowerment (Oflaz et al., 2008; Watkins et al., 2018). Psychoeducation represents a fundamental method of increasing adherence to therapy for mental disorders. It is considered a key part of cognitive-behavioral therapy (Richardson, 2017), which is, as reported by the American Psychological Association, a form of psychological treatment strongly evidence-based that emphasizes the strict connection between behavior, cognition, and emotions and on helping individuals learn to be autonomous. Psychoeducation can also be an intervention of its own, educating patients on their diseases and how to deal with them (Coulthard et al., 2013; Taylor-Rodgers and Batterham, 2014; Zhang and Ho, 2016).
Virtual reality (VR) represents a potentially very useful tool for psychoeducational programs, thanks especially to the greater involvement of users in the learning process compared to a standard lesson, such as learning from a textbook, slideshow, or lecture (Bailenson et al., 2008; De Freitas et al., 2010; Webster, 2016; Parong and Mayer, 2018). In particular, empirical studies suggest that this technology leads to better outcomes than slideshows or textbooks presented on desktop displays (Kozhevnikov and Gurlitt, 2013; Webster, 2016; Orman et al., 2017). Thanks to virtual reality individuals can experience a fully immersive sensory experience that may help them engage in deeper learning (Parong and Mayer, 2018).
To understand how the learning process takes place in virtual environments, it is essential to refer to the constructivist principles approach (Chittaro and Ranon, 2007; Dimitropoulos et al., 2007; Virvou and Katsionis, 2008; Huang et al., 2010). According to this paradigm, learning is situated: knowledge is based on active experience and is obtained through the construction of meaning in a certain learning environment. Learners take an active role in their learning, and the environment will affect the learner (Dewey, 1916). Immersive environments offer a variety of situated learning experiences different from what learners encounter in a traditional classroom or text-based learning. In virtual reality, learners can construct knowledge through interaction with objects and events in the artificial world (Chittaro and Ranon, 2007; Huang et al., 2010; Shin, 2017; Gomes et al., 2020). This technology creates a strong sense of presence, which in turn motivates and thereby causes the learner to cognitively process the learning material more deeply. In other words, learners begin actively learning (Huang et al., 2010).
Despite the important role that virtual reality could play in providing information regarding mental suffering conditions, only a few previous studies to date have investigated the possibilities of virtual reality for psychoeducation (Rizzo et al., 2009, 2010; Riva et al., 2012; Tielman et al., 2017; Migoya-Borja et al., 2020). For example, a virtual environment called “Learning Island” and developed in Second Life, was created to host an interactive course on stress management (Riva et al., 2012). Also, a virtual reality-based psychoeducational tool for individuals suffering from depression called “VRight” has been shown to increase depressive symptoms awareness (Migoya-Borja et al., 2020). Within this context, MIND-VR’s main goal is to develop an immersive experience offering psychoeducation on stress and anxiety. This virtual experience, as described below, was built following a UCD approach and will be tested on a sample of Italian hospital healthcare personnel involved in the COVID-19 pandemic emergency.
Development and Evaluation Protocol of the MIND-VR Psychoeducational Virtual Experience
Starting in July 2020, a team from the University of Milano-Bicocca, in collaboration with the authors of this paper and an Italian start-up specializing in the development of immersive solutions (AnotheReality), partnered to develop MIND-VR. Development funds (approximately $5,000) were raised thanks to a fundraising campaign that took place between April and June 2020 and was promoted by the University of Milano-Bicocca in collaboration with the platform Produzioni dal Basso.
MIND-VR is intended to be released immediately, to be easy-to-use, enjoyable, and educational. Thus, principles and practices of UCD were followed in its design and development. According to the International Organization for Standardization (ISO), UCD, often used synonymously with “human-centered design,” is defined as “an approach to systems design and development that aims to make interactive systems more usable by focusing on the use of the system and applying human factors/ergonomics and usability knowledge and techniques” (Woodson et al., 1992). UCD can be considered a multistage problem-solving approach that puts the product end-users, both expert and naïve, at the center of its design and development (Fischer and Coutellier, 2007; Barbieri et al., 2018).
This approach is used successfully in many application fields such as mobile and web application development (Taylor et al., 2011; De Paula et al., 2014; Lopes et al., 2018; Llema and Vilela-Malabanan, 2019), and healthcare technology (DeSmet et al., 2016; Schnall et al., 2016; Verschueren et al., 2019; Farao et al., 2020; Vandekerckhove et al., 2020). For example, a UCD process has been adopted to develop a digital platform offering psychological support to patients suffering from paranoia (Hardy et al., 2018). It has also been used to create video games to alleviate anxiety and depression symptoms (Dekker and Williams, 2017), and to promote physical exercise in older adults (Brox et al., 2017).
Regarding virtual reality, it has been demonstrated that, by optimizing the virtual environment’s architecture, the use of UCD in the development of virtual experiences offers a more efficient, satisfying, user-friendly experience (Gabbard et al., 1999; Jerald, 2015; Barbieri et al., 2018; Gladden, 2018). It has been employed to design and evaluate virtual experiences in museums (Barbieri et al., 2018), tourism (Poux et al., 2020), automotive industries (de Clerk et al., 2019), and military (Hix et al., 1999), and medical fields (Vailland et al., 2019). Furthermore, it has been used to develop virtual reality content for mental health (Fidopiastis et al., 2010; Koenig et al., 2012; Boger et al., 2018; Eisapour et al., 2018; Flobak et al., 2019; Pizzoli et al., 2019), particularly virtual environments for cognitive assessment and rehabilitation (Fidopiastis et al., 2010; Koenig et al., 2012), and to promote physical activity among people with mild cognitive impairment (MCI) (Boger et al., 2018; Eisapour et al., 2018). The UCD approach has recently been proposed for prototyping virtual reality scenarios for exposure therapy in patients with fear of public speaking (Flobak et al., 2019), and for promoting relaxation (Pizzoli et al., 2019). Regarding virtual experiences of psychoeducation, to the best of our knowledge, only one previous work conducted a requirements analysis involving end-users to develop an immersive game for older adults with chronic back pain (Stamm et al., 2020).
Therefore, MIND-VR represents one of the first few published attempts to adopt the UCD approach for the development of a virtual reality-based psychoeducational experience. Specifically, in the development of this virtual experience, the recommendations ISO 13407 (ISO/IEC, 1999) for a UCD project were adopted, and the main design phases were followed according to this approach, which will be described in detail in the following paragraphs. Psychologists with high expertise in creating virtual content for managing stress and anxiety follow developed the requirements, selected the architectural paradigm for the design of the virtual reality experience, and created content and evaluation protocol of MIND-VR in collaboration with psychotherapists who have assisted health professionals following the situation triggered by the COVID-19 pandemic and co-authored this paper.
Requirement Specifications
According to the ISO 9241–210:2010 (Woodson et al., 1992), before the design phase in a UCD approach, it is fundamental to take into account the gathering of requirements and specifications to better define the context of use and the needs of the end-users (Brox et al., 2017; Barbieri et al., 2018). For MIND-VR to be efficiently designed, end-users’ needs and expertise were collected via semi-structured interviews conducted virtually in April 2020 with two psychotherapists who also co-authored this paper. The purpose was to find out the expectations, desires, preferences, and barriers to adopt a virtual reality experience for the management of stress and anxiety for healthcare workers. The interviews included the following main content categories: experience with treating stress and anxiety of medical staff during the COVID-19 pandemic; virtual reality knowledge and expertise; acceptance and experience with virtual reality system for the management of stress and anxiety for healthcare workers. The interviews lasted about 40 min. Subsequently, they were transcribed and analyzed to identify the main categories. A summarizing content analysis by Marying (Mayring, 2015) was applied for the data analysis. The transcripts were coded manually and were carried out by two scientists, using the four-eyes-principle to ensure reliable coding.
Based on the funds available for development and taking into account the interviews with the psychotherapists, MIND-VR had to satisfy four fundamental requirements:
• Low cost: the funds for the development of this virtual experience were very limited (about 5,000 euros). Consequently, the best virtual content development solution—in terms of interaction mode, home construction, user data recording, etc.—had to be functionally relevant, cost-effective, and technically possible.
• Easy to use: since the target of MIND-VR is represented by users who are not necessarily experts or with previous experience of virtual reality systems, (i.e. doctors and nurses), usability—namely, “the extent to which a product can be used, by specified users, to achieve specified goals with effectiveness, efficiency, and satisfaction in a specified context of use” (Malone, 1980, 1982; Nielsen, 1994)—was considered a key quality of the virtual experience. To ensure high usability of the content, various evaluation phases with the involvement of end-users were planned during development (see also Evaluation).
• Pleasant and informative experience: in the development of MIND-VR, in addition to a strong focus on a simple and intuitive mode of interaction, the computer-generated graphics, and the related visual experience have been rendered with high quality to make the experience engaging and captivating. The conceptual framework defined as “emotional design” has been followed in the design of the virtual content (Norman, 2007). This framework highlights that an effective design is composed not only of a modality of interaction perceived as simple and intuitive, but also of the involvement of users’ affective/visceral level through the use of physical features such as sight, touch, and sound. This theoretical approach has been adopted largely in the context of developing interactive content like video games (Baharom et al., 2014), and many other technological products (Desmet and Hekkert, 2007; Triberti et al., 2017). Particular attention was paid to the choice of colors and the selection of the fonts of all text present in the experience. Besides, MIND-VR was developed with extreme consideration of the sound aspect of the experience and its quality. For this purpose, the psychoeducational content includes audio recorded by professional actors and high-quality ambient sounds.
• Easily accessible: MIND-VR had to run on some currently more widespread, user-friendly, and low-cost head-mounted display (HMD): Oculus Quest and Oculus Quest 2 (see also Implementation: MIND-VR hardware and software). Furthermore, since November 2020 the virtual experience can be downloaded for free, both in Italian and English, on the project website (https://mind-vr.com/free-download/).
Selection of the Architectural Paradigm for the Design of the Virtual Reality Experience
This part of the design process started with an in-depth analysis of the literature on psychoeducation in programs for stress and anxiety (Romano, 1992; Houghton and Saxon, 2007; Donker et al., 2009; van Daele et al., 2012), and with the collection of information about products (commercial and non-commercial) in virtual reality built for educational purposes. Also, two psychotherapists (who are among the authors of this paper) were interviewed to explore: psychoeducation on stress and anxiety for healthcare workers; the essential element to be included in a virtual experience to offer basic education on stress and anxiety. The interviews lasted about 40 min and afterward were transcribed and analyzed. Then, a summarizing content analysis (Mayring, 2015) was applied. Based on the information and the scientific literature collected (Romano, 1992; Houghton and Saxon, 2007; Donker et al., 2009; van Daele et al., 2012), the architecture and the content of the experience were written.
At the beginning of MIND-VR, the user is in a home, where three different icons are presented: “Tutorial,” “Project information,” and “A brief guide on stress and anxiety”. The tutorial was inserted to offer the basic information necessary for the use of psychoeducational content (i.e., navigation mode through teleportation and the possibility of using it both standing and sitting on a swivel chair). In the “Project Information” section, on the other hand, there is information about the history and purpose of MIND-VR. Furthermore, since it has been shown that listening to some types of music can promote relaxation and increase positive emotions (Kuan et al., 2018; de Witte et al., 2020), a classical music piece has been included in the home of MIND-VR. In particular, the music is taken from Frederick Delius’ Florida Suite: III “Near the Plantation,” a piece that has proven effective in reducing stress and anxiety according to a previous study (Kuan et al., 2018).
As for the psychoeducational content, since the scientific literature suggests that naturalistic virtual environments cause a relaxing effect and an increase in positive emotions (Kahn et al., 2009; Anderson et al., 2017; Browning et al., 2020; Litleskare et al., 2020), a small tropical island was chosen as the setting. The prevailing colors are blue and green, as they are associated with positive affective states and neutral levels of emotional activation (Kurt and Osueke, 2014; Minguillon et al., 2017; Briki and Majed, 2019). The elements on the island, (i.e. palm trees, vegetation, a small lake over which there is a bridge, etc.) were inserted taking care that they are not too numerous to avoid elements that are excessively distracting for the user compared to the psychoeducational experience (Figure 1).
FIGURE 1. Screenshot of the virtual island developed for MIND-VR: a bridge on a small lake in the center of the island that users cross in the third path.
The design of the experience is based on the theoretical framework of architectural paradigms for the UCD of virtual environments developed by Gladden (Gladden, 2018). By considering the unique possibilities for structuring and experiencing space within virtual environments and reinterpreting Norberg-Schulz’s schemas in the context of virtual environment design (Norberg-Schulz, 1980), this framework formulated six fundamental architectural paradigms available to designers of virtual environments that could easily be incorporated as an additional step within the UCD process. In the design of the MIND-VR psychoeducational experience, the paradigm of the “Virtual Museum” was selected. This architectural paradigm can be used to create many types of virtual environments that are not literally meant to function as “museums,” creating for users the atmosphere of a gallery whose visual and textual information are being displayed and appreciated. Inside these virtual museums, the person can follow a predetermined path or freely explore the environment (Gladden, 2018).
Based on this paradigm, the virtual island is divided into three areas, each focused on different aspects related to stress and anxiety and presented to the user through a map at the beginning of the content (Figure 2). In each area, the users have the opportunity to follow a predetermined path along which they will find information in text, audio, and image form on conditions of mental suffering. In particular, in some stages there is a wooden lectern above which a three-dimensional object is presented that represents the theme of the stage, (e.g. psychology symbol, DNA, etc.). By clicking on the object, the user is presented with information on stress and anxiety, both in text form, (i.e., written on the lectern) and audio (Figure 3). The user can advance along the path and explore the island through teleportation, which has been evaluated as the simplest and most intuitive way of navigation (Bozgeyikli et al., 2016; Boletsis and Cedergren, 2019).
FIGURE 2. Map of the three areas into which the virtual island is divided (“An introduction on stress and anxiety disorders”, “Causes and symptoms” “Main Treatments”).
FIGURE 3. Content of a stage along one of the three paths, where the user is provided with information on stress and anxiety disorders in both text and audio form.
Concerning the specific content of the three paths, based on the information collected and the literature on psychoeducation in programs for stress and anxiety (Romano, 1992; Houghton and Saxon, 2007; Donker et al., 2009; van Daele et al., 2012), three fundamental issues emerged as the most relevant to be explored within the virtual experience. For each of these topics, the storyboard for the paths inside the island was created (see Table 1). The island is designed to be completed in three sessions, each trying a different path. Each of the routes takes about 15 min to be ended, for a total of about 45 min to finish the whole experience. The decision to limit the duration of the virtual experience to 15 min per session was made following the recommendations of the psychotherapists collected during the interviews. This also matches the health and safety warnings of the Oculus Quest (Oculus, 2020).
TABLE 1. Content of the three paths into which the virtual island is divided.
Implementation: MIND-VR Hardware and Software
Based on the elements that emerged in the previous phases, the psychologists who followed the MIND-VR design created a document for developers describing the basic requirements of the experience and its architecture. Besides, psychologists wrote the storyboards. They included a detailed description of all the visual elements (icons, objects, texts) and sound elements to be incorporated in the various parts of the content. Finally, the audio scripts created for each virtual island path were delivered to professional actors and recorded in mp3 format.
MIND-VR was developed over three months, between July and October 2020. This virtual experience was developed using Unity Engine, with scenes composed with 3D graphics created with 3DSMax, ZBrush, Substance Painter and thoroughly optimized to maintain high and stable framerate on the Oculus Quest. Logic and interactivity were programmed with C# using Microsoft Visual Studio 2019 and based on the SPACS framework.
The virtual experience has been designed to run on the headset Oculus Quest. It is also compatible with Oculus Quest 2. Oculus Quest can be used wirelessly as a standalone device with its integrated hardware. This HMD uses two diamond Pentile OLED displays, each with an individual resolution of 1,440 × 1,600 and a refresh rate of 72 Hz. It utilizes Oculus Touch controllers, which are tracked via an array of cameras embedded in the front of the headset. Like its predecessor, the Oculus Quest 2 (released in October 2020) is a standalone headset with an internal, Android-based operating system. It’s lighter than the first-generation Quest and uses a single fast-switch LCD panel with a 1832 × 1920 per eye resolution, and it supports a 90 Hz refresh rate.
Evaluation
Following the UCD design process for optimizing the virtual experience, various phases of MIND-VR testing are envisaged. The studies have received ethical approval from the Ethics Committee of the University of Milano-Bicocca and the Foundation IRCCS Neurological Institute Carlo Besta. In the coming months, the experiments will be extended to include other Italian hospitals. All data collected will be stored anonymously in a web Central database Repository. Security and privacy issues will consider the respective situation and the participant's consent. Specifically, the study will follow the World Medical Association Declaration of Helsinki regarding ethical principles for medical research involving human subjects (World Medical Association, 2013). Concerning the COVID-19 infection prevention, specific safety and hygiene procedures will be adopted (see Table 2).
TABLE 2. Safety and hygiene protocol to prevent COVID-19 infection during the experimental sessions.
Pilot Study one
The first step will involve testing MIND-VR in a pilot study with a small portion of subjects recruited from the target population to verify the usability of and satisfaction with the mock-up (Alpha version) of the virtual content. The pilot study will be used to identify any problems during the interaction with users and to collect data that can be used in the development of the final version of the virtual experience (Beta version).
• Study design and setting: This study will be using a small sample size and a mixed-methods design. Psychometric outcomes will serve as quantitative variables, while participant interviews will provide qualitative data. In the room where the experiments will be conducted, a second HMD will be available if the first one encounters technical problems to avoid problems with the experimental procedure and participants' disappointment.
• Participants: Twenty healthcare workers will be recruited as a convenience sample. The participants' eligibility will be verified through an interview with a psychologist who will explain the purpose and procedures of the training and conduct an assessment to identify the participants who can enter the study. On this occasion, if the psychologist detects levels and symptoms of stress and/or anxiety of clinical interest, the participant will be sent to the Institutional Psychological Help Desk and will be excluded from the study. The study inclusion criteria will be: 1) maximum age of 65; 2) having worked on the front lines during the COVID-19 emergency; 3) absence of medical disorders (heart disease or blood pressure, neurological disorders, epilepsy); 4) absence of pharmacotherapy that could interfere with the measured data (psychoactive drugs, anti-hypertensive, anti-depressants); 5) no significant visual impairment (all with normal visual acuity or corrected to normal). Participants will be informed about the possibility to participate in the study with oral communication and a formal email from the institutional study referent. They should confirm their intent to participate in the study by answering that email and the referent will set an appointment for the screening interview, as described above. Eligible participants will be given written information about the procedure and asked to sign the consent form indicating their willingness to participate in the study. The participants will also be informed about the safety and hygiene procedures to prevent COVID-19 infection (see Table 2).
• Measures: Participants will complete a demographic questionnaire (age, years of education, profession, hospital, work department). They will also complete an ad hoc questionnaire assessing their use of technological solutions and virtual reality (see Table 3). To study the usability of MIND-VR, all participants will complete the following questionnaires after the virtual experience: System Usability Score (SUS) (Bangor et al., 2008); Slater-Usoh-Steed Presence Questionnaire (SUS-II) (Slater et al., 1994); Igroup Presence Questionnaire (IGP) (Schubert et al., 2001); Net Promoter Score (NPS) (Reichheld, 2003). These questionnaires have good internal consistency, with a Cronbach α coefficient ranging from 0.74 to 0.92 (Lewis and Sauro, 2009; Panahi-Shahri et al., 2009; Sauro and Lewis, 2011; Lotko, 2015; Schwind et al., 2019). Also, to measure changes in the affective states of users, the following scales will be administered: State-Trait Anxiety Inventory Y-1 (STAI Y-1) (Spielberger et al., 1983; Spielberger 2010); Visual Analogue Scale (VAS) (Aitken, 1969; Flint et al., 2000). The STAI-Y1 scale has high internal consistency, with a Cronbach α coefficient ranging from 0.86 to 0.95 (Spielberger, 2010; Julian, 2011; Rossi and Pourtois, 2012). Regarding the VAS, several studies have confirmed its reliability and validity (e.g. Luria, 1975; Gift, 1989; Abend et al., 2014). Finally, at the end of the experimental session, the participants will be interviewed for about 15 min. The questions will be based on the methodology used in a recent study (Migoya-Borja et al., 2020): Were there any features that you especially liked? Were there any features you did not like or found useless? Have you learned anything after using MIND-VR? Were there any features more useful than others? Would you like to have more sessions with MIND-VR? Do you have any questions or would like to add anything?
• Procedure: At the beginning of the experimental session, participants will be asked to answer a questionnaire on their demographic, level of knowledge of technology and virtual reality, and the STAI-Y1 and VAS. Subsequently, they will be able to use MIND-VR freely for about 15 min. The participants will then fill out questionnaires on the content's usability (SUS, SUS-II, IGP, NPS) and, again, the STAI-Y1 and VAS. Finally, a short interview of about 15 min will be conducted. The session will last approximately 40 min in total.
• Data analysis strategy: A descriptive statistical analysis of the usability questionnaires will be performed to evaluate the participants’ acceptability of MIND-VR. Besides, the differences between the final and initial STAI-Y1 and VAS scores will be calculated. A Wilcoxon rank test will be conducted to determine whether the age, gender, or the level of knowledge of technology and virtual reality of the participants influences the usability questionnaires score or the change in the STAI-Y1 and VAS scores.
TABLE 3. The ad hoc questionnaire on technological solutions and virtual reality will be adopted in Pilot Study one and 2.
Pilot Study two
In the second step, MIND-VR (Beta version) will be tested on a small sample of target individuals to evaluate the learning experience of the basic information on stress and anxiety. Previous empirical studies suggest that virtual reality leads to better outcomes than slideshows or textbooks presented on desktop displays (Kozhevnikov and Gurlitt, 2013; Webster, 2016; Orman et al., 2017). However, since no study has yet specifically investigated learning psychoeducational content, the goal of Pilot Study two is to compare MIND-VR and a non-immersive modality (i.e. a slideshow on a computer screen) in the acquisition of information on stress and anxiety through MIND-VR and a non-immersive modality (i.e. a slideshow on a computer screen).
• Study design and setting: This pilot study will adopt within-subject repeated measures (2 × 4) design. Participants will be randomly assigned to one of the two groups, the “VR Group” that will try MIND-VR, and the “Non-VR group” that will try a non-immersive (2D) slideshow. Data will be recorded on four-time points, including three follow-ups, at 1 week, 1 month, and 6 months to assess the long-term learning and its generalization. To avoid the disappointment of the participants who will try the slideshow, at the end of the experiment, individuals in the "Non-VR Group" will be offered the possibility to try MIND-VR for about 15 min.
• Participants: Thirty healthcare workers will be recruited as a convenience sample. The recruitment methods and inclusion criteria will be the same as those adopted in Pilot Study 1. The participants in this experiment will be different from those in Pilot Study 1.
• Measures: Participants will complete a questionnaire assessing their demographic characteristics and the use of technological solutions and virtual reality (see Pilot Study one for details). They will also fill in an ad hoc questionnaire measuring their background knowledge of stress and anxiety (see Table 4). We will use a self-reported background knowledge questionnaire rather than a pretest about the material presented in the study because we want to avoid a testing effect in which the pretest would serve as a form of instruction to prime learners to learn the answers before the actual lesson (Fiorella and Mayer, 2015). Furthermore, to measure subjective indices of the learning level of psychoeducational information, an ad hoc questionnaire on the learning level of psychoeducational contents will be administered at the end of the experimental session. The questions will be based on the methodology used in a recent study (Parong and Mayer, 2018) (see Table 5). Also, participants will be asked to complete a posttest containing 10 questions, eight factual questions in multiple-choice format, and two conceptual questions in short-answer format, assessing the learning outcome of participants based on their experience of the VR. An example of a factual question is, “Which is the first phase of the “General Adaptation Syndrome” (A) Resistance phase (B) Alarm phase (C) Exhaustion phase. An example of a conceptual question is, “What are the main symptoms of stress?”.
• Procedure: First, the experimenter will describe the study, and the participants will sign the informed consent form. Second, the participants will complete the demographic questionnaires, the ad hoc questionnaire on the use of technological solutions and virtual reality, and the background knowledge of stress and anxiety. Third, the psychoeducational material on stress and anxiety will be presented. In the VR condition, the participants will use MIND-VR for about 15 min and complete the island's first path (i.e. “Introduction on stress and anxiety”). In the Non-VR condition, each participant will try a 2D slideshow, which will present the same content as that included in the first path of MIND-VR. The slideshow will be presented on a desktop computer with a 20-inch color screen. Fourth, participants will complete a post-questionnaire about their experiences and a posttest on the material they viewed, with no time limit. Finally, participants will be thanked and dismissed. The session will last approximately 1 h. The same tests made at the end of the experimental session will be conducted also after 1 week, 1 month, and 6 months in the presence of the researcher, to evaluate long-term learning.
• Data analysis strategy: The presence of spurious associations between demographic variables and learning scores will be evaluated using a chi-square test of independence and t-test analyses for independent samples. An analysis of covariance (ANCOVA) will be conducted on the posttest learning scores of the VR and non-VR groups with background stress and anxiety-knowledge scores as covariates.
TABLE 4. The ad hoc questionnaire on the background knowledge of stress and anxiety will be adopted in Pilot Study 2.
TABLE 5. The ad hoc questionnaire on the learning level of psychoeducational contents will be adopted in Pilot Study 2.
Conclusion
To the best of our knowledge, MIND-VR represents one of the first few published virtual reality content designed specifically to be used for psychological support for healthcare workers affected by the health crisis triggered by the COVID-19 pandemic. Once the ongoing pilot studies have been concluded, the data collected will allow to verify the usability of the first version of this virtual experience and to collect important data for the development of its final version. Besides, the collected data will permit to make more general reflections on the effect of virtual reality for psychoeducation in terms of short and long-term learning.
Interestingly, since MIND-VR has been developed not only in Italian but also in English, its use can be extended beyond Italian national borders and become a tool available free of charge worldwide for psychological support to doctors and nurses involved in the COVID-19 pandemic. Furthermore, this virtual reality psychoeducational experience on stress and anxiety can also be used once the COVID-19 emergency is over within prevention programs to prevent and alleviate stress and anxiety in healthcare personnel. In particular, MIND-VR can be included as a part of psychoeducation within cognitive-behavioral skills training for the medical personnel. This type of program is considered among the most effective approaches in reducing stress and anxiety among hospital staff (Clough et al., 2017; Alkhawaldeh et al., 2020), and it could benefit from integrating new technologies, including virtual reality (Richardson, 2017; Pallavicini and Bouchard, 2019; Weerasekara and Smedberg, 2019).
In addition to these aspects, other elements make MIND-VR potentially very relevant. First, this virtual psychoeducational content can be used not only with doctors and nurses but also with all other categories of people who need support in the management of stress and anxiety. For example, MIND-VR can be used within programs for the prevention and treatment of such conditions of mental suffering for all those people forced into isolation and/or quarantine due to COVID-19 infection. Also, in the next months, this psychoeducational virtual content will be tested in collaboration with an Italian hospital as a tool for psychoeducation on stress and anxiety within psychological support programs for families of patients with dementia, another typology of people greatly in need of new support tools for mental well-being.
Second, since MIND-VR was created to be very simple to use and to run on some currently more widespread, user-friendly, and low-cost HMDs (ie, Oculus Quest and Oculus Quest 2), such an experience could be also used autonomously by people and offered to provide psychological assistance remotely. This fact appears important because, due to the limitations imposed in many countries of the world in response to the COVID-19 pandemic emergency, many people have difficulty accessing face-to-face consultations and new tools appear necessary to increase the effectiveness of the therapies offered. remotely. Interestingly, in a recent study, it was reported that virtual reality-based remote therapy is more effective than treatments through video conferencing using programs such as Skype or Zoom, increasing the level of engagement, reducing dropout rates, and producing positive clinical outcomes (Pedram et al., 2020).
Finally, given how few formative research studies and protocols are published for similar projects, the methodology used for MIND-VR could provide important knowledge for determining the feasibility of adopting virtual reality-based psychoeducational experience, expanding the range of virtual contents that can be created to address mental suffering. The methodology used for the development of MIND-VR, in fact, can also be used for the creation of other virtual psychoeducational experiences, providing important guiding elements in the creation of such virtual contents. While we adopted this specific process, others exist for developing interactive entertainment for behavior change (Mummah et al., 2016; Gotsis and Jordan-Marsh, 2018) and can serve as examples of the process in a field where there is a shortage of HCI professionals working in healthcare, as well as a shortage of creatives who are trained in UCD.
Data Availability Statement
The original contributions presented in the study are included in the article. Further inquiries can be directed to the corresponding author.
Author Contribution
FP conceived the work, wrote the first draft of the manuscript, and led the development of requirements, architectural paradigm, content, and evaluation protocol of MIND-VR. FM supervised the work in its scientific aspect. EO, SS, AP, and GV as highly experienced psychotherapists and psychologists, supervised the clinical features of MIND-VR. All the authors contributed to manuscript revision, read, and approved the submitted version of the manuscript.
Conflict of Interest
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
Acknowledgments
The authors would like to thank Univale Onlus and the more than fifty supporters who, through the fundraising campaign, have allowed the realization of MIND-VR. This article is dedicated to them.
References
Abend, R., Dan, O., Maoz, K., Raz, S., and Bar-Haim, Y. (2014). Reliability, validity and sensitivity of a computerized visual analog scale measuring state anxiety. J. Behav. Ther. Exp. Psychiatr. 45, 447–453. doi:10.1016/j.jbtep.2014.06.004
Aitken, R. C. (1969). Measurement of feelings using visual analogue scales. Proc. Roy. Soc. Med. 62, 989–993. doi:10.1177/003591576906201005
Alkhawaldeh, J. M. A., Soh, K. L., Mukhtar, F. B. M., Peng, O. C., and Anshasi, H. A. (2020). Stress management interventions for intensive and critical care nurses: a systematic review. Nurs. Crit. Care. 25, 84–92. doi:10.1111/nicc.12489
American Psychiatric Association (2013). Diagnostic and statistical manual of mental disorders. 5th Edn. Arlington, TX, United States: American Psychiatric Association
Anderson, A. P., Mayer, M. D., Fellows, A. M., Cowan, D. R., Hegel, M. T., and Buckey, J. C. (2017). Relaxation with immersive natural scenes presented using virtual reality. Aerosp. Med. Hum. Perform. 88, 520–526. doi:10.3357/AMHP.4747.2017
Babore, A., Lombardi, L., Viceconti, M. L., Pignataro, S., Marino, V., Crudele, M., et al. (2020). Psychological effects of the COVID-2019 pandemic: perceived stress and coping strategies among healthcare professionals. Psychiatr. Res. 293, 113366. doi:10.1016/j.psychres.2020.113366
Baharom, S. N., Tan, W. H., and Idris, M. Z. (2014). Emotional design for games: a framweork for player-centric approach in the game design process. Int. J. Multimed. Ubiquitous Eng. 9, 387–398. doi:10.14257/ijmue.2014.9.10.37
Bailenson, J. N., Yee, N., Blascovich, J., Beall, A. C., Lundblad, N., and Jin, M. (2008). The use of immersive virtual reality in the learning sciences: digital transformations of teachers, students, and social context. J. Learn. Sci. 17, 102–141. doi:10.1080/10508400701793141
Bangor, A., Kortum, P. T., and Miller, J. T. (2008). An empirical evaluation of the system usability scale. Int. J. Hum. Comput. Interact. 24, 574–594. doi:10.1080/10447310802205776
Barbieri, L., Bruno, F., and Muzzupappa, M. (2018). User-centered design of a virtual reality exhibit for archaeological museums. Int. J. Interact. Des. Manuf. 12, 561–571. doi:10.1007/s12008-017-0414-z
Boger, J., Eisapour, M., Cao, S., and Domenicucci, L. (2018). “Participatory design of a virtual reality exercise for people with mild cognitive impairment.” Conference on human factors in computing systems—proceedings, 1–9, Montreal, QC, Canada, 2018: Association for Computing Machinery). doi:10.1145/3170427.3174362
Bohlken, J., Schömig, F., Lemke, M. R., Pumberger, M., and Riedel-Heller, S. G. (2020). COVID-19 pandemic: stress experience of healthcare workers—a short current review. Psychiatr. Prax. 47, 190–197. doi:10.1055/a-1159-5551
Boletsis, C., and Cedergren, J. E. (2019). VR locomotion in the new era of virtual reality: an empirical comparison of prevalent techniques. Adv. Human-Computer Interact. 20197420781. doi:10.1155/2019/7420781
Bozgeyikli, E., Raij, A., Katkoori, S., and Dubey, R. (2016). “Point and Teleport locomotion technique for virtual reality.” Proceedings of the 2016 annual symposium on computer-human interaction in play (CHI PLAY 2016). New York, NY, United States: Association for Computing Machinery, Inc), 205–216
Briki, W., and Majed, L. (2019). Adaptive effects of seeing green environment on psychophysiological parameters when walking or running. Front. Psychol. 10, 252. doi:10.3389/fpsyg.2019.00252
Brooks, S. K., Webster, R. K., Smith, L. E., Woodland, L., Wessely, S., Greenberg, N., et al. (2020). The psychological impact of quarantine and how to reduce it: rapid review of the evidence. Lancet. 395, 912–920. doi:10.1016/S0140-6736(20)30460-8
Browning, M. H. E. M., Mimnaugh, K. J., van Riper, C. J., Laurent, H. K., and LaValle, S. M. (2019). Can simulated nature support mental health? Comparing short, single-doses of 360-degree nature videos in virtual reality with the outdoors. Front. Psychol. 10, 2667. doi:10.3389/fpsyg.2019.02667
Brox, E., Konstantinidis, S. T., and Evertsen, G. (2017). User-centered design of serious games for older adults following 3 Years of experience with exergames for seniors: a study design. JMIR Serious Games. 5, e2. doi:10.2196/games.6254
Cai, X., Hu, X., Ekumi, I. O., Wang, J., An, Y., Li, Z., et al. (2020a). Psychological distress and its correlates among COVID-19 survivors during early convalescence across age groups. Am. J. Geriatr. Psychiatr. 28, 1030–1039. doi:10.1016/j.jagp.2020.07.003
Cai, H., Tu, B., Ma, J., Chen, L., Fu, L., Jiang, Y., et al. (2020b). Psychological impact and coping strategies of frontline medical staff in hunan between january and march 2020 during the outbreak of coronavirus disease 2019 (COVID-19) in Hubei, China. Med. Sci. Mon. Int. Med. J. Exp. Clin. Res. 26, e924171–16. doi:10.12659/MSM.924171
Carmassi, C., Foghi, C., Dell'Oste, V., Cordone, A., Bertelloni, C. A., Bui, E., et al. (2020). PTSD symptoms in healthcare workers facing the three coronavirus outbreaks: what can we expect after the COVID-19 pandemic. Psychiatr. Res. 292, 113312. doi:10.1016/j.psychres.2020.113312
Chen, Q., Liang, M., Li, Y., Guo, J., Fei, D., Wang, L., et al. (2020). Mental health care for medical staff in China during the COVID-19 outbreak. Lancet Psychiatry. 7, e15–e16. doi:10.1016/S2215-0366(20)30078-X
Cheng, P., Xia, G., Pang, P., Wu, B., Jiang, W., Li, Y. T., et al. (2020). COVID-19 epidemic peer support and crisis intervention via social media. Community Ment. Health J. 56, 786–792. doi:10.1007/s10597-020-00624-5
Chirico, F., Nucera, G., and Magnavita, N. (2020). Protecting the mental health of healthcare workers during the COVID-19 emergency. BJPsych Int., 1–2. doi:10.1192/bji.2020.39
Chittaro, L., and Ranon, R. (2007). Web3D technologies in learning, education and training: motivations, issues, opportunities. Comput. Educ. 49, 3–18. doi:10.1016/j.compedu.2005.06.002
Clough, B. A., March, S., Chan, R. J., Casey, L. M., Phillips, R., and Ireland, M. J. (2017). Psychosocial interventions for managing occupational stress and burnout among medical doctors: a systematic review. Syst. Rev. 6, 144. doi:10.1186/s13643-017-0526-3
Conway, P. M., Campanini, P., Sartori, S., Dotti, R., and Costa, G. (2008). Main and interactive effects of shiftwork, age and work stress on health in an Italian sample of healthcare workers. Appl. Ergon. 39, 630–639. doi:10.1016/j.apergo.2008.01.007
Coulthard, K., Patel, D., Brizzolara, C., Morriss, R., and Watson, S. (2013). A feasibility study of expert patient and community mental health team led bipolar psychoeducation groups: implementing an evidence based practice. BMC Psychiatr. 13, 301. doi:10.1186/1471-244X-13-301
de Clerk, M., Dangelmaier, M., Schmierer, G., and Spath, D. (2019). User centered design of interaction techniques for VR-based automotive design reviews. Front. Robot. AI. 6, 13. doi:10.3389/frobt.2019.00013
De Freitas, S., Rebolledo-Mendez, G., Liarokapis, F., Magoulas, G., and Poulovassilis, A. (2010). Learning as immersive experiences: using the four-dimensional framework for designing and evaluating immersive learning experiences in a virtual world. Br. J. Educ. Technol. 41, 69–85. doi:10.1111/j.1467-8535.2009.01024.x
De Paula, D. F. O., Menezes, B. H. X. M., and Araújo, C. C. (2014). “Building a quality mobile application: a user-centered study focusing on design thinking, user experience and usability,” in Design, user experience, and usability. User experience design for diverse interaction platforms and environments. Lecture notes in computer science. Editor A. Marcus (Cham, Switzerland): Springer), 8518, 313–322. doi:10.1007/978-3-319-07626-3_29
de Witte, M., Spruit, A., van Hooren, S., Moonen, X., and Stams, G. J. (2020). Effects of music interventions on stress-related outcomes: a systematic review and two meta-analyses. Health Psychol. Rev. 14, 294–324. doi:10.1080/17437199.2019.1627897
Dekker, M. R., and Williams, A. D. (2017). The use of user-centered participatory design in serious games for anxiety and depression. Game. Health J. 6, 327–333. doi:10.1089/g4h.2017.0058
Dellve, L., Hadzibajramovic, E., and Ahlborg, G. (2011). Work attendance among healthcare workers: prevalence, incentives, and long-term consequences for health and performance. J. Adv. Nurs. 67, 1918–1929. doi:10.1111/j.1365-2648.2011.05630.x
DeSmet, A., Thompson, D., Baranowski, T., Palmeira, A., Verloigne, M., and De Bourdeaudhuij, I. (2016). Is participatory design associated with the effectiveness of serious digital games for healthy lifestyle promotion? A meta-analysis. J. Med. Internet Res. 18, e94. doi:10.2196/jmir.4444
Desmet, P., and Hekkert, P. (2007). Framework of product experience. Int. J. Des. 1, 57–66. doi:10.1162/074793602320827406
Di Tella, M., Romeo, A., Benfante, A., and Castelli, L. (2020). Mental health of healthcare workers during the COVID-19 pandemic in Italy. J. Eval. Clin. Pract. 26, 1583–1587. doi:10.1111/jep.13444
Dimitropoulos, K., Manitsaris, A., and Mavridis, I. (2007). Building virtual reality environments for distance education on the web: a case study in medical education. Int. J. Soc. Sci. 2, 62–70. doi:10.5281/zenodo.1330935
Donker, T., Griffiths, K. M., Cuijpers, P., and Christensen, H. (2009). Psychoeducation for depression, anxiety and psychological distress: a meta-analysis. BMC Med. 7, 79. doi:10.1186/1741-7015-7-79
Du, J., Dong, L., Wang, T., Yuan, C., Fu, R., Zhang, L., et al. (2020). Psychological symptoms among frontline healthcare workers during COVID-19 outbreak in Wuhan. Gen. Hosp. Psychiatr. 67, 144–145. doi:10.1016/j.genhosppsych.2020.03.011
Eisapour, M., Cao, S., Domenicucci, L., and Boger, J. (2018). “Participatory design of a virtual reality exercise for people with mild cognitive impairment.” in Extended abstracts of the 2018 CHI conference on human factors in computing systems, NY United States, 2018. Editors R. Mandryk, and M. Hancock (New York, NY, United States: Association for Computing Machinery)), 1–9
Farao, J., Malila, B., Conrad, N., Mutsvangwa, T., Rangaka, M. X., and Douglas, T. S. (2020). A user-centred design framework for mHealth. PLoS One. 15, e0237910. doi:10.1371/journal.pone.0237910
Fidopiastis, C. M., Rizzo, A. A., and Rolland, J. P. (2010). User-centered virtual environment design for virtual rehabilitation. J. NeuroEng. Rehabil. 7, 11. doi:10.1186/1743-0003-7-11
Fiorella, L., and Mayer, R. E. (2015). Learning as a generative activity: eight learning strategies that promote understanding. Cambridge, United Kingdom: Cambridge University Press
Fischer, X., and Coutellier, D. (2007). Research in interactive design, 2. Paris, France: Springer-Verlag Paris
Flint, A., Raben, A., Blundell, J. E., and Astrup, A. (2000). Reproducibility, power and validity of visual analogue scales in assessment of appetite sensations in single test meal studies. Int. J. Obes. Relat. Metab. Disord. 24, 38–48. doi:10.1038/sj.ijo.0801083
Flobak, E., Wake, J. D., Vindenes, J., Kahlon, S., Nordgreen, T., and Guribye, F. (2019). “Participatory design of VR scenarios for exposure therapy.” Conference on human factors in computing systems—proceedings, 1–12, New York, United States, 2019. Association for Computing Machinery)
Gabbard, J. L., Hix, D., and Swan, J. E. (1999). User-centered design and evaluation of virtual environments. IEEE Comput. Graph. Appl. 19, 51–59. doi:10.1109/38.799740
Gaggioli, A., Pallavicini, F., Morganti, L., Serino, S., Scaratti, C., Briguglio, M., et al. (2014). Experiential virtual scenarios with real-time monitoring (interreality) for the management of psychological stress: a block randomized controlled trial. J. Med. Internet Res. 16, e167. doi:10.2196/jmir.3235
Gandi, J. C., Wai, P. S., Karick, H., and Dagona, Z. K. (2011). The role of stress and level of burnout in job performance among nurses. Ment. Health Fam. Med. 8, 181–194
Gerardi, M., Rothbaum, B. O., Ressler, K., Heekin, M., and Rizzo, A. (2008). Virtual reality exposure therapy using a virtual Iraq: case report. J. Trauma Stress. 21, 209–213. doi:10.1002/jts.20331
Gift, A. G. (1989). Visual analogue scales: measurement of subjective phenomena. Nurs. Res. 38, 286–288
Gladden, M. (2018). A phenomenological framework of architectural paradigms for the user-centered design of virtual environments. Multimodal Technol. Interact. 2, 80. doi:10.3390/mti2040080
Gomes, N., Lou, Y., Patwardhan, V., Moyer, T., Vavala, V., and Barros, C. (2020). The effects of virtual reality learning environments on improving the retention, comprehension, and motivation of medical school students. Advances in intelligent systems and computing. Human Interaction and Emerging Technologies. Paris, France: Springer-Verlag, 289–296. doi:10.1007/978-3-030-25629-6_45
Gorini, A., Capideville, C. S., De Leo, G., Mantovani, F., and Riva, G. (2011). The role of immersion and narrative in mediated presence: the virtual hospital experience. Cyberpsychol., Behav. Soc. Netw. 14, 99–105. doi:10.1089/cyber.2010.0100
Gotsis, M., and Jordan-Marsh, M. (2018). “Calling HCI professionals into health research: patient safety and health equity at stake.” Proceedings of the 22nd pan-hellenic conference on informatics—PCI ’18, New York, NY, United States, 2018, Association for Computing Machinery), 213–218
Grace, S. L., Hershenfield, K., Robertson, E., and Stewart, D. E. (2005). The occupational and psychosocial impact of SARS on academic physicians in three affected hospitals. Psychosomatics. 46, 385–391. doi:10.1176/appi.psy.46.5.385
Gradl, S., Wirth, M., Zillig, T., and Eskofier, B. M. (2018). “Visualization of heart activity in virtual reality: a biofeedback application using wearable sensors.” IEEE 15th international conference on wearable and implantable body sensor networks (BSN). Las Vegas, NV, United States, 2018, IEEE). 152–155
Greenberg, N., Docherty, M., Gnanapragasam, S., and Wessely, S. (2020). Managing mental health challenges faced by healthcare workers during covid-19 pandemic. BMJ. 368, m1211. doi:10.1136/bmj.m1211
Gulliver, A., Griffiths, K. M., and Christensen, H. (2010). Perceived barriers and facilitators to mental health help-seeking in young people: a systematic review. BMC Psychiatr. 10, 113–119. doi:10.1186/1471-244X-10-113
Hardy, A., Wojdecka, A., West, J., Matthews, E., Golby, C., Ward, T., et al. (2018). How inclusive, user-centered design research can improve psychological therapies for psychosis: development of SlowMo. JMIR Ment. Heal. 5, e11222. doi:10.2196/11222
Hix, D., Swan, J. E., Gabbard, J. L., McGee, M., Durbin, J., and King, T. (1999). User-centered design and evaluation of a real-time battlefield visualization virtual environment. Proceedings virtual Reality. Annu. Int. Symp., 96–103. doi:10.1109/vr.1999.756939
Houghton, S., and Saxon, D. (2007). An evaluation of large group CBT psycho-education for anxiety disorders delivered in routine practice. Patient Educ. Counsel. 68, 107–110. doi:10.1016/j.pec.2007.05.010
Huang, H. M., Rauch, U., and Liaw, S. S. (2010). Investigating learners’ attitudes toward virtual reality learning environments: based on a constructivist approach. Comput. Educ. 55, 1171–1182. doi:10.1016/j.compedu.2010.05.014
Huang, J. Z., Han, M. F., Luo, T. D., Ren, A. K., and Zhou, X. P. (2020). Mental health survey of 230 medical staff in a tertiary infectious disease hospital for COVID-19. Zhonghua Lao Dong Wei Sheng Zhi Ye Bing Za Zhi. 38, E001. doi:10.3760/cma.j.cn121094-20200219-00063
Imperatori, C., Dakanalis, A., Farina, B., Pallavicini, F., Colmegna, F., Mantovani, F., et al. (2020). Global storm of stress-related psychopathological symptoms: a brief overview on the usefulness of virtual reality in facing the mental health impact of COVID-19. Cyberpsychol., Behav. Soc. Netw. 23, 782–788. doi:10.1089/cyber.2020.0339
ISO/IEC (1999). 13,407 Human-centred design processes for interactive systems Available at:Accessed: https://www.iso.org/standard/21197.html October 1, 2020)
Italian National Institute of Health (2020). Stress management among healthcare workers. Available at: https://www.epicentro.iss.it/en/coronavirus/sars-cov-2-stress-management-healthcare-workers
Jerald, J. (2015). The VR book: human-centered design for virtual reality. New York NY United States: Association for Computing Machinery and Morgan and Claypool
Johns Hopkins Coronavirus Resource Center (2020). COVID-19 map Available at:Accessed: https://coronavirus.jhu.edu/map.html May 19, 2020)
Jorm, A. F. (2000). Mental health literacy. Public knowledge and beliefs about mental disorders. Br. J. Psychiatry. 177, 396–401. doi:10.1192/bjp.177.5.396
Julian, L. J. (2011). Measures of anxiety: state-trait anxiety inventory (STAI), beck anxiety inventory (bai), and hospital anxiety and depression scale-anxiety (HADS-A). Arthritis Care Res. 63 Suppl 11, S467–S472. doi:10.1002/acr.20561
Kahn, P. H., Severson, R. L., and Ruckert, J. H. (2009). The human relation with nature and technological nature. Curr. Dir. Psychol. Sci. 18, 37–42. doi:10.1111/j.1467-8721.2009.01602.x
Kang, L., Li, Y., Hu, S., Chen, M., Yang, C., Yang, B. X., et al. (2020). The mental health of medical workers in Wuhan, China dealing with the 2019 novel coronavirus. Lancet Psychiatry. 7, e14. doi:10.1016/S2215-0366(20)30047-X
Kazdin, A. E. (2000). Encyclopedia of psychology. New York, NY, United States: Oxford University Press
Koenig, S. T., Krch, D., Chiaravalloti, N., Lengenfelder, J., Nikelshpur, O., Lange, B. S., et al. (2012). “User-centered development of a virtual reality cognitive assessment.” Proceedings of international conference on disability, virtual reality and associated technologies, Laval, France, 2012. 10–12
Kosunen, I., Salminen, M., Järvelä, S., Ruonala, A., Ravaja, N., and Jacucci, G. (2016). “RelaWorld: neuroadaptive and immersive virtual reality meditation system.” Proceedings of the 21st international conference on intelligent user interfaces, New York, NY, United States, 2016: Association for Computing Machinery), 208–217
Kozhevnikov, M., and Gurlitt, J. (2013). “Immersive and non-immersive virtual reality system to learn relative motion concepts.” Proceedings of the 3rd interdisciplinary engineering design education conference, Las Vegas, NV, United States, 2013. IEEE), 168–172
Krystal, J. H., and McNeil, R. L. (2020). Responding to the hidden pandemic for healthcare workers: stress. Nat. Med. 26, 639. doi:10.1038/s41591-020-0878-4
Kuan, G., Morris, T., Kueh, Y. C., and Terry, P. C. (2018). Effects of relaxing and arousing music during imagery training on dart-throwing performance, physiological arousal indices, and competitive state anxiety. Front. Psychol. 9, 14. doi:10.3389/fpsyg.2018.00014
Kurt, S., and Osueke, K. K. (2014). The effects of color on the moods of college students. SAGE Open. 4, 215824401452542. doi:10.1177/2158244014525423
Lai, J., Ma, S., Wang, Y., Cai, Z., Hu, J., Wei, N., et al. (2020). Factors associated with mental health outcomes among health care workers exposed to coronavirus disease 2019. JAMA Netw. open. 3, e203976. doi:10.1001/jamanetworkopen.2020.3976
Lewis, J. R., and Sauro, J. (2009). “The factor structure of the system usability scale,” in Human centered design. HCD 2009. Lecture notes in computer science. Editor M. Kurosu (Berlin, Germany: Spinger)), Vol5619, 94–103
Lindner, P., Miloff, A., Hamilton, W., and Carlbring, P. (2019). The potential of consumer-targeted virtual reality relaxation applications: descriptive usage, uptake and application performance statistics for a first-generation application. Front. Psychol. 10, 132. doi:10.3389/fpsyg.2019.00132
Litleskare, S., Macintyre, T. E., and Calogiuri, G. (2020). Enable, reconnect and augment: a new era of virtual nature research and application. Int. J. Environ. Res. Publ. Health. 17, 1738. doi:10.3390/ijerph17051738
Liu, C.-Y., Yang, Y., Zhang, X.-M., Xu, X., Dou, Q.-L., and Zhang, W.-W. (2020). The prevalence and influencing factors for anxiety in medical workers fighting COVID-19 in China: a cross-sectional survey. SSRN Electron. J. 148, e98. doi:10.2139/ssrn.3548781
Llema, C. F., and Vilela-Malabanan, C. M. (2019). Design and development of MLERWS: a user-centered mobile application for English reading and writing skills. Proc. Comp. Science. 161, 1002–1010. doi:10.1016/j.procs.2019.11.210
Lopes, A., Valentim, N., Moraes, B., Zilse, R., and Conte, T. (2018). Applying user-centered techniques to analyze and design a mobile application. J. Softw. Eng. Res. Dev. 6, 5. doi:10.1186/s40411-018-0049-1
Lotko, A. (2015). Classifying customers according to NPS index: cluster analysis for contact center services. Central European Review of Economics and Finance. 8 (2), 5–30
Luria, R. E. (1975). The validity and reliability of the visual analogue mood scale. J. Psychiatr. Res. 12, 51–57. doi:10.1016/0022-3956(75)90020-5
Ma, Y. F., Li, W., Deng, H. B., Wang, L., Wang, Y., Wang, P. H., et al. (2020). Prevalence of depression and its association with quality of life in clinically stable patients with COVID-19. J. Affect. Disord. 275, 145–148. doi:10.1016/j.jad.2020.06.033
Malone, T. W. (1980). “What makes things fun to learn? heuristics for designing instructional computer games,” in Proceedings of the 3rd ACM SIGSMALL symposium and the first SIGPC symposium on Small systems. Editors P. Lehot, L. Loop, and G. W. Gorsline (New York, NY, United States: Association for Computing Machinery)), 162–169
Malone, T. W. (1982). “Heuristics for designing enjoyable user interfaces,” in Proceedings of the 1982 conference on human factors in computing systems. Editors J. A. Nichols, and M. L. Schneider (New York, NY, United States: Association for Computing Machinery)), 63–68
Maples-Keller, J. L., Bunnell, B. E., Kim, S. J., and Rothbaum, B. O. (2017). The use of virtual reality technology in the treatment of anxiety and other psychiatric disorders. Harv. Rev. Psychiatr. 25, 103–113. doi:10.1097/HRP.0000000000000138
Mayring, P. (2015). “Qualitative content analysis: theoretical background and procedures,” in In approaches to qualitative research in mathematics education. Editors A. Bikner-Ahsbahs, C. Knipping, and N. Presmeg N (Dordrecht, Netherlands: Springer), 365–380
McAlonan, G. M., Lee, A. M., Cheung, V., Cheung, C., Tsang, K. W., Sham, P. C., et al. (2007). Immediate and sustained psychological impact of an emerging infectious disease outbreak on health care workers. Can. J. Psychiatr. 52, 241–247. doi:10.1177/070674370705200406
Migoya-Borja, M., Delgado-Gómez, D., Carmona-Camacho, R., Porras-Segovia, A., López-Moriñigo, J.-D., Sánchez-Alonso, M., et al. (2020). Feasibility of a virtual reality-based psychoeducational tool (VRight) for depressive patients. Cyberpsychol., Behav. Soc. Netw. 23, 246–252. doi:10.1089/cyber.2019.0497
Minguillon, J., Lopez-Gordo, M. A., Renedo-Criado, D. A., Sanchez-Carrion, M. J., and Pelayo, F. (2017). Blue lighting accelerates post-stress relaxation: results of a preliminary study. PLoS One. 12, e0186399. doi:10.1371/journal.pone.0186399
Müller, M. P., Hänsel, M., Fichtner, A., Hardt, F., Weber, S., Kirschbaum, C., et al. (2009). Excellence in performance and stress reduction during two different full scale simulator training courses: a pilot study. Resuscitation. 80, 919–924. doi:10.1016/j.resuscitation.2009.04.027
Mummah, S. A., Robinson, T. N., King, A. C., Gardner, C. D., and Sutton, S. (2016). IDEAS (integrate, design, assess, and share): a framework and toolkit of strategies for the development of more effective digital interventions to change health behavior. J. Med. Internet Res. 18 (12), e317. doi:10.2196/jmir.5927
Navarro-Haro, M. V., López-Del-Hoyo, Y., Campos, D., Linehan, M. M., Hoffman, H. G., García-Palacios, A., et al. (2017). Meditation experts try Virtual Reality Mindfulness: a pilot study evaluation of the feasibility and acceptability of Virtual Reality to facilitate mindfulness practice in people attending a Mindfulness conference. PLoS One. 12, e0187777. doi:10.1371/journal.pone.0187777
Nielsen, J. (1994). “Usability inspection methods,” in Conference companion on human factors in computing systems (CHI ’94). Editor C. Plaisant (New York, NY, United States: Association for Computing Machinery), 413–414
Norberg-Schulz, C. (1980). Genius loci: towards a phenomenology of architecture. New York, NY, United States: Rizzoli
Norman, D. (2007). Emotional design: why we love (or hate) everyday things. New York, NY, United States: Basic Books
Oculus, V. R. (2020). Health and safety warnings. Accessed: https://www.oculus.com/legal/health-and-safety-warnings/December 01, 2020)
Oflaz, F., Hatipoğlu, S., and Aydin, H. (2008). Effectiveness of psychoeducation intervention on post-traumatic stress disorder and coping styles of earthquake survivors. J. Clin. Nurs. 17, 677–687. doi:10.1111/j.1365-2702.2007.02047.x
Oing, T., and Prescott, J. (2018). Implementations of virtual reality for anxiety-related disorders: systematic review. JMIR serious games. 6, e10965. doi:10.2196/10965
Orman, E. K., Price, H. E., and Russell, C. R. (2017). Feasibility of using an augmented immersive virtual reality learning environment to enhance music conducting skills. J. Music Teach. Educ. 27, 24–35. doi:10.1177/1057083717697962
Pallavicini, F., Algeri, D., Repetto, C., Gorini, A., and Riva, G. (2009). Biofeedback, virtual reality and mobile phones in the treatment of generalized anxiety disorder (gad): a phase-2 controlled clinical trial. J. Cyber Ther. Rehabil. 2, 315–327
Pallavicini, F., Gaggioli, A., Raspelli, S., Cipresso, P., Serino, S., Vigna, C., et al. (2013). Interreality for the management and training of psychological stress: study protocol for a randomized controlled trial. Trials. 14, 191. doi:10.1186/1745-6215-14-191
Pallavicini, F., and Bouchard, S. (2019). Editorial: assessing the therapeutic uses and effectiveness of virtual reality, augmented reality and video games for emotion regulation and stress management. Front. Psychol. 10, 2763. doi:10.3389/fpsyg.2019.02763
Pallavicini, F., and Pepe, A. (2020). Virtual reality games and the role of body involvement in enhancing positive emotions and decreasing anxiety: within-subjects pilot study. JMIR serious games. 8, e15635. doi:10.2196/15635
Pallavicini, F., Pepe, A., Ferrari, A., Garcea, G., Zanacchi, A., and Mantovani, F. (2020). What is the relationship among positive emotions, sense of presence, and ease of interaction in virtual reality systems? An on-site evaluation of a commercial virtual experience. PRESENCE: Virtual and Augmented Reality. 27 (2), 183–201. doi:10.1162/pres_a_00325
Panahi-Shahri, M., Fathi-Ashtiani, A., and Azad-Fallah, P. (2009). Reliability and validity of Igroup presence questionnaire (IPQ). J. Behav. Sciences. 3 (1), 27–34
Parong, J., and Mayer, R. E. (2018). Learning science in immersive virtual reality. J. Educ. Psychol. 110, 785–797. doi:10.1037/edu0000241
Parsons, T. D., and Rizzo, A. A. (2008). Affective outcomes of virtual reality exposure therapy for anxiety and specific phobias: a meta-analysis. J. Behav. Ther. Exp. Psychiatr. 39, 250–261. doi:10.1016/j.jbtep.2007.07.007
Pedram, S., Palmisano, S., Perez, P., Mursic, R., and Farrelly, M. (2020). Examining the potential of virtual reality to deliver remote rehabilitation. Comput. Hum. Behav. 105, 106223. doi:10.1016/J.CHB.2019.106223
Pizzoli, S. F. M., Mazzocco, K., Triberti, S., Monzani, D., Alcañiz Raya, M. L., and Pravettoni, G. (2019). User-centered virtual reality for promoting relaxation: an innovative approach. Front. Psychol. 10, 479. doi:10.3389/fpsyg.2019.00479
Poux, F., Valembois, Q., Mattes, C., Kobbelt, L., and Billen, R. (2020). Initial user-centered design of a virtual reality heritage system: applications for digital tourism. Rem. Sens. 12, 2583. doi:10.3390/RS12162583
Qiu, J., Shen, B., Zhao, M., Wang, Z., Xie, B., and Xu, Y. (2020). A nationwide survey of psychological distress among Chinese people in the COVID-19 epidemic: implications and policy recommendations. Gen. psychiatry. 33, e100213. doi:10.1136/gpsych-2020-100213
Reichheld, F. F. (2003). The one number you need to grow. Harv. Bus. Rev. 81, 46Available at:Accessed: https://hbr.org/2003/12/the-one-number-you-need-to-grow December 1, 2020)
Repetto, C., Gorini, A., Vigna, C., Algeri, D., Pallavicini, F., and Riva, G. (2009). The use of biofeedback in clinical virtual reality: the INTREPID project. JoVE. 12, 1554. doi:10.3791/1554
Richardson, K. M. (2017). Managing employee stress and wellness in the new millennium. J. Occup. Health Psychol. 22, 423–428. doi:10.1037/ocp0000066
Rickwood, D., Deane, F. P., Wilson, C. J., and Ciarrochi, J. (2005). Young people’s help-seeking for mental health problems. Aust. e-Journal Adv. Ment. Heal. 4, 218–251. doi:10.5172/jamh.4.3.218
Riva, G., Vigna, C., Grassi, A., Raspelli, S., Cipresso, P., Pallavicini, F., et al. (2012). Learning Island: the development of a virtual reality system for the experiential training of stress management. Stud. Health Technol. Inf. 173, 369–371. doi:10.3233/978-1-61499-022-2-369
Riva, G., Mantovani, F., and Wiederhold, B. K. (2020). Positive technology and COVID-19. Cyberpsychol., Behav. Soc. Netw. 23, 581–587. doi:10.1089/cyber.2020.29194.gri
Riva, G., and Wiederhold, B. K. (2020). How cyberpsychology and virtual reality can help us to overcome the psychological burden of coronavirus. Cyberpsychol., Behav. Soc. Netw. 23, 277–279. doi:10.1089/cyber.2020.29183.gri
Rizzo, A., Newman, B., Parsons, T., Difede, J., Reger, G., Holloway, K., et al. (2009). Development and clinical results from the virtual Iraq exposure therapy application for PTSD. 2009 virtual rehabilitation international conference, Haifa, Israel, 2009. 8–15. IEEE. doi:10.1109/ICVR.2009.5174198
Rizzo, A., Difede, J., Rothbaum, B. O., Reger, G., Spitalnick, J., Cukor, J., et al. (2010). Development and early evaluation of the Virtual Iraq/Afghanistan exposure therapy system for combat-related PTSD. Ann. N. Y. Acad. Sci. 1208, 114–125. doi:10.1111/j.1749-6632.2010.05755.x
Rogers, J. P., Chesney, E., Oliver, D., Pollak, T. A., McGuire, P., Fusar-Poli, P., et al. (2020). Psychiatric and neuropsychiatric presentations associated with severe coronavirus infections: a systematic review and meta-analysis with comparison to the COVID-19 pandemic. Lancet Psychiatry. 7, 611–627. doi:10.1016/S2215-0366(20)30203-0
Romano, J. L. (1992). Psychoeducational interventions for stress management and well-being. J. Counsel. Dev. 71, 199–202. doi:10.1002/j.1556-6676.1992.tb02200.x
Rossi, V., and Pourtois, G. (2012). Transient state-dependent fluctuations in anxiety measured using STAI, POMS, PANAS or VAS: a comparative review. Hist. Philos. Logic. 25, 603–645. doi:10.1080/10615806.2011.582948
Sauro, J., and Lewis, J. R. (2011). “When designing usability questionnaires, does it hurt to be positive?. Conference on human factors in computing systems—proceedings,” New York, NY, United States, 2011: ACM Press), 2215–2223. doi:10.1145/1978942.1979266
Schnall, R., Rojas, M., Bakken, S., Brown, W., Carballo-Dieguez, A., Carry, M., et al. (2016). A user-centered model for designing consumer mobile health (mHealth) applications (apps). J. Biomed. Inf. 60, 243–251. doi:10.1016/j.jbi.2016.02.002
Schubert, T., Friedmann, F., and Regenbrecht, H. (2001). The experience of presence: factor Analytic insights. Presence Teleoperators Virtual Environ. 10, 266–281. doi:10.1162/105474601300343603
Schwind, V., Knierim, P., Haas, N., and Henze, N. (2019). “Using presence questionnaires in virtual reality,”. Proceedings of the 2019 CHI conference on human factors in computing systems (CHI '19). Editors S. Brewster, and G. Fitzpatrick (New York, NY, United States: Association for Computing Machinery)), 1–12
Seabrook, E., Kelly, R., Foley, F., Theiler, S., Thomas, N., Wadley, G., et al. (2020). Understanding how virtual reality can support mindfulness practice: mixed methods study. J. Med. Internet Res. 22, e16106. doi:10.2196/16106
Serrano, B., Baños, R. M., and Botella, C. (2016). Virtual reality and stimulation of touch and smell for inducing relaxation: a randomized controlled trial. Comput. Hum. Behav. 55, 1–8. doi:10.1016/J.CHB.2015.08.007
Shechter, A., Diaz, F., Moise, N., Anstey, D. E., Ye, S., Agarwal, S., et al. (2020). Psychological distress, coping behaviors, and preferences for support among New York healthcare workers during the COVID-19 pandemic. Gen. Hosp. Psychiatr. 66, 1–8. doi:10.1016/j.genhosppsych.2020.06.007
Shin, D. H. (2017). The role of affordance in the experience of virtual reality learning: technological and affective affordances in virtual reality. Telematics Inf. 34, 1826–1836. doi:10.1016/j.tele.2017.05.013
Slater, M., Usoh, M., and Steed, A. (1994). Depth of presence in virtual environments. Presence Teleoperators Virtual Environ. 3, 130–144. doi:10.1162/pres.1994.3.2.130
Spielberger, C. D., Gorsuch, R. L., Lushene, P. R., Vagg, P. R., and Jacobs, A. G. (1983). Manual for the state-trait anxiety inventory (form Y). Palo Alto: Consulting Psychologists Press
Spielberger, C. D. (2010). State‐Trait anxiety inventory. Hoboken, NJ, United States: John Wiley and Sons
Spoorthy, M. S., Pratapa, S. K., and Mahant, S. (2020). Mental health problems faced by healthcare workers due to the COVID-19 pandemic-A review. Asian J. Psychiatr. 51, 102119. doi:10.1016/j.ajp.2020.102119
Stamm, O., Dahms, R., and Müller-Werdan, U. (2020). Virtual reality in pain therapy: a requirements analysis for older adults with chronic back pain. J. NeuroEng. Rehabil. 17, 129. doi:10.1186/s12984-020-00753-8
Stetz, M. C., Kaloi-Chen, J. Y., Turner, D. D., Bouchard, S., Riva, G., and Wiederhold, B. K. (2011). The effectiveness of technology-enhanced relaxation techniques for military medical warriors. Mil. Med. 176, 1065–1070. doi:10.7205/milmed-d-10-00393
Tam, C. W., Pang, E. P., Lam, L. C., and Chiu, H. F. (2004). Severe acute respiratory syndrome (SARS) in Hong Kong in 2003: stress and psychological impact among frontline healthcare workers. Psychol. Med. 34, 1197–1204. doi:10.1017/S0033291704002247
Taylor, H. A., Sullivan, D., Mullen, C., and Johnson, C. M. (2011). Implementation of a user-centered framework in the development of a web-based health information database and call center. J. Biomed. Inf. 44, 897–908. doi:10.1016/j.jbi.2011.03.001
Taylor-Rodgers, E., and Batterham, P. J. (2014). Evaluation of an online psychoeducation intervention to promote mental health help seeking attitudes and intentions among young adults: randomised controlled trial. J. Affect. Disord. 168, 65–71. doi:10.1016/j.jad.2014.06.047
Tielman, M. L., Neerincx, M. A., Van Meggelen, M., Franken, I., and Brinkman, W. P. (2017). How should a virtual agent present psychoeducation? Influence of verbal and textual presentation on adherence. Technol. Health Care. 25, 1081–1096. doi:10.3233/THC-170899
Triberti, S., Chirico, A., La Rocca, G., and Riva, G. (2017). Developing emotional design: emotions as cognitive processes and their role in the design of interactive technologies. Front. Psychol. 8, 1773. doi:10.3389/fpsyg.2017.01773
Vagni, M., Maiorano, T., Giostra, V., and Pajardi, D. (2020). Hardiness, stress and secondary trauma in Italian healthcare and emergency workers during the COVID-19 pandemic. Sustainability. 12, 5592. doi:10.3390/su12145592
Vailland, G., Grzeskowiak, F., Devigne, L., Gaffary, Y., Fraudet, B., Leblong, E., et al. (2019). User-centered design of a multisensory power wheelchair simulator: towards training and rehabilitation applications. IEEE Int. Conf. Rehabil. Robot. 2019, 77–82. doi:10.1109/ICORR.2019.8779496
van Daele, T., Hermans, D., van Audenhove, C., and van den Bergh, O. (2012). Stress reduction through psychoeducation: a meta- analytic review. Health Educ. Behav. 39, 474–485. doi:10.1177/1090198111419202
Vandekerckhove, P., De Mul, Ma., Bramer, W. M., and De Bont, A. A. (2020). Generative participatory design methodology to develop electronic health interventions: systematic literature review. J. Med. Internet Res. 22, e13780. doi:10.2196/13780
Verschueren, S., Buffel, C., and Vander Stichele, G. (2019). Developing theory-driven, evidence-based serious games for health: framework based on research community insights. JMIR serious games. 7, e11565. doi:10.2196/11565
Villani, D., and Riva, G. (2008). Presence and Relaxation: a preliminary controlled study. PsychNol. J. 6, 7–25
Villani, D., Lucchetta, M., Preziosa, A., and Riva, G. M. (2009). The role of interactive media features on the affective response: a virtual reality study. E-Minds. Interational J. Hum. Comput. Interact. 1, 1–35
Vindegaard, N., and Eriksen Benros, M. (2020). COVID-19 pandemic and mental health consequences: systematic review of the current evidence. Brain Behav. Immun. 89, 531–542. doi:10.1016/j.bbi.2020.05.048
Vinstrup, J., Jakobsen, M. D., and Andersen, L. L. (2020). Perceived stress and low-back pain among healthcare workers: a multi-center prospective cohort study. Front. Public Heal. 8, 297. doi:10.3389/fpubh.2020.00297
Virvou, M., and Katsionis, G. (2008). On the usability and likeability of virtual reality games for education: the case of VR-Engage. Comput. Educ. 50, 154–178. doi:10.1016/j.compedu.2006.04.004
Watkins, L. E., Sprang, K. R., and Rothbaum, B. O. (2018). Treating PTSD: a review of evidence-based psychotherapy interventions. Front. Behav. Neurosci. 12, 258. doi:10.3389/fnbeh.2018.00258
Webster, R. (2016). Declarative knowledge acquisition in immersive virtual learning environments. Interact. Learn. Environ. 24, 1319–1333. doi:10.1080/10494820.2014.994533
Wechsler, T. F., Kümpers, F., and Mühlberger, A. (2019). Inferiority or even superiority of virtual reality exposure therapy in phobias?—a systematic review and quantitative meta-analysis on randomized controlled trials specifically comparing the efficacy of virtual reality exposure to gold standard in vivo exposure in agoraphobia, specific phobia, and social phobia. Front. Psychol. 10, 1758. doi:10.3389/FPSYG.2019.01758
Weerasekara, M., and Smedberg, A. (2019). “Design practices and implications in information and communication technology supported occupational stress management interventions.” 2019 international conference on advanced computer science and information systems, New York, NY, United States, 2019. IEEE, 285–294. doi:10.1109/ICACSIS47736.2019.8979813
Weiner, L., Berna, F., Nourry, N., Severac, F., Vidailhet, P., and Mengin, A. C. (2020). Efficacy of an online cognitive behavioral therapy program developed for healthcare workers during the COVID-19 pandemic: the REduction of STress (REST) study protocol for a randomized controlled trial. Trials. 21, 870. doi:10.1186/s13063-020-04772-7
Wilson, W., Raj, J. P., Rao, S., Ghiya, M., Nedungalaparambil, N. M., Mundra, H., et al. (2020). Prevalence and predictors of stress, anxiety, and depression among healthcare workers managing COVID-19 pandemic in India: a nationwide observational study. Indian J. Psychol. Med. 42, 353–358. doi:10.1177/0253717620933992
Woodson, W., Tillman, P., and Tillman, B. (1992). Human factors design handbook. Fremont, CA, United States: Woodson
World Medical Association (2013). World Medical Association declaration of Helsinki: ethical principles for medical research involving human subjects. J. Am. Med. Assoc. 310, 2191–2194. doi:10.1001/jama.2013.281053
Wu, K. K., Chan, S. K., and Ma, T. M. (2005). Posttraumatic stress, anxiety, and depression in survivors of severe acute respiratory syndrome (SARS). J. Trauma Stress. 18, 39–42. doi:10.1002/jts.20004
Zhang, M. W., and Ho, R. C. (2016). Tapping onto the potential of smartphone applications for psycho-education and early intervention in addictions. Front. Psychiatr. 7, 40. doi:10.3389/fpsyt.2016.00040
Zhao, H., He, X., Fan, G., Li, L., Huang, Q., Qiu, Q., et al. (2020). COVID-19 infection outbreak increases anxiety level of general public in China: involved mechanisms and influencing factors. J. Affect. Disord. 276, 446–452. doi:10.1016/j.jad.2020.07.085
Zheng, W. (2020). Mental health and a novel coronavirus (2019-nCoV) in China. J. Affect. Disord. 269, 201–202. doi:10.1016/j.jad.2020.03.041
Keywords: virtual reality, stress, anxiety, healthcare workers, COVID-19, psychoeducation, user-centered design
Citation: Pallavicini F, Orena E, di Santo S, Greci L, Caragnano C, Ranieri P, Vuolato C, Pepe A, Veronese G, Dakanalis A, Rossini A, Caltagirone C, Clerici M and Mantovani F (2021) MIND-VR: Design and Evaluation Protocol of a Virtual Reality Psychoeducational Experience on Stress and Anxiety for the Psychological Support of Healthcare Workers Involved in the COVID-19 Pandemic. Front. Virtual Real. 2:620225. doi: 10.3389/frvir.2021.620225
Received: 22 October 2020; Accepted: 04 January 2021;
Published: 02 February 2021.
Edited by:
Marientina Gotsis, University of Southern California, United StatesReviewed by:
Maryalice Jordan-Marsh, University of Southern California, United StatesAryabrata Basu, Emory University, United States
Laura Garcia, University of Southern California, United States
Copyright © 2021 Pallavicini, Orena, di Santo, Greci, Caragnano, Ranieri, Vuolato, Pepe, Veronese, Dakanalis, Rossini, Caltagirone, Clerici and Mantovani. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
*Correspondence: Federica Pallavicini, federica.pallavicini@unimib.it