ORIGINAL RESEARCH article
Sec. Virtual Reality and Human Behaviour
Volume 4 - 2023 | https://doi.org/10.3389/frvir.2023.1174701
Designing immersive stories with novice VR creators: a study of autobiographical VR storytelling during the COVID-19 pandemic
- 1Department of Film and Media, DAN School of Drama and Music, Queen’s University, Kingston, ON, Canada
- 2Media Production and Design, School of Journalism and Communication, Carleton University, Ottawa, ON, Canada
- 3School of Computing and Information Systems, University of Melbourne, Parkville, VIC, Australia
Virtual reality (VR) is increasingly being used as a tool for eliciting empathy and emotional identification in fact-based stories. However, it may not be clear whether VR stories authentically deliver the protagonists’ perspectives if the works are not created by or with the protagonists themselves. Therefore, it is crucial for the VR community to explore effective methods for democratizing VR storytelling, and to support novice VR designers in creating autobiographical stories. In this paper, we report findings from a collaborative design research project that aimed to create autobiographical stories with novice VR designers who lacked experience in VR storytelling. We collaborated with university students in Canada to design eight individual VR stories that expressed each student’s experiences of lockdown, during the early stages of the COVID-19 pandemic. We conducted interviews with the students to understand how VR contributed to conveying their individual experiences. Our findings demonstrate how immersive VR can be used as a meaningful tool for sharing autobiographical stories by delivering the character’s feelings, creating a sense of confinement and isolation, expressing inner worlds, and showing environmental details. Our discussion draws attention to the significance of careful camera positioning and movement in VR story design, the meaningful use of limited interaction and disorienting components, and the balance between spatial and temporal information in a three-dimensional environment. Our study highlights the potential of VR as an autobiographical storytelling tool and demonstrates how VR stories can be created through iterative collaboration between VR experts and novices.
Immersive virtual reality (VR) technologies that use head-mounted displays (HMDs) have become increasingly affordable and widely adopted, enabling storytellers and artists to create stories and narrative experiences using VR. The characteristic of the first-person view associated with VR enables the viewer to embody virtual characters in real or fictional situations, making them feel as though they are present within the story’s environment (Ryan, 2015; Bucher, 2017; Tricart, 2017). Based on these developments, many artists, journalists and documentary filmmakers have attempted to create VR works that provoke emotional engagement in fact-based stories. These efforts have mainly focused on the notion of empathy, most highlighted by ‘empathy machine’ — a term first used by Chris Milk at the 2015 TED conference (Milk, 2015). In this talk, Milk presented a 360° VR documentary, Clouds over Sidra (2015), that tells the story of a 13-year-old girl living in a refugee camp in Jordan. He argued that since the viewer can explore the world where Sidra actually lives, they can empathize with her more deeply. Another example is the immersive journalism model pioneered by Nonny de la Peña, which involves the recreation of news events in a virtual 3D environment to enable viewers to perceive the events from the first-person perspective, as if they were physically present (De la Peña et al., 2010).
Although one important facet of VR storytelling is the aforementioned ability to elicit empathy or emotional identification with protagonists in the story, it may not be clear whether the VR work authentically delivers the protagonists’ perspectives. This is especially true if the work is not created by or with the protagonists themselves, but rather by a third-party developer. Although VR devices allow viewers to look around and interact with the virtual environment freely (Vosmeer and Schouten, 2014), it is ultimately the VR designer who determines the position of physical or virtual cameras and the environment design, both of which have the potential to reflect the designer’s perspectives about the work more than the protagonists’ actual perceptions of their lived experience or situation. In other words, if the protagonist is too far removed from the creative process, their story may be mis-represented, even if unintentionally, by the creative choices of the designer.
Therefore, it is imperative that the VR community looks to effective methods to democratize VR storytelling. We argue this not only because everyone should be able to leverage VR for creative purposes, but also to empower people to authentically share their own perspectives and feelings with others. Given that there are autobiographical VR storytelling projects by professional VR creators, we see an opportunity to support novice storytellers leveraging VR to share autobiographical stories. Although VR technologies have become increasingly ubiquitous, creating a VR story still requires specialized skill sets, knowledge and technical abilities, resulting in a steep learning curve. In addition, designing a VR story requires a deep understanding of techniques related to 3D modeling, animation, environment design and cinematic consideration of the virtual environment. These issues heighten the complexity of creating VR stories, especially with creators who may still be building their technical skills and storytelling abilities.
In this context, this article aims to investigate the value of immersive VR as an autobiographical storytelling tool, and describes our experience of support novice VR designers in creating autobiographical VR stories. We conducted a collaborative design research project (Kvan, 2000) to create autobiographical stories with novice VR designers who lack experience and skills in creating VR stories. Participants in the study were Canadian university students who were enrolled in an online course about immersive storytelling taught by the first author. The students were novice VR creators in the sense that while they had some knowledge of basic concepts related to storytelling and 3D environment construction, they had not learnt how to design and implement a VR story due to the limitations of technical infrastructure and having their taken classes online. Based on iterative design meetings with these eight student participants, we collaboratively created eight immersive VR stories about each participant’s experiences of the COVID-19 pandemic. The VR stories are 360° video format made with 3D graphic environments using the Unity game engine. The stories aim to convey participants’ experiences of the 2021 lockdowns in Canada, and focus on issues such as social isolation, loneliness, and the feeling of disconnection from society.
Through the collaborative design research, we identified how immersive VR can be mobilized as a meaningful tool to support autobiographical storytelling by representing the protagonist’s feelings, and by communicating their experiences of confinement and disconnection through narrative design, environmental details and perspective. This research provides insight into the value of democratizing VR autobiographical storytelling in practice, which could allow protagonists to create their own VR stories in collaboration with VR experts and apply those insights to specific design implications for novice VR creators.
2 Related work
2.1 VR, empathy and subjectivity
The notion that VR can operate as an ‘empathy machine’, along with the proliferation of commercial VR headsets, has inspired governments, global non-profit organizations, and charities to explore the potential of VR as a tool to create empathetic experiences. Additionally, many filmmakers and artists have attempted to use VR technologies to create empathetic experiences, and the use of VR as an empathetic tool continues to gain traction among practitioners (Herson, 2016; Jones and Dawkins, 2018; McRoberts, 2018). Advocates of the ‘empathy machine’ idea tend to directly connect the sense of presence and embodiment with empathetic engagements and responses.
Presence is understood as the subjective experience of being in one environment, even if one is physically in another (Lee, 2004). In VR, presence has been defined as the psychological sense of ‘being there’ in the virtual world, which causes people to act as if the world is real even though it is known to be an illusion (Witmer and Singer, 1998). The question of VR embodiment is chiefly related to virtual body experiences, such as how and to what extent a user perceives a virtual body representation as their own physical being (Kilteni et al., 2012; Gall et al., 2021). Embodiment is closely related to the concept of presence in a virtual environment. While embodiment is concerned with the relationship between one’s self and one’s body, presence focuses on the relationship between one’s self and the environment (Kilteni et al., 2012). However, the depth of empathy is not directly connected to the presence or the sense of embodiment (Bahng, 2020).
The concept of VR as an empathy machine has been criticized and has provoked debates about what empathy really means in VR (Bollmer, 2017; Crawford-Holland, 2018; Nash, 2018). Bollmer radically criticized the general empathy industry, including the concept of the empathy machine, because empathy can be used to deny the differences between oneself and others, which can be ethically irresponsible in the name of sameness (Bollmer, 2017). Nash argued that VR carries the risk of people witnessing others’ pain with improper distance (Nash, 2018). Furthermore, although people still understand that VR is not reality, the illusion can be overwhelming for some viewers due to the intensity of presence. Therefore, people can easily emotionally identify with (or project themselves onto) virtual characters without critical awareness. Crawford-Holland critiques VR documentaries such as Clouds over Sidra for their use of VR’s enhanced capacity for perceptual realism to represent images of suffering (Crawford-Holland, 2018).
We agree that VR could potentially create the illusion of emotional identification, simultaneously objectifying or witnessing the virtual character’s suffering at a safe distance. Nonetheless, we do not completely discard VR’s potential to provoke empathetic experiences. VR’s methods of creating empathy might be varied and complex; depending on stories, aesthetic techniques, ethical points of view, and so on (Dziekan, 2018). We emphasise the importance of delivering the protagonist’s or main character’s feelings through their subjective engagement and decision-making in VR storytelling to genuinely share their feelings or generate empathy.
In nonfiction stories, autobiographical storytelling reveals the creator’s subjective engagement in the story or the event. The storytelling mode emphasizes the subjective representation of the real-world situations or events rather than delivering objective information by directly involving the creator’s personal stories or perspective (Lane, 2002; Dowmunt, 2013). While there are a few autobiographical VR storytelling projects by professional VR creators, we are not aware of any research about supporting novice VR creators or storytellers in creating their own stories using VR. There are several examples of artists who have created their own autobiographical stories using VR. Oscar Raby’s Assent (2013) can be considered the first autobiographical 3D VR documentary. This work is a story about the artist’s father, who witnessed executions during the Chilean dictatorship. Raby heard this story from his father while he was growing up, and it affected his life. Using VR, he reconstructed his and his father’s memories. Sojung Bahng’s Floating Walk (2017) is an autobiographical 360° VR documentary of her experiences as a young Korean woman living in Australia. She argued that 360° video could be an effective tool for self-confrontation and self-acceptance for creators because 360° VR could allow them to observe their lives without framing and filtering them from a virtual space perspective (Bahng, 2020).
2.2 Collaborative design and VR storytelling
Collaborative design (Kvan, 2000), co-design (Zamenopoulos and Alexiou, 2018) and participatory design (Hansen et al., 2019) are similar concepts that engage users in the design process. Participatory design is rooted in political origins, emphasizing the democratic engagement of participants and users in knowledge production and design outcomes (Hansen et al., 2019). Co-design embraces the broader practice of people conceptually developing and creating things together (Zamenopoulos and Alexiou, 2018). Collaborative design is not significantly different from co-design but does not necessarily require equal participation (Kvan, 2000). The aim is simply to enable the involved parties to contribute to the design process in a meaningful way, and for their contributions to have an impact on the end product.
In the context of VR, participatory design has been mobilized to create VR experiences for people with mild cognitive impairment (Eisapour et al., 2018) and dementia (Hodge et al., 2018). Participatory methods have also been mobilized to design VR experiences for therapeutic purposes, including for exposure therapy for fear of public speaking (Flobak et al., 2019), military sexual trauma (Mozgai et al., 2020) and re-entry training in a woman’s prison (Teng et al., 2019). Presently, we are not aware of any previous research focused specifically on collaborative or participatory design methods to support autobiographical VR storytelling for novices. Here, we consider novices to be any potential stakeholder with little to no VR design or development expertise. Although some scenario-creation processes have been studied in the aforementioned examples, the aim of creating stories for VR experiences is focused on helping certain groups of people or those in specific situations rather than supporting autobiographical storytelling.
Supporting novices in VR storytelling can be connected to the development and teaching of VR courses. Extensive research has been conducted on curriculum design for VR storytelling and pedagogical approaches to interactive digital narratives using VR (Appel et al., 2021; Fisher and Samuels, 2021; Kim et al., 2021; Mohamed and Sicklinger, 2022). For instance, Fisher and Samuels proposed undergraduate course curricula that integrate interactive narrative design, VR project management, and technical implementation, aiming to provide an introductory understanding of interactive narrative in VR (Fisher and Samuels, 2021). Additionally, there has been a study exploring the integration of virtual and augmented reality curricula to teach design students from diverse backgrounds, including product and graphic design (Mohamed and Sicklinger, 2022). However, while these studies primarily focus on teaching aspects, our main emphasis in this research lies in collaborating with students as novice VR creators. Specifically, our research adopted a collaborative design approach to support autobiographical storytelling for novice users.
We focused on participants’ own stories related to their experiences in the first year of the COVID-19 pandemic. Our participants were undergraduate students who had recently taken a course on immersive storytelling and were assisted by a VR expert who was also the professor for the aforementioned course. Research indicates that when teachers engage in collaboration with their students, it empowers them to improve their self-determination, problem-solving, decision-making, goal-setting, self-knowledge and self-regulation (Villa et al., 2010). Furthermore, the COVID-19 pandemic had significant impact on mental health issues in North American university students, including increased levels of stress, anxiety, and depression. Research has shown that these issues have resulted in negative outcomes including difficulty concentrating, disruptions to sleeping patterns, decreased social interaction, and concerns about academic performance (Son et al., 2020). Our collaborative process helped our participants side-step a number of technological barriers to effectively expressing their experiences of early lockdowns during the pandemic. We chose to focus on autobiographical VR storytelling due to its ability to immerse the audience in the position of the storyteller. Our motivation was to help our participants see that they were not alone in their experiences and to provide them with the opportunity to deliver their feelings.
3 VR story design process: Method
3.1 Participants and setting
Participants in this study had been previously enrolled in and completed an online course about immersive storytelling taught by the first author in 2021 at Carleton University in Canada. Most lacked any experience with 3D modelling, animation, interaction design and VR before taking the course. In the course, students were asked to create a 3D environment containing some basic animations in Unity about their pandemic experience. They learned how to import free 3D models into the Unity game engine, set up the environment, and add audio files such as music, background sound, and voice-over narration. The students were able to view the environment using a desktop computer, but it was not yet a full VR story and was not viewable in 360°. Due to pandemic restrictions, both the course and the study had to be conducted online. As such, the students were not able to gain physical access to the VR lab and VR prototyping software. Their technical skills and knowledge by the end of the course in VR design were very basic and at the novice level.
Following the conclusion of the course, students were invited to participate in a research phase that formed the basis for our study. A total of eight students from the course decided to join in the study. Our aim was to collaboratively translate the 3D environment into a full VR story so that it authentically conveyed the students’ experiences of pandemic lockdown, and to explore how this could be achieved when the lockdown restrictions were in place. The collaborative design meetings were conducted using the 3D immersive environment they created for the course as a starting point for the study. The 1-h meetings were held four times via Zoom for each participant for a total of 8 weeks. Three participants were located in China, and five were located in Canada. Carleton University’s ethics committee granted ethical approval for this study, and participants signed consent forms before participating in the study. Each participant received a $30 Amazon gift card for their involvement as well as a cardboard VR device. This device was used by participants to view and provide feedback on their collaborative VR short films, allowing them to experience them via the cardboard HMD. In the following sections, we refer to the student participants as P1–P8 to describe the resultant films.
3.2 Design meetings
We conducted one-on-one design meetings via Zoom with the aim of translating participants’ 3D environment into a working 360° video VR story, while also helping them refine or change their projects’ audiovisual aesthetics. Due to the social distancing restrictions related to the COVID-19 pandemic at the time of the study, it was not possible to conduct the meetings in person. Thus, we designed a longitudinal, collaborative study utilizing videoconferencing and cardboard HMDs to support distance methods.
We held four design meetings with each participant. All of these meetings were video recorded for later analysis. Prior to the first meeting, participants sent the Unity project folders that they had created during the course to the lead researcher, and the researcher connected the VR headset (Oculus Quest 2) to their project. This allowed us to begin the first design meeting by demonstrating how their work could be experienced through a VR headset in real time. More specifically, the researcher shared her screen via Zoom and showed a first-person view of the 3D environment created by the participants through a virtual camera connected to the Oculus Quest 2. Based on that demonstration, the remainder of the meeting time was used to discuss how to refine and develop participants’ 3D environments into a 360° video.
In the second and third meeting, we collected the participants’ ideas, comments, and feedback on their projects and the collaborative design process. Between and after meetings, we focused on the technical implementation based on the feedback. Specifically, we refined audiovisual aesthetics and added design details to transfer their projects to the 360° video format. Finally, we exported their work as a 360° video using VR panorama software from the Unity asset store. We used a 360° video instead of other real-time VR formats because 360° video is most accessible to students. As noted above, we sent cardboard head-mounted displays (HMD) to our participants as part of their remuneration, allowing them to easily experience the final outcome of their projects with an HMD.
In the final meeting, each student viewed their project using the cardboard HMD. Following this, we conducted semi-structured interviews with the participants and focused on their responses to their experience viewing their individual revised VR pieces using the cardboard HMDs. For the purpose of the study, we only let each participant view their own VR piece as the aim of this research was to explore the storytellers’ perspectives in creating autobiographical VR stories.
Our research contribution is based on artistic practice-based research, where phenomenological and subjective learning contributes to knowledge through original creative practice (Candy, 2006; Morley, 2008). We used qualitative analysis method to understand the participants’ subjective and personal thoughts and feelings in creating autobiographical VR storytelling about the COVID-19 pandemic and how immersive VR 360° can be mobilized as a meaningful device for telling their autobiographical stories.
We transcribed the audio from video recordings of the design meetings and then analyzed the data using an inductive thematic analysis following Braun and Clarke’s six phase approach (Braun and Clarke, 2006). Having transcribed, read and re-read the data (Phase 1), we conducted open coding (Phase 2) in which the first author labelled printed transcripts to create codes inductively and developed these codes into themes (Phase 3). For example, the “get to experience what the other person is feeling” and “gives people a chance to have a similar feeling to those people who experienced” codes were merged into the theme of “delivering the character’s feelings”. Phases 4 through 6 involved refining the themes through thematic mapping, consolidation of codes and discussion among the authors.
3.4 VR story design
3.4.1 3D environment reconstruction
During the VR story creation process, there were several issues that we had to negotiate with the participants. Here we provided some examples to show we worked collaboratively with the participants to realise their vision. First, some participants created 3D environments without considering the 360° perspectives, in which the scene can be seen by the viewer from all directions in VR. Two participants created outdoor environments that only worked for a 2D-based traditional screen, so there was a lot of empty space outside a specific frame. Based on the discussions with the participant, we made the surrounding environment by adding more 3D objects, such as buildings and a mountain. Furthermore, one participant only used 2D textures to design land and grass, which did not look good with the perspectives from multiple directions. Based on the agreement with the participant, we replaced the flat 2D texture with 3D objects such as 3D terrain and grass. Figure 1 shows before and after images of the 3D environment.
FIGURE 1. The left image shows the flat 2D texture that represents land and grass; the right image shows the 3D terrain and grass modifications based on the design meeting. Created with Unity Editor®. Unity is a trademark or registered trademark of Unity Technologies.
3.4.2 Camera position and movement
Another consideration for the participants was how the virtual camera would be positioned in the environment. We showed participants how their 3D environment could be seen through a virtual camera connected to the VR headset (Oculus Quest 2) via screen sharing. Based on the demonstration, we discussed the role of a virtual camera and how the position and movement of the camera affect the VR story. For example, the lead researcher and the participants experimented with camera positions and movements that could effectively deliver participants’ stories and feelings. Five participants decided to use slow and consistent camera movements within an eye-level position to represent their movement in the scene. Two participants used jump-cut scenes between different views rather than moving cameras so as not to cause motion sickness and to show sudden changes, confusion and disorientation intentionally. Three participants decided to position cameras extremely low or high to exaggerate the surreal or fantastic components of the environment or to make the viewer feel overwhelmed or surprised.
3.4.3 Audio design
Lastly, we had to work collaboratively with participants to create the audio for the VR story. The most significant part of the audio design was the participants’ voice narration. Although participants added their voice-over narration in their initial 3D environment, the structure and timing of the narration needed to be changed due to the camera position and movement for the 360° format. Based on the discussion, most participants re-recorded the audio narration considering the virtual camera and sent it to us. We added the new audio narration clips on the virtual cameras. Moreover, participants did not consider the features of 3D spatial audio, which enables viewers to hear three-dimensional audio in VR. The lead researcher and participants experimented with how spatial audio can be used impactfully based on the virtual camera position and movement. After this, participants sent us more audio effects that can add environmental information, and we added the sounds that came from specific directions in the 3D environment, such as a knocking sound behind a wall, the sound of a toilet flushing in the bathroom, and television sounds coming from the position of a television.
4.1 VR projects
Based on the collaborative design meetings, we created eight VR stories that portray each participant’s autobiographical experiences during the first year of the COVID-19 pandemic. All eight participants expressed their pandemic experiences in creatively different ways and provided the narration using their own voices. They represented how their daily lives during the pandemic became repetitive, lonely, inescapable, confined, or surreal. Audiovisual aesthetics in each of the projects are diverse and depend on participants’ expression, including realistic scenes, low-poly cartoony aesthetics, and fantastic or surreal environment construction. Each work’s running time is 2–5 min in length. Figures 2, 3 show still images from each VR story.
FIGURE 2. Still images from a collection of 3D VR autobiographical stories about the COVID-19 pandemic from university undergraduate students in Canada. Clockwise from top-left: Greenhouse, Sinkhole, Loop, and Pandemic Dream. Created with Unity Editor®.
FIGURE 3. Clockwise from top-left: The City of Dawn, Confusion & Life, COVID Confinement, and Solitary Confinement FT COVID-19 Pandemic. Created with Unity Editor®.
Greenhouse was made by P1. She created a transparent room that looks like a glass box. To express her yearning to connect to the world and the loneliness arising from her disconnection during the pandemic poetically, she used a fantasy-like aesthetic to show the beauty of nature and the desire to escape from reality. The story begins inside the glass room, and she escapes from the room into nature.
Sinkhole is based on P2’s inner experience of the pandemic. He tested positive for COVID-19 and experienced several symptoms, including tiredness, anxiety, disorientation and confusion. He expressed his status as a sick person in a motel room metaphorically, and he used surreal and abstract aesthetics to artistically express his inner view, which cannot be conveyed realistically.
Loop was created by P3, a Chinese international student. Similar to P1, she described her repetitive and isolated life during the pandemic in Canada. To represent the room in which she lives, she took photos of objects in her room and used them as textures for 3D models. She attempted to replicate her room exactly to allow the viewer to feel as if they are actually in her room (see Figure 4). She represented the change in seasons through changes in lighting, the color of trees and the presence of snow outside the window.
FIGURE 4. A screenshot of Loop (left) and a photo of P3’s real room (right); the image was used with permission. Created with Unity Editor®.
Pandemic Dream is based on P4’s bizarre dreams during the pandemic. He expressed his surreal dream experience related to a grocery store artistically. The work lacks a concrete story structure; instead, it visually presents various surreal moments and perspectives such as lucid dream-like experiences. The work uses multiple cut scenes to show sudden changes in perceptions and unexpected feelings.
The City of Dawn by P5 represents the Wuhan lockdown. As a Chinese international student, she did not experience the Wuhan lockdown directly. Instead, she built an environment based on images of Wuhan that she saw in news and documentaries. Although this was not her direct experience, she considered her work autobiographical because she believes the Wuhan lockdown might have directly influenced her life and many other people’s lives. Thus, the work is a reconstruction of the Wuhan lockdown based on her perspectives and feelings.
Confusion & Life was made by P6, a Chinese international student, to describe his repetitive and lonely routine in his studio apartment in Canada. He used realistic aesthetics to simulate his real daily life in his apartment. As the story begins, he wakes up and goes to the bathroom. He then plays games until late night and eats unhealthy food, which deepens his unhealthy routine and causes him to lose his sense of time.
COVID Confinement represents P7’s pandemic experience as if he is in a prison cell. As time passes, the door disappears, the ceiling descends, and the light becomes red. He uses a low-poly cartoony style aesthetic to convey the sense that he is in a game that is not escapable. Compared to other students, he did not add many details to the environment. Instead, he used a minimal aesthetic to focus on the sense of emptiness and prison-like confinement.
Solitary Confinement FT COVID-19 Pandemic was created by P8; he created many different rooms to convey his life experience during the pandemic. Between rooms, there are outdoor areas that allow him to be liberated from interior spaces, but he goes back into a different room due to another lockdown. The room size gradually becomes smaller to represent his sense of confinement and feelings of entrapment.
4.2.1 Delivering the character’s feelings
We found that participants perceived VR as beneficial for expressing their feelings and that immersion in the virtual environment helped to convey feelings they had experienced during the pandemic. During the interviews and testing sessions, we observed that participants often reacted with surprise when they experienced their projects in 360° VR format using an HMD. All eight participants said that they actually felt as though they were in their rooms or the environments they created and re-experienced the feelings they had felt during the pandemic lock-downs. Six participants noted that VR could allow viewers to experience someone else’s feelings in a way that cannot be accomplished with other more common formats, such as 2D video. This finding makes sense given that VR brings a sense of presence through its ability to put the audience in the first-person perspective of the storyteller. However, in the context of our study where we wanted to explore collaborative methods to support autobiographical storytelling for novice users, this finding is important because our participants felt as though their feelings were accurately represented in the resultant VR shorts through our collaborative process:
“I think in a 2D video, the audience is an outsider; they cannot experience feelings similar to the subject, who experienced something in person. But a VR video gives people a chance to experience feelings similar to those of people who experienced the lockdown.” (P1)
“I think a lot more stories can be shown that touch a more emotional side if they’re using VR compared to a regular screen.” (P6)
Four participants felt that the immersive quality of VR is especially powerful to deliver personal and autobiographical stories surrounding the pandemic. This further demonstrates the value of the project as a personal or subjective archive that contains people’s diverse feelings and experiences, which may be experienced differently through other media formats.
“It being a pandemic, there has not been a single life on this planet that has not been changed, so everyone has their own very personal story.” (P4)
“I think VR is the most effective medium for telling pandemic stories because you actually get to experience what the other person is feeling.” (P7)
We also found that camera position and movement were important factors to consider to express and deliver participants’ feelings. P1 said she felt like he had actually been lifted when the camera panned up, and P6 mentioned he was lying on his bed when the camera was positioned on a bed. P5 noted that she felt as though she was one of the cars when the camera moved quickly along the road she designed.
P8 described his feelings about the camera movement:
“It showed how there was no time to stop and think, it was just moving on from each stage of the pandemic. It was at a pace where you could still look around, but it was always in the back of my mind that I was moving forward and I had no control over it. I thought it progressed the story nicely compared to having to move yourself.” (P8)
4.2.2 Creating a sense of confinement and isolation
We found that VR can be used as a storytelling tool to represent the sense of confinement and isolation that participants experienced during COVID-19. Participants felt that immersion in the virtual environment helped to convey the feeling of being isolated from others during lockdown, and that immersion contributed to the sense of loneliness and disconnection from the real world. This speaks to the utility of VR as a storytelling device, in the sense that the intrinsic qualities of VR in combination with the story design may lend themselves to accurately conveying these kinds of feelings.
Six students used VR to explore their feelings of being trapped in their rooms. These participants all mentioned their intentions were successfully conveyed by the ability to design a 3D graphic representation of their personal space and strategically position a camera in that space to let the viewer freely look around, while also being confined in the same way the participant was. This was expressed by two participants:
“I felt pretty isolated. The glass walls especially helped communicate this. Despite feeling isolated, I still felt like I could see other people living life, but you did not get to see the life around you. You saw things, but not really the people.” (P1)
“The main theme of what I’m trying to express is that I was imprisoned in my room. You can see the seasons and the light changing, you can see sun rise and the sun set, but you cannot go outside.” (P3)
These findings speak to the way that non-navigable 360° VR was well aligned with participants’ intention to convey their autobiographical stories. In non-navigable VR where locomotion is not possible, viewers cannot move in the virtual environment and cannot control anything except for looking around. They can only experience movement in the same way our participants did. This may enhance the sense of confinement and loss of control that our student participants experienced, because viewers cannot escape from the room while observing things happening inside or outside of it, and they cannot reach outside or other people. They must simply stay there and watch what is happening.
Two students designed their VR experiences so that the physical space in the rooms became smaller over time to deliver the ongoing and gradual sense of isolation and suffocation. P7 made the ceiling slowly descend as the VR experience played and P8 created different rooms, with the camera moving to a smaller room each time. In VR, the change in the room’s size can be physically experienced by the viewer, so the shrinking of the space and sense of suffocation can be conveyed effectively.
“It gave me a good opportunity to showcase how I felt mentally and emotionally trapped. Even the simple things in my project like each room getting smaller were meant to show how everything came crashing down to one point.” (P8)
4.2.3 Expressing inner worlds
We found that the students perceived VR as a way to effectively express their inner worlds and perspectives such as feelings of a dreamlike state, subconscious desires or even fantasy elements.
Five students mentioned that VR was particularly helpful to express their personal pandemic experiences in more creative or surreal ways. Two participants intentionally used surreal aesthetics to express their inner worlds and perspectives (see Figure 5). In the following quote, P4 noted that VR is an effective tool to express dreamlike experiences due to the surreal visual components of VR and the freedom of creative expression granted by the game engine.
“I think VR makes it easier to connect to a story or put yourself in a character’s dreamlike experience. VR helped add to the nightmarish or dreamlike element. You cannot put someone really low to the ground with a normal camera—or rather, you can do it, but it’s not the same. It’s an exaggerated way to show my worldview as well as my dreams.” (P4)
P2 mentioned using VR to convey his deep feelings of confusion and anxiety in a surreal way, as symptoms of COVID-19 cannot be described in realistic ways.
“VR itself is disorienting; the attributes contribute to overall feeling that I was going through during the pandemic. A lot of confusion, a sense of disorientation and unescapable feelings made me imagine myself as a sick person living in a motel room. It is hard to describe my experience with words or realistic images, so a more surreal way of expression works well.” (P2)
In addition to this, our participants noted that the use of VR allowed them to express their inner worlds beyond realistic representations to accurately portray their feelings of stress during lockdown.
“The VR experience was like that of a dream, and a dream is better for describing the pressure on your heart, your inner self. Your heart’s pressure is hidden, but in your dreams it becomes real. In daily life, it does not show as well. So, I think the VR version of my life showed the actual mental pressure I felt.” (P6)
P6 built a realistic environment, not an abstract or imaginary one. Nevertheless, he articulated that the VR experience is dreamlike. He stated that it gave him an opportunity to express his inner self, which is rarely found in daily life. P1 mentioned she often played a VR game to escape from reality and enter a fantasy world, and that experience inspired her to build a fantastical scene. In short, VR allowed the students to create their inner perspectives or worlds by using fantastical or dreamlike elements as artistic devices.
4.2.4 Showing environmental details
We found that adding environmental detail in the virtual environment helped our participants convey a personal story. All eight participants were surprised by how well VR highlighted many details of the virtual environments they had created, and how much these details added to a strong sense of presence. P5 felt that being able to look around the 360° environment and focus on details added to the weight of the story for conveying her experience:
“I can look around, I can look at objects behind me. It provides me more detailed visual information than normal video. This is overwhelming, but powerful.” (P5)
P8 was also surprised by the impact of visual details added during the design meeting. He mentioned that a few changes for aesthetic consistency, such as texture changes or small visual effects like fog, could change the atmosphere or mood of the scene. He noted that he would focus more on visual details to express his feelings about the pandemic if he could create a new project using this new knowledge.
“Surprisingly, it hit me harder than I thought it would. When you put it on and you can actually look around yourself and notice little details, it makes it way cooler.” (P8)
P3 mentioned that the most important part of autobiographical VR storytelling is to create a detailed environment. P2 and P6 even created new environments by adding more objects, visual effects and textures after the first design meeting. They noted that the design of environmental details could impact the effectiveness of the VR scenes for representing their personal lives, so they decided to add more 3D components to their projects for more effective storytelling. P6 said that adding objects such as books, plates, food and cups could reveal how he lived and felt during the pandemic. P2 added more detailed textures to the walls, floor, blanket and ceiling, mentioning how the change of textures added more surreality. He said the changes created more disorientation because it seems as though we can feel the details of textures, even though they are virtual. This intensified his intention to use surreal components to convey his own pandemic experience.
P7 said that the main element of VR storytelling is environmental storytelling, in which the environment itself is part of the story itself and not just a backdrop. Compared to other students, he used minimal aesthetics without adding a lot of detail. P7 said that simple visual elements let the viewer focus on certain important objects that he wanted to emphasize.
In short, the advantages of showing environmental details in VR storytelling do not simply mean providing a lot of visual information is best. We argue that environmental details and objects can be used as main storytelling components because the viewer has the opportunity to look around and explore the environments to understand someone else’s personal life during the pandemic.
5 Discussion and implications
Our study demonstrates how VR can be used as an effective storytelling tool to create and share autobiographical stories related to isolation, loneliness and disconnection. In this case, we focused on the experiences of students during COVID-19 lockdowns and explored the ways that VR enabled them to convey their feelings and experiences. In addition, we developed a collaborative design process that successfully supported novice creators in delivering their own feelings through the medium of VR. While we did not assess whether the VR pieces generated empathy from viewers, our evaluation suggested that the participants did view the pieces as capable of delivering the creator’s feelings from the storyteller’s perspectives. There is always a possibility that a collaborative process can introduce variations in how the story is presented in the final product. There is also a risk that narrative elements that are important to the storyteller can be incorrectly re-imagined or even omitted from the final project. In the present study, our method aimed to alleviate this issue by including participants in all major points of the iterative development cycle, promoting the representation of novice VR storytellers’ own feelings and lived stories.
Findings from the study drew attention to the significance of camera positioning for conveying participants’ experiences. As storytellers, the question of how to position and move the virtual camera is linked to the identities and activities of the subject in the stories themselves. For example, if the camera is placed on a bed, the viewer may experience it like they are present on the bed. If the camera is positioned in front of the desk, the viewer could feel like they are sitting in front of the desk. When the camera moves forward, the viewer may perceive the sensation of walking forward. Extreme camera angles, such as a very low angle, could also make people feel like being small entities within the virtual world. If the camera position is very high, they may think that they can observe events unfolding from a distance.
Within traditional film and screen fields, camera angles and movements are recognised as crucial elements for creating the atmosphere of works and conveying the emotions of its characters (Zettl, 2016). However, in the realm of VR, the camera position and movements have been considered less significant because most research focuses on the sense of presence in the virtual environment (Vosmeer and Schouten, 2014). Although there is the absence of a frame in the virtual environment, the camera position and movement can be experienced more physically in VR, so the virtual camera should be carefully adjusted to deliver a VR story. Particularly in the context of autobiographical storytelling, the camera position can be directly linked to the subject’s perspectives and feelings in the stories. Thus, when novice storytellers design an autobiographical story, they need to consider how they can effectively use the camera position and movement to express their perceptions and feelings.
Our findings also reinforce the idea that VR can effectively create a sense of confinement and disconnection. This finding shows that VR can be used to deliver people’s personal stories specifically related to feelings of helplessness, frustration and isolation. There are many VR experiences about solitary confinement (Plager et al., 2020), death (Bahng et al., 2020a), narcolepsy (Bahng et al., 2020b) and bipolar disorder (Kim, 2018). All these examples are about conditions or situations that we cannot easily control or wherein we lose autonomy or agency. Many people use VR to create a fantasy and escape from the real world, freely exploring other worlds; however, it is important to be aware that current VR technologies cannot deliver a complete, fully immersive experience due to the limitations of interactivity and immersive technology. Although VR technologies are becoming increasingly ubiquitous, and affordable technologies like cardboard HMDs make it possible for more people to experience passive VR, commercial VR hardware is still rife with limitations. For example, we cannot freely interact and navigate in the virtual environment - there are always hardware and software constraints that limit our interactions in VR. Due to this, ironically, VR can be a powerful tool to explore stories that focus on loss of movement or agency. Therefore, it would be very important for a novice storyteller to consider how these limitations impact storytelling, and decide how to leverage them effectively as a technique for autobiographical storytelling.
We also found that VR was used meaningfully to express participants’ inner worlds and perspectives. The findings may have been different if we had used 360° cameras rather than computer-generated imagery (CGI). Using a 360° camera would capture the participant’s environment “as-is”, creating a more direct replication with some similar qualities to those of our participants’ stories, e.g., non-navigability. However, using CGI to create the virtual environment can give VR storytellers more artistic freedom to create their story environments, adjusting the virtual camera position and movement in dramatic ways. Many VR creators try to use VR to represent reality accurately and mimic real world situations as if people are in reality; however, VR can be used meaningfully to deliver subconscious, fantastic and dreamy worlds that cannot be easily expressed in the real world.
As viewers in VR experiences, we often feel as if we are in the virtual environment, but we cannot see our own bodies in VR, which can make the experience surreal, disorienting, dreamlike and imaginary. This can sometimes be an unintentional side-effect of VR. However, sometimes this can be leveraged by VR storytellers as a powerful tool for conveying feelings associated with surrealism or a dreamlike state. For novice autobiographical VR storytellers, it would be useful to consider how to use the surreal and disorienting components of VR to express the inner worlds or unconscious perspectives that are rarely expressed in the real world.
Lastly, our findings articulated how VR was a useful tool for autobiographical storytelling by enabling storytellers to include environmental details in three-dimensions. Through the collaborative design process we used, our participants became increasingly aware of the importance of the environment and its role in storytelling. If we imagine our real lives, all the details of objects, textures and materials in our surroundings bring various sensations, thoughts and imagination. A virtual environment has the potential to evoke the same response, so the novice VR storyteller should carefully introduce objects, textures and materials according to whether they can express the feelings, sensations or stories around them.
Furthermore, novice storytellers should consider the balance between spatial and temporal information. It has been observed that an excessive amount of simultaneous spatial and temporal information in VR can result in increased mental effort for the viewer (Gödde et al., 2018). Given the VR shows a lot of environmental detail, viewers may miss the significant narrative-related information or other aspects of the stories. Therefore, adding more detail to the VR environment is not necessarily good; intentional and strategic delivery of visual information using environmental details will be the crucial element to effective storytelling.
Overall, this research suggests that VR has the potential to be an autobiographical storytelling tool for novice storytellers by providing opportunities to deliver their feelings, create a sense of confinement, express inner worlds and show environmental details. This study highlights the importance of VR as an autobiographical tool for sharing perspectives and feelings and the useful methods for people to create their own stories using VR. We have also shown how the VR design process can be democratized to support the involvement of novice creators who cannot implement the VR application. In our study, the COVID-19 pandemic prevented the storytellers from gaining the necessary skills to implement their stories in VR. A key lesson for future research is that a carefully designed collaborative process can be used to convey the experiences of novice VR creators. We anticipate that similar situations may arise in which a person wishes to convey a personal experience through the medium of VR but does not have the skills to implement the technology. Our work emphasises that this can be achieved through an iterative and collaborative process in which the VR designer works with the storyteller to poetically convey their experience.
6 Limitations and future work
While our research process worked well given the circumstances of the pandemic, this study does have several limitations. First, collaborating on a VR story over distance was challenging in terms of enabling participants to view the VR story throughout the process. The students did not have the opportunity to check their work through virtual headsets until we sent the cardboard HMD to where they were located. The impossibility of checking the development process using HMDs may cause difficulties and challenges for novices in creating VR autobiographical stories. Nevertheless, these situations make it more important to collaborate with VR experts when designing their own stories using VR. Without support or assistance from VR experts, it would be harder to create VR storytelling during the pandemic because many novice storytellers do not have VR equipment and other required technology.
Second, our project had an initial aim of exploring how students learned the process of VR creation through collaboration. However, due to the constraints of online collaboration, their roles for VR design became limited because they could not use VR equipment to translate their work to a VR format. Therefore, the collaborative design process was focused on a VR expert translating their ideas.
Third, we only examined autobiographical storytelling related to COVID-19. However, it is important to deal with different topics and media technologies to understand how we can tell our own stories in various ways. Future research will include a variety of themes in autobiographical storytelling using various forms of VR technology, such as room-scale VR and real-time interaction.
This research investigated autobiographical VR storytelling based on a collaborative design process with novice VR creators. The study also investigated the participants’ responses to those stories in terms of how well they felt VR conveyed their lockdown experience. For practical examination, we created eight autobiographical VR pieces about COVID-19 by collaborating with university students in Canada. We conducted one-to-one collaborative design meetings with eight students to explore how VR can be a useful and meaningful tool for creating stories about the COVID-19 pandemic. The discussions and interviews from the meetings revealed four elements—delivering the character’s feelings, creating a sense of confinement and isolation, showing environmental details, and expressing inner worlds—that were used by novices as methods of autobiographical VR storytelling. Based on our findings, we illustrated the potential autobiographical storytelling design of VR for creating and sharing the subject’s feelings and perspectives in meaningful, effective ways.
Data availability statement
The datasets presented in this study can be found in online repositories. The names of the repository/repositories and accession number(s) can be found in the article/Supplementary Material.
The studies involving human participants were reviewed and approved by Carleton University’s Research Ethics Board (CUREB-A). The patients/participants provided their written informed consent to participate in this study.
SB and VM contributed to the conception and design of the study. SB mainly conducted the practice-based research under the supervision of VM and RK advised on the data analysis. SB wrote the first draft of the manuscript, and VM and RK developed and revised all sections. All authors contributed to the article and approved the submitted version.
The research has been mainly supported and sponsored by Carleton University, collaborating with Queen's University and the University of Melbourne; FPA Postdoctoral Fellowship Award.
The VR films described in this paper began as a class project for ITEC 3100: Immersive Storytelling–a required course in the third year of the Media Production and Design program at Carleton University. This project was directed by SB and produced by VM at Include Lab at Carleton University.
Conflict of interest
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.
The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/frvir.2023.1174701/full#supplementary-material
Appel, L., Peisachovich, E., and Sinclair, D. (2021). Cvrriculum program: Benefits and challenges of embedding virtual reality as an educational medium in undergraduate curricula. Int. J. Innov. Educ. Res. 9, 219–236. doi:10.31686/ijier.vol9.iss3.2992
Bahng, S., Kelly, R. M., and McCormack, J. (2020a). “Reflexive VR storytelling design beyond immersion: Facilitating self-reflection on death and loneliness,” in Proceedings of the 2020 CHI conference on human factors in computing systems, Germany, April 2020 (IEEE).1–13.
Bahng, S., Rajcic, N., McCormack, J., and Lee, S. (2020b). “Sleeping eyes: Experiencing narcolepsy through the duality of virtual embodiment,” in Proceedings of the Fourteenth International Conference on Tangible, Embedded, and Embodied Interaction, New York, February 9 - 12, 2020 (Association for Computing Machinery), 639–645.
De la Peña, N., Weil, P., Llobera, J., Giannopoulos, E., Pomés, A., Spanlang, B., et al. (2010). Immersive journalism: Immersive virtual reality for the first-person experience of news. Presence Teleoperators virtual Environ. 19, 291–301. doi:10.1162/pres_a_00005
Eisapour, M., Cao, S., Domenicucci, L., and Boger, J. (2018). “Participatory design of a virtual reality exercise for people with mild cognitive impairment,” in Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems, New York, April 2018 (IEEE).1–9.
Fisher, J. A., and Samuels, J. T. (2021). “A proposed curriculum for an introductory course on interactive digital narratives in virtual reality,” in Interactive Storytelling: 14th International Conference on Interactive Digital Storytelling, ICIDS 2021, Tallinn, Estonia, December 7–10, 2021 (Springer), 462–477.14.
Flobak, E., Wake, J. D., Vindenes, J., Kahlon, S., Nordgreen, T., and Guribye, F. (2019). “Participatory design of vr scenarios for exposure therapy,” in Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, Glasgow Scotland Uk, 02 May 2019 (IEEE).1–12.
Gall, D., Roth, D., Stauffert, J.-P., Zarges, J., and Latoschik, M. E. (2021). Embodiment in virtual reality intensifies emotional responses to virtual stimuli. Front. Psychol. 12, 674179. doi:10.3389/fpsyg.2021.674179
Gödde, M., Gabler, F., Siegmund, D., and Braun, A. (2018). “Cinematic narration in VR–rethinking film conventions for 360 degrees,” in International Conference on Virtual, Augmented and Mixed Reality, 15-20 July, Las Vegas, FL, USA (Springer), 184–201.
Hansen, N. B., Dindler, C., Halskov, K., Iversen, O. S., Bossen, C., Basballe, D. A., et al. (2019). “How participatory design works: Mechanisms and effects,” in Proceedings of the 31st Australian Conference on Human-Computer-Interaction, Fremantle WA Australia, December 2 - 5, 2019 (HFESA).30–41
Herson, B. (2016). Empathy engines: How virtual reality films may (or may not) revolutionize educationClouds over Sidra produced by socrates kakoulides, gabo arora, and Chris Milk 360° virtual reality film distributed on the within channel and Milk VR USA 8 minutes, 35 seconds On the Brink of famine directed by marcelle hopkins and produced by frontline 360° virtual reality film. Comp. Educ. Rev. 60 (4), 853–862. doi:10.1086/688582
Hodge, J., Balaam, M., Hastings, S., and Morrissey, K. (2018). “Exploring the design of tailored virtual reality experiences for people with dementia,” in Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, Montreal QC Canada, April 21 - 26, 2018 (Association for Computing Machinery).1–13
Kim, E. S. (2018). “Virtual reality dances about adolescent bipolar disorder: Accommodating patient and audience concerns,” in ACM SIGCHI Conference on Designing Interactive Systems (DIS 2018): Design and Diversity, Hong Kong China, June 9 - 13, 2018 (IEEE).
Mozgai, S., Hartholt, A., Leeds, A., and Rizzo, A. (2020). “Iterative participatory design for vret domain transfer: From combat exposure to military sexual trauma,” in Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems, Germany, April 2020 (IEEE).1–8.
Plager, T., Zhu, Y., and Blackmon, D. A. (2020). Creating a vr experience of solitary confinement. In 2020 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops 22-26 March 2020, New York, (VRW (IEEE, 692–693.
Son, C., Hegde, S., Smith, A., Wang, X., and Sasangohar, F. (2020). Effects of Covid-19 on college students’ mental health in the United States: Interview survey study. J. Med. internet Res. 22, e21279. doi:10.2196/21279
Teng, M. Q., Hodge, J., and Gordon, E. (2019). “Participatory design of a virtual reality-based reentry training with a women’s prison,” in Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems, Glasgow Scotland Uk, May 4 - 9, 2019 (IEEE).1–8.
Keywords: autobiographical VR storytelling, virtual reality, immersive journalism, VR nonfiction, collaborative design
Citation: Bahng S, McArthur V and Kelly RM (2023) Designing immersive stories with novice VR creators: a study of autobiographical VR storytelling during the COVID-19 pandemic. Front. Virtual Real. 4:1174701. doi: 10.3389/frvir.2023.1174701
Received: 27 February 2023; Accepted: 30 May 2023;
Published: 14 July 2023.
Edited by:Joshua A. Fisher, Ball State University, United States
Reviewed by:Lei Zhang, Kennesaw State University, United States
James Calvert, Torrens University Australia, Australia
David Antognoli, Columbia College Chicago, United States
Copyright © 2023 Bahng, McArthur and Kelly. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
*Correspondence: Sojung Bahng, email@example.com