HYPOTHESIS AND THEORY article
From Readership to Usership: Communicating Heritage Digitally Through Presence, Embodiment and Aesthetic Experience
- 1Foundation for Research and Technology (FORTH), Institute of Computer Science (ICS), Heraklion, Greece
- 2Milieux Institute for Arts, Culture and Technology, Concordia University, Montreal, QC, Canada
- 3School of Literature, Faculty of Philosophy, Aristotle University of Thessaloniki, Thessaloniki, Greece
- 4ORamaVR SA, Heraklion, Greece
- 5Department of Computer Science, School of Sciences and Engineering, University of Crete, Heraklion, Greece
The primary mission of cultural institutions, including heritage sites and museums, is to perform and perpetuate Cultural Heritage (CH) by ideally transforming audiences into stewards of that heritage. In recent years, these institutions have increasingly turned to Mixed Reality (MR) technologies to expand and democratize public access to Cultural Heritage—a trend that is called upon to accelerate with COVID-19—because these technologies provide opportunities for more remote outreach, and moreover, can make partial remains or ruins more relatable to the public. But as emerging evaluations indicate, existing MR intangible and tangible Digital Cultural Heritage (DCH) applications are largely proving inadequate to engaging audiences beyond an initial fascination with the immersive 3D visualization of heritage sites and artefacts owing in part to misguided storytelling or non-compelling narratives. They fail to effectively communicate the significance of Cultural Heritage to audiences and impress upon them its value in a lasting way due to their overreliance on an education-entertainment-touristic consumption paradigm. Building on the recent case made for Literature-based MR Presence, this article examines how the literary tradition of travel narratives can be recruited to enhance presence and embodiment, and further elicit aesthetic experiences in Digital Cultural Heritage applications by drawing on recent findings from the fields of Extended Reality (XR), cognitive literary science and new museology. The projected effects of this innovative approach are not limited to an increase in audience engagement on account of a greater sense of presence and embodiment. This approach is also expected to prompt a different kind of public involvement characterized by a personal valuation of the heritage owing to aesthetic experience. As the paper ultimately discusses, this response is more compatible both with MR applications’ default mode of usership, and with newly emerging conceptions of a user-centered museum (e.g., the Museum 3.0), thereby providing a narrative roadmap for future Virtual Museum (VM) applications better suited to the primary mission of transmitting and perpetuating Cultural Heritage.
With the exponential uptake of consumer Virtual Reality (VR) technology since 2014, cultural institutions have increasingly turned to Mixed Reality (MR) applications to expand and democratize public access to Cultural Heritage (CH). However, recent findings have shown that existing MR intangible and tangible digital cultural heritage applications by and large fail to adequately engage audiences beyond an initial fascination with the immersive 3D visualization of cultural sites and artefacts. This far-from-optimal public engagement is due in part to misguided storytelling or non-compelling narratives (Vassiliadi et al., 2018), which hinder the effective transmission of the heritage by leaving audiences cold. In response, designers have increasingly introduced gamification into Digital Cultural Heritage (DCH) applications to offset the public’s lack of sustained captivation by way of interaction. Indeed, gamified Mixed Reality applications often recreate a virtual environment featuring interactive tasks to orchestrate an immersive experience (Zikas et al., 2016). However, reliance on gamification as a compensatory device does not remedy the content-based, root causes of this unsatisfactory public engagement vis-à-vis Digital Cultural Heritage.
As a solution, the concept of Literature-based Mixed Reality Presence was recently introduced to address the “content-based” shortcomings of modern MR intangible and tangible digital heritage storytelling applications (Vassiliadi et al., 2018). The notion was first established on the basis of literary myth’s potential to enhance presence in Digital Cultural Heritage immersive applications owing to its multi-temporal and multi-cultural features. As argued, these features are coextensive with Mixed Reality applications’ multimodal functions, and have a beneficial impact on the user. In effect, they amplify presence not only through a more engaging narrative than the straightforward transmission of informational content, but also through the phenomenon of literary transportation.
Building on this approach, the following paper is the first analysis to show that the use of literature to enhance storytelling in Mixed Reality intangible and tangible digital heritage applications presents an alternative to the current education-entertainment-touristic consumption paradigm in prevailing Digital Cultural Heritage applications. This paper offers a rare challenge to the widespread and enduring assumption that didactic content packaged in an entertaining form—as captured by the notion of “edutainment”—and disseminated via touristic channels is the most effective means of transmitting and perpetuating Cultural Heritage to diverse local and international audiences. Certainly, informative content is often indispensable when situating heritage sites and artefacts in their historical context. Meanwhile, entertainment has proven its worth as an effective means of capturing and sustaining public attention. In addition, touristic channels provide an established network that can be utilized to vastly expand outreach. But if the ultimate goal of digital heritage storytelling applications, in line with cultural institutions’ main mission, is to not only provide access to Cultural Heritage, but to turn patrons into stewards of that Cultural Heritage (McKenzie 2015), then this prevailing paradigm is insufficient to the task.
In order to demonstrate the departure and benefits of using literature to perform and perpetuate Cultural Heritage via new Mixed Reality media, this paper begins by exploring how literary transportation can heighten the user’s sense of presence in Virtual Environments (VE). It goes on to examine the importance of embodiment in Virtual Environments and how literature can increase this sense of embodiment through certain perceptual phenomena, including mental imagery. Mental imagery can effectively engage the user’s wider sensorium beyond Mixed Reality technologies’ limited spectrum (as primarily visual and auditory media) through simulation. The paper then turns to the popular literary genre of travel narratives, rooted in the 17th–18th century tradition of the Grand Tour. It examines two excerpts drawn from Virginia Wolfe’s oeuvre, which describe encounters with the Ancient Greek ruins of Acropolis using motor and haptic imagery. These excerpts are used to illustrate the sensorial impact of mental imagery, and how it might be recruited to enhance presence and user engagement vis-à-vis Cultural Heritage sites within Mixed Reality applications. The paper ultimately argues that by inciting aesthetic experience around Cultural Heritage by means of a literary narrative, Literary-based MR Presence re-orients public engagement from an amusing, pedagogical reception towards the formulation of a value judgment. The claim is that this approach can more effectively transform patrons into stewards of that heritage through the production of incremental, personal value. As discussed in the final section, this approach is better suited to fulfilling cultural institutions’ primary mission all the while implementing new museology’s usership principles, thus providing a narrative roadmap for future Virtual Museum applications effectively tailored to Cultural Heritage.
Presence and Literary Transportation
The initial case for Literary-based MR Presence was first established based on three main considerations: 1) storytelling’s effectiveness in comparison to didactic content (i.e., the attention-grabbing ability of captivating narratives); 2) the multi-temporal and multi-cultural aspects of literary myths (i.e., fiction’s compatibility with Mixed Reality’s multimodal affordances); and 3) the impact of literary transportation (Vassiliadi et al., 2018). Of the three features originally ascribed to the concept, literary transportation is of particular importance for “sense of presence” in virtual environments. In discussion of Extended Reality, “sense of presence” is generally described as a subjective sensation of “being there.” As Schuemie et al. observe in their survey of presence in virtual environments: “presence as discussed in literature related to immersive VR can most often be characterized by the concept of presence as transportation: people are usually considered “present” in an immersive VR when they report a sensation of being in the virtual world (“you are there”)” (2001). The same immersive principles also apply in Augmented Reality (AR) environments where True AR elements blend the real with the virtual world (Geronikolakis et al., 2020).
By definition, literary transportation is “a convergent process, where all mental systems and capacities become focused on events occurring in the narrative” (Green and Brock, 2000), thereby providing a means to greatly enhance a user’s engagement with a given simulation. It should be noted that this phenomenon is also coherent with alternative understandings of presence in Virtual Environments, including Slater’s Psi, whereby presence corresponds to the illusion that the environment exhibited in VR is actually taking place (Slater and Sanchez-Vives 2016). This is owing to the fact that literary transportation can simultaneously offset the sensorimotor constraints of the VR system while increasing the credibility of the scenario. Moreover, literary transportation is associated with intense aesthetic involvement attributted to the activation of the brain’s default mode network (Starr 2013, 59–63): a pattern of cognitive activity linked, among other, to internal mentation (Andrews-Hanna 2011), self-reflection and eudemonia (personal growth and well-being) (Stark et al., 2018), as well as to interactive virtual humans in cultural heritage (Arnold et al., 2008; Ioannides et al., 2017). As follows, the impact of literary transportation has far-reaching consequences for the user beyond heightened involvement and immersion in Mixed Reality environments. It can render the experience both more appealing and more credible. Literary transportation has also been shown to increase empathy skills over time across fictional genres in contrast to non-fiction (Bal and Veltkamp 2013), thus offering promising connections with emerging empathy training in VR (Schutte and Stilinović 2017; Bertrand et al., 2018).
In addition, beyond providing captivating narratives and triggering transportation, the use of literary texts in Mixed Reality intangible and tangible digital heritage storytelling applications can also contribute to amplifying presence through embodiment. Specifically, literature has been shown to elicit embodied responses in readers through several means, including multisensory imagery, as examined below. Before delving further into these processes, however, it is necessary to first consider embodiment’s role in immersive digital applications so as to further situate and assess the impact of the proposed approach in relation to the existing scholarship.
Presence and Embodiment in Extended Reality
If “presence is typically seen in academic research as the aim of virtual reality environments” (Pujol and Champion 2012), embodiment constitutes a vital yet still under investigated condition for its successful attainment. As enactivism, embodied cognition and situated cognition theories and empirical findings have shown over the past few decades—counter to prior cognitivist approaches and mind-body dualism—embodiment is an integral part of cognitive processing. With respect to VR, the chief implication stemming from this new understanding is that it bridges the relation of the self to its surrounding environment (i.e., external involvement and immersion) with the relation of the self to the body (i.e., internal sensations such as self-location and body ownership), which are both fundamental to achieving a satisfactory level of presence in Virtual Environments (Papaefthymiou et al., 2018).
Biocca remarks, as early as 1997, that embodiment is expected to have a direct impact on different types of presence (Biocca 1997). Citing this research, Schubert et al., 1999 soon after propose the notion of “embodied presence,” which they attribute to meshed patterns of action primarily oriented towards navigation and interaction with objects in Virtual Environments (1999). In a similar vein, ecological views of presence greatly indebted to Gibson have explored the bearing of situated affordances, perception-action coupling and “ready-to-hand” tools for VR (Schuemie et al., 2001). However, broadly speaking, the field has produced only limited research into the potential impact of embodied cognition in the context of Virtual Environments.
In a rare paper leveraging embodied cognition towards advancing “a new set of relationships between dimensions of embodiment and forms of presence,” Costa et al. theorize that increased embodiment is correlated with higher levels of psycho-physiological responses to Virtual Environments. It can therefore serve the purpose of improving the user’s sense of presence (2013). Citing structural coupling’s model of mutual affect between organism and environment, the authors affirm that: Practically speaking, we can say a user is embodied in a virtual reality environment (VRE) if changes in the VRE affect the user (emotionally, cognitively, and/or physiologically), and the user can affect the VRE (move objects, interact with others, etc.) (Costa et al., 2013).
Notwithstanding, as their paper reveals, research into the interconnection between embodiment and presence in virtual environments remains primarily focused on: sensory input, possible action and interaction in the Virtual Environment, as well as on the user’s relationship to their avatar. For instance, how the aforementioned innate senses of self-location and of body-ownership might be induced vis-à-vis an artificial body (Kilteni et al., 2012). When it comes to sensory input, as Costa et al. outline, empirical studies have concentrated on the impact of visual scale and dimensionality, the auditory effects of higher quality ambient and action driven sounds, and the haptic feedback offered by different controllers on the user (2013). In parallel to sensory engagement, the authors propose the notion of “afforded embodiment” to capture the specific sense arising from avatar manipulation and customization. They rely on research demonstrating an increased sense of embodiment as a result of: 1) greater motor control (i.e., the degree to which the user can effectively control the avatar’s movements); and 2) psycho-social afforded embodiment (i.e., “the degree to which the user can modify and/or manipulate their avatar to reflect or express their identity”) (Costa et al., 2013). From a technical perspective, glGA (Papagiannakis et al., 2014) proposes a lightweight, open source, shader-based framework used in various graphics projects, among others, VR exhibitions and rapid reconstruction of Cultural Heritage monuments in Mixed Reality environments. A similar approach presents an Augmented Reality system which relies purely on passive techniques to solve the real-time registration problems of combining a VR component-based simulation framework with computer vision techniques to generate AR experiences (Vacchetti et al., 2004).
In this way, sensory input, action and interaction and afforded embodiment are vital to amplifying presence in Mixed Reality applications. Notwithstanding, the discussions cited above fail to address how narrative content can also prime users and induce a heightened sense of embodiment and therefore presence in Virtual Environments (though it should be noted here that Schubert et al., 1999 explicitly acknowledge dramatic content’s measurable impact on presence in the context of screen-based 3D games (1999). Examining narrative content’s function is particularly decisive in the case of virtual heritage projects, which necessitate added intellectual and emotional engagement to sustain “cultural presence” (Pujol and Champion, 2012, 98). As the following section will examine, new research in the fields of neuroaesthetics, cognitive cultural studies and cognitive semantics expressly focused on reader responses to literary texts is beginning to provide new leads in this direction.
Literature, Embodiment and Mental Imagery
By focusing on the first half of Costa et al.’s account according to which “a user is embodied in a virtual reality environment (VRE) if changes in the VRE affect the user (emotionally, cognitively, and/or physiologically)” (2013), the case for Literary-based MR Presence becomes plain. To wit, the integration of literary excerpts into Digital Cultural Heritage applications’ narrative content can naturally heighten emotional response, and reconcile a certain mediated external perception of heritage with a likely internal reaction. Hereof, as Starr relevantly points out in her reading of John Keats’s “Ode on a Grecian Urn”: “while the urn can “express. . . more sweetly than. . . rhyme,” poetry can evoke the visible surface of the urn as well as the internal response of a viewer—a sense of puzzlement at the urn’s mysteries” (2013, 12). Beyond reflexively bridging outer engagement and mentation, the incorporation of literary texts into Digital Cultural Heritage applications can also play a seminal role in amplifying the user’s sense of embodiment, and therefore presence in Virtual Environments, through simulation (Oatley 2011; Gibbs, 2017). Indeed, emerging research from the empirical turn in literary studies increasingly points towards embodied responses to poetry and fiction as a result of various phenomena, including multisensory imagery.
Multisensory imagery is of particular significance to Literary-based MR Presence given poetry and fiction’s distinctly vivid and sensuous portrayals. Beyond a certain baseline of literature-induced embodied simulation (whereby readers mirror protagonists’ actions and emotions), these textual depictions have the ability to trigger powerful mental images: i.e., “the subjective experience of sensation without corresponding sensory input” (Starr, 2010, 276). As Kosslyn et al., 2006 observe, mental imagery occurs “when a representation of the type created during the initial phases of perception is present but the stimulus is not actually being perceived; such representations preserve the perceptible properties of the stimulus and ultimately give rise to the subjective experience of perception” (2006, 4). In other words, when people experience mental imagery, the same areas of the brain that are involved in the perception of actual sensory input become active, and moreover operate in commensurate organizational patterns. For instance, auditory imagery is organized temporally, whereas visual and haptic images typically mirror spatial detail and relation (Starr, 2013, 75). This phenomenon is observed across sensory modes. In fact, evidence points to many different types of mental imagery, including object-based imagery (e.g., shapes and colours), spatial imagery (e.g., of locations), auditory imagery and motor imagery: i.e., kinaesthetic and/or proprioceptive images (Moulton and Kosslyn 2009). In the context of Virtual Environments, this suggests that it is not only direct sensory stimulation (e.g., geographically encoded sound effects—see previous section) that can be mobilized towards increased embodiment. In addition, specific forms of narrative content, such as auditory images (in keeping with the example), can also contribute to amplifying this sense by activating many of the same areas of the brain in similar organizational patterns (Starr, 2010).
Among the different types of sensory imagery, motor imagery has been put forward as the most paradigmatic case (Starr, 2013, 81), in accordance with scientific evidence indicating a robust perception/action coupling when it comes to kinaesthetic and proprioceptive images. Indeed, neuroimaging studies have found significant overlap in the neural circuitry involved in action execution and in the observation of another person’s motions (e.g., seeing another person smile activates the same facial muscles at a sub-threshold level in the viewer), as well as in imagining both one’s own actions and another person’s actions (Decety and Jackson, 2004). Reproducing images of somatic and motoric components is a means through which individuals recognize other people’s emotions. Furthermore, such reverse mapping has been linked to the architecture of empathy (Decety and Jackson, 2004). Accordingly, motor imagery has been found to play a central role in theory-of-mind. Based on motion, that is, both the perception of biological motion and static images of motion, individuals infer other people’s mental states: e.g., attribute intention—a crucial survival skill that has consistently been observed as early as preverbal infancy (Blackemore and Decety, 2001). When it comes to Mixed Reality applications, the suggestion here is that by engaging the sensorium not directly stimulated by these technologies (which remain primarily visual and auditory media) by way of simulation, and by amplifying actual stimuli, mental imagery can augment these technologies’ impact by means of content-based triggers.
It is no wonder then, as Starr remarks, that: “Multisensory imagery, especially the multisensory imagery of motion, is centrally important to a variety of aesthetic pleasures in part because it gives us access not to the “real” complexity of experience but to certain powerfully connected aspects of the ways our minds internally represent experiences and objects.” (Starr, 2013, 91) As suggested, sensory imagery is not only linked to embodiment and social interaction (as seen in the case of motor imagery). It is also involved, by means of aesthetic experience, in the production of associative knowledge, triggering redefinitions and revaluations of what we feel and what we know (Starr, 2013, 92).
Virginia Woolf and the Acropolis
A characteristic example of literary multisensory imagery directly related to Cultural Heritage can be found in Virginia Woolf’s recurrent references to the Acropolis, both in her private journals as well as in her works of fiction. Woolf is among a number of prominent writers and thinkers who have traveled to and subsequently written about the renowned Ancient Greek monument throughout history, including Ernest Renan, Sigmund Freud, Henry Miller and Jacques Derrida. These remarkable accounts belong to the modern literary tradition of travel narratives going back to the 17th–18th century tradition of the Grand Tour, which has produced an extensive body of artistic and literary works around Cultural Heritage. Traveling, and especially the ‘Eurocentric’ travel tradition, in that sense, is part of an educational project always focused on a solid historical framework. Yet even as it theoretically follows the traces of cultural origins, ‘voyage literature’ often functions as a mirror of the self. As Porter (1991) points out in Haunted Journeys, if traveling, through the centuries, becomes a protracted act of understanding the world, travel writing is a process of understanding oneself:
“the most interesting writers of nonfictional travel books have managed to combine explorations in the world with self-explanation. They submitted themselves to the challenge of travel and, in the process, managed if not always, to make themselves over, then at least to know themselves differently.” (p. 5)
As follows, although travel writing is based on realistic conventions (i.e. explanatory narratives, explicit notes and general information), journeys to archaeological remains, such as the Acropolis, the Colosseum or Pompeii, are often related to a bodily, shared and ineffable experience of empirical reality received through the senses.
Accordingly, among the numerous allusions to the Acropolis that appear throughout Virginia Woolf’s writings, the two passages included below were chosen on account of the manifest intensity of their multisensory imagery. Nonetheless, they were also deliberately procured from divergent sources by the same author so as to demonstrate how different literary perspectives can be recruited towards inducing embodiment and aesthetic experience vis-à-vis Cultural Heritage. The first excerpt is drawn from the author’s early private journals, where she recounts her personal experience of visiting the Acropolis at sunset. The second is a short passage lifted from her famous satirical novel Orlando: A Biography (Woolf, 1928) that alludes to the main character’s transgendered metamorphosis on her perilous journey East.
As is immediately evident from these two excerpts, they are both rich in visual, haptic and above all, motor imagery, owing to Woolf’s perception of the Acropolis as a mighty soundless monument, in stark contrast to the noisy hustle and bustle of the Athenian streets below it. As the author writes in her early novel Jacob’s Room: “the Parthenon is really astonishing in its silent composure” (Woolf, 1922, 105). This enduring impression of the monument gives rise, throughout Woolf’s oeuvre, to vivid object-based, spatial, tactile and kinaesthetic images, as the following passage demonstrates.
“The Temple glows red; the whole west pediment seems kindled, as if for the first time, in the sunset opposite: its rays light and heat, while the other temples burn with a white radiance. No place seems more lusty and alive than this platform of ancient dead stone. The fat Maidens who bear the weight of the Erechtheum on their heads, stand smiling tranquil ease, for their burden is just meet for their strength. They glory in it; one foot just advances, their hands, one conceives, loosely curled at their sides. And the warm blue sky flows into all the crevices of the marble; yet they detach themselves, and spring in to the air, with crisp edges, unblunted, and still virile and young.
But it is the Parthenon that over comes you; it is so large, and so strong, and so triumphant. You feel warmed through and through, as though you walked by some genial hearth. But perhaps the most lovely picture in it—at least it is the most detachable—is that which you receive when you stand where the great Statue used to stand. She looked straight through the long doorway, made by the curved lines of the columns, and saw a long slice of Attic mountain and sky and plain, and a shinning strip of the sea. It is like a panel, let in to the Parthenon to complete its beauty. It is soft, and soon grows dark, though the water still gleams; then you see that the white columns are ashy pale, and the warmth of the Parthenon ebbs from her.
A bell rings down below, and once more the Acropolis is left quite alone. (Woolf, 1991, 321–22)
As is palpable straightaway, this description of the Acropolis at sunset generates a highly embodied experience through the multisensory combination of dramatic visual and haptic imagery of burning and of warmth: from the flaming red sunset, to the day’s accumulated heat emanating from the dead, ashy pale stones, to the radiating brilliance of the mighty white-hot temple, tempered to the comforting smolder of a genial hearth. In addition, this rich multisensory depiction is further animated—making the site come alive—by vivid motor imagery of weight carried (fat and marble) with a smile, of “one foot forward,” of loosely curled hands and a sprightly leap; but also, of standing, curving, sawing and slicing until a single auditory image, the toll of a lone bell at nightfall, prevails upon, dispatches and ultimately dissipates the whole sensuous vision-occurrence.
By the same token, even though sourced from a completely different genre, the second passage from Woolf’s oeuvre included below makes similar use of multisensory imagery. In this case, to capture Orlando’s distant encounter with the Ancient monument. Only this time, the multisensory imagery is not employed to induce the portentous experience of a walk through, of a visit to the site of Acropolis. Instead, it is recruited to evoke the rapture of a spiritual and transformative journey.
“There were mountains; there were valleys; there were streams. She climbed the mountains; roamed the valleys; sat on the banks of streams. She likened the hills to ramparts, to the breasts of doves, and the flanks of kine. She compared the flowers to enamel and the turf to Turkey rugs worn thin. The trees were withered hags, and sheep were grey boulders. Everything, in fact, was something else. She found the tarn on the mountain-top and almost threw herself in to seek the wisdom she thought lay hid there; and when, from the mountain-top, she beheld far off, across the Sea of Marmara the plains of Greece, and made out (her eyes were admirable) the Acropolis with a white streak or two which must, she thought, be the Parthenon, her soul expanded with her eyeballs, and she prayed she might share the majesty of the hills, know the serenity of the plains, etc. etc., as all such believers do.” (Woolf 1928/2007, 468)
Here again, Woolf enlists potent haptic and motor imagery to sustain a highly sensate voyage: a metaphorical passage, at once physical and spiritual, that requires one to climb, roam, sit and seek, at the risk of throwing one’s self in, so as to behold at last, and feel one’s soul and one’s eyeballs swell, in prayer for sharing, knowing and believing. In this passage, Orlando’s inner and outer metamorphosis is mirrored by the fact that everything is something else, and that something can be grasped to the touch: solid rugged ramparts, soft belly feathers and silky cow hide; jagged petal tooth enamel and threadbare Turkey rugs; craggy branches and cold hefty boulders. Until, in the end, the rapid-fire succession of haptic and motor imagery gives way to a clear visual image: Acropolis as a white streak or two in the far-off distance—a revelation born out of a transformative experience.
Overall, these two excerpts provide a sense of the wealth of cultural heritage content that can be gleaned from the travel narrative genre. The suggestion is that these literary accounts can be recruited to enhance the narrative content of Mixed Reality, Digital Cultural Heritage applications and increase user engagement through literary transportation and simulation, which additionally augments the range of sensory perceptions within the XR experience. Owing to these phenomena, literary accounts substantially differ from the informative, edutainment narratives that are typically used to communicate the significance of cultural heritage in such applications. In contrast to this didactic content, not much can be learned from Woolf’s excerpts in terms of the Acropolis’s long history and socio-cultural bearing. Nor are these two literary passages overtly entertaining—though there is certainly pleasure involved. No gaming can be immediately fathomed from their narratives. Instead, these excerpts suggest a different type of experience of Cultural Heritage: one that is not inherently grounded in touristic consumption—notwithstanding the first passage’s plain recollection of a foreigner’s sightseeing visit.
As argued in the previous sections, the rich multisensory imagery that these literary excerpts contain provides a more embodied response to the virtual monument, even when the latter is merely imagined through the simple act of reading the words on the page. Moreover, aside from increasing user engagement through a more compelling and evocative narrative, such excerpts open up more creative avenues for the design of novel Digital Cultural Heritage applications, which currently privilege realistic renderings of cultural artefacts and sites corresponding to their didactic, informational content. The integration within Mixed Reality’s multimodal environment of literary travel narratives, which oscillate between real and virtual places and objects through the use of symbolism, metaphor and allegory, introduces the possibility of more creative visual renderings that might accordingly alternate between realistic and more artistic depictions.
Additionally, when it comes to the notion of presence in virtual environments, it should be noted that beyond transportation and sensory simulation, this oscillation between different registers (i.e., literal, symbolic, metaphorical, allegorical) is conceivably be better suited to users’ fluctuating attention between immersion and cognizance of the experience’s artificial nature. Indeed, presence in XR experiences depends in part on the allocation of attentional resources. If literary transportation tends to heighten and secure focus, artistic narratives’ alternation between real and imaginary spaces might render the technologies’ inherent conflict between “real” and virtual experience less jarring, thus reducing disengagement. In this sense, the excerpt from Woolf’s Orlando is particularly indicative. It relies on material metaphors to evoke a fantastical landscape where “everything is something else.” This feature could be recruited, for instance, to allude to the artificiality of a Virtual Environment all the while reinforcing its credibility and captivation, thus reconciling the users’ different levels of awareness and alternating focus. Moreover, in accordance with the previous allusion to empathy training, the impact of this second literary passage is not only limited to added value around the cultural heritage monument. It could also be used, for instance, to simultaneously induce empathy for a female transgender voice without the typical reliance on an avatar.
To be clear, the suggestion here is not that informative content should be abandoned altogether. Travel literature excerpts could be integrated in a polyvocal narrative combining more poetic and didactic content, as previously suggested. Moreover, this storytelling approach does not preclude the use of interactive features. In fact, the symbolic, metaphorical and allegorical content of literary passages can open up novel opportunities for more imaginative and wonderous interactions within Virtual Environments that are not just amusing, but also grounded in a cogent worldview. Nevertheless, the main argument remains that interactive features prove less necessary for user engagement when narrative shortcomings are adequately addressed. What is more, it is important to recognize that the inclusion of literary passages in Mixed Reality applications is inevitably bound to circumscribe interactivity, to a certain extent, owing to the well-established narrative paradox between predetermined structure and emergent interaction (Riggs, 2019). In terms of the advocated approach, user agency is not based on a capacity to direct the narrative within the virtual environment, but instead on the ability to formulate a judgment of a particular kind. Far from the education-entertainment-touristic consumption paradigm at the core of a majority of Digital Cultural Heritage applications, which typically utilize gamification elements to offer a unique storytelling experience combining entertainment and education (Ioannides et al., 2017), the use of multisensory imagery in these two excerpts is rather designed to elicit an individual (re)valuation of Cultural Heritage through aesthetic experience.
Narrative content’s ability to amplify presence through embodiment akin to actual sensory input presents significant advantages for Digital Cultural Heritage applications both when it comes to the Virtual Reality Environment and the user. As demonstrated, such narrative content can sustain accrued presence, thus fulfilling the Virtual Reality Environment’s central aim (see Presence and Embodiment in Extended Reality). At the same time, it can provide effective empathy and social training for the user by recruiting literature’s proven faculty to increase social abilities and alter selfhood, even in hard-to-reach individuals (Oatley et al., 2012). But beyond these notable effects, the use of Literary-based MR Presence can also provide a distinct benefit for actual heritage sites, monuments and artefacts. Indeed, while enhanced presence, embodiment and empathy may be crucial to maintaining attention and engagement in virtual heritage applications, these factors do not ensure adequate reception of the cultural heritage itself. As Pujol and Champion observe in their study on “cultural presence”: “ability to navigate and complete tasks in a virtual environment is no guarantee that relevant cultural learning has taken place” (2012, 97), attesting to the limited impact of interaction and gamification when it comes to digital cultural heritage.
Correspondingly, a recent critical survey of the state of the art on Virtual Museums has found that existing Digital Cultural Heritage applications fail to live up to their radical potential owing, in large part, to their reductive pedagogical transmission of didactic content: viz. descriptive and explanatory commentary entirely disconnected from visitor expectations, personal agendas and emotional involvement (Perry et al., 2017). In response, Perry et al. advocate instead for an emotional, participatory, interactive and social engagement of the public through the use of a collaborative and affective user-centered design methodology aligned with the latest approaches in museum studies. Of particular relevance to the discussion at hand, the authors note that such user-centered interfaces are uniquely valuable when it comes to antiquities because: “many heritage sites have few remnants that are either visible or relatable to the broad public. As such, they may not have enough resonance to engage visitors on their own or through standard interpretational means.” In other words, existing Virtual Museums’ explanatory information and empty reconstructions are unable to bring heritage sites and artefacts back to life for lack of emotionally evocative content characteristic of storytelling media such as film and literature (Perry et al., 2017).
When considered in the light of Perry et al.’s approach, Literary-based MR Presence shares many of the same objectives and operative modalities as their advocated model of “emotive storytelling,” correspondingly described as an interactive, story-based (as opposed to object-based) approach involving a dramatic and affective narrative. Both clearly aim to surpass an outdated model of “educative leisure,” fostering instead more curiosity, attentiveness, empathy and personal transformation. Notwithstanding, Literature-based MR Presence differs on a fundamental level from Perry et al.’s affect-based approach on account of its distinct mode of reception, which is grounded in aesthetic experience owing to its reliance on an art form as part of its narrative approach, thus highlighting the aesthetic qualities of the heritage.
For in effect, aesthetic experience implies, first and foremost, a process of valuation: i.e., a value judgment. It “works to produce new value in what we see and what we feel” (Starr, 2013, 66). To be clear, value is here explicitly understood as the outcome of an evaluative judgment, and not an a priori standard, rule, criterion, norm, goal, or ideal that one might use to formulate such a judgment (Sánchez-Fernández and Iniesta-Bonillo, 2007, 429). It consists of a surplus created by the user, as opposed to describing either a quality intrinsic to a cultural object, or the product of an institutional process of legitimation. Certainly, emotions are implicitly involved in the cognitive processes of directing attention, passing judgement (e.g., allowing individuals to make rapid decisions as to whether something is beneficial or harmful) and aesthetic experience. However, evidence indicates that the neural activity of everyday emotions differs from the one associated with aesthetic, emotional involvement (Starr, 2013, 42). Indeed, aesthetic judgment and transportation’s activation of the brain’s default mode network suggests a more long-lasting and profound valuation. As follows, Literature-based MR Presence can provide a direct benefit for cultural sites, monuments and artefacts, beyond a narrower focus on interactive user design. It essentially opens the possibility of recruiting the user’s attention and personal transformation towards the creation of incremental value for Cultural Heritage.
Building on the notion of Literary-based MR Presence, this paper has examined the advantages of drawing on the literary arts to enrich the narrative content of MR intangible and tangible Digital Cultural Heritage applications, which include a heightened sense of presence as a result of literary transportation and increased embodiment, as well as a different type of public response rooted in personal value judgments. When considered from the broader perspective of museology, these findings have important ramifications for the design of Virtual Museums. Indeed, if the impact of these new digital applications is not entirely aligned with cultural institutions’ primary mission, then these institutions are unlikely to fully adopt MR technologies in the long run despite their obvious benefits for access and outreach, especially given the milieu’s enduring, if unfounded, fear that broad uptake of Virtual Museums might threaten in-person visits vital to cultural institutions’ survival.
McKenzie (2015) eloquently articulates the fundamental mission and social contribution of cultural institutions today. She writes:
“What kind of emergency services do museums actually perform? Museums enable diverse communities to discover, perform and perpetuate heritage over the long term. (…) Museums can offer the best conditions for “affective germination” — stirring meanings and emotional responses to things, places and ideas — to remind people why heritage matters (…) so that they can “become stewards of heritage”.”
Naturally, it is important to recognize that this notion of turning viewers into stewards of Cultural Heritage is a fairly recent idea in museology, which has only just begun to supersede the enduring traditional, authoritative museum model that continues to underpin many Digital Cultural Heritage applications. This contemporary idea is in fact linked to a specific understanding of usership arising from new thinking at the junction of museology and social practice.
In recent years, cutting edge curatorial and educational practices have increasingly turned towards visitor-centered approaches, characterized by the unprecedented attribution of equal importance to the collection as to the public (Samis and Michaelson, 2017). This has generated novel institutional methodologies implemented via physical (haptic), immersive, emotive, cognitive and co-creative, as well as meta-cognitive procedures (Samis and Michaelson, 2017). Among the most radical of these visitor-centered conceptions, Arte Útil’s (Useful Art’s) notion of the Museum 3.0 presents unique parallels with Digital Cultural Heritage applications owing to its basis in the attention economy. As philosopher Stephen Wright (2013) remarks in Toward a Lexicon of Usership, a majority of cultural institutions today have made strides to implement elements of 2.0 Culture into their operative modalities. While they have mostly retained the top-down gatekeeping mechanisms involved in the determination and dissemination of content, they have concurrently adapted their model of legitimation to incorporate visitor experience, feedback and input (2013). However, as Wright accurately observes, if museums have integrated some aspects of user-generated content, by and large the value of this contribution is far from mutualized. This is hardly surprising given that such a radical conception of usership, in contradistinction to the more commonly used term “participation”, poses a direct challenge to three fundamental aspects integral to traditional cultural institutions, namely: spectatorship, expert culture and ownership (Wright 2013).
Based on its uncompromising ideal of usership, the Museum 3.0 model recognizes, in essence, that user engagement generates value, and that this entitles users to share said value through a form of remunerated (though not necessarily monetized) exchange. Otherwise stated, this new vision of the museum sees usership as an engagement—or “cognitive privilege”—that produces value as opposed to consuming it: “usership is creation socialized, and as such engenders a surplus” (Wright 2013). While Wright explicitly states that he is referring to physical institutions rather than digital media and Virtual Museums, his proposed model for a Museum 3.0 essentially bridges the gap between Digital Cultural Heritage designers’ protracted efforts to emulate museum practices on the one hand, and virtual heritage applications’ default mode of usership on the other. As previously stated, in contrast to traditional forms of spectatorship tantamount to democratizing elite privilege through education, this radical understanding of public engagement is rooted in the attention economy, which basically views human attention as a scarce commodity. In Virtual Environments, this precious resource is the decisive condition for “presence,” as evidenced by the fact that “attention” is one of the rare factors common to all varying definitions of the term (Schuemie et al., 2001, 4). Based on this idea of the Museum 3.0, rather than monetizing attention, a Virtual Museum 3.0 might alternatively channel it, at least in part, toward the creation of an ever-renewed and shared cultural heritage commons.
To that end, Literary-based MR Presence can provide a useful tool. Prevailing Digital Cultural Heritage applications’ focus on education, entertainment and touristic consumption have generally aligned them with cultural institutions’ traditional gatekeeping role. This gatekeeping function was designed to legitimate cultural sites and artefacts all the while inculcating a wide and diverse audience as to their significance and value. In contrast, by eliciting a personal value judgment through aesthetic experience, Literary-based MR Presence has the potential to perform what McKenzie (2015) describes as museums’ “emergency service”: i.e., transform patrons into stewards of Cultural Heritage. To date, efforts within cultural institutions to fulfill this social service through digital means have, to a great extent, relied on open access strategies designed to enable patrons to creatively reuse digital assets. Literary-based MR Presence offers a different approach based on what Samis and Michaelson (2017) describe as immersive, emotive and cognitive procedures.
From a literary perspective, such an approach equally benefits this art form by transforming its traditional readership into a usership, thus actualizing Roland Barthes’ conception of the reader as an author in their own right by ascribing value to the contribution resulting from the public’s engagement. Instead of monetizing this value, a Virtual Multimodal Museum 3.0 using Literary-based MR Presence might otherwise focus it on a collective appreciation of Cultural Heritage. Overall, what this suggests is that in letting go of ownership and exclusive top-down expertise, the institutional guardians of heritage stand to gain surplus value for the patrimony in their care, not at the outcome of endless institutional efforts at legitimization, education and propagation, but instead through transformative usership.
Naturally, there are certain trade-offs to such an approach when compared with more common design methodologies geared towards inclusionary access, didactic communication and leisurely consumption. Aesthetic experience is highly variable from one individual, and from one cultural context, to the next (Starr, 2013, 57). It is not a universal mode of reception, constrained instead to a limited audience. Certainly, personal appreciation of any literary work used in a Digital Cultural Heritage application will have an undeniable impact on the encounter, and therefore, on the resulting impression of the cultural site, monument or artefact at hand. Moreover, literature is always saddled with the problem of translation, thus implying a much more situated and audience-specific Virtual Museum model. While such audience specificity may not be synonymous with geographical locality (e.g., people from different parts of the world might have a positive aesthetic experience around the same literary text), it is not global. Therefore, richness of multisensory imagery and embodiment potential will always have to be balanced out against public familiarity with the author, and the wide-audience appeal of a text (in contrast to more obscure niche genres) in order to maximize access and impact. Notwithstanding these trade-offs, the fact remains that in reconciling new museology and digital usership, Literature-based MR Presence can contribute to a uniquely integrated and socially beneficial Virtual Museum approach, capable of insuring the ever-renewed valuation of cultural heritage.
If heritage is to be effectively preserved, it needs to be valued by people living today. What is more, if it holds no contemporary significance for present generations, no living currency, then it consequently plays no actual role in the social fabric. The idea inherited from the Enlightenment that the public should be educated as to culture and patrimony’s worth through didactic modes of transmission fails to recognize the process by which things come to be intimately valued. It is an outdated paradigm, ill-advisedly carried over and replicated in Digital Cultural Heritage applications—otherwise integrally predicated on usership—especially at a time when cultural institutions are increasingly turning towards more participatory and collaborative modes of public engagement. Virtual Museums’ prevalent pedagogical approaches are thus insufficient, often necessitating recourse to gamification techniques to compensate for a lack of user enjoyment and engagement.
Instead of seeing public attention and involvement as wanting in information, they should alternatively be esteemed as a precious resource capable of generating surplus value around Cultural Heritage. Such valuation, however, is not automatically guaranteed at the outcome of basic exposure or access to cultural heritage through digital immersion. It is contingent upon a compelling and lasting transformative experience, which Literature-based MR Presence can provide by way of transportation, embodiment and aesthetic experience. As shown, aesthetic experience is not only a means to augment interest and captivation, it also involves a profound evaluative judgment, being “the result not so much of perceiving the outside world as becoming aware of our own judgment of what matters to us” (Starr, 2013, 16).
In conclusion, this paper opens the door to a more targeted multidisciplinary approach judiciously recruiting different forms of knowledge from the arts, the humanities and new digital technologies towards generating specific effects when it comes to the public reception of Cultural Heritage. It proposes an alternative to current approaches that essentially present didactic content in an entertaining package punctuated by gamification elements with potentially shorter-lived instructional and amusement impacts. Immediate offshoots of this paper might examine the possibility of also integrating or drawing on other artforms, like the visual arts in the design of Digital Cultural Heritage applications: e.g., in keeping with the example presented in this paper, the numerous paintings of the Acropolis realized as part of Grand Tours during the 17th-18th century, but also contemporary artworks that continue to represent the monument today, tacitly suggesting enduring value and custodianship passed down from generation to generation. As demonstrated, such an approach can contribute to creating a more expanded sensory experience, extending MR media’s current limitations, while inciting value judgments that have the potential to transform viewers and readers into true users and stewards of Cultural Heritage.
FIGURE 1. Illustrating an AR virtual museum. On the left, the 3D reconstructed and animated priest of Asinou church in Cyprus (Geronikolakis et al., 2020).
Data Availability Statement
The original contributions presented in the study are included in the article/supplementary material, further inquiries can be directed to the corresponding author.
MV and GP originally conceived the main idea behind the paper. SB expanded this idea and wrote the first draft of the manuscript. MV, PZ, and GP wrote sections of the manuscript. All authors contributed to manuscript revision, read, and approved the submitted version.
SB is the recipient of a H2020 Marie Sklodowska-Curie Action Global Fellowship, grant number 893454, currently funding her postdoctoral research into virtual museums. The Foundation for Research and Technology Hellas (FORTH) will cover the open access publication fees.
Conflict of Interest
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
An earlier version of this manuscript has been released as a pre-print at arxiv.org, SB, MV, PZ, EG, and GP (Bertrand et al., 2019). We would like to acknowledge and warmly thank the reviewers for their insightful feedback and constructive suggestions, which contributed to enhancing the quality of this manuscript.
Arnold, D., Day, A., Glauert, J., Haegler, S., Jennings, V., Kevelham, B., et al. (2008). “Tools for Populating Cultural Heritage Environments with Interactive Virtual Humans,” in EPOCH Conference on Open Digital Cultural Heritage Systems 2008, Rome, February 1, 2008. Available at: https://research.brighton.ac.uk/en/publications/tools-for-populating-cultural-heritage-environments-with-interact.
Bal, P. M., and Veltkamp, M. (2013). How Does Fiction Reading Influence Empathy? an Experimental Investigation on the Role of Emotional Transportation. PLoS ONE. 8 (1), e55341. doi:10.1371/journal.pone.0055341
Bertrand, P., Guegan, J., Robieux, L., McCall, C. A., and Zenasni, F. (2018). Learning Empathy through Virtual Reality: Multiple Strategies for Training Empathy-Related Abilities Using Body Ownership Illusions in Embodied Virtual Reality. Front. Robot. AI. 5, 26. doi:10.3389/frobt.2018.00026
Bertrand, S., Vassiliadi, M., Zikas, P., Geronikolakis, E., and Papagiannakis, E. (2019). From Readership to Usership and Education, Entertainment, Consumption to Valuation: Embodiment and Aesthetic Experience in Literature-Based MR Presence. arXiv. [Epub ahead of print]. Available at: https://arxiv.org/abs/1910.10019v1.
Costa, M. R., Kim, S. Y., and Biocca, F. (2013). “Embodiment and Embodied Cognition,” in Virtual Augmented and Mixed Reality. Designing and Developing Augmented and Virtual Environments. VAMR 2013. Editor R. Shumaker (Berlin, Heidelberg: Springer), 8021. Lecture Notes in Computer Science.
Geronikolakis, E., Zikas, P., Kateros, P., Lydatakis, N., Georgiou, S., Kentros, M., et al. (2020). “A True AR Authoring Tool for Interactive Virtual Museums,” in Visual Computing for Cultural Heritage. Springer Series on Cultural Computing. Editors F. Liarokapis, A. Voulodimos, N. Doulamis, and A. Doulamis (Cham: Springer). doi:10.1007/978-3-030-37191-3_12
Gibbs, R. W. (2017). “Embodied Dynamics in Literary Experience,” in Cognitive Literary Science: Dialogues between Literature and Cognition. Editors M. Burke, and E. T. Troscianko (New York: Oxford University Press)
McKenzie, B. (2015). ‘Towards the Sociocratic Museum’. Medium. Available at: https://medium.com/code-words-technology-and-theory-in-the-museum/towards-the-sociocratic-museum-223390e2a00b. doi:10.1057/9781137446770
Oatley, K., Mar, R. A., and Djikic, M. (2012). “The Psychology of Fiction: Present and Future,” in Cognitive Literary Studies: Current Themes and New Directions. Jaén, Isabel and Simon. Editor J. Jacques (Austin: The University of Texas Press).
Papaefthymiou, M., Kanakis, M. E., Geronikolakis, E., Nochos, A., Zikas, P., and Papagiannakis, G. (2018). “Rapid Reconstruction and Simulation of Real Characters in Mixed Reality Environments,” in Digital Cultural Heritage. Lecture Notes in Computer Science. Editors M. Ioannides (Cham: Springer), 10605. doi:10.1007/978-3-319-75826-8_22
Papagiannakis, G., Papanikolaou, P., Greassidou, E., and Trahanias, P. (2014). “glGA: an OpenGL Geometric Application Framework for a Modern, Shader-Based Computer Graphics Curriculum,” in Eurographics 2014 - Education Papers (Geneva: The Eurographics Association).
Perry, S., Rousseau, M., Economou, M., Young, H., and Pujol, L. (2017). ‘Moving beyond the Virtual Museum: Engaging Visitors Emotionally’. 23rd International Conference on Virtual Systems and Multimedia. Dublin: VSMM, 1–8.
Schubert, T., Friedmann, F., and Regenbrecht, H. (1999). “Embodied Presence in Virtual Environments,” in Visual Representations and Interpretations. Editors R. Paton, and I. Neilson (London: Springer)
Schuemie, M. J., Van Der Straaten, P., Krijn, M., and Van Der Mast, C. A. P. G. (2001). Research on Presence in Virtual Reality: A Survey. Cyberpsychology Behav. 4, 183–201. N. 2 (April). doi:10.1089/109493101300117884
Stark, E. A., Vuust, P., and Kringelbach, M. L. (2018). Music, Dance, and Other Art Forms: New Insights into the Links between Hedonia (Pleasure) and Eudaimonia (Well-being). Prog. Brain Res. 237, 129–152. doi:10.1016/bs.pbr.2018.03.019
Vacchetti, L., Lepetit, V., Ponder, M., Papagiannakis, G., Fua, P., Thalmann, D., et al. (2004). A Stable Real-Time AR Framework for Training and Planning in Industrial Environments. London: Springer, 129–145. doi:10.1007/978-1-4471-3873-0_8
Wright, S. (2013). Toward a Lexicon of Usership. Editors S. Wright, and N. Aitkens (Eindhoven: Van Abbemuseum). Available at: https://museumarteutil.net/wp-content/uploads/2013/12/Toward-a-lexicon-of-usership.pdf. doi:10.4324/9781315079295
Zikas, P., Bachlitzanakis, V., Papaefthymiou, M., Kateros, S., Georgiou, S., Lydatakis, N., and Papagiannakis, G. (2016). Mixed Reality Serious Games and Gamification for Smart Education in European Conference on Games Based Learning, Parsley, Scotland 2016 (Academic Conferences International Limited).
Keywords: aesthetic experience, communication, education, embodiment, mixed reality, multisensory imagery, storytelling, usership
Citation: Bertrand S, Vassiliadi M, Zikas P, Geronikolakis E and Papagiannakis G (2021) From Readership to Usership: Communicating Heritage Digitally Through Presence, Embodiment and Aesthetic Experience. Front. Commun. 6:676446. doi: 10.3389/fcomm.2021.676446
Received: 05 March 2021; Accepted: 23 June 2021;
Published: 07 July 2021.
Edited by:Stephen Croucher, National Research University Higher School of Economics, Russia
Reviewed by:Paromita Pain, University of Nevada, Reno, United States
Marko Siitonen, University of Jyväskylä, Finland
Copyright © 2021 Bertrand, Vassiliadi, Zikas, Geronikolakis and Papagiannakis. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
*Correspondence: Stéphanie Bertrand, email@example.com