Mini Review ARTICLE
Enhancing the Ecological Validity of fMRI Memory Research Using Virtual Reality
- 1Department of Psychology, University of California, Los Angeles, Los Angeles, CA, United States
- 2Department of Psychiatry & Biobehavioral Sciences, University of California, Los Angeles, Los Angeles, CA, United States
- 3Division of Biology and Biological Engineering, California Institute of Technology, Pasadena, CA, United States
- 4Department of Neurosurgery, University of California, Los Angeles, Los Angeles, CA, United States
- 5Division of Child and Adolescent Psychiatry, Johns Hopkins Children’s Center, Johns Hopkins Medicine, Baltimore, MD, United States
Functional magnetic resonance imaging (fMRI) is a powerful research tool to understand the neural underpinnings of human memory. However, as memory is known to be context-dependent, differences in contexts between naturalistic settings and the MRI scanner environment may potentially confound neuroimaging findings. Virtual reality (VR) provides a unique opportunity to mitigate this issue by allowing memories to be formed and/or retrieved within immersive, navigable, visuospatial contexts. This can enhance the ecological validity of task paradigms, while still ensuring that researchers maintain experimental control over critical aspects of the learning and testing experience. This mini-review surveys the growing body of fMRI studies that have incorporated VR to address critical questions about human memory. These studies have adopted a variety of approaches, including presenting research participants with VR experiences in the scanner, asking participants to retrieve information that they had previously acquired in a VR environment, or identifying neural correlates of behavioral metrics obtained through VR-based tasks performed outside the scanner. Although most such studies to date have focused on spatial or navigational memory, we also discuss the promise of VR in aiding other areas of memory research and facilitating research into clinical disorders.
Virtual reality (VR) is a term used to encompass any computer-generated experience that induces a sense of presence – the feeling of being transported to and inhabiting a place different from one’s immediate surroundings (Steuer, 1992; McCreery et al., 2013). Given the intimate relationship between context and memory (Godden and Baddeley, 1975; Smith, 1988; Ramirez et al., 2013), VR offers a powerful means to enhance the ecological validity of memory research by providing realistic virtual environments (VEs) in which participants can learn information and/or draw upon past memories to guide their behavior. These VEs can be highly customized to meet the needs of a wide variety of tasks and offer experimental control over the learning experience. Given these characteristics, along with the recent surge in VR technological development and accessibility (Figure 1A), it is unsurprising that cognitive neuroscientists interested in the brain mechanisms of memory have increasingly found ways to incorporate VR into their fMRI studies.
FIGURE 1. (A) A limited showcase of currently available VR technologies. Devices are sorted as a function of their ability to provide the participant with a sense that they are “in” a virtual environment (immersiveness; x-axis) and the system’s affordability (y-axis). “Window on World” refers to a traditional desktop and monitor setup. CAVE = cave automatic virtual environment – a real world room that leverages projectors and motion capture to create room-size virtual experiences. MR-safe equipment (joystick and buttonbox: Current Design, Inc., Philadelphia, PA, United States, goggles: cinemavision.biz) can be used during MR scans. (B) Examples of common perspectives presented to participants while actively navigating VEs or during spatial memory tests. Both first- and third-person viewpoints provide an egocentric perspective whereas a bird’s eye view provides an allocentric one.
Experimental designs employing VR and fMRI to study memory predominantly fall into three categories: (1) having participants actively engage in VR experiences in the scanner while functional neuroimaging data are acquired, (2) scanning participants as they are prompted to retrieve information previously acquired in a VE, and (3) identifying structural or functional correlates of behavioral metrics obtained through the use of VR (Figure 2). One virtue of VR as an experimental tool is its ability to enable the translation of research paradigms that have been used extensively in animal research, which may not otherwise translate readily to human participants for ethical or technical reasons. For example, a direct human analog of the Morris water maze – dropping a participant into a pool of cloudy water in search of an invisible platform – would likely raise the ethical eyebrows of any Institutional Review Board, yet such a task paradigm can be implemented in VR. Likewise, VR empowers neuroscientists to create experiments that would either be impossible or impractical without the use of VR (e.g., imposing invisible boundaries, altering/morphing environmental features, or teleporting a participant between contexts).
FIGURE 2. Examples of VR-fMRI experimental paradigms. (A) MR-compatible joysticks/gamepads and 3D stereoscopic goggles allow for participants to enter a VE while laying supine in the scanner. By time-locking events of interest (e.g., a participant’s heading direction while traversing the world) to the corresponding fMRI signals, researchers can identify neural correlates associated with specific task conditions or behaviors. In this example, entorhinal cortex activity is associated with the grid-cell-like property of hexagonal symmetry during navigation. Figures adapted with permission from Doeller et al. (2010). (B) Participants can perform VR-based learning tasks outside of the scanner, and their memory for information encoded within a VE can later be tested in the scanner using traditional fMRI task paradigms. In this example, trials can be coded based on each object’s properties within the VE (e.g., whether or not the object was located at a pertinent decision-point) to reveal incidental neural differences during retrieval as a function of the encoding experience. Figures adapted with permission from Janzen and Weststeijn (2007). (C) Just as questionnaires and computer tasks reveal individual differences in a host of behavioral metrics, VR can serve as an instrument to gather unique behavioral data points (e.g., number of times a participant revisited a particular location). Researchers can then examine whether these performance metrics can account for variance in brain activity or connectivity measured in a completely different context (e.g., while participants are simply resting in the scanner). Figures adapted with permission from Wong et al. (2014).
Researchers may go to great lengths to increase the ecological validity of their tasks, given the growing appreciation that laboratory-encoded stimuli and real-world events tend to evoke different brain activation profiles (Roediger and McDermott, 2013; Chen et al., 2017; Chow et al., 2018). For instance, wearable cameras can be used to capture photographs of participants’ real-world experiences so that memories for these events can later be probed in the scanner (Chow and Rissman, 2017). A related approach involves having participants engage in real-world navigation tasks. In one such study, Schinazi and Epstein (2010) created a 3-km outdoor walking course for participants to traverse. Later, fMRI data were collected while participants were tested on their recollection for buildings encountered on the route. While the fMRI results revealed interesting effects within visuospatial processing regions such as the retrosplenial cortex, reflecting the interplay between landmark-identification and route direction at navigationally pertinent decision points, the authors acknowledged that their behavioral results were largely consistent with those of a similarly designed VR study by Janzen and Weststeijn (2007). A subsequent fMRI study then showed that comparable neuroimaging findings could be obtained used a VR-based route navigation task (Wegman and Janzen, 2011). Although real-world task paradigms will continue to have value in memory research, VR paradigms have the potential to provide a less labor-intensive and more highly controlled investigational medium that sacrifices relatively little in terms of neural processing and experimental outcome.
While VR allows for precise control over stimuli and contexts, providing greater consistency across participants than can typically be attained in real-world designs, it is not without its caveats. Recently, there has been debate as to whether VR-based navigation should be considered true navigation (Taube et al., 2013; Minderer et al., 2016). One of the most crucial arguments against the fusion of VR and fMRI is that when lying in a scanner, vestibular self-motion (idiothetic) cues cannot match external landmark-based (allothetic) cues since otolith organs will persistently relay a signal that the individual is supine. Decoupling of cues can cause a reorientation (Wang and Spelke, 2002) and force one system into domination (Golledge, 1998; Dolins and Mitchell, 2010). Further adding to these complications, visual cues alone have proven insufficient to elicit accurate distance measurements (Witmer and Kline, 1998) and turn responses (Riecke et al., 2012), which can lead to impaired navigation. Meanwhile, on a neuronal level, the activity pattern of cells implicated in spatial representation, such as place cells, grid cells, and head-direction cells (Buzsáki and Moser, 2013) have been shown to differ between real-world environments and VEs (Chen et al., 2013; Ravassard et al., 2013; Aghajan et al., 2015).
Nevertheless, the neural responses of spatially selective cells in VR resemble those observed in real navigation under certain circumstances (Domnisoru et al., 2013; Aronov and Tank, 2014; Killian and Buffalo, 2018). Additionally, VR navigation has been shown to maintain hippocampal theta rhythms (Ekstrom et al., 2005), albeit with some differences from real-world navigation (Jacobs, 2014; Aghajan et al., 2017; Bohbot et al., 2017). Various VR accessories, including head-mounted displays (HMD) can be used to increase participants’ immersion (Figure 1A; Dede, 2009) and, subsequently, spatial understanding (Ruddle et al., 1997; Bowman and McMahan, 2007). Importantly, Ganesh et al. (2012) found that increasing participants’ self-identification with an avatar resulted in increased engagement of left inferior parietal lobe regions associated with self-identification and improved recognition memory for traits associated with their avatar. Furthermore, brain activity patterns expressed during recall remain similar despite encoding in real-world vs. fictional environments (Spiers and Maguire, 2006). Even navigation through digital folders (Benn et al., 2015) and abstract conceptual space (Constantinescu et al., 2016) recruits similar brain structures and processes.
Given that the overarching goal of cognitive neuroscience research is to understand the brain mechanisms that give rise to our thoughts and behaviors, VR affords researchers with the ability to execute task paradigms that more closely mimic the way we use our cognition as we dynamically engage with our environment. This mini-review surveys the burgeoning neuroimaging literature on VR applications to memory research. In so doing, we hope to illustrate some creative ways in which researchers have leveraged VR to increase the ecological validity of memory experiments and conduct studies that would be relatively infeasible without the use of VR.
Harnessing the Affordances of VR to Aid Memory Research
Although neural recordings from freely moving rodents have provided crucial insights into spatial memory functioning, ethical and physical limitations have prevented a direct replication of these studies in human participants. However, VR offers researchers boundless, safe, and controllable environments to conduct analogs of foundational experimental paradigms like the Morris water maze (MWM; Morris, 1984), radial arm maze (RAM; Olton et al., 1977), and random foraging tasks. Indeed, when combined with fMRI, VR has afforded researchers with the ability to quickly iterate manipulations of different MWM task features (e.g., distal vs. no cues; visible vs. invisible platforms) to determine hippocampal dependence (Shipman and Astur, 2008; Kolarik et al., 2016), identify compensatory mechanisms following scopolamine injection (Antonova et al., 2011), examine functional connectivity changes (Woolley et al., 2015), and investigate the different neural patterns recruited when using egocentric vs. allocentric navigation strategies (Rodriguez, 2010a). A research group even recently replicated their rodent body-behavior findings in humans using a VR version of the MWM (Müller et al., 2018).
Virtual variations of the RAM have equipped researchers to study working memory and decision-making in both win-shift (Demanuele et al., 2015) and win-stay (Cyr et al., 2016) paradigms. VR also allows for real-time changes to RAM and similar tasks. For instance, shuffling distal cues and providing visual navigational guidance (e.g., following arrows on the ground) has made it possible to disentangle cognitive decision-making from other processes of interest (Marsh et al., 2010). The ability to “teleport”, restrict access to certain areas with virtual “walls”, and track the precise location of the subject within the VE permit researchers to tease apart place-based and sequence-based strategies (Igloi et al., 2015). VR versions of the RAM were also used to assess the integrity of the hippocampus – predicting risk or severity in a variety of psychiatric disorders (Astur et al., 2005; Wilkins et al., 2017). Such insights are in line with the growing trend of using VR to provide objective diagnostic metrics (Cogné et al., 2017; van Bennekom et al., 2017). For instance, Migo et al. (2016) identified behavioral and neural correlates of completing the RAM task in patients with amnestic mild cognitive impairment (MCI), which extends upon the work of King et al. (2002) who showed that when changing virtual viewpoints, MCI patients could not recall the positions of objects. Similar spatial memory tests have been conducted on athletes following mild traumatic brain injury (Slobounov et al., 2010).
Given the expanse of possibilities afforded by VR, experimental paradigms can move beyond the replication of rodent studies. By familiarizing participants with a VE, experimenters can probe a participant’s spatial memory by asking them to navigate from one location to another – a general paradigm that also can be used to test orientation, route-learning, and viewpoint-dependence (Brown et al., 2014; Stokes et al., 2015; Dimsdale-Zucker et al., 2018). Indeed, many such studies have used VEs to examine the neural correlates supporting navigation under different manipulations such as: using one landmark vs. many (Wegman et al., 2014), finding one’s way vs. following a visible path (Hartley et al., 2003), relying on coarse vs. global strategies (Evensmoen et al., 2013), leveraging survey vs. route knowledge (Gillner and Mallot, 1998; Wolbers et al., 2004), tracking paths and distances (Wolbers et al., 2007; Chrastil et al., 2015), varying head directions (Shine et al., 2016), egocentric and/or allocentric related manipulations (Wolbers et al., 2008; Suthana et al., 2009), and navigating towards a goal in healthy (Rodriguez, 2010b; Brown et al., 2016) and clinical populations (Thomas et al., 2001). Embedding several such manipulations within a single VR study, Dhindsa et al. (2014) utilized fMRI to measure signal fluctuations as participants oriented themselves towards a learned location in a VE that lost critical features one-by-one. Their results provided empirical evidence in support of the Byrne et al. (2007) model of orientation and navigation, which emphasizes the translation of egocentric representations in parietal cortex to allocentric representations in the hippocampus. Furthermore, virtual renditions of familiarized real-world environments can allow researchers to probe memory for real-world objects using virtual cues – a technique previously used to examine the neural correlates of egocentric representations for objects outside of one’s visual field (Schindler and Bartels, 2013).
The use of concurrent fMRI and VR also begets an opportunity to examine the neural underpinnings of spatial information that is being encoded incidentally. For example, following periods of egocentric navigation, researchers can provide participants with a spatial memory test using a bird’s eye view of the environment (Figure 1B) – a metric of allocentric memory that has been used to explain differences in navigational ability (Pine et al., 2002). Other examples come from fMRI studies looking for evidence of pattern separation and pattern completion processes (Yassa and Stark, 2011). By having participants complete the same relative distance task across different, but visually similar, environments, Kyle et al. (2015) found that the more distinguishable a neural representation is of an environment (i.e., successful pattern separation), the less the interference of competing memories will hinder performance. Relatedly, a human analog of the attractor dynamic model of mnemonic processing (Leutgeb et al., 2007) was demonstrated by Steemers et al. (2016): hippocampal responses to VEs that were constructed by linearly morphing two previously known VEs exhibited non-linear (sigmoid-like) response properties indicative of pattern completion, despite participants’ behavioral reports that they consciously perceived linear morphs. By leveraging multivoxel pattern analysis in the hippocampus to decode a participant’s location within a virtual environment, Hassabis et al. (2009) corroborated the classic function of hippocampal place cells (O’Keefe and Dostrovsky, 1971), albeit at a far less granular level. VR-based random foraging tasks have also been used to identify population-based grid-cell-like activity patterns in human entorhinal cortex (Doeller et al., 2010) – a measurement whose consistency over time could be prognostic of Alzheimer’s Disease risk (Kunz et al., 2015) – and 3D place coding representations in the human hippocampus (Kim et al., 2017).
VEs can also be utilized to systematically, and quantitatively, investigate processes that rely on imagined navigation. For example, Legge et al. (2012) familiarized participants with a VE that they were later instructed to use as a “memory palace” while they implemented the Method of Loci mnemonic strategy of mentally “placing” a set of to-be-remembered items along a route within an imagined environment. In this way, the authors matched the size, detail, and exposure time to the environment – properties that are often confounded in traditional implementations of this mnemonic technique (Yates, 1966). Further, the use of imagined virtual navigation has revealed fMRI signals that exhibit grid-cell-like properties (Bellmund et al., 2016; Horner et al., 2016) and activity patterns associated with location and facing direction (Marchette et al., 2014). Equalizing environments used for imagination tasks is particularly relevant in the domain of prospective memory (the ability to maintain a representation of intended tasks and execute them at the appropriate time and place). For instance, VR has recently been used in conjunction with high-resolution fMRI to index the degree to which specific goal and sub-goal locations are represented within hippocampal activity patterns during route planning, reflecting prospective coding of navigational intentions (Brown et al., 2016). Additionally, Kalpouzos and Eriksson (2013) familiarized participants to a VE and subsequently collected fMRI data while they mentally executed intended tasks within the imagined VE – a design that reduced variability in neural representation for environment.
Given that a time-course of fMRI activity can be collected during virtual navigation, it is possible to examine the different temporal phases of navigation behavior (Demanuele et al., 2015). Previous work has examined: planning vs. execution (Xu et al., 2010), encoding vs. retrieval (Suthana et al., 2011), periods of object manipulation (Baumann et al., 2003a), and active vs. guided periods (Baumann et al., 2003b). Persson et al. (2013) measured hippocampal activity as participants navigated through a virtual maze and found that males and females show dissociable recruitment of left and right hippocampus during active navigation relative to orientation judgments made at maze end-points. Additionally, events that occur within VR (e.g., encountering another avatar who dispenses objects) can be dissociated from their visual scene context by using different approach routes (Burgess et al., 2001). Even metrics like memory for heading direction (Baumann and Mattingley, 2010) and environmental size/complexity (Baumann and Mattingley, 2013) can be investigated by examining fMRI activity levels at relevant task time points (e.g., when the participant is facing North; Figure 2A), without explicitly probing the participant.
In addition to navigation studies, VEs can be employed to study object-place associative memory. VR can be used to efficiently change the constellation of objects and their identities, with respect to locations within the VEs [e.g., shuffling object identities (Wong et al., 2014), modulating their saliency (Buchy et al., 2014), or altering the environment boundaries (Lee et al., 2016)]. Object-place memory tasks have also shown that emotion is bound to places by examining how the co-occurrence of task-irrelevant emotional events alongside encoding can heighten subsequent retrieval activity(Chan et al., 2014) – extending findings that show place cells remapping once an environment becomes associated with fear (Moita et al., 2004). VR allows for object-place experiments to be conducted with high precision, immersion, and repeatability – a set of capabilities that make it particularly useful for obtaining diagnostic metrics in clinical populations (e.g., schizophrenia patients; Hawco et al., 2015).
Performance on VR-based tasks can also serve as a useful measuring instrument for examining factors outside of the learning experience that may affect behavior. For instance, Rauchs et al. (2008) investigated the neurocognitive effects of sleep deprivation on a series of virtual navigation tests. Researchers can also examine how fMRI signals measured in one setting (e.g., during resting fixation) might predict individual differences in performance on VR-based tasks performed outside the scanner. For example, Wong et al. (2014) identified patterns of resting-state activity and functional connectivity that correlated with participants’ memory for objects that had been learned in a room-scale VE the day before. In another study, Wegman and Janzen (2011) scanned participants while passively viewing a route through a VE to identify brain regions associated with navigation-based decision points, later using the functional connectivity profile of those regions during resting-state to account for individual differences in spatial memory.
While fMRI has served as a powerful tool in human memory research, it requires participants to be placed in a context that is far from naturalistic – a potential confound for many memory studies. The inclusion of VR in fMRI memory investigations allows researchers to utilize immersive and navigable contexts for stimulus presentation both inside and outside the scanner (Figure 1A). Moreover, it affords researchers a medium in which to conduct experiments that is both replicable and controllable.
Facets unique to VR position it as an indispensable toolkit for specific types of investigations. For instance, creating invisible walls that restrict movement, but retain the visibility of distal cues would not be feasible outside of a VE (Lee et al., 2016). Work by Bergouignan et al. (2014), which used VR to induce out-of-body experiences in the scanner while examining the role of perceiving the world from the perspective of one’s own body for successful episodic encoding of real-life events, would not have been possible without the use of VR. The same concept applies to VR’s ability to “blend” VEs (Steemers et al., 2016) or shift participants’ perspective within the same VE (Sulpizio et al., 2016). Additionally, VR has the capacity to even the playing field in experiments that hinge on the use of imagination (e.g., Legge et al., 2012): it provides a common virtual space instead of relying on familiar real-world environments that could vary across individuals as a function of their pre-experimental exposure to the environment.
VR technologies can also bolster the ecological validity of fMRI for researchers and clinicians to obtain objective diagnostic metrics for patient populations (King et al., 2002; Plancher et al., 2012; Cogné et al., 2017; van Bennekom et al., 2017). With HMDs, cross-institutional collaboration can be facilitated as participants immersed in VR will not be cognizant of the real-world environmental cues. Such attributes are particularly advantageous for the examination of disorders that are highly context-dependent (e.g., post-traumatic stress disorder). For instance, researchers have utilized VR to induce context-specific fear-conditioning (Huff et al., 2011; Tröger et al., 2012; Ewald et al., 2014) and fear extinction (Dunsmoor et al., 2014; Ahs et al., 2015) – dramatically extending current treatment methods which often require therapy to occur in a context that is dissimilar from where the fear was acquired (for review see Bohil et al., 2011; Maples-Keller et al., 2017). Furthermore, compared to many real-world tasks, VR-based experimental techniques can be replicated in shorter time spans.
The utilization of VR in fMRI studies need not be daunting nor expensive; open-source software such as OpenSimulator1 and equipment found in most scanner suites (Figure 1A), such as MR-compatible stereoscopic goggles and joysticks/joypads, make it increasingly accessible. Nonetheless, VR research is still in its infancy and not without limitations. Given the visual–vestibular disconnect of most setups, some participants may experience nausea and be unable to complete the study (Sharples et al., 2008). However, advances in HMD technology are already helping to alleviate motion-sickness concerns. Devices that increase immersion through haptic feedback (e.g., Tesla Suit) and stationary locomotion (e.g., Omni Treadmill) or setups that create room-scale environments (e.g., cave automatic virtual environment; Figure 1A) afford researchers with the ability to employ encoding paradigms that increasingly resemble “real life” circumstances, making the neural correlates associated with the formation and recall of such memories more likely to generalize to real-world behaviors.
NR, JE, ZA, AT, JM, NS, and JR conducted the literature review and wrote the manuscript. AT, NR, and JR generated figures.
This work was supported by a Defense Advanced Research Project Agency (DARPA) Research Grant awarded to JR (Grant No. D13AP00057), National Science Foundation (NSF) Graduate Research Fellowships awarded to NR (DGE-1650604) and JE (DGE-1144087), and a National Institute of Neurological Disorders and Stroke UO1 Grant awarded to NS (NS103802).
Conflict of Interest Statement
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
The authors extend thanks to Avery Bedows for his comments on this review.
Aghajan, Z., Schuette, P., Fields, T. A., Tran, M. E., Siddiqui, S. M., Hasulak, N. R., et al. (2017). Theta oscillations in the human medial temporal lobe during real-world ambulatory movement. Curr. Biol. 27, 3743.e3–3751.e3. doi: 10.1016/j.cub.2017.10.062
Aghajan, Z. M., Acharya, L., Moore, J. J., Cushman, J. D., Vuong, C., and Mehta, M. R. (2015). Impaired spatial selectivity and intact phase precession in two-dimensional virtual reality. Nat. Neurosci. 18, 121–128. doi: 10.1038/nn.3884
Ahs, F., Kragel, P. A., Zielinski, D. J., Brady, R., and LaBar, K. S. (2015). Medial prefrontal pathways for the contextual regulation of extinguished fear in humans. Neuroimage 122, 262–271. doi: 10.1016/j.neuroimage.2015.07.051
Antonova, E., Parslow, D., Brammer, M., Simmons, A., Williams, S., Dawson, G. R., et al. (2011). Scopolamine disrupts hippocampal activity during allocentric spatial memory in humans: an fMRI study using a virtual reality analogue of the Morris Water Maze. J. Psychopharmacol. 25, 1256–1265. doi: 10.1177/0269881110379285
Astur, R. S., St Germain, S. A., Baker, E. K., Calhoun, V., Pearlson, G. D., and Constable, R. T. (2005). fMRI hippocampal activity during a virtual radial arm maze. Appl. Psychophysiol. Biofeedback 30, 307–317. doi: 10.1007/s10484-005-6385-z
Baumann, S., Neff, C., Fetzick, S., Stangl, G., Basler, L., Vereneck, R., et al. (2003a). A virtual reality system for neurobehavioral and functional MRI studies. Cyberpsychol. Behav. 6, 259–266. doi: 10.1089/109493103322011542
Benn, Y., Bergman, O., Glazer, L., Arent, P., Wilkinson, I. D., Varley, R., et al. (2015). Navigating through digital folders uses the same brain structures as real world navigation. Sci. Rep. 5:14719. doi: 10.1038/srep14719
Bohbot, V. D., Copara, M. S., Gotman, J., and Ekstrom, A. D. (2017). Low-frequency theta oscillations in the human hippocampus during real-world and virtual navigation. Nat. Commun. 8:14415. doi: 10.1038/ncomms14415
Brown, T. I., Carr, V. A., LaRocque, K. F., Favila, S. E., Gordon, A. M., Bowles, B., et al. (2016). Prospective representation of navigational goals in the human hippocampus. Science 352, 1323–1326. doi: 10.1126/science.aaf0784
Brown, T. I., Hasselmo, M. E., and Stern, C. E. (2014). A high-resolution study of hippocampal and medial temporal lobe correlates of spatial context and prospective overlapping route memory. Hippocampus 24, 819–839. doi: 10.1002/hipo.22273
Buchy, L., Hawco, C., Bodnar, M., Izadi, S., Dell’Elce, J., Messina, K., et al. (2014). Functional magnetic resonance imaging study of external source memory and its relation to cognitive insight in non-clinical subjects. Psychiatry Clin. Neurosci. 68, 683–691. doi: 10.1111/pcn.12177
Burgess, N., Maguire, E. A., Spiers, H. J., and O’Keefe, J. (2001). A temporoparietal and prefrontal network for retrieving the spatial context of lifelike events. Neuroimage 14, 439–453. doi: 10.1006/nimg.2001.0806
Byrne, P., Becker, S., and Burgess, N. (2007). Remembering the past and imagining the future: a neural model of spatial memory and imagery. Psychol. Rev. 114, 340–375. doi: 10.1037/0033-295X.114.2.340
Chan, E., Baumann, O., Bellgrove, M. A., and Mattingley, J. B. (2014). Negative emotional experiences during navigation enhance parahippocampal activity during recall of place information. J. Cogn. Neurosci. 26, 154–164. doi: 10.1162/jocn_a_00468
Chen, H.-Y., Gilmore, A. W., Nelson, S. M., and McDermott, K. B. (2017). Are there multiple kinds of episodic memory? An fMRI investigation comparing autobiographical and recognition memory tasks. J. Neurosci. 37, 2764–2775. doi: 10.1523/JNEUROSCI.1534-16.2017
Chow, T. E., and Rissman, J. (2017). Neurocognitive mechanisms of real-world autobiographical memory retrieval: insights from studies using wearable camera technology: wearable cameras and autobiographical memory. Ann. N. Y. Acad. Sci. 1396, 202–221. doi: 10.1111/nyas.13353
Chow, T. E., Westphal, A. J., and Rissman, J. (2018). Multi-voxel pattern classification differentiates personally experienced event memories from secondhand event knowledge. Neuroimage 176, 110–123. doi: 10.1016/j.neuroimage.2018.04.024
Chrastil, E. R., Sherrill, K. R., Hasselmo, M. E., and Stern, C. E. (2015). There and back again: hippocampus and retrosplenial cortex track homing distance during human path integration. J. Neurosci. 35, 15442–15452. doi: 10.1523/JNEUROSCI.1209-15.2015
Cogné, M., Taillade, M., N’Kaoua, B., Tarruella, A., Klinger, E., Larrue, F., et al. (2017). The contribution of virtual reality to the diagnosis of spatial navigation disorders and to the study of the role of navigational aids: a systematic literature review. Ann. Phys. Rehabil. Med. 60, 164–176. doi: 10.1016/j.rehab.2015.12.004
Cyr, M., Wang, Z., Tau, G. Z., Zhao, G., Friedl, E., Stefan, M., et al. (2016). Reward-based spatial learning in teens with bulimia nervosa. J. Am. Acad. Child Adolesc. Psychiatry 55, 962.e3–971.e3. doi: 10.1016/j.jaac.2016.07.778
Demanuele, C., Bähner, F., Plichta, M. M., Kirsch, P., Tost, H., Meyer-Lindenberg, A., et al. (2015). A statistical approach for segregating cognitive task stages from multivariate fMRI BOLD time series. Front. Hum. Neurosci. 9:537. doi: 10.3389/fnhum.2015.00537
Dhindsa, K., Drobinin, V., King, J., Hall, G. B., Burgess, N., and Becker, S. (2014). Examining the role of the temporo-parietal network in memory, imagery, and viewpoint transformations. Front. Hum. Neurosci. 8:709. doi: 10.3389/fnhum.2014.00709
Dimsdale-Zucker, H. R., Ritchey, M., Ekstrom, A. D., Yonelinas, A. P., and Ranganath, C. (2018). CA1 and CA3 differentially support spontaneous retrieval of episodic contexts within human hippocampal subfields. Nat. Commun. 9:294. doi: 10.1038/s41467-017-02752-1
Dunsmoor, J. E., Ahs, F., Zielinski, D. J., and LaBar, K. S. (2014). Extinction in multiple virtual reality contexts diminishes fear reinstatement in humans. Neurobiol. Learn. Mem. 113, 157–164. doi: 10.1016/j.nlm.2014.02.010
Evensmoen, H. R., Lehn, H., Xu, J., Witter, M. P., Nadel, L., and Haberg, A. K. (2013). The anterior hippocampus supports a coarse, global environmental representation and the posterior hippocampus supports fine-grained, local environmental representations. J. Cogn. Neurosci. 25, 1908–1925. doi: 10.1162/jocn_a_00436
Ewald, H., Glotzbach-Schoon, E., Gerdes, A. B. M., Andreatta, M., Müller, M., Mühlberger, A., et al. (2014). Delay and trace fear conditioning in a complex virtual learning environment—Neural substrates of extinction. Front. Hum. Neurosci. 8:323. doi: 10.3389/fnhum.2014.00323
Ganesh, S., van Schie, H. T., de Lange, F. P., Thompson, E., and Wigboldus, D. H. J. (2012). How the human brain goes virtual: distinct cortical regions of the person-processing network are involved in self-identification with virtual agents. Cereb. Cortex 22, 1577–1585. doi: 10.1093/cercor/bhr227
Hartley, T., Maguire, E. A., Spiers, H. J., and Burgess, N. (2003). The well-worn route and the path less traveled: distinct neural bases of route following and wayfinding in humans. Neuron 37, 877–888. doi: 10.1016/S0896-6273(03)00095-3
Hassabis, D., Chu, C., Rees, G., Weiskopf, N., Molyneux, P. D., and Maguire, E. A. (2009). Decoding neuronal ensembles in the human hippocampus. Curr. Biol. 19, 546–554. doi: 10.1016/j.cub.2009.02.033
Hawco, C., Buchy, L., Bodnar, M., Izadi, S., Dell’Elce, J., Messina, K., et al. (2015). Source retrieval is not properly differentiated from object retrieval in early schizophrenia: an fMRI study using virtual reality. Neuroimage Clin. 7, 336–346. doi: 10.1016/j.nicl.2014.08.006
Huff, N., Alba Hernandez, J., Fecteau, M., Zielinski, D., Brady, R., and LaBar, K. S. (2011). Revealing context-specific conditioned fear memories with full immersion virtual reality. Front. Behav. Neurosci. 5:75. doi: 10.3389/fnbeh.2011.00075
Igloi, K., Doeller, C. F., Paradis, A.-L., Benchenane, K., Berthoz, A., Burgess, N., et al. (2015). Interaction between hippocampus and cerebellum crus I in sequence-based but not place-based navigation. Cereb. Cortex 25, 4146–4154. doi: 10.1093/cercor/bhu132
Jacobs, J. (2014). Hippocampal theta oscillations are slower in humans than in rodents: implications for models of spatial navigation and memory. Philos. Trans. R. Soc. Lond. B Biol. Sci. 369:20130304. doi: 10.1098/rstb.2013.0304
Kolarik, B. S., Shahlaie, K., Hassan, A., Borders, A. A., Kaufman, K. C., Gurkoff, G., et al. (2016). Impairments in precision, rather than spatial strategy, characterize performance on the virtual Morris Water Maze: a case study. Neuropsychologia 80, 90–101. doi: 10.1016/j.neuropsychologia.2015.11.013
Kunz, L., Schröder, T. N., Lee, H., Montag, C., Lachmann, B., Sariyska, R., et al. (2015). Reduced grid-cell-like representations in adults at genetic risk for Alzheimer’s disease. Science 350, 430–433. doi: 10.1126/science.aac8128
Kyle, C. T., Stokes, J. D., Lieberman, J. S., Hassan, A. S., and Ekstrom, A. D. (2015). Successful retrieval of competing spatial environments in humans involves hippocampal pattern separation mechanisms. eLife 4:e10499. doi: 10.7554/eLife.10499
Lee, C.-H., Ryu, J., Lee, S.-H., Kim, H., and Lee, I. (2016). Functional cross-hemispheric shift between object-place paired associate memory and spatial memory in the human hippocampus. Hippocampus 26, 1061–1077. doi: 10.1002/hipo.22587
Legge, E., Madan, C. R., Ng, E. T., and Caplan, J. B. (2012). Building a memory palace in minutes: equivalent memory performance using virtual versus conventional environments with the Method of Loci. Acta Psychol. 141, 380–390. doi: 10.1016/j.actpsy.2012.09.002
Maples-Keller, J. L., Yasinski, C., Manjin, N., and Rothbaum, B. O. (2017). Virtual reality-enhanced extinction of phobias and post-traumatic stress. Neurotherapeutics 14, 554–563. doi: 10.1007/s13311-017-0534-y
Marchette, S. A., Vass, L. K., Ryan, J., and Epstein, R. A. (2014). Anchoring the neural compass: coding of local spatial reference frames in human medial parietal lobe. Nat. Neurosci. 17, 1598–1606. doi: 10.1038/nn.3834
Marsh, R., Hao, X., Xu, D., Wang, Z., Duan, Y., Liu, J., et al. (2010). A virtual reality-based FMRI study of reward-based spatial learning. Neuropsychologia 48, 2912–2921. doi: 10.1016/j.neuropsychologia.2010.05.033
Migo, E. M., O’Daly, O., Mitterschiffthaler, M., Antonova, E., Dawson, G. R., Dourish, C. T., et al. (2016). Investigating virtual reality navigation in amnestic mild cognitive impairment using fMRI. Neuropsychol. Dev. Cogn. B Aging Neuropsychol. Cogn. 23, 196–217. doi: 10.1080/13825585.2015.1073218
Moita, M. A. P., Rosis, S., Zhou, Y., LeDoux, J. E., and Blair, H. T. (2004). Putting fear in its place: remapping of hippocampal place cells during fear conditioning. J. Neurosci. 24, 7015–7023. doi: 10.1523/JNEUROSCI.5492-03.2004
Müller, N., Campbell, S., Nonaka, M., Rost, T. M., Pipa, G., Konrad, B. N., et al. (2018). 2D:4D and spatial abilities: from rats to humans. Neurobiol. Learn. Mem. 151, 85–87. doi: 10.1016/j.nlm.2018.04.012
Persson, J., Herlitz, A., Engman, J., Morell, A., Sjölie, D., Wikström, J., et al. (2013). Remembering our origin: gender differences in spatial memory are reflected in gender differences in hippocampal lateralization. Behav. Brain Res. 256, 219–228. doi: 10.1016/j.bbr.2013.07.050
Pine, D. S., Grun, J., Maguire, E. A., Burgess, N., Zarahn, E., Koda, V., et al. (2002). Neurodevelopmental aspects of spatial navigation: a virtual reality fMRI study. Neuroimage 15, 396–406. doi: 10.1006/nimg.2001.0988
Plancher, G., Tirard, A., Gyselinck, V., Nicolas, S., and Piolino, P. (2012). Using virtual reality to characterize episodic memory profiles in amnestic mild cognitive impairment and Alzheimer’s disease: influence of active and passive encoding. Neuropsychologia 50, 592–602. doi: 10.1016/j.neuropsychologia.2011.12.013
Rauchs, G., Orban, P., Schmidt, C., Albouy, G., Balteau, E., Degueldre, C., et al. (2008). Sleep modulates the neural substrates of both spatial and contextual memory consolidation. PLoS One 3:e2949. doi: 10.1371/journal.pone.0002949
Ravassard, P., Kees, A., Willers, B., Ho, D., Aharoni, D., Cushman, J., et al. (2013). Multisensory control of hippocampal spatiotemporal selectivity. Science 340, 1342–1346. doi: 10.1126/science.1232655
Rodriguez, P. F. (2010a). Human navigation that requires calculating heading vectors recruits parietal cortex in a virtual and visually sparse water maze task in fMRI. Behav. Neurosci. 124, 532–540. doi: 10.1037/a0020231
Ruddle, R. A., Payne, S. J., and Jones, D. M. (1997). Navigating buildings in “desk-top” virtual environments: experimental investigations using extended navigational experience. J. Exp. Psychol. Appl. 3, 143–159. doi: 10.1037/1076-898X.3.2.143
Sharples, S., Cobb, S., Moody, A., and Wilson, J. R. (2008). Virtual reality induced symptoms and effects (VRISE): comparison of head mounted display (HMD), desktop and projection display systems. Displays 29, 58–69. doi: 10.1016/j.displa.2007.09.005
Shine, J. P., Valdés-Herrera, J. P., Hegarty, M., and Wolbers, T. (2016). The human retrosplenial cortex and thalamus code head direction in a global reference frame. J. Neurosci. 36, 6371–6381. doi: 10.1523/JNEUROSCI.1268-15.2016
Slobounov, S. M., Zhang, K., Pennell, D., Ray, W., Johnson, B., and Sebastianelli, W. (2010). Functional abnormalities in normally appearing athletes following mild traumatic brain injury: a functional MRI study. Exp. Brain Res. 202, 341–354. doi: 10.1007/s00221-009-2141-6
Steemers, B., Vicente-Grabovetsky, A., Barry, C., Smulders, P., Schröder, T. N., Burgess, N., et al. (2016). Hippocampal attractor dynamics predict memory-based decision making. Curr. Biol. 26, 1750–1757. doi: 10.1016/j.cub.2016.04.063
Stokes, J., Kyle, C., and Ekstrom, A. D. (2015). Complementary roles of human hippocampal subfields in differentiation and integration of spatial context. J. Cogn. Neurosci. 27, 546–559. doi: 10.1162/jocn_a_00736
Sulpizio, V., Committeri, G., Lambrey, S., Berthoz, A., and Galati, G. (2016). Role of the human retrosplenial cortex/parieto-occipital sulcus in perspective priming. Neuroimage 125, 108–119. doi: 10.1016/j.neuroimage.2015.10.040
Suthana, N., Ekstrom, A., Moshirvaziri, S., Knowlton, B., and Bookheimer, S. (2011). Dissociations within human hippocampal subregions during encoding and retrieval of spatial information. Hippocampus 21, 694–701. doi: 10.1002/hipo.20833
Suthana, N. A., Ekstrom, A. D., Moshirvaziri, S., Knowlton, B., and Bookheimer, S. Y. (2009). Human hippocampal CA1 involvement during allocentric encoding of spatial information. J. Neurosci. 29, 10512–10519. doi: 10.1523/JNEUROSCI.0621-09.2009
Thomas, K. G., Hsu, M., Laurance, H. E., Nadel, L., and Jacobs, W. J. (2001). Place learning in virtual space. III: investigation of spatial navigation training procedures and their application to fMRI and clinical neuropsychology. Behav. Res. Methods Instr. Comput. 33, 21–37. doi: 10.3758/BF03195344
Tröger, C., Ewald, H., Glotzbach, E., Pauli, P., and Mühlberger, A. (2012). Does pre-exposure inhibit fear context conditioning? A virtual reality study. J. Neural Transm. 119, 709–719. doi: 10.1007/s00702-011-0757-8
van Bennekom, M. J., de Koning, P. P., and Denys, D. (2017). Virtual reality objectifies the diagnosis of psychiatric disorders: a literature review. Front. Psychiatry 8:163. doi: 10.3389/fpsyt.2017.00163
Wegman, J., and Janzen, G. (2011). Neural encoding of objects relevant for navigation and resting state correlations with navigational ability. J. Cogn. Neurosci. 23, 3841–3854. doi: 10.1162/jocn_a_00081
Wilkins, L. K., Girard, T. A., Herdman, K. A., Christensen, B. K., King, J., Kiang, M., et al. (2017). Hippocampal activation and memory performance in schizophrenia depend on strategy use in a virtual maze. Psychiatry Res. 268, 1–8. doi: 10.1016/j.pscychresns.2017.07.007
Wolbers, T., Hegarty, M., Büchel, C., and Loomis, J. M. (2008). Spatial updating: how the brain keeps track of changing object locations during observer motion. Nat. Neurosci. 11, 1223–1230. doi: 10.1038/nn.2189
Wolbers, T., Weiller, C., and Buchel, C. (2004). Neural foundations of emerging route knowledge in complex spatial environments. Brain Res. Cogn. Brain Res. 21, 401–411. doi: 10.1016/j.cogbrainres.2004.06.013
Wolbers, T., Wiener, J. M., Mallot, H. A., and Büchel, C. (2007). Differential recruitment of the hippocampus, medial prefrontal cortex, and the human motion complex during path integration in humans. J. Neurosci. 27, 9408–9416. doi: 10.1523/JNEUROSCI.2146-07.2007
Wong, C. W., Olafsson, V., Plank, M., Snider, J., Halgren, E., Poizner, H., et al. (2014). Resting-state fMRI activity predicts unsupervised learning and memory in an immersive virtual reality environment. PLoS One 9:e109622. doi: 10.1371/journal.pone.0109622
Woolley, D. G., Mantini, D., Coxon, J. P., D’Hooge, R., Swinnen, S. P., and Wenderoth, N. (2015). Virtual water maze learning in human increases functional connectivity between posterior hippocampus and dorsal caudate. Hum. Brain Mapp. 36, 1265–1277. doi: 10.1002/hbm.22700
Xu, J., Evensmoen, H. R., Lehn, H., Pintzka, C. W. S., and Haberg, A. K. (2010). Persistent posterior and transient anterior medial temporal lobe activity during navigation. Neuroimage 52, 1654–1666. doi: 10.1016/j.neuroimage.2010.05.074
Keywords: functional magnetic resonance imaging (fMRI), virtual reality (VR), memory, ecological validity, context
Citation: Reggente N, Essoe JK-Y, Aghajan ZM, Tavakoli AV, McGuire JF, Suthana NA and Rissman J (2018) Enhancing the Ecological Validity of fMRI Memory Research Using Virtual Reality. Front. Neurosci. 12:408. doi: 10.3389/fnins.2018.00408
Received: 01 March 2018; Accepted: 25 May 2018;
Published: 15 June 2018.
Edited by:Emiliano Macaluso, Claude Bernard University Lyon 1, France
Reviewed by:Martha Reeves Forloines, University of California, Davis, United States
Chris Bird, University of Sussex, United Kingdom
Copyright © 2018 Reggente, Essoe, Aghajan, Tavakoli, McGuire, Suthana and Rissman. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
*Correspondence: Nicco Reggente, firstname.lastname@example.org