This article was submitted to Technologies for VR, a section of the journal Frontiers in Virtual Reality
This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
This study identifies that increasing the fidelity of terrain representation does not necessarily increase overall understanding of the terrain in a simulated mission planning environment using the Battlefield Visualization and Interaction software (BVI; formerly known as ARES (M. W. Boyce et al., International Conference on Augmented Cognition, 2017, 411–422). Prior research by M. Boyce et al. (Military Psychology, 2019, 31(1), 45–59) compared human performance on a flat surface (tablet) versus topographically-shaped surface (BVI on a sand table integrated with top-down projection). Their results demonstrated that the topographically-shaped surface increased the perceived usability of the interface and reduced cognitive load relative to the flat interface, but did not affect overall task performance (i.e., accuracy and response time). The present study extends this work by adding BVI onto a Microsoft HoloLens™. A sample of 72 United States Military Academy cadets used BVI on three different technologies: a tablet, a sand table (a projection-based display onto a military sand table), and on the HoloLens™ in a within-subjects design. Participants answered questions regarding military tactics in the context of conducting an attack in complex terrain. While prior research (Dixon et al., Display Technologies and Applications for Defense, Security, and Avionics III, 2009, 7327) suggested that the full 3D visualization used by the Hololens™ should improve performance relative to the sand table and tablet, our results demonstrated that the HoloLens™ performed relatively worse than the other modalities in accuracy, response time, cognitive load, and usability. Implications and limitations of this work will be discussed.
Military planning is a time and resource intensive process; there currently exists a bottleneck in the soldier being able to consume, analyze, and communicate the myriad multi-sensory sources of incoming information, which leads to information processing challenges that have been a part of research for decades (
Specific to training, a challenge is to simulate realistic operations in a controlled environment and be able to deploy these training exercises at scale to the operational force (
Extended Reality (XR) applications in the military exist in many fields, including aviation, wargaming, weapons training, and human agent teaming applications (
The Army has leveraged commercial XR technologies to support improved skill acquisition for complex environments. An immediate advantage to training with XR is the ability to effectively visualize diverse elements and techniques, evaluating threats/targets, gauging spatial proximity, and inferring information relevant to friendly or enemy ground and airspace units. Conversely, each of the components described above require the operator to maintain situational awareness of the battlespace that directly affects attention management, visual search, spatial cognition, aggregation of information, and working memory (
The Army has developed its own government-owned off the shelf architecture known as Battlespace Visualization and Interaction or BVI. BVI is a java-based platform agnostic mission planning tool that was first developed to support military sand table briefings and known as ARES (Augmented Reality Sandtable;
Sample applications of BVI.
Prior research has examined the appropriateness of display type to task (
The research using ARES/BVI software began in 2015–2016 to investigate the role of displays on training tactics a line of research was developed starting in a pilot study to provide empirical data. A pilot study (N = 19 ROTC Cadets) compared a single projected terrain onto a military sand table to a flat surface display. Results indicated self-assessment manikin scores pre- and post- experiment split between conditions indicated a significant difference for participants in the non flat condition on the arousal and dominance scales, indicating higher interest and feeling of control. There was also a marginal trend towards improved post test performance (
A follow-on study was ran in 2017–2018 to see if terrain type or style had an effect (
The objective of this iteration was to understand how the addition of a mixed reality device would influence military tactics assessment in terms of performance usability and cognitive load. While the BVI sand table is similar to a 3D display in that it provides additional elevation (e.g., height) information as well as length and width, the Microsoft HoloLens™ provides an immersive 3D representation of the terrain features.
Using a military tactics training methodology inspired by (
The Above Represent the Visualizations That Each Participant Experiences: Tablet, Sand table, and HoloLens™.
We hypothesize that the present study will replicate the results of
Seventy-seven cadets at the United States Military Academy were awarded extra course credit for their participation. All participants had normal or corrected-to-normal vision. Participants were recruited via the SONA system. A sample size of 60 participants was required based on an
The present study utilized three different types of technologies. A Samsung Galaxy Tab tablet which is used for mission planning and can also be used for tactical decision games, scenario authoring, and classroom exercises (
The BVI Software was used to display the scenarios to the participants. For each block, in a given trial participants use one of the technologies (tablet, sand table, or Microsoft HoloLens™) to answer multiple choice questions on a tactical scenario (see
Sample scenario of an operational attack.
Each participant sees four scenarios from which they have to coordinate conducting an attack: a training mission plus one scenario per technology. All participants were trained
The NASA Task Load Index (NASA-TLX) is a multidimensional scale for measuring workload consisting of six sub-scales representing independent variable clusters: effort, frustration, mental demand, performance, physical demand, and temporal demand (
The System Usability Scale (SUS) uses a Likert scale format consisting of 10 questions that range with five responses from “strongly agree” to “strongly disagree” (
This experiment was conducted in the laboratory. Participants read, agreed to, and signed the informed consent document. Once consenting it was explained how they would be working with three different technologies (tablet, sand table, and HoloLens™) and answering questions associated with each. The participant was also told that they would be giving their answers verbally while using the technologies and then answering surveys (NASA-TLX and SUS) about the technologies in between using a computer running the surveys in the experiment room
Technologies from User’s perspective.
To ensure that the participant understood the task a they first participated in a training round, where they could ask questions. It was explained that they were to take the role of a commander who has been told by his leadership to make decisions about conducting a platoon-level attack based on the information provided to them. The task of conducting an attack was specifically chosen since West Point cadets will assume the role of a platoon leader when they are out in the Army. As a part of their West Point training, they receive platoon leader instruction which makes this task both practical and important to support cadet learning. In an attack, friendly forces seek to move and shape the direction of enemy forces (called control measures) such that the enemy is at a disadvantage and can be defeated or destroyed (
Once training was complete and the participant felt comfortable they would begin the first of three experimental conditions
After each set of 9 questions and surveys they would move to the next technology. The participants were told that the whole process would take around an hour, and they were debriefed at the end.
BVI Tablet: the User Experience(UX) with the tablet consisted of viewing the map, as well as having the ability to pan and zoom to see the tactical picture. The tablet was held in whatever position the participant felt comfortable, usually resting on their lap from a seated position. When a participant was working with the tablet, the research team ensured that the other conditions were not powered on to avoid using one technology to help answer questions from another.
BVI Sandtable: the sand table condition consisted of an experience of the participant standing in front of a downward projected topographic map. Participants were allowed to move physically around the table as they answered the questions. There was no physical interaction with the table, it was solely a viewing task.
BVI HoloLens: the HoloLens condition was a standard Microsoft HoloLens (first edition) with a custom app which was written in Unity, displaying the terrain with elevation in addition to having tactical graphics attached to the terrain. The location of the terrain was pre-set by the research team before allowing the participant to view it. The participant was allowed to use zoom and pan gestures to move around the terrain and also had freedom of movement to which they could change their perspective (crouching down, moving to a 90°angle).
Two separate 3-way (display type: tablet, sand table, HoloLens™) repeated measure ANOVAs were conducted for accuracy (i.e., proportion correct on multiple-choice questions) and response time (in ms). When Mauchley’s test of sphericity was violated, Greenhouse-Geisser adjusted values were reported. All post-hoc analyses are reported with pairwise Bonferroni-adjusted values. For Usability and Workload measures, Related-Samples Friedman’s Two-Way Analysis of Variance by Ranks was used because the Likert data was ordinal and non-normally distributed.
The main effect of display type on accuracy was significant with a small effect size F(2, 142) = 3.84, MSE = 0.031,
The main effect of display type on response time was significant with a moderate effect size, F(1.75, 123.99) = 18.93, MSE = 6.70,
System usability was statistically significantly different using the different technology,
Global Workload was statistically significantly different using the different technology,
The goal of this study was to determine whether the fully 3D immersive interface of the Microsoft HoloLens™ would relatively improve performance and reduce workload compared to the topographic 2.5D sand table and the flat 2D tablet. Surprisingly, our hypothesis was not supported. Using the HoloLens™ led to consistently worse performance and workload measures when compared with the sand table and tablet. Performance and workload measures between the sand table and tablet were comparable and consistent with
The fact that participants took significantly longer to respond to the questions (and were less accurate) solely for the HoloLens™ is telling. Despite receiving some task training, the lack of familiarity with the technology could have impacted their performance. It may also be the case that the more immersive environment, while higher fidelity, required more effort to process the environment. This was seen in the relatively higher workload for the HoloLens™ when compared with the sand table and tablet. While participants may have engaged with more exploratory learning, the current software was not designed to assess any exploration metrics or measures such as gaze tracking. Practically, in a military environment it would be challenging to allow enough time to achieve a desired level of exploration due to the time constraints of the participants.
The usability scores (as assessed by medians) were the same for both the tablet and the sand table, which is not consistent with the results from our previous study (
Overall, this study identifies that moving from a topological 2.5D display to an immersive 3D display does not necessarily confer any benefits without requiring additional training. Perhaps if the immersion would have been equivalent (e.g., fully holographic) then the participants would not have had the additional physical requirements of wearing the headset, which may have confounded how usable they found the actual image. Further research is required to tease apart how much the physical technology of the HoloLens™ confounded with participants experiencing of the scenario terrain, and with additional familiarity, would the HoloLens™ be a relatively more effective training tool (in both performance and portability) when compared with the BVI sand table.
This study has a few limitations that could be addressed in future research. Perhaps the largest issue is that participants were only familiarized with the technology and were not trained to a common level of basic proficiency. If the participants were instead allowed to train and test to a certain level of proficiency with each of the technologies then we could be sure that it is differences based on the person rather than differences on the technology. A second limitation is that the HoloLens ™ was a an early research build. Since this study, the HoloLens 2™ was released which is lighter, more powerful, and has a broader field of view.
A limitation of our population was that it was drawn from the freshman population at a Military Academy. The content was tailored to the ability levels of those cadets by military subject matter experts, but this knowledge was not the same as that of the greater operational force. Though academically that sounds compelling, in the Active Duty Army there are a wide range of skills (military occupational specialities), roles, and experience level all of which impact the nature of the content presented. It may not be the case that there is a “one size fits all” solution for training these disparate roles.
The present study provided evidence that the Microsoft HoloLens™ is not as effective a training device as the BVI sand table or tablet, with the caveat that more specialized training may be required to compensate for the lack of familiarity and physical limitations of the HoloLens™ itself. The impetus for this work was the Army’s Synthetic Training Environment Cross Functional Team’s emphasis on using technologies like HoloLens™ for training. As the Department of Defense focuses on technology as a means of addressing the future of Joint Force and Multi Domain Operations (e.g., air, land, maritime, space, and cyberspace) and training, research focused on technologies’ impact on being able to scale training using XR to reduce the need for large-scale physical training areas, allowing for operational training which could be taken on-site. Similar to what this research has found, recently released was the news that the Army’s Integrated Visualization Augmentation System (IVAS) is being delayed for roll out for army training due to challenges in technology capabilities (
The raw data supporting the conclusion of this article will be made available by the authors, without undue reservation.
The studies involving human participants were reviewed and approved by United States Army Research Laboratory Institutional Review Board. The patients/participants provided their written informed consent to participate in this study.
MB—Research Conceptualization and Design, Protocol Writing, Data Collection, Statistical Analysis, and Manuscript Preparation RT, JC—Study Coordination, Statistical Analysis, Manuscript Preparation and Review DF—Study conceptualization and content development, IRB and manuscript writing, data analysis CS—Study Conceptualization, Data Collection, Manuscript Preparation and Review, Research Coordination JF—Manuscript Preparation, statistical analysis review. CA, JE—Data collection and initial writeups to support manuscript development—built study materials and created content. ER, CA—Study Conceptualization, coordination and facilitation support, manuscript and data analysis review.
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.
1. Which is the best location for a possible LZ? 2. What key terrain in enemy territory is most likely Between 3. What is the enemy’s most deadly course of action? 4. What is the best position for a blue force sniper? 5. What is the best location for CCP? 6. If comms don’t work on objective where is the best location to re-establish comms? 7. Where is the best location to cross the LDA on 8. Where is the best location for reconsolidation/PZ after the mission? 9. Where is the best location for fires plan to suppress the enemy?
As a reminder, participants were military academy cadets with specific training in viewing these kinds of scenarios. Questions were developed in consultation with Army Officers.