ORIGINAL RESEARCH article

Front. Virtual Real., 26 November 2020

Sec. Virtual Reality in Medicine

Volume 1 - 2020 | https://doi.org/10.3389/frvir.2020.528810

Assessing the Value of a Novel Augmented Reality Application for Presurgical Planning in Adolescent Elbow Fractures

  • 1. Department of Radiology and Biomedical Imaging, University of California, San Francisco, San Francisco, CA, United States

  • 2. Division of Pediatric Orthopedic Surgery, Department of Orthopedic Surgery, Benioff Children's Hospital, University of California, San Francisco, San Francisco, CA, United States

  • 3. Division of Pediatric Orthopedic Surgery, Department of Orthopedic Surgery, Benioff Children's Hospital-Oakland, University of California, San Francisco, Oakland, CA, United States

  • 4. Department of Radiology and Biomedical Imaging, Benioff Children's Hospital, University of California, San Francisco, San Francisco, CA, United States

Abstract

Objectives: We retrospectively assess the potential impact of a novel, investigational Augmented Reality (AR) software application, Radiology with Holographic Augmentation (RadHA), on pediatric orthopedic surgeon's confidence in surgical planning, hardware selection, hardware fit, and estimated potential intraoperative time savings in the setting of complex adolescent elbow fractures.

Methods: After study selection, 12 individual cases of complex elbow fractures in adolescent pediatric patients were identified for review. AR models were generated for each case derived from the patient's CT. Five fellowship-trained pediatric orthopedic surgeons reviewed each case for a total of 60 separate observations. Surgeons reviewed clinical data, radiologic imaging, and AR models and then answered Likert Scale questions on measures of confidence in presurgical planning and projected potential time savings. These data were reviewed and analyzed using various statistical tools.

Results: Surgeons reported high confidence in the quality of the AR models created. Additionally, surgeons reported increased confidence in their surgical plan, increased confidence in hardware selection, and increased confidence in hardware fit. Within the sub-analysis of complex (comminuted) fractures, surgeons reported greater expected increases in confidence of their surgical plan and hardware fit. Overall, surgeons estimated potential intraoperative time savings, averaging 17.3 min for all fracture types and 17.6 min for complex fractures.

Conclusions: Preoperative planning using AR-based models can increase surgeon confidence in preoperative planning, hardware selection, and confidence in hardware fit.

Highlights

  • – Evaluated the utility of AR holograms in pre-operative planning within the use of case of pediatric elbow fractures.

  • – Characterized the potential impact of AR on key measures, such as surgeon confidence and potential operative time savings.

Introduction

Repair of complex fractures in the pediatric orthopedic patient can be challenging, as often the exact type and approach of repair is dependent on the degree, pattern, and relationships of osseous fragmentation (Houshian et al., ; Abzug and Herman, ). Distal humeral fracture patterns, in particular, differ in adolescent patients, who often demonstrate a more complex “adult-type” fracture, compared to juveniles, who are more likely to have a straightforward supracondylar, lateral condyle, or medial epicondyle fracture pattern, all of which have standard treatment algorithms. These “adult-type” adolescent distal humerus fractures may require open reduction and internal fixation with plating, possible olecranon osteotomy; more typical juvenile patterns often require no more than closed reduction and pin fixation (Handelsman et al., ; Howard et al., ). For complex intra-articular cases, cross-sectional imaging plays an integral role in preoperative workup, including evaluating fracture patterns, determining a specific surgical approach, and selecting appropriate hardware (Koh et al., ; Park et al., ). The addition of three-dimensional (3D) imaging has been shown to be of potential benefit in other areas of orthopedic surgical planning (Wicky et al., ; Schepers et al., ; Fotouhi et al., ). However, conventional 3D reconstructions displayed on a two-dimensional flat screen may not always provide sufficient demonstration of complex, comminuted fractures and the relationships of various fracture fragments (Atesok and Schemitsch, ). These reformations are typically generated by the CT technologist and are limited to arbitrary planes of rotation, often not demonstrating the relevant plane the orthopedic surgeon requires for operative planning. Furthermore, these reconstructions are typically performed at the time of acquisition, and generating post-hoc reconstructions can be challenging and cumbersome.

Interventions that can help reduce operative time and increase surgeon confidence are of utmost importance in the pediatric patient. Within healthcare, a variety of different technologies have been developed to help physicians leverage patient-specific imaging (Wicky et al., ; Schepers et al., ). These include the utilization of 3D reconstructions that are usually provided at the time of image acquisition, 3D printing, and digital technologies such as Virtual Reality (VR), Augmented Reality (AR), and other types of Mixed Reality (MR) visualizations, where digital information is displayed on varying levels of digital backgrounds. Augmented Reality has been utilized in a variety of different applications, including military, industry, and increasingly, within healthcare (Fotouhi et al., ; Wake et al., ). Many of these applications leverage AR's ability to superimpose digital images on the physical world and use this superimposition to help users achieve specific tasks. AR applications have the potential to enhance visualization of complex medical imaging and add higher degrees of interactivity than conventional 3D reformations on two-dimensional flat screens (Vávra et al., ). Improved visualization of complex imaging can potentially be beneficial in reducing operating room times and complication rates and increasing physician confidence prior to performing an operation, specifically by improving user understanding and awareness of complex anatomy that can sometimes be lost in translation with current visualization standards (Oishi et al., ; Hassani et al., ; Schepers et al., ). Cost savings in terms of more-efficient use of hardware has also been demonstrated using digital templating software, where preoperative planning can help in anticipating appropriate hardware needs specific to a patient, and prior to the intervention, preventing the need to open and potentially waste numerous mal-sized hardware during a procedure (Hsu et al., ).

These improvements in visualization can become even more important in the pediatric patient, where reductions in OR time, length of anesthesia, and long-term sequelae of repair can have significant consequences (Mehlman et al., ; Ramachandran et al., ; Howard et al., ). It is critical to utilize available tools to try and limit time under anesthesia, particularly in pediatric patients, who may still be undergoing cognitive development. While time for precise characterization and surgical planning is available in non-emergent surgical cases, this time is finite and can be shortened if users are provided with the appropriate tools and imaging to best plan for a procedure (Leet et al., ). In many instances, given the precise nature of complex repairs, having an accurate 3D representation of the underlying anatomy is critical.

Previous work with quality control phantoms have demonstrated a high degree of accuracy and precision of AR models in comparison with gold standard measurements (, ). Early preliminary work has demonstrated preplanning with 3D models has the potential to decrease operating room times (Jacobs et al., ; Zhang et al., ; Lethaus et al., ) and improve accuracy in hardware placement (Teber et al., ; Elmi-Terander et al., ; Zhang et al., ).

In this study, we retrospectively assess the potential impact of a novel AR software application, Radiology with Holographic Augmentation (RadHA), on pediatric orthopedic surgeons' confidence in surgical planning, hardware selection, hardware fit, and estimated potential intraoperative time savings in the setting of complex adolescent elbow fractures.

Materials/Methods

Approval for the study was obtained through our institutional review board. A total of 12 cases of intra-articular elbow fractures of varying levels of complexity in pediatric patients 10–18 years of age were identified through retrospective review using specified search criteria (Figure 1). Search criteria included patients <18 years of age, undergoing CT, with reports including “comminuted,” “intra-articular,” and “elbow.” Digital Imaging and Communications in Medicine (DICOM) images (Figure 2) were converted to 3D.stl mesh files using OSIRIX MD (Pixmeo SARL, Bernex Switzerland) and optimized for AR using Blender (Stichting Blender Foundation, Amsterdam, Netherlands) (Figure 3). The 3D images were adapted (Figures 4A,B) and loaded onto the Microsoft HoloLens (Microsoft Corporation, Redmond, WA, USA) platform using a novel, investigational software application written in C# programming language-based code on the Unity Platform (Unity Technologies, San Francisco, CA, USA), which was developed at our institution. Abstracted pertinent clinical information (Table 1) and conventional CT DICOM data of the 12 models were reviewed by five separate fellowship-trained attending pediatric orthopedic surgeons (averaging 4.6 years post-fellowship completion, ranging between 2 and 8 years). Subsequently, the same surgeons evaluated an AR model derived from the same CT DICOM data. Surgeons then completed a survey with multiple Likert scale questions regarding use of AR models for planning (Table 2), and included their own subjective estimation of time savings based on use of these AR models (Figure 5 and Table 3). A one sample, two-tailed t test was performed to compare the mean Likert responses to a null hypothesis of no significant change upon review of AR models (Likert = 3) among all and comminuted fracture types (Table 4), with all responses treated as independent measures. A Kruskal–Wallis test was performed to compare differences between Likert responses among surgeons (Table 5), given that our collected ordinal data did not have a normal distribution, both among all and comminuted fracture patterns.

Figure 1

Figure 2

Figure 3

Figure 4

Table 1

PatientFracture typeMechanismPatient age (years)Patient weight (kg)Patient gender
1Sequela of prior elbow dislocation with anteriorly displaced medial condyle avulsion fracture and small impaction fracture at the posterior aspect of the lateral condyleFall on an outstretched hand1278.5Female
2Left distal humeral lateral condylar fracture with superior displacement of the capitellumFall on an outstretched hand1240Female
3Supracondylar fracture with posterior displacement and disruption of the anterior and posterior cortexFall onto left elbow, axial load, swing1455Male
4Comminuted bicondylar intra-articular distal humerus fracture with ventral subluxation of the capitellumFall onto outstretched left arm1350.1Male
5Mildly displaced intra-articular fracture through the left olecranon at its trochlear notchAxial load1138.6Male
6Comminuted intra-articular fracture of the distal left humerus with intra-articular fragmentsMotor vehicle collision1659Male
7Intra-articular olecranon fractureFall1456.7Male
8Comminuted olecranon fractureFall onto elbow14UnavailableFemale
9Proximal ulnar fracture with intra-articular extensionFall14UnavailableMale
10Capitellar fractureunavailable13UnavailableMale
11Comminuted supracondylar fracture of the distal left humerusFall16UnavailableMale
12Intra-articular supracondylarunavailable17UnavailableFemale

Abstracted patient clinical information.

Table 2

1. Does this model accurately display the bone structures as portrayed by the CT dataset?
Mark only one oval.
2. How would review of this model before surgery change your surgical plan?
Mark only one oval.
Negatively impact my plan (5)
Minor negative deviation in my plan (4)
No impact on my surgical plan (3)
Minor improvement in my surgical plan (2)
Major positive change in my surgical plan (1)
3. Would review of this model before surgery improve your confidence in your surgical plan?
Mark only one oval.
Negatively affect my confidence (5)
Minor negative affect in my confidence (4)
No impact on my confidence in my surgical plan (3)
Minor improvement in confidence in my surgical plan (2)
Major improvement in confidence in my surgical plan (1)
4. Would review of this model before surgery improve your confidence in your hardware selection?
Mark only one oval.
Negatively affect my confidence (5)
Minor negative affect in my confidence (4)
No impact on my confidence in my hardware selection (3)
Minor improvement in confidence in my hardware selection (2)
Major improvement in confidence in my hardware selection (1)
5. Would review of this model before surgery improve your confidence in your hardware fit?
Mark only one oval.
Negatively affect my confidence (5)
Minor negative affect in my confidence (4)
No impact on my confidence in hardware fit (3)
Minor improvement in confidence in hardware fit (2)
Major improvement in confidence in hardware fit (1)
6. Is review of A.R. models useful for presurgical planning?
Mark only one oval.
Very unhelpful (5)
Mildly unhelpful
Neither helpful nor unhelpful Mildly helpful
extremely helpful (1)
7. If this would change time, how much time do you estimate (in minutes)? (use + for time added in the O.R. and - for time reduced in the O.R.)

Survey questions posed to orthopedic surgeon raters following Augmented Reality (AR) model review.

Figure 5

Table 3

All elbow fracture types17.3 min
Complex (comminuted) fractures17.6 min

Average time savings estimated for all and complex fracture types.

Table 4

Survey questionAll fracturesComminuted fractures
P-valueP-value
(degrees of freedom)(degrees of freedom)
Matching reality<0.0001 (59)<0.0001 (29)
Change in surgical plan<0.0001 (59)<0.0001 (29)
Confidence with surgical plan<0.0001 (59)<0.0001 (29)
Confidence with hardware selection<0.0001 (50)<0.0001 (29)
Confidence in Hardware Fit<0.0001 (49)<0.0001 (29)
Utility for Planning<0.0001 (58)<0.0001 (29)

One sample two-tailed t test evaluating improvement of Augmented Reality (AR) model in Likert survey questions for all fractures, comparing average responses to a neutral change.

Table 5

Survey questionAll fracturesComminuted
P-valuefractures
Matching reality0.14720.0567
Change in surgical plan0.9830.4771
Confidence with surgical plan0.39370.6774
Confidence with hardware selection0.66340.4281
Confidence in hardware fit0.36690.0659
Utility for planning0.576230.3552

Kruskal–Wallis evaluating for significant difference between reviewers for all and comminuted fracture types.

Results

Of the 25 total reports found using the specified search criteria, 13 unique patients were present. A single study was excluded on the basis of absence of acute imaging. A total of 12 patients were therefore included in the study [average age, 13.8 (range, 11–17), 8:4 M/F, mean weight = 54 kg]. Further clinical details on each case is provided in Table 1.

Summary of responses to survey questions (Figures 6A,B) and summary of responses separated by reviewer (Figures 7A,B) are provided. Surgeons rated models “highly accurate” (median = 1, mode = 1, interquartile range = 0). Raters noted no overall change in surgical plan (median = 3, mode = 3, interquartile range = 1). Raters noted improved confidence in surgical plan (median = 2, mode = 1, interquartile range = 1). Confidence scores for hardware selection were on average mildly increased (median = 2, mode = 2, interquartile range = 2), as were confidence scores in hardware fit (median = 2, mode = 2, interquartile range = 1), and utility for planning for specific fracture type (median = 2, mode = 2, interquartile range = 1). Four of the five orthopedic surgeons felt comfortable providing time estimations (the surgeon who most recently completed training abstained, Reviewer 3). Overall average estimated intraoperative time savings for all fracture types = 17.3 min (Table 3). When asked to clarify/expand on how AR models would impact case or affect time estimates, surgeons responded with “better understanding of alignment/fracture read,” “better understanding of surgical approach,” “can see extent better,” “retractor positioning,” and “better understanding of relationships of fragments” (Table 6). A Kruskal–Wallis test was performed to evaluate if there was any significant difference among between different raters and their survey responses. Overall, we did not find a statistical difference between respondents, both in the case of all fractures and in the case of comminuted fracture types (Table 5). This is also graphically depicted (Figures 7A,B), where mean Likert responses and 95% confidence intervals are displayed between reviewers, visually confirming the Kruskal–Wallis test results. Additionally, a t-test (Table 4) allowed us to reject our null hypothesis that the review of AR models would result in no difference among surgeons preparing for pediatric elbow fracture repair.

Figure 6

Figure 7

Table 6

–Can see the comminution better and where the fragments are located better than on CT
–Help with decision for tension band vs. screw; helps you decide the size and the obliquity for that decision
–Useful for hardware fit; would use a plate
–Would go both medial and lateral; would have done just lateral, but can see there is another fragment on the model
–Would have changed from pinning to open; it makes the fracture clearer
–Would change her “read” of the fracture; would change from anterior to superior approach; might lengthen surgery, but would improve overall outcome
–Would change the how I'd approach; looking at this makes it more likely I'd have recognized the need for bone grafting; allograft, decide how to approach first
–Shows where you will look for your fracture reduction read; looking for the apex of a triangle, shows that nicely.
–Helpful to envision where your fracture reads are going to be, what you will look for to line up; where your plates will fit
–Complex intra-articular with a bad CT, gives you more info on how it will fit together, relationships of fragments and where you will get your fracture “read”
–Helping understanding for planning
–What I didn't appreciate on the CT scan is the coronal alignment; don't find the flat screen 2D reformat; time savings in doing an arthrogram; or doing a fluoroscopy to understand the alignment; limited movement on the 3D flat screen
–Saving time in the planning portion; doesn't know that fracture
–Give me more confidence on which hardware, more intuitive to look at this than the CT
–Could see the little displaced articular/posterior fragment better in the model than CT. Can't appreciate it on the X-ray, very hard to see. Have to be looking for it. Focused on the lateral condylar part. Use this for education
–“This is awesome” just really like it!
–Would cut out of the fluoroscopy element; so don't have to do that; and potential need for arthrogram
–Saves time in review ad in the OR; cool
–Obvious for this type
–Saves time in review of the model; probably best in distal humeral
–Would find helpful if you can expand it out; an expand feature; would like to reduce the fracture; if I could reduce the pieces, then plan out where hardware needs to go
–Would like better quality, a little spiculated (likely low dose technique)
–Good example of a case where it would be helpful to separate the fragments and piece it back together. Make each fracture component piece a distinct piece. Fracture bone (humerus), each piece independent to move, manipulate those pieces. Place hardware on to the fixed model. Once I've reconstructed the fracture, to put the place on. Going through the motions of the surgery via the model

Responses to “any other comments or clarification on how these AR models change your management or potentially help reduce operative time?”

When subanalyzed in the cases of complex fractures (fractures more than simple, non-displaced fractures, n = 6), surgeons rated: subjective assessment of model accuracy “highly accurate” = (scale 1–5, median = 1, mode = 1, interquartile range = 0); confidence in surgical plan “major increase” = (median = 1, mode = 1, interquartile range = 1); confidence in hardware selection “mild increase” = (median = 2, mode = 2, interquartile range = 1); confidence in hardware fit “mild increase” = (median = 2, mode = 2, interquartile range = 1). Utility for planning for specific fracture type was rated “high” = (median = 1, mode = 1, interquartile range = 1). In the cases of comminuted fracture types, average estimated intraoperative time savings was 17.6 min.

Discussion

In this study, we evaluated orthopedic surgeons' assessment of 3D AR models for presurgical planning in complex elbow fractures. Surgeons reported high confidence in the quality of the AR models created. Additionally, surgeons reported increased confidence in their surgical plan, increased confidence in hardware selection, and increased confidence in hardware fit. Within the subanalysis of comminuted fractures, there was a greater overall increase in surgeon confidence in surgical plan. Overall, there was a projected intraoperative time savings (averaging 17.3 min for all fracture types and 17.6 min for comminuted fractures). The most experienced of the surgeons interestingly estimated that the use of this technology could have up to a 30-min time reduction in the cases of complex fractures.

The improved 3D visual spatial awareness of the fracture fragments and fracture patterns, as cited by the surveyed surgeons, is similar with previously reported studies of 3D imaging use in orthopedics (Sasso and Garrido, ; Citak et al., ; Yu et al., ). This includes a study of 3D modeling for pelvic acetabular fractures, which allowed for improved hardware placement and reduced soft tissue trauma (Yu et al., ). Notably, reviewers in this study found that AR models provided additional value over standard 3D reformations through improved ability to interact with and freely manipulate the AR model (as opposed to the fixed horizontal and vertical spin directions of rotation provided on routine clinical imaging). Furthermore, the ability to interact with real world objects with the AR models was reported as an advantage, providing the potential opportunity for hardware sizing.

Time reduction of ~18 min per case in cases of complex fractures in our study, while an estimation, is in keeping with other studies of time reduction using 3D visualization (Sasso and Garrido, ; Citak et al., ). Physicians reported a variety of mechanisms for time reduction (Table 6). These included enhanced visualization of relevant anatomy, reducing total preoperative review time; prevention of unnecessary procedures during a case, such as an arthrogram or fluoroscopy to understand anatomy; precise hardware selection, reducing time spent deciding between hardware types; aid in appropriate surgical technique, either in approach or specific technique, potentially reducing intraprocedural time; and aids operator “read” of fracture pattern, possibly reducing time spent reviewing CT prior to surgery. Reported operating times for repair of complex distal humeral fractures in a study in the adult population averaged 2.5 h (Frankle et al., ). Furthermore, reported costs per minute of operating room time range from $36.50/min to $133.12, averaging $66/min (Shippert, ; Macario, ; Childers and Maggard-Gibbons, ). While an estimation, using the provided estimates of our study, this could represent a potential 12% overall decrease in time per case and a reduction in cost of $1,122 per case.

The time to segment, process, and optimize orthopedic models for this AR application is <5 min, as the processing is done electronically. This increases the potential for such applications to be used more broadly than alternative options such as 3D printing that require more resources and infrastructure. Further, the opportunity for interaction between physical orthopedic hardware and scale 3D holographic models provides a degree of interactivity that is beyond conventional 3D-rendered reformats displayed on a 2D monitor.

Our study was not without limitations. In addition to the retrospective nature of the study, our study was limited by the subjective assessment of projected time savings and by the differences in training and degree of experience by the surgeons reviewing models. Further, while the models available for review had limitations in degree of interactivity in that the fracture fragments on the tested version could not be manipulated or repositioned, this was a feature that reviewing surgeons stated would have been useful in planning. Within these limitations, we do believe that our study not only provides critical information on the improvement in surgeon confidence in plan and hardware choice but also provides an early preliminary window into the potential economic impact that this technology can provide, which may influence further adoption.

Preoperative planning using AR-based models statistically increases surgeon confidence in preoperative planning, hardware selection, and confidence in hardware fit. This study lays a foundation for further, larger-scale prospective trials assessing AR technology for planning in a wider variety of orthopedic fracture types. Additionally, further testing should be focused on the safety and barriers of using intraoperative AR systems for surgical guidance and reference.

Statements

Data availability statement

The datasets generated for this study are available on request to the corresponding author.

Ethics statement

This work was conducted with approval of our University's Institutional Review Board.

Author contributions

BL: manuscript preparation, statistical analysis, and figure legends. KL: study design, survey design input, performed measurements, and manuscript preparation. RB, JJ, NP, and CS: manuscript preparation and performed measurements. JC: study design, manuscript preparation, figure annotation, video editing, statistical analysis, and project oversight. All authors contributed to the article and approved the submitted version.

Funding

This work was funded by the Clinical and Translational Science Institute (CTSI) and UCSF Catalyst Program of Innovation Ventures Fund #5018, Project #2001449.

Acknowledgments

Preliminary findings of this work were presented at the 2018 Annual Meeting of the American Roentgen Ray Society, Washington, DC. Microsoft HoloLens v1 was used with permission from Microsoft (this equipment was purchased and owned by the BL). We thank Dr. Ivor Asztalos for statistical guidance.

Conflict of interest

BL and JC are founders of Sira Medical, pre-revenue Augmented Reality startup. BL is a co-founder of an AR company Sira Medical. The remaining authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Supplementary material

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/frvir.2020.528810/full#supplementary-material

References

Summary

Keywords

Hololens, radiology, DICOM, Augmented Reality (AR), computed tomography

Citation

Laguna B, Livingston K, Brar R, Jagodzinski J, Pandya N, Sabatini C and Courtier J (2020) Assessing the Value of a Novel Augmented Reality Application for Presurgical Planning in Adolescent Elbow Fractures. Front. Virtual Real. 1:528810. doi: 10.3389/frvir.2020.528810

Received

21 January 2020

Accepted

15 September 2020

Published

26 November 2020

Volume

1 - 2020

Edited by

Holger Thomas Regenbrecht, University of Otago, New Zealand

Reviewed by

Christopher M. Laine, University of Southern California, United States; Ali Fardinpour, Wise Realities Institute for Healthcare Emerging Technologies Research, Australia

Updates

Copyright

*Correspondence: Jesse Courtier

This article was submitted to Virtual Reality in Medicine, a section of the journal Frontiers in Virtual Reality

Disclaimer

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article or claim that may be made by its manufacturer is not guaranteed or endorsed by the publisher.

Outline

Figures

Cite article

Copy to clipboard


Export citation file


Share article

Article metrics