Mental workload during endoscopic sinus surgery is associated with surgeons’ skill levels

1. Introduction

Burnout among medical professionals has been recognized as “a crisis” in modern health care even before Covid-19 (1). A recent nation-wide study in the US demonstrated that 44.0% of physicians had experienced burnout at least once in their career (2). The cost of healthcare related to physician burnout is estimated at between 2.6 to 6.3 billion USD per year (3).

Identified as one of the associated risk factors of burnout in healthcare workers is a high mental workload, along with age, gender, practice setting, specialty, and hours of work per week (2). Mental workload is defined as the total cognitive work needed to accomplish a specific task (4). The National Aeronautics and Space Administration-Task Load Index (NASA-TLX) is designed to evaluate mental workload and widely accepted across many specialties including healthcare (4–7). NASA-TLX allows for evaluation not only of the overall scale of mental workload but also its subscales: mental demand, physical demand, temporal demand, perceived performance, effort, and frustration.

Among various medical procedures, surgery is particularly associated with a high mental and physical burden on surgeon (8–10). It is no wonder that endoscopic sinus surgery (ESS) burdens surgeons mentally because of the risk due to proximity to orbit and brain and requiring fine psychomotor skills (9, 11, 12). So far, however, only few studies focused on surgeons’ mental workload during ESS (9, 11, 12) have been published. One of the reasons is a lack of standardized conditions to evaluate the workload. In general, surgeon’s perceived mental workload is affected by the task’s difficulties (13). The difficulty of ESS largely depends on the complexity of paranasal sinus anatomy, which greatly varies in every patient. Thus, precise assessment of mental workload during ESS has been challenging.

Recently, we reported a high-fidelity ESS simulation surgery using newly designed 3D printed-sinus models, with sufficient face, content, and construct validity (14). With recent advanced 3D-printing technology and the high quality of the printing materials applied, the tactile “real-life” feel of the tissues is reproduced in the models. The mass producibility of the 3D-printed sinus models allows for multiple dissections by surgeons with the exact same anatomy, which is impossible in actual clinical situations. This provides the standardization for comparison among intra- and inter-individuals on several aspects of ESS such as efficiency, efficacy, and safety of surgeries (14, 15).

In this study, we examined surgeons’ mental workload during ESS using NASA-TLX under the standardized condition provided by the 3D-printed sinus models. Specifically, we focused on the possible association between the level of surgical skills and the mental workload. We also examined whether repetitive training could alleviate trainees’ mental workload. This is the first study examining surgeons’ mental workload during ESS under standardized conditions.

2. Materials and methods

2.1. Participants

This study was conducted concurrently with the previously published studies regarding the validation of 3D-printed sinus models for ESS training (14). Forty-seven otolaryngologists voluntarily took part in the study. Participants were explained the purpose and design of the present study in advance. The written informed consent was obtained from all participants.

2.2. Simulation surgeries

For simulation surgeries, a 4-mm rigid nasal endoscope and a monitor (Telepac, Storz, Tuttlingen, Germany), standard ESS instruments (Storz), and a powered microdebrider (Medtronic, Jacksonville, FL) and 3D-printed sinus models (Fusetec, Adelaide, South Australia) were prepared as previously reported (14). The models were 3D-printed from the axial CT scans of actual patients with chronic rhinosinusitis. Infrared reflective markers were attached to several surgical instruments for a motion capture study, although it was not the focus of the present study. The participants were allocated 45 min to complete a unilateral full house ESS (maxillary antrostomy, sphenoethmoidectomy, and frontal sinusotomy). The detail of the simulation surgeries was described in the previous study (14).

All participants performed the surgeries for the 3D-printed sinus models (Model 2 Right side). Further, 10 otolaryngology registrars among the participants repeated the exercise six times as part of repetitive training. In the 1st and the final training (2nd, and 7th surgeries in total, respectively), Model 2 Left was used for intra-individual comparison. Details of the simulation training have been previously reported (14).

2.3. Assessment of surgical skill levels

As there is no standard objective method to evaluate the ESS levels, the following index were used in this study; board certification by the Japanese Otolaryngology Society, the number of previous ESS cases performed and the objective structured assessment of technical skills (OSATS) score (16). The scoring system is designed to score surgical performance in each specific procedures during FESS on the 5-likert scale, from the scale of one (unable to perform) to five (performs easily with good flow). A score of three or more in each checklist is considered competent for the task. The OSATS score for ESS was assessed by two attending rhinologists (MS and YN). More details are provided in the previous study (14). Data on the board certification and the number of prior ESS cases were obtained from a questionnaire survey performed after the simulation surgeries. As the definition of ESS experts has not been established, in this study, experts were defined as certified surgeons both within the top 1/3 experienced cases and with top 1/3 OSATS scores. More details of the simulation training and the assessment were described in the previous report (14).

2.4. Mental workload assessment

National aeronautics and space administration-task load index (NASA-TLX) were utilized to evaluate mental workload assessment. NASA-TLX is the most widely utilized subjective questionnaire for mental workload assessment and consists of the following six subscales: mental demand, physical demand, temporal demand, perceived performance, effort, and frustration (4–7). After the simulation surgeries, the participants answered the following questions using a 20-point visual analog scale based on the six subscales.

• How much mental activity was required for the surgery? (Mental demand, 1: not at all, 20: extremely high)

• How much physical activity was required for the surgery? (Physical demand, 1: not at all, 20: extremely high)

• How much time pressure did you feel for the surgery? (Temporal demand, 1: not at all, 20: extremely high)

• How successful do you think you were in completing the surgery? (Perceived performance, 1: perfectly completed, 20: nothing achieved at all)

• How hard did you have to work during the surgery? (Effort, 1: not at all, 20: extremely high)

• How insecure, discouraged, irritated, stressed, and annoyed did you feel during the surgery? (Frustration, 1: not at all, 20: extremely high)

2.5. Analysis and statistics

Data were shown in median (interquartile range). Shapiro–Wilk tests were applied to evaluate whether the data fitted a normal distribution curve. For analysis on experienced cases, participants were classified into three groups according to the number of experienced cases (the top 1/3, the middle 1/3, and the bottom 1/3 of the experienced cases). As well, they were also classified into three groups according to their OSATS score (the top 1/3, the middle 1/3, and the bottom 1/3 of the score). The total NASA-TLX score were calculated as sum of the 6 subscales. For comparison among three or more groups, data were assessed with the Kruskal-Wallis test followed by the Mann–Whitney U. Principal component analysis (PCA) was performed using the number of experienced cases, OSATS score, and the subscales of NASA-TLX. The paired Wilcoxon test was utilized for comparison before and after the repetitive training. p values of less than 0.05 were considered statistically significant. For comparison among three groups or six subscales, the Bonferroni correction was applied and p values of less than 0.017 for three groups (i.e., 0.05/3 = 0.017), and 0.0033 for six subscales (i.e., 0.05/15 = 0.0033) were considered statistically significant, respectively. All the analyses were performed using JMP 11 (SAS Institute Inc.).

3. Results

3.1. Characteristics and the mental workload of the participants

Table 1 shows the characteristics of the 47 participants. Among them, 28 were board-certified members of the Japanese otolaryngology society, and the other 19 were otolaryngology registrars in training. The total NASA-TLX score was 62 (49–75). In the subscales of NASA-TLX, mental demand, physical demand, temporal demand, perceived performance, effort, and frustration were 7 (5–10), 8 (5–10), 16 (10–20), 10 (6–16), 15 (10–20), and 5 (4–10), respectively (Table 1). Temporal demand and effort were significantly higher than the other four subscales (Figure 1).

TABLE 1

Table 1. Characteristics of participants in the present study.

FIGURE 1

Figure 1. The subscales of mental task load evaluated with NASA-TLX. Forty-seven otolaryngologists performed ESS for 3D-printed sinus models and evaluated their mental workload during ESS, in a point of mental demand, physical demand, performance, effort, and frustration. ESS, endoscopic sinus surgeries; NASA-TLX, national aeronautics and space administration-task load index. ***p < 0.001.

3.2. The surgical skill level was associated with mental workload during ESS

The association between the mental workload during ESS and the surgical skill level was examined. First, the mental workload was compared between the registrars and the certified otolaryngologists. Registrars’ response showed significantly higher total NASA-TLX score (z = −2.8410, p = 0.0048), higher temporal demand (z = −2.7900, p = 0.0054), higher performance (z = −4.2320, p < 0.0001), and higher effort (z = −2.5390, p = 0.0115) for the simulation surgeries than certified otolaryngologists, while no significant differences were found in mental demand (z = −0.1530, p = 0.8867), physical demand (z = −0.2520, p = 0.8093), and frustration (z = −0.4460, p = 0.6631, Figure 2 and Table 1).

FIGURE 2

Figure 2. Comparison of mental workload during endoscopic sinus surgery (ESS) between otolaryngology registrars and board-certified otolaryngologists. Total score of national aeronautics and space administration-task load index (NASA-TLX) (A), mental demand (B), physical demand (C), temporal demand (D), performance (E), effort (F), and frustration (G) were compared between otolaryngology registrars and board-certified otolaryngologists. ESS, Endoscopic sinus surgeries; NASA-TLX, national aeronautics and space administration-task load index. *p < 0.05, **p < 0.01, ***p < 0.001.

Next, the mental workload was analyzed in association with the number of the experienced cases. The bottom 1/3 of the cases showed significantly, higher total NASA-TLX score (z = −3.3623, p = 0.0008), higher temporal demand (z = −4.3109, p < 0.0001), higher performance (z = −4.1756, p = 0.0026), and higher effort (z = −3.7960, p = 0.0006) than the top 1/3, while there was no significant difference in mental demand (z = 3.2176, p = 0.7179), physical demand (z = 0.2799, p = 0.7795), and frustration (z = −1.7878, p = 0.2406), between the top 1/3, middle 1/3, and bottom 1/3 of the surgeons who were experienced (Supplementary Figure 1) and Table 1.

For the OSATS score, the bottom 1/3 had a significantly higher total NASA-TLX score (z = −3.835, p = 0.0001), higher temporal demand (z = −3.7034, p = 0.0002), higher performance (z = −4.4203, p = 0.0016), and higher effort (z = −3.6567, p = 0.0002) than the top 1/3 (Supplementary Figure 2 and Table 2). There were no significant differences in physical demand (z = 0.0599, p = 0.9895), and frustration (z = −1.9276, p = 0.0837) among the groups. In mental demand, although the Kruskal Wallis test showed a low value of p among the groups (p = 0.0360), the comparison between the groups did not reach statistical significance.

The mental workload was compared between the experts and non-experts. The experts showed significantly lower score in total NASA-TLX score (z = −4.1354, p < 0.0001), temporal demand (z = −4.48429, p < 0.0001), performance (z = −4.0808, p < 0.0001), effort (z = −3.9414, p < 0.0001), and frustration (z = −1.9941, p = 0.0461), compared to non-experts (Figure 3 and Supplementary Table 3). These results suggest that surgeons’ mental workload during ESS is associated with surgical skill level, similar to what has been previously reported in other surgical disciplines (17). This was especially true for temporal demand, performance and effort.

FIGURE 3

Figure 3. The comparison of mental workload during ESS between experts and non-experts. Total score of NASA-TLX (A), mental demand (B), physical demand (C), temporal demand (D), performance (E), effort (F), and frustration (G) were compared between experts and non-experts. ESS, endoscopic sinus surgeries; OSATS, objective structured assessment of technical skills; NASA-TLX, national aeronautics and space administration-task load index. *p < 0.05, **p < 0.01, ***p < 0.001.

Next, the index related to the surgical skill level (the number of previous surgical cases and OSATS score) and the subscales in NASA-TLX were subjected to PCA test (Figure 4 and Table 2). PCA demonstrated that surgeons with low mental workload were mainly distributed to the left side of the principal component score plot and those who had a middle and high mental workload to the center and the right of the plot. The 1st, 2nd, and 3rd principal components explained 45.8, 19.2, and 10.6% of the total variance. The 1st principal component was the most strongly affected by temporal demand (loading 0.86), followed by OSATS score (−0.85), effort (0.84), performance (0.83) and the number of prior ESS cases (−0.71). The three subscales in the 1^st component, temporal demand, effort, and performance, were negatively correlated to the index, prior surgical cases and OSATS score (Table 2). The 2nd principal component consisted of mental demand (loading 0.83) and physical demand (0.81). The two subscales are independent from both the number of cases and OSATS score (Table 2). The 3rd principal component was affected by frustration (loading 0.64). This implies that the mental workload that the surgeon felt during ESS consisted of a different type of burden with a part of the burden associated with the surgical skill level other parts were independent of the surgeons’ experience.

FIGURE 4

Figure 4. Principal component analysis regarding the surgical skill level and mental workload. (A) The principal component score plot on each surgeon. The color represents the total score of NASA-TLX (red; high, green; middle, and blue low). (B) The Loading plots of 1st and 2nd principal components. NASA-TLX, national aeronautics and space administration-task load index; Exp. Cases, experienced cases; OSATS, objective structured assessment of technical skills.

TABLE 2

Table 2. Correlation coefficient among the surgical skill level of ESS and the subscales of NASA-TLX.

3.3. Changes noted in the registrars’ mental workload after repetitive ESS training

Ten registrars among the participants performed mock surgeries an additional six times as part of a repetitive training program. Their skill improvement was investigated in detail and confirmed in the previous study (14). Briefly, their OSATS score significantly increased after the program (z = −2.8031, p < 0.001). The residual bony septation within paranasal sinuses, evaluated with CT examination, was also significantly decreased (z = −3.326, p = 0.013) (14). The NASA-TLX was compared in the 2nd and final training, where the same 3D model was used. Although there were no significant differences in the total NASA-TLX, most subscales significantly changed. Mental and Physical demand significantly increased after the repetitive training, while temporal demand, performance, and effort were significantly decreased (Figure 5 and Supplementary Table 4). There was no significant change in frustration after the training.

FIGURE 5

Figure 5. Registrars’ mental workload during ESS before and after the repetitive training. After the first simulation surgeries, 10 registrars repeated surgeries more six times. Total score of NASA-TLX (A), mental demand (B), physical demand (C), temporal demand (D), performance (E), effort (F), and frustration (G) were compared between 2nd training and the final training, which the same models were used. ESS, Endoscopic sinus surgeries; NASA-TLX, national aeronautics and space administration-task load index. 2nd, the 2nd training; Final, the final training.

4. Discussion

So far, mental workload during ESS has not been able to be fully evaluated due to the significant variation in anatomy between patients. The first paper reported by Alobid, et al., studied 15 novice surgeons performing ESS on actual patients and evaluated their mental workload including the surgeons’ anxiety score, cardiovascular index, and serum cortisol level (9). They found that the surgeons had a high anxiety score with increased blood pressure and cortisol levels during ESS (11). Stelter et al. investigated four experienced rhinologists’ mental workload during transnasal endoscopic surgeries for variety of diseases including chronic rhinosinusitis, mucoceles of frontal sinuses, cerebral spinal fluid leaks, and skull-base surgeries (12). Mental workload also has been investigated as part of studies on utility of new devices, such as augmented reality (AR) image guidance (18), Virtual reality (VR) (19), and flexible endoscopes for ESS (20). In addition, the role of surgical ergonomics on the surgeons’ mental workload was evaluated (21, 22). However, the major limitation common to all these studies was the limited number of participants and the unstandardized conditions due to the great diversity of paranasal sinus’ anatomies in the patients or cadaveric materials on which the surgery was performed.

This study was performed with 47 participants under the standardized conditions provided by the 3D sinus model. This allows for a more detailed analysis on mental workload during ESS. We found that total mental workload was significantly associated with surgeons’ skill level. As for the subscales, temporal demand, effort, and performance, these were significantly associated with surgical experience (number of prior surgeries), while the mental and physical demand had no relationship with experience. PCA demonstrated that the surgeons’ mental burden during ESS could be classified into three categories: (1) the surgical experience-dependent factor (temporal demand, effort, and performance), (2) the surgical experience-independent factor (mental and physical demand), and (3) the in-between factor (frustration). Interestingly, these three categories significantly changed after repetitive training; (1) the experience dependent factors were alleviated, (2) the mental and physical factors improved, and (3) the in-between factor (frustration) did not change. As a result, the total NASA-TLX was unchanged after the training. However, training improved the most burdensome factors of temporal demand and effort. Interestingly, the mental and physical demand was increased after the repetitive training. This suggests that some mental burdens are not always alleviated after training but remain similar or even worsen even when surgeons gain surgical experience. A higher physical demand was found in surgeons with previous burnout experience (p = 0.048, data not shown). Evaluation of surgical skills and the routine assessment of mental burden should be part of surgical training to allow the continued well being of trainees’ mental health.

Surgical residents are continuously exposed to a significant amount of mental burden (23, 24). The assessment of mental workload should be considered when residency programs are designed (23) and appropriate adjustments made. A high mental workload in surgery increases the risk of complications (25, 26). As the development of residency programs continue to evolve so the mental stress of the program should form a significant part of improving the program and consequently the health of the residents. Therefore, it is important that there is a standardized method of assessing mental health.

The significant improvement in the ability of 3D printers to generate models from actual patients for residents to perform surgery in temporal bones (27–29), paranasal sinuses (30–33), skull base (30, 34–37), kidney, renal pelvis, ureter (38, 39), spine (40, 41), mandibula (30), aorta (42) and heart (43) has been established. The 3D-printed models are not only useful for surgical training (44), but also provide a platform to assess surgeons’ mental workload during surgery (39–41). With this improved technology there is now a high degree of similarity and reality for surgeons as has been shown in recent studies (39). The 3D-printed sinus models used in this study have been previously studied and have shown satisfactory face, content, and construct validity (14). There is a significantly high correlation between the assessment of the 3D-printed sinus models when compared with cadaveric materials both in terms of skill levels and surgical efficiency (skill levels; r = 0.828 and efficiency; r = 0.953) (14). The high fidelity of the 3D-printed sinus models also allows for the prediction of mental workload in ESS for actual patients in advance. The other significant advantage of the 3D-printed sinus models is that they are plastic and do not any ethical and transport restrictions, unlike cadaveric materials. This is illustrated by a recent published study for a remote surgical training course held simultaneously in Japan and Australia with combination of the 3D models and a web conferencing systems (15). This provides a standardized surgical exercise on complex anatomy that can be used to evaluate surgeons’ performance, skill level and can include mental workload. There is now the ability with the number of different anatomies available for surgical boards to be able to use these standardized yet complex anatomies to evaluate actual surgical skill rather than just the theory of surgery.

This study has limitations because although simulation training using the 3D sinus model has been shown to have a high fidelity and has been previously validated (14), there are still small differences between the models and actual ESS on patients. The models reduce the fear of intraoperative bleeding and the risk of operative complications, and this can affect the surgeon’s mental workload. In addition, in this study all surgeons were asked to perform the surgery within a specified time and this may add to the temporal demand. It has been previously shown that time pressure is the biggest stressor for surgeons during ESS (12). Some differences only had small effect sizes despite their statistically significant (cf. a difference in frustration between experts and non-experts, Supplementary Table 3). However, most other statistically significant differences analyzed in this study also held sufficient effect sizes. The total NASA-TLX score in this study was not a weighted score but just a summation of raw figures of the subscales, although the raw total score was validated as sensitive as the weighted score (45). It is also to be investigated if the changes in the mental workload found after the short-term training are equal to the differences in the workload found among the surgeons with various skill levels produced by their long-term experience. Further, our study lacks the objective method to evaluate the workload such as heart rate, blood pressure, catecholamines or cortisol in blood and saliva (46). Despite the limitations, this study is the first step to assess the mental workload during ESS under the standardized conditions and can help with the development of surgical training curricula in the future.

5. Conclusion

The level of surgical skill significantly affected surgeons’ mental workload during ESS, especially in temporal demand, performance, and effort.

Data availability statement

The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.

Ethics statement

Ethical review and approval was not required for the study on human participants in accordance with the local legislation and institutional requirements. The patients/participants provided their written informed consent to participate in this study.

Author contributions

MS, TA, AK, and AH designed the project. MS, KMi, RW, TS, AN, YN, and AH organized the simulation training and collected data. MS and YN analyzed OSATS score. KMa, and AK analyzed NASA-TLX score. MS, DH, AP, and PW wrote the draft. All authors contributed to the article and approved the submitted version.

Funding

This work was partially supported by JSPS Grant-in-Aid for Scientific Research JP18H04102, 22 K16923, and 22 K10599.

Acknowledgments

We thank Aya Honma, Shogo Kimura, Yuu Manabe, Shizuka Sugawara, Yuka Masuta, and Yumiko Kimura for their support in the surgical training course and in the writing of the manuscript. We would also like to thank KARL STORZ, Medtronic, and Stryker for providing the endoscopes, surgical equipment, instruments and the Building Block software.

Conflict of interest

PW: consultant for Fusetec and receiving royalties from Fusetec. AP: consultant for Fusetec, Medtronic, ENT technologies, Tissium, and Aerin Medical, shareholder of Chitogel, and speaker’s bureau for Sequiris.

The remaining authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher’s note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Supplementary material

The Supplementary material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fmed.2023.1090743/full#supplementary-material

SUPPLEMENTARY FIGURE S1 | The comparison of mental workload during ESS in association with surgeons’ experienced ESS cases. Participants were classified into three groups according to the number of prior surgeries performed (the top 1/3, the middle 1/3, and the bottom 1/3). Total score of NASA-TLX (A), mental demand (B), physical demand (C), temporal demand (D), performance (E), effort (F), and frustration (G) were compared among the three groups. ESS, Endoscopic sinus surgeries; NASA-TLX, National Aeronautics and Space Administration-Task Load Index. **p < 0.01, ***p < 0.001.

SUPPLEMENTARY FIGURE S2 | The comparison of mental workload during ESS in association with OSATS score. Participants were classified into three groups according to their OSATS score (the top 1/3, the middle 1/3, and the bottom 1/3). Total score of NASA-TLX (A), mental demand (B), physical demand (C), temporal demand (D), performance (E), effort (F), and frustration (G) were compared amongst these three groups. ESS, Endoscopic sinus surgeries; OSATS, Objective Structured Assessment of Technical Skills; NASA-TLX, National Aeronautics and Space Administration-Task Load Index. **p < 0.01, ***p < 0.001.

References

1. Murthy, VH. Confronting health worker burnout and well-being. N Engl J Med. (2022) 387:577–9. doi: 10.1056/NEJMp2207252

PubMed Abstract | CrossRef Full Text | Google Scholar

2. Harry, E, Sinsky, C, Dyrbye, LN, Makowski, MS, Trockel, M, Tutty, M, et al. Physician task load and the risk of burnout among us physicians in a National Survey. Jt Comm J Qual Patient Saf. (2021) 47:76–85. doi: 10.1016/j.jcjq.2020.09.011

PubMed Abstract | CrossRef Full Text | Google Scholar

3. Tanios, M, Haberman, D, Bouchard, J, Motherwell, M, and Patel, J. Analyses of burn-out among medical professionals and suggested solutions—a narrative review. Journal of Hospital Management and Health Policy. (2021) 6. doi: 10.21037/jhmhp-20-153

CrossRef Full Text | Google Scholar

4. Longo, L, Wickens, CD, Hancock, G, and Hancock, PA. Human mental workload: a survey and a novel inclusive definition. Front Psychol. (2022) 13:883321. doi: 10.3389/fpsyg.2022.883321

PubMed Abstract | CrossRef Full Text | Google Scholar

5. Lowndes, BR, Forsyth, KL, Blocker, RC, Dean, PG, Truty, MJ, Heller, SF, et al. Nasa-Tlx assessment of surgeon workload variation across specialties. Ann Surg. (2020) 271:686–92. doi: 10.1097/sla.0000000000003058

PubMed Abstract | CrossRef Full Text | Google Scholar

6. Hart, SG, and Staveland, LE. Development of Nasa-Tlx (task load index): results of empirical and theoretical research. In: PA Hancock and N Meshkati, editors. Advances in Psychology. Amsterdam, Netherlands: Elsevier. (1988) 52:139–83.

Google Scholar

7. Carswell, CM, Clarke, D, and Seales, WB. Assessing mental workload during laparoscopic surgery. Surg Innov. (2005) 12:80–90. doi: 10.1177/155335060501200112

CrossRef Full Text | Google Scholar

8. Berguer, R, Smith, WD, and Chung, YH. Performing laparoscopic surgery is significantly more stressful for the surgeon than open surgery. Surg Endosc. (2001) 15:1204–7. doi: 10.1007/s004640080030

PubMed Abstract | CrossRef Full Text | Google Scholar

9. Theodoraki, MN, Ledderose, GJ, Becker, S, Leunig, A, Arpe, S, Luz, M, et al. Mental distress and effort to engage an image-guided navigation system in the surgical training of endoscopic sinus surgery: a prospective. Randomised Clinical Trial Eur Arch Otorhinolaryngol. (2015) 272:905–13. doi: 10.1007/s00405-014-3194-0

PubMed Abstract | CrossRef Full Text | Google Scholar

10. Fan, X, Forsman, M, Yang, L, Lind, CM, and Kjellman, M. Surgeons’ physical workload in open surgery versus robot-assisted surgery and nonsurgical tasks. Surg Endosc. (2022) 36:8178–94. doi: 10.1007/s00464-022-09256-0

PubMed Abstract | CrossRef Full Text | Google Scholar

11. Alobid, I, de Pablo, J, Mullol, J, Centellas, S, Parramon, G, Carrasco, J, et al. Increased cardiovascular and anxiety outcomes but not endocrine biomarkers of stress during performance of endoscopic sinus surgery: a pilot study among novice surgeons. Arch Otolaryngol Head Neck Surg. (2011) 137:487–92. doi: 10.1001/archoto.2011.60

PubMed Abstract | CrossRef Full Text | Google Scholar

12. Stelter, K, Theodoraki, MN, Becker, S, Tsekmistrenko, V, Olzowy, B, and Ledderose, G. Specific stressors in Endonasal Skull Base surgery with and without navigation. Eur Arch Otorhinolaryngol. (2015) 272:631–8. doi: 10.1007/s00405-014-3154-8

PubMed Abstract | CrossRef Full Text | Google Scholar

13. Hu, JS, Lu, J, Tan, WB, and Lomanto, D. Training improves laparoscopic tasks performance and decreases operator workload. Surg Endosc. (2016) 30:1742–6. doi: 10.1007/s00464-015-4410-8

PubMed Abstract | CrossRef Full Text | Google Scholar

14. Suzuki, M, Miyaji, K, Watanabe, R, Suzuki, T, Matoba, K, Nakazono, A, et al. Repetitive simulation training with novel 3d-printed sinus models for functional endoscopic sinus surgeries. Laryngoscope Investig Otolaryngol. (2022) 7:943–54. doi: 10.1002/lio2.873

PubMed Abstract | CrossRef Full Text | Google Scholar

15. Suzuki, M, Vyskocil, E, Ogi, K, Matoba, K, Nakamaru, Y, Homma, A, et al. Remote training of functional endoscopic sinus surgery with advanced manufactured 3d sinus models and a telemedicine system. Front Surg. (2021) 8:746837. doi: 10.3389/fsurg.2021.746837

PubMed Abstract | CrossRef Full Text | Google Scholar

16. Lin, SY, Laeeq, K, Ishii, M, Kim, J, Lane, AP, Reh, D, et al. Development and pilot-testing of a feasible, reliable, and valid operative competency assessment tool for endoscopic sinus surgery. Am J Rhinol Allergy. (2009) 23:354–9. doi: 10.2500/ajra.2009.23.3275

PubMed Abstract | CrossRef Full Text | Google Scholar

17. Yurko, YY, Scerbo, MW, Prabhu, AS, Acker, CE, and Stefanidis, D. Higher mental workload is associated with poorer laparoscopic performance as measured by the Nasa-Tlx tool. Simul Healthc. (2010) 5:267–71. doi: 10.1097/SIH.0b013e3181e3f329

PubMed Abstract | CrossRef Full Text | Google Scholar

18. Dixon, BJ, Chan, H, Daly, MJ, Vescan, AD, Witterick, IJ, and Irish, JC. The effect of augmented real-time image guidance on task workload during endoscopic sinus surgery. Int Forum Allergy Rhinol. (2012) 2:405–10. doi: 10.1002/alr.21049

PubMed Abstract | CrossRef Full Text | Google Scholar

19. Haerle, SK, Daly, MJ, Chan, H, Vescan, A, Witterick, I, Gentili, F, et al. Localized intraoperative virtual endoscopy (live) for surgical guidance in 16 Skull Base patients. Otolaryngol Head Neck Surg. (2015) 152:165–71. doi: 10.1177/0194599814557469

PubMed Abstract | CrossRef Full Text | Google Scholar

20. Legrand, J, Ourak, M, Van Gerven, L, Vander Poorten, V, and Vander, PE. A miniature robotic steerable endoscope for maxillary sinus surgery called Plient. Sci Rep. (2022) 12:2299. doi: 10.1038/s41598-022-05969-3

PubMed Abstract | CrossRef Full Text | Google Scholar

21. Ramakrishnan, VR, and Milam, BM. Ergonomic analysis of the surgical position in functional endoscopic sinus surgery. Int Forum Allergy Rhinol. (2017) 7:570–5. doi: 10.1002/alr.21911

PubMed Abstract | CrossRef Full Text | Google Scholar

22. Arrighi-Allisan, AE, Garvey, KL, Wong, A, Filip, P, Shah, J, Spock, T, et al. Ergonomic analysis of functional endoscopic sinus surgery using novel inertial sensors. Laryngoscope. (2022) 132:1153–9. doi: 10.1002/lary.29796

PubMed Abstract | CrossRef Full Text | Google Scholar

23. Kahol, K, Smith, M, Mayes, S, Deka, M, Deka, V, Ferrara, J, et al. The Effect of Fatigue on Cognitive and Psychomotor Skills of Surgical Residents. Foundations of Augmented Cognition. Berlin, Heidelberg: Springer Berlin Heidelberg (2007).

Google Scholar

24. Serenari, M, Cucchetti, A, Russo, PM, Fallani, G, Mattarozzi, K, Pinna, AD, et al. Burnout and psychological distress between surgical and non-surgical residents. Updat Surg. (2019) 71:323–30. doi: 10.1007/s13304-019-00653-0

PubMed Abstract | CrossRef Full Text | Google Scholar

25. Prabhu, A, Smith, W, Yurko, Y, Acker, C, and Stefanidis, D. Increased stress levels may explain the incomplete transfer of simulator-acquired skill to the operating room. Surgery. (2010) 147:640–5. doi: 10.1016/j.surg.2010.01.007

PubMed Abstract | CrossRef Full Text | Google Scholar

26. Yamaguchi, K, and Kanemitsu, S. Surgeons’ stress from surgery and night duty: a multi-institutional study. Arch Surg. (2011) 146:271–8. doi: 10.1001/archsurg.2010.250

PubMed Abstract | CrossRef Full Text | Google Scholar

27. Chien, WW, da Cruz, MJ, and Francis, HW. Validation of a 3d-printed human temporal bone model for otology surgical skill training. World J Otorhinolaryngol Head Neck Surg. (2021) 7:88–93. doi: 10.1016/j.wjorl.2020.12.004

PubMed Abstract | CrossRef Full Text | Google Scholar

28. Da Cruz, MJ, and Francis, HW. Face and content validation of a novel three-dimensional printed temporal bone for surgical skills development. J Laryngol Otol. (2015) 129:S23–9. doi: 10.1017/s0022215115001346

PubMed Abstract | CrossRef Full Text | Google Scholar

29. Mooney, MA, Cavallo, C, Zhou, JJ, Bohl, MA, Belykh, E, Gandhi, S, et al. Three-dimensional printed models for lateral Skull Base surgical training: anatomy and simulation of the Transtemporal approaches. Oper Neurosurg (Hagerstown). (2020) 18:193–201. doi: 10.1093/ons/opz120

PubMed Abstract | CrossRef Full Text | Google Scholar

30. Chan, HH, Siewerdsen, JH, Vescan, A, Daly, MJ, Prisman, E, and Irish, JC. 3d rapid prototyping for otolaryngology-head and neck surgery: applications in image-guidance, surgical simulation and patient-specific modeling. PLoS One. (2015) 10:e0136370. doi: 10.1371/journal.pone.0136370

PubMed Abstract | CrossRef Full Text | Google Scholar

31. Chang, DR, Lin, RP, Bowe, S, Bunegin, L, Weitzel, EK, McMains, KC, et al. Fabrication and validation of a low-cost, medium-Fidelity silicone injection molded endoscopic sinus surgery simulation model. Laryngoscope. (2017) 127:781–6. doi: 10.1002/lary.26370

PubMed Abstract | CrossRef Full Text | Google Scholar

32. Alrasheed, AS, Nguyen, LHP, Mongeau, L, Funnell, WRJ, and Tewfik, MA. Development and validation of a 3d-printed model of the Ostiomeatal complex and frontal sinus for endoscopic sinus surgery training. Int Forum Allergy Rhinol. (2017) 7:837–41. doi: 10.1002/alr.21960

PubMed Abstract | CrossRef Full Text | Google Scholar

33. Barber, SR, Jain, S, Son, YJ, and Chang, EH. Virtual functional endoscopic sinus surgery simulation with 3d-printed models for mixed-reality nasal endoscopy. Otolaryngol Head Neck Surg. (2018) 159:933–7. doi: 10.1177/0194599818797586

PubMed Abstract | CrossRef Full Text | Google Scholar

34. Tai, BL, Wang, AC, Joseph, JR, Wang, PI, Sullivan, SE, McKean, EL, et al. A physical simulator for endoscopic Endonasal drilling techniques: technical note. J Neurosurg. (2016) 124:811–6. doi: 10.3171/2015.3.jns1552

PubMed Abstract | CrossRef Full Text | Google Scholar

35. Hsieh, TY, Cervenka, B, Dedhia, R, Strong, EB, and Steele, T. Assessment of a patient-specific, 3-dimensionally printed endoscopic sinus and Skull Base surgical model. JAMA Otolaryngol Head Neck Surg. (2018) 144:574–9. doi: 10.1001/jamaoto.2018.0473

PubMed Abstract | CrossRef Full Text | Google Scholar

36. Zheng, JP, Li, CZ, Chen, GQ, Song, GD, and Zhang, YZ. Three-dimensional printed Skull Base simulation for Transnasal endoscopic surgical training. World Neurosurg. (2018) 111:e773–82. doi: 10.1016/j.wneu.2017.12.169

PubMed Abstract | CrossRef Full Text | Google Scholar

37. Ding, CY, Yi, XH, Jiang, CZ, Xu, H, Yan, XR, Zhang, YL, et al. Development and validation of a multi-color model using 3-dimensional printing Technology for Endoscopic Endonasal Surgical Training. Am J Transl Res. (2019) 11:1040–8.

PubMed Abstract | Google Scholar

38. Cheung, CL, Looi, T, Lendvay, TS, Drake, JM, and Farhat, WA. Use of 3-dimensional printing technology and silicone modeling in surgical simulation: development and face validation in pediatric laparoscopic Pyeloplasty. J Surg Educ. (2014) 71:762–7. doi: 10.1016/j.jsurg.2014.03.001

PubMed Abstract | CrossRef Full Text | Google Scholar

39. Monda, SM, Weese, JR, Anderson, BG, Vetter, JM, Venkatesh, R, Du, K, et al. Development and validity of a silicone renal tumor model for robotic partial nephrectomy training. Urology. (2018) 114:114–20. doi: 10.1016/j.urology.2018.01.030

PubMed Abstract | CrossRef Full Text | Google Scholar

40. Bohl, MA, McBryan, S, Pais, D, Chang, SW, Turner, JD, Nakaji, P, et al. The living spine model: a biomimetic surgical training and education tool. Oper Neurosurg (Hagerstown). (2020) 19:98–106. doi: 10.1093/ons/opz326

PubMed Abstract | CrossRef Full Text | Google Scholar

41. Bohl, MA, McBryan, S, Spear, C, Pais, D, Preul, MC, Wilhelmi, B, et al. Evaluation of a novel surgical skills training course: are cadavers still the gold standard for surgical skills training? World Neurosurg. (2019) 127:63–71. doi: 10.1016/j.wneu.2019.03.230

PubMed Abstract | CrossRef Full Text | Google Scholar

42. Hussein, N, Honjo, O, Barron, DJ, and Yoo, SJ. Supravalvular aortic stenosis repair: surgical training of 2 repair techniques using 3d-printed models. Interact Cardiovasc Thorac Surg. (2021) 33:966–8. doi: 10.1093/icvts/ivab198

PubMed Abstract | CrossRef Full Text | Google Scholar

43. Yıldız, O, Köse, B, Tanıdır, IC, Pekkan, K, Güzeltaş, A, and Haydin, S. Single-center experience with routine clinical use of 3d Technologies in Surgical Planning for pediatric patients with complex congenital heart disease. Diagn Interv Radiol. (2021) 27:488–96. doi: 10.5152/dir.2021.20163

PubMed Abstract | CrossRef Full Text | Google Scholar

44. Langridge, B, Momin, S, Coumbe, B, Woin, E, Griffin, M, and Butler, P. Systematic review of the use of 3-dimensional printing in surgical teaching and assessment. J Surg Educ. (2018) 75:209–21. doi: 10.1016/j.jsurg.2017.06.033

CrossRef Full Text | Google Scholar

45. Hart, SG. Nasa-task load index (Nasa-Tlx); 20 years later. Proceedings of the Human Factors and Ergonomics Society Annual Meeting; 2006. 50. p. 904–908.

Google Scholar

46. Lean, Y, and Shan, F. Brief review on physiological and biochemical evaluations of human mental workload. Human Factors Ergonom Manuf Serv Indust. (2012) 22:177–87. doi: 10.1002/hfm.20269

CrossRef Full Text | Google Scholar