Assessment of a novel patient-specific 3D printed multi-material simulator for endoscopic sinus surgery

Background: Three-dimensional (3D) printing is an emerging tool in the creation of anatomical models for surgical training. Its use in endoscopic sinus surgery (ESS) has been limited because of the difficulty in replicating the anatomical details. Aim: To describe the development of a patient-specific 3D printed multi-material simulator for use in ESS, and to validate it as a training tool among a group of residents and experts in ear-nose-throat (ENT) surgery. Methods: Advanced material jetting 3D printing technology was used to produce both soft tissues and bony structures of the simulator to increase anatomical realism and tactile feedback of the model. A total of 3 ENT residents and 9 ENT specialists were recruited to perform both non-destructive tasks and ESS steps on the model. The anatomical fidelity and the usefulness of the simulator in ESS training were evaluated through specific questionnaires. Results: The tasks were accomplished by 100% of participants and the survey showed overall high scores both for anatomy fidelity and usefulness in training. Dacryocystorhinostomy, medial antrostomy, and turbinectomy were rated as accurately replicable on the simulator by 75% of participants. Positive scores were obtained also for ethmoidectomy and DRAF procedures, while the replication of sphenoidotomy received neutral ratings by half of the participants. Conclusion: This study demonstrates that a 3D printed multi-material model of the sino-nasal anatomy can be generated with a high level of anatomical accuracy and haptic response. This technology has the potential to be useful in surgical training as an alternative or complementary tool to cadaveric dissection.


Introduction
Endoscopic sinus surgery (ESS) represents the current gold standard technique for several pathologies, ranging from chronic rhinosinusitis to benign and malignant neoplasms (Weber and Hosemann, 2015). The paranasal sinuses are an anatomically complex district, located in close relationship with noble structures, such as the brain, the orbit, and the internal carotid artery. The presence of several anatomical variants makes the surgical approaches strictly dependent on the patient's specific configuration (Beale et al., 2009;Vaid and Vaid, 2015). Furthermore, being a one-handed technique, ESS requires hand-eye coordination and proper instruments handling in a restricted surgical area; despite technical advancements, such as the use of neuronavigation, complications from ESS can still be very serious, even fatal (Stammberger et al., 1995;Keerl et al., 1999;Humphreys and Hwang, 2015).
These peculiarities account for the importance of specific training for this type of surgery, aimed to both comprehensively understand sino-nasal anatomy and develop the required surgical skills in a risk-free environment, before operating on a patient (Acar et al., 2010;Laeeq et al., 2013;Stew et al., 2018).
The restrictions of traditional training based on cadaveric dissections have led to the investigation of alternative training methods, including the use of synthetic anatomical models created with 3-dimensional (3D) printing. The emerging advanced additive manufacturing technologies have lately expanded the possibilities of representing 3D patient-specific anatomies, which are extremely useful for both surgical planning, training (Ganguli et al., 2018;Battaglia et al., 2019;Bianchi et al., 2019;Schiavina et al., 2019;Tejo-Otero et al., 2019;Bianchi et al., 2020) and medical education (Cercenelli et al., 2022). Also, in the field of ENT surgery, 3D modeling and printing technologies allow the synthesis of patient-specific anatomical models that can be used in simulation to train on surgical steps of ESS, similarly to standard training on the cadaver (Tack et al., 2016;Sander et al., 2017). The use of 3D printing in sinus surgery is not recent, but it has been limited by some technical difficulties in printing the intricate details of the sinus anatomy and in reproducing the haptic feedback of soft tissues. New opportunities may be offered by the material jetting 3D printing technology that enables the printing of several polymers with varying properties, such as color and hardness, to be mixed at customizable ratios in a 3D space within the same model.
The aim of this study was to develop a patient-specific simulator for ESS using the 3D printing multi-material jetting technology, which reproduces both soft tissues and bony structures, thus increasing the anatomical realism and tactile feedback of the model. Moreover, a preliminary validation of this simulator as a training tool for ESS among a group of ear-nosethroat (ENT) residents and experts, is described.

Model creation
The design and fabrication of the simulator consisted of three steps: A) Image segmentation and 3D modeling of the anatomical components; B) Design of the simulator box and the external face mold; C) 3D printing of the simulator components.
The simulator was then assembled and tested by the surgeons in the operation room.

Image segmentation and 3D modeling
The preoperative computed tomography (CT) scan of a patient with no evidence of sino-nasal pathologies was acquired. The Digital Imaging and Communications in Medicine (DICOM) data were imported into D2P TM (3D Systems Inc., Rock Hill, SC, United States), which is a certified software for medical 3D modeling. The anatomical structures of interest, i.e., the midface bony structures, the turbinates, and nasal septum were segmented in collaboration with a radiologist with expertise in head and neck anatomy. The segmented mask for the turbinates was slightly reduced, in accordance with the surgical team, to simulate the effect of decongestants used before the ESS. 3D meshes were then generated from all the segmented masks and saved in STL format; For converting the segmentation mask to 3D mesh, we used the standard settings provided by D2P software. In detail, these settings include selection of: "Maintain original morphology" method (to preserve all segmented voxels of the mask in the mesh file) and selection of "Simplify and smooth" (to decimate and apply extra smoothing to the mesh surface).
The meshes were imported and processed into 3-Matic Medical software (Materialise, Leuven, Belgium): all the structures were cut coronally into two sections to ease the cleaning of the internal parts after the printing process. According to the surgeons' requests, the medial periorbital surface and the dura mater of the anterior skull base were selected as relevant anatomical landmarks for surgery, and they were enlarged in order to be more visible in the simulator. Moreover, the nasolacrimal duct and the lacrimal sac were reproduced, filling the segmented bony duct and the lacrimal sac fossa (Figures 1, 2). Finally, the inferior turbinates were hollowed with a shell of 1 mm to allow the subsequent filling with silicone, thus achieving more compressible structures. All these refinements of the 3D model were performed following the surgeon's indications.
As final step, the Fix Wizard tool of 3-Matic software was applied to the meshes to make sure they were printable.

Design of the simulator box and the external face mold
A box including the reconstructed patient-specific paranasal sinuses model was created using the open-source software Frontiers in Bioengineering and Biotechnology frontiersin.org FreeCAD. The box was designed as a hollow rectangle combined with a larger base in order to secure it to the operating table during the simulation.
The portion of the facial skin was segmented in D2P TM , then converted into a mesh and exported as STL file to be further processed in MeshMixer software (Autodesk Inc., CA, US).

FIGURE 1
Segmentation and 3D rendering of the bony structures (white), the turbinates (red) and both bony and cartilaginous nasal septum (green). Using the negative of the facial skin model, a rectangular mold was designed to allow the subsequent silicone casting ( Figure 3).
Lastly, the box was provided with a closing lid that was designed in FreeCAD. The patient-specific portion of skin was combined with the lid to let its borders assume the shape of the patient-specific face.

3D printing of the simulator components
The patient-specific model of the paranasal sinuses was printed using the J720 Dental 3D printer (Stratasys Ltd., Eden Prairie, MN). This printer is based on material jetting technology, a process that resembles inkjet paper printing since the material is dropped through small diameter nozzles. In this case, the printing material is a photopolymeric resin subsequently hardened by a UV lamp.
In detail, the bony structures of the simulator were 3D printed using a rigid white opaque material (VeroWhitePlus resin), which was previously used for reproducing bones in other 3D printed phantoms of thorax, pelvis and prosthetic fingers (Cabibihan, 2011;Jahya et al., 2013;Mayer et al., 2015). Considering the cartilage mechanical properties, that lie between bone and soft tissue (Richmon et al., 2005;Richmon et al., 2006;Griffin et al., 2016), the cartilaginous nasal septum was printed via a mixing of rigid resins (VeroMagenta) and a rubber-like material from the Polyjet photopolymer family (Agilus30) with a resulting Shore-A 50. Also, the membranes replicating the orbital content and the dura mater as well as the middle, the superior turbinates, and the inferior turbinates were printed combining Agilus30 and rigid resin VeroMagenta with Shore-A 50 and Shore-A 30 respectively. The lacrimal sac was printed by mixing VeroYellow and Agilus30 with a Shore-A 35 ( Figure 4). Particularly, the specific hardness for turbinates and the lacrimal sac was chosen in agreement with the surgeons, after several trials on prototypes.
Regarding the other components of the simulator, the box and the lid were both printed using a stereolithography (SLA) 3D printer (Form 3, Formlabs, Somerville, MA, United States) with White resin.
The components of the patient-specific facial skin mold were also printed using Form 3 SLA printer but with a Clear resin so that, during the silicone injection, both the level of silicone and the presence of air bubbles could be seen.

Silicone casting and assembly of the simulator
The patient-specific facial skin was realized by a silicone casting through the mold: the selected silicone was Dragon Skin FX-Pro (Smooth-On Inc., Texas, United States) which is a bicomponent silicone frequently used in literature to reproduce human skin and surgical simulation (Francesconi et al., 2015;Teyssier et al., 2019;Ock et al., 2020;Ock et al., 2021). The silicone is available as a liquid that is separated in two components, that were mixed in ratio 1:1 by volume; then some pigments (Silc-Pig, Smooth-On Inc., Texas, United States) were added to achieve the realism of the final product ( Figure 5).
After the cleaning of the two parts of the model, the inferior turbinates were filled up with Dragon Skin FX-Pro, to make them more compressible. The two parts were then glued together shaping the entire model. To replicate the nasal mucosa, some layers of silicone were added over the nasal septum. Lastly, the skin portion was placed on the lid and the box was closed ( Figure 6).

Augmented Reality feature
As an additional feature, we implemented an Augmented Reality (AR) view to be integrated into the simulator. This allows   Frontiers in Bioengineering and Biotechnology frontiersin.org 05 a "hybrid" training, i.e. the fusion of digital and physical tools, that may improve the potentialities of the simulator.
The anatomical virtual models and some cutting planes planned by the surgeon on the reconstructed anatomy (Figures 7B,C) were imported into Unity 3D (Unity Technologies, San Francisco, CA, United States), a software with a specific development kit for creating AR apps (Vuforia Engine package, PTC, Inc., Boston, MA, United States).
To track the physical simulator, we designed and printed a tracker with a well-contrasted texture, which was then anchored to the external box of the simulator ( Figure 7A).
The tracking algorithm and the registration between the digital models and the real scene were implemented using the Image Target function of Vuforia engine: the engine detects the image texture of the tracker by comparing extracted natural features from the camera image against the known image target (i.e., the texture previously selected for the tracker). Once the image target is detected, Vuforia engine will track the image target on the tracker and augment the scenario with digital content anchored to the target itself.
The AR application was built as a UWP (Universal Windows Platform) app deployed on Microsoft HoloLens 2 smart glasses. The user just needs to look at the tracker on the simulator while wearing the AR glasses, until the image target matching occurs.

FIGURE 6
The final assembly of the simulator (A) and the view of the internal part (B).

FIGURE 7
The AR view as displayed to the surgeon via HoloLens 2 smart glasses (A) to generate the holographic overlays of anatomical inner structures and cutting planes (B,C) on the 3D printed simulator.
Frontiers in Bioengineering and Biotechnology frontiersin.org After that, tracking of the real object and virtual content overlay can begin. The AR application generates holographic overlays on the physical model, by rendering the underlying anatomical structures (i.e., turbinates, nasal septum), not visible from the outside of the simulator, together with the cutting planes which may help the trainee in understanding the deep anatomical structures while navigating the physical model with instruments (see Supplementary Video S1).
Interactable user interface toggles (check boxes) were added to turn off and on the rendering of each virtual anatomical structure and cutting planes. In particular, cutting planes following progressive lines perpendicular to the nasal bones were created to convey the anatomical configuration that the surgeon has to bear in mind during the endoscopic approaches to the frontal sinus (commonly considered by surgeons in training a highly complex region to reach endoscopically). The possibility to virtually remove the middle turbinate allows for realistic visualization of the intraoperative field across the steps to performing DRAF procedures.
Voice commands to show/hide the virtual structures were also implemented in order to provide a completely hands-free AR application. (see Supplementary Video S1).

Validation of the simulator as a training tool for ESS
The process to assess the fidelity and utility of the simulator as a training tool for ESS consisted of two phases which are described below.

Phase one: Face and anatomical content validation
First, the anatomical realism of the nasal cavity and the reproducibility of the instruments handling inside it were evaluated.
This phase involved 12 participants with different expertise levels: 3 ENT residents from the third and fourth year of residency, 3 young ENT specialists (less than 3 years of practice), and 6 ENT surgeons experienced in ESS (more than 10 years of practice). Each participant was asked to perform the following non-destructive tasks on the simulator using the standard instruments ( Figure 8, Figure 9), in a timetrial mode: 1 Identification of normal anatomy landmarks on one nasal cavity (the inferior, middle and superior turbinate, the inferior, middle and superior meatus, the choana, the uncinate process and the hiatus semilunaris, the sphenoid ostium); 2 Removal of a foreign body (small plastic segment) from the choana; 3 Medialization of the middle turbinate and insertion of a cottonoid into the middle meatus; 4 Incision of the simulated nasal septum mucosa with a curved scalpel and its detachment from the underlying simulated cartilage with a Cottle dissector-elevator.
The completion of each task (yes/no) and the time needed to complete it were annotated by an external observer not involved in this phase (N.E.). A 5-point Likert scale questionnaire  Table S1).

Phase two: Task-specific content validation
This phase was aimed at assessing the feasibility of different ESS procedures on the simulator and its potential as a training tool for ESS. Only four expert surgeons participated in this second phase. They were asked to simulate all the basic procedures of ESS, namely uncinectomy, maxillary antrostomy, anterior and posterior ethmoidectomy, sphenoidotomy (Supplementary Video S2) and DRAF I procedure, on previously untouched models (one side per surgeon). Dacriocystorhinostomy was also required, either as the first or last surgical step (Supplementary Video S3). Two of the four surgeons were asked to complete frontal sinus surgery with DRAF IIa, IIb, and eventually DRAF III (Supplementary Video S4) at the end of dissections on both sides.
After completing these tasks, the experts were required to fill out the task-specific questionnaire (Supplementary Table S2), using a Likert 5-point scale. A space for free comments and/or suggestions for improving the simulator was available at the end of the form.

Statistical analysis
All measured times needed to complete the tasks were reported as mean values and standard deviation (SD). t-test for unpaired data was used to compare mean times for each pair of participant groups (residents vs young experts, young experts vs experts, residents vs experts) for each task.
All statistical tests were performed using SPSS software, version 26.0 (SPSS Inc., Chicago, IL, United States), and a p-value of <0.05 was considered statistically significant.

Face and anatomical content validation
All the 12 participants completed the face and anatomical content validation of the model and answered the survey. The non-destructive tasks were accomplished by 100% of participants ( Figure 9). The survey results are reported in detail in Figure 10 and the scores stratified per group of participants are shown in Figure 11.
A total of 92% (n = 11) of the participants agreed or strongly agreed about the high anatomical fidelity of the simulator and the tactile feedback of both the turbinates and bony structures. On the other hand, 25% (n = 3) of the participants were neutral about the tactile feedback of the nostrils and nose tip and the use of Frontiers in Bioengineering and Biotechnology frontiersin.org instruments and endoscope inside the cavities. All the participants gave positive feedback (100% agreed or strongly agreed) regarding the role of the simulator in learning anatomy and improving accuracy and precision in endoscopic settings. Moreover, the utility of the simulator in developing hand-eye coordination received the highest scores (100% strongly agreed) from all responders. Figure 12 shows the minimum, maximum, mean, and standard deviation of the operative times recorded for each non-destructive task per groups (resident, young, and expert). Despite no statistical FIGURE 10 5-point Likert scale rating from face and anatomical content validation questionnaire. 1: strongly disagree; 2: disagree; 3: neutral; 4: agree; 5: strongly agree. In bold are reported the percentages for the different ratings obtained for every question.

FIGURE 11
5-point Likert scale ratings divided by groups of participants. "Resident": ENT residents from the third and fourth year; "Young": ENT specialists with less than 3 years of practice; "Expert": experienced ENT surgeons with more than 10 years of practice. 1: strongly disagree; 2: disagree; 3: neutral; 4: agree; 5: strongly agree. In bold are reported the percentages for each rating given by the different members of the groups. In gray dots are reported the mean and standard deviation of the scores obtained for every group.
Frontiers in Bioengineering and Biotechnology frontiersin.org significance was found, results suggested that the experts completed the tasks faster than the residents.

Task-specific content validation
All 4 ENT experts were able to realize the required steps on the simulator with standard instruments set for ESS and drill ( Figures  13, 14). Then they answered the 15-question survey, whose percentage results are summarized in Figure 15A.
Similar to the results obtained for the face and anatomical content validity questionnaire, all participants in this phase agreed or strongly agreed regarding the anatomical fidelity of the paranasal sinuses. Dacryocystorhinostomy, medial antrostomy and turbinectomy were rated as accurately replicable on the simulator. Also for ethmoidectomy and DRAF I the 75% of participants agreed or strongly agreed for accurate replication. The replication of sphenoidotomy, on the contrary, received more neutral ratings.
Moreover, both the experts performing DRAF IIa, b and III, strongly agreed that the model could accurately replicate these procedures ( Figure 15B).

Discussion
The increasing number of medical trainees compared to the limited availability of body donors, and the ethical and biological issues related to ex-vivo dissection, have promoted research on alternative training methods to traditional cadaver dissection. Simulation-based training with 3D printed models has been favorably accepted in different surgical specialties as they provide an additional hands-on experience to improve anatomy learning, simulate surgical procedures in a risk-free setting, and eventually improve technical and cognitive skills (Meyer-Szary et al., 2022).
Regarding endoscopic sinus surgery (ESS), the training has been traditionally performed either on ovine models or cadaveric heads, which both lack from pathology and specific anatomical variants, and could be dissected just once. The development of synthetic models for ESS training has encountered several difficulties in relation to the replication of the complex anatomy and of the variable haptic feedback from different parts of the nose and paranasal sinuses.
The first low-fidelity models date back to 10 years ago, and despite having been demonstrated to improve basic endoscopic skills, the lack of anatomical and haptic fidelity hampered the possibility to simulate ESS procedures (Wais et al., 2012;Chang et al., 2017). More recently, Hsieh et al. (2018) developed a mono-material 3D printed endoscopic sinus and skull base surgical model with reported high anatomical and haptic accuracy, despite the lack of soft tissues (Hsieh et al., 2018). Zhuo et al. (2019) were the first to provide a pediatric nasal cavity and a post-surgical paranasal sinuses model, with positive results on anatomic realism and tactile feedback. However, the models were

FIGURE 12
Representation of mean recorded operative times for each non-destructive tasks per group. Task 1: identification of normal anatomy landmarks; Task 2: removal of a foreign body (small plastic segment) from the choana; Task 3: medialization of the middle turbinate and insertion of a cottonoid into the middle meatus; Task 4: incision of the simulated nasal septum mucosa.
Frontiers in Bioengineering and Biotechnology frontiersin.org created by assembling several cuboid components subsequently fixed with double-sided tape so that only nondestructive tasks were simulated on their model, preventing its validation for ESS procedures. On the contrary, Alrasheed et al. (2017) developed and validated a 3D printed multimaterial model for basic endoscopic skill training, as well as uncinectomy, anterior ethmoidectomy and frontal sinus dissection. The models had high anatomical fidelity and proved realistic haptic feedback, especially for bony dissection. Although soft structures were reproduced, the weakest feature reported was the replication of the mucosa. Alwani et al. (2020) validated the PHACON Sinus Trainer, one of the ESS simulators available on the market. Unlike the previous models, this simulator received low scores for the anatomical fidelity of both the mucosa and the bone, with only 30% and 42% of the participants rating them realistic, respectively. The unmet needs of 3D printed simulators for ESS encouraged our group to develop a 3D printed multimaterial and multi-color patient-specific simulator for ESS, using the Polyjet printing technology. The potentiality and the precision of this recent technology allowed us to create a model with bony and soft structures simultaneously, reaching a high level of anatomical fidelity and an overall satisfactory realistic haptic feedback, as suggested by our results. Indeed, the technology has many benefits, including excellent resolution (up to 0.016 mm), smooth surfaces, and the ability to print multi-material and multi-color parts, thanks to its multiple print heads. Furthermore, it is possible to dynamically mix materials and create "digital materials" with new characteristics. Indeed, it is possible to print rubber-like materials targeting a specific Shore-A hardness, as well as to mix up rigid and rubber-like materials, creating new materials with hybrid properties.
However, we decided not to use the rubber-like Agilus Stratasys material, which has a minimum achievable Shore-A 30, to reproduce the facial skin, since preliminary tests with surgeons confirmed that the obtained 3D printed part is still too rigid if compared to the real facial skin. Moreover, using Agilus material at the minimum Shore-A 30, the parts can be printed only in white, so it is not possible to achieve a realistic color for the skin. Many papers in the literature report the use of the Dragon Skin FX-Pro, especially for the replication of the human skin using specific pigments to have different realistic skin coloration (Francesconi et al., 2015;Teyssier et al., 2019;Ock et al., 2020;Ock et al., 2021).
Regarding the turbinates, we decided to fill them with silicone because the filling with Agilus Shore-A 30 made them too rigid. Conversely, the silicone filling allows the turbinates to be more compressible even if they still maintain a greater elasticity if compared to the real ones.
Moreover, the silicone has realistic feedback to cut, and we decided to use it also for simulating the nasal mucosa; also in this case, using an additional layer of Agilus Shore-A 30 on the nasal septum is not a good solution to achieve a realistic replication of the mucosa.
In our simulator, we were able to replicate the mucosa on the nasal septum so that the simulation of mucosal incision and mucoperichondrial flap elevation could be successfully performed on the 3D printed model by all participants. Alreefi et al., 2017 were the first to propose a training model for endoscopic septoplasty, using the TangoPlus FLX930 material from Stratasys to replicate the nasal mucosa. The procedure was scored as realistic, but the Frontiers in Bioengineering and Biotechnology frontiersin.org haptic feedback and membrane coloring received the lowest ratings. In our model, silicone was used to manually cover the nasal cartilage, obtaining a soft layer that could be realistically incised with a sickle knife or scalpel and a tissue-to-tissue surface dissectible with a Freer elevator.
To the best of our knowledge, this study is the first to consider the feasibility of endoscopic lacrimal surgery on a 3D printed model. The nasolacrimal duct and sac were realized with a different material (VeroYellow and Agilus30 with Shore-A 35), obtaining a color quite similar to the surrounding bone, as it is in real anatomy. The consistency of the sac resulted quite fragile so that in one case it was accidentally drilled before identifying it.
Moreover, in our model some high-risk anatomical landmarks such as the lamina papyracea and the dura mater of the anterior skull base were replicated with layers of material that, despite differences in color from the real structures, resulted not easily identifiable during endoscopic dissection, as in a reallife situation during ESS (both in vivo and on cadaver).
We also provide the replication of the interface between the endonasal structures and the orbital and intracranial content. Such boundaries are fundamental in understanding the limits of anatomical dissection and in preventing major complications of ESS, thus their reproduction should be as accurate as possible. Regarding this, the anatomical fidelity of our model was confirmed by the results of the surveys.
Our efforts to fabricate an increasingly realistic patientspecific simulator are in line with another recent experience of advanced ENT simulators based on 3D printing technology (Suzuki et al., 2021;Suzuki et al., 2022) which demonstrates how this kind of training tool may be beneficial to practice on difficult anatomies such as the ones in ESS field. The introduction of AR feature may have an incremental value for the simulator, since it combines the benefits of virtual and physical tools.
Other than its educational role in facilitating the understanding of the anatomy of the sino-nasal district, this application could be of particular interest in the surgical planning of complex ESS procedures, where the navigation across a pathological model could help the surgeon to decide upon a specific endoscopic approach and possibly to foresee critical intraoperative steps to avoid complications.
According to the face and anatomical content validity assessment, our simulator showed anatomic realism and utility in learning anatomy and developing psychomotor skills. We noticed that across the simulations of nondestructive tasks, the silicone of the facial skin changed its consistency resulting softer and softer. The neutral positions of some users in judging the haptic feedback of the nostrils and nose tip may be attributed to the first attempts on the model. Similarly, the realism in using the instruments and endoscope inside the nasal cavity received heterogeneous scores. Other than the changing rigidity of the external nose, an influencing factor could be the impossibility to fracture the turbinates and/or the nasal septum, and to use decongestant agents to enlarge the endonasal space, as routinely performed during ESS. The turbinates showed greater elasticity than the real structures; this caused a more difficult passage of the endoscope and instruments, either lateral or medial to them, inside the nasal cavity due to the spring back effect of the elastic material.
Overall, the operative time for non-destructive tasks was longer for residents than the experts as previously reported by other studies (Alrasheed et al., 2017;Zhuo et al., 2019). However, this difference was not significant, probably for the paucity of participants.
Ethmoidectomy and sphenoidotomy were scored more variably than other ESS steps among the experts, according to the task-specific validation questionnaire. Reviewing these results and the video recording of the procedures, we found that FIGURE 15 (A) 5-point Likert scale ratings for each question given by all participants (ENT experts). 1: strongly disagree; 2: disagree; 3: neutral; 4: agree; 5: strongly agree. (B) 5-point Likert scale ratings for DRAF II a, b and DRAF III given by the two experts. 1: strongly disagree; 2: disagree; 3: neutral; 4: agree; 5: strongly agree. In bold are reported the percentages for each rating given by the different members of the groups.
Frontiers in Bioengineering and Biotechnology frontiersin.org the inferior scores were given from the two experts dissecting the right sides. A higher number of ethmoidal cells filled with the printing support material were found on the right ethmoid compared to the contralateral. This made the dissection of this area more difficult and longer than in real ESS. Since both rightside dissectors chose to perform transethmoidal sphenoidotomy, these participants rated the access to the sphenoid less realistic. On the opposite side, sphenoidotomy which was performed through a parasettal route by one participant and transethmoidal route by the other, received the highest score in terms of replication. The experts agreed on the utility of the simulator during ENT residency, both for those naïve and those with some experience in cadaveric dissection, and confirmed the idea of including this kind of training tools in ENT residency programs.
The simulator we developed is primarily addressed to provide a training tool for ESS procedures mainly for residents, as an alternative or complementary solution to cadaveric dissection.
For this primary intended use, our idea is to select various exemplary cases of normal and pathologic sino-nasal anatomies and to reconstruct from them exemplary simulators that can be used by residents for ESS training at different levels of complexity. Therefore, the simulators are fabricated starting from patient-specific anatomy, thus requiring time for patient data collection, 3D model reconstruction, and then printing. However, in this case, the short production time is not a strict requirement as in the case of using the simulator for preoperative rehearsal the day before going to the operating room.
In the future, it could be interesting to foresee producing, for complex surgical cases, a patient-specific simulator to be used by the surgeon for preoperative rehearsal before surgery. In that case, surely the whole process should be speeded up.
The average time it took to obtain the current ESS simulator was about 16 h for the patient-specific anatomical part (8 h for image segmentation/3D model refinement and 8 h for printing process and cleaning), and about 20 h for the design and printing of the external box and face mold.
A strategy to reduce these times is to take advantage of the modular design of the simulator, in which the internal part relating to the patient-specific paranasal sinuses anatomy can be changed and customized from time to time, while the external part (box and face mold) can be reused.

Limitations
Despite all our efforts in the cleaning of the paranasal sinuses model, some residual printing support material remained inside some ethmoidal cells, reducing the anatomical reliability and the performance of selected ESS steps. Future studies will include improvements in the design of the model to entirely clean it and reproduce the variable degree of pneumatization of the paranasal sinuses.
The performance of ESS procedures was evaluated by a subjective Likert scale applied to not-validated questionnaires. However, for the purposes of this preliminary study, these methods could be considered acceptable in identifying positive feedback from the users. This limitation is shared with several other validation studies on the same topic (Alrasheed et al., 2017;AlReefi et al., 2017).
Our results are promising but the evaluation was performed on a restricted number of participants.
A future study will be planned to compare the ESS training based on our simulator and the one based on cadaver dissection, involving a higher number of trainees.
The AR feature we introduced can certainly be improved, e.g. by implementing the possibility to manipulate the virtual objects in addition to making them visible or not. Although we have received enthusiastic feedback from surgeons who have also experimented with the AR feature, its educational and training benefits must be more systematically evaluated.

Conclusion
This study demonstrated that a detailed multi-material 3D printed model of sinus anatomy can be generated with a high level of anatomical accuracy and an overall satisfactory haptic response. This technology has the potential to be used in surgical training of ESS procedures, as an alternative or complementary training solution to cadaveric dissection. It could also potentially play a significant role in preoperative planning, since it may provide an individualized preoperative design on which the surgeon can rehearse before surgery, to shorten operation duration and ensure safe and effective surgery in patients.

Data availability statement
The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.

Ethics statement
Ethical review and approval was not required for the study on human participants in accordance with the local legislation and institutional requirements. The patients/participants provided their written informed consent to participate in this study. Written informed consent was obtained from the individual(s) for the publication of any potentially identifiable images or data included in this article.
Frontiers in Bioengineering and Biotechnology frontiersin.org