Impact of nursing robotic assisted surgery education on Kirkpatrick's four levels

Introduction: As the robotic surgical burden of diseases increases, the development of skilled surgeons and nursing surgery teams is of fundamental importance. The Kirkpatrick Model is a widely recognised evaluation framework that helps educators assess the impact and effectiveness of educational interventions. However, there is little evidence regarding the effectiveness of robotic surgical training programmes for nursing teams. The study aimed to develop an organisational improvement model to identify the profile of a nurse expert in Robotic Assisted Surgery (RAS) and to define a suitable training programme according to Kirkpatrick's evaluation levels. The model also aimed to increase the number of active robotic rooms while maintaining the same number of staff.

Methods: From January 2023 to December 2024, a single hospital institution developed an organisational improvement project consisting of the following steps:- Identification of the catalogue of competences of the nurse expert in RAS, using the contributions of personnel already active in this field;- Mapping of competences;- Start of a training programme, evaluated according to the Kirkpatrick model;- Remapping of competences at the end of the training;- Gradual increase in the number of active robotic operating theatres by 2024.

Results: A total of 58 operating room nurses, including the 15 experienced nurses already on staff, had their competencies mapped. At the end of the training programme, 81% of these nurses had achieved the level of competency required to operate a robotic operating room independently, resulting in an overall increase of 154% in the number of active robotic operating rooms per week.

Discussion: Detailed planning of training activities dedicated to existing operating theatre nurses, eliminating the need for additional recruitment, together with periodic monitoring of newly acquired skills, has enabled the number of active robotic operating theatres to increase, significantly impacting the company's organisational model.

1 Introduction

Robotic Assisted Surgery (RAS) is one of the most significant innovations in surgical procedures, offering advantages such as greater precision, minimally invasive techniques and faster recovery times for patients. However, the success of these advanced techniques also hinges on the preparation and expertise of the support team, especially nurses, who are instrumental in ensuring the safety and efficacy of these interventions. Moreover, there is no single benchmark to identify the profile of a nurse expert in RAS or define a suitable training programme. Many studies analysed learning curves in robotics using at least 20 surgical procedures (1). It is therefore possible to argue that perioperative staff, including nurses, benefit from a minimum exposure of about 20 cases to attain a baseline skill level.

For these reasons, adequate and targeted training of nurses in RAS is essential, as it improves technical skills, optimises operating procedures, and ensures high-quality care (2, 3). According to Kirkpatrick's model (4), the effectiveness of this training can be evaluated in terms of not only satisfaction and learning, but also changes in professional behaviour and tangible results for patients.

This ensures that the training investment produces lasting and tangible results, improving clinical practice and safety in the operating theatre, and fostering better care quality, resource management and organisational effectiveness and innovation.

The main objectives of the study were therefore to develop an organisational improvement model to identify the profile of an expert nurse in RAS, implement a training programme for operating room nurses to acquire transversal skills in robotics and increase the number of active robotic rooms while maintaining the same number of staff.

2 Materials and methods

An educational training programme was implemented in a single Italian Tertiary Hospital from January 2023 to December 2024 with the aim of improving the RAS skills of operating room nurses. This project involved the following steps: (1) To define the nursing skills profile in RAS (for the Da Vinci, Versius and Hugo platforms) by creating a special catalogue with the help of nurses who are already working in this field. (2) To map the skills of operating theatre personnel. (3) Planning all training activities, including lectures, video viewing on an intranet platform, simulator-based activities in the operating theatre, shadowing experienced personnel, and individual mapping, for both newly recruited nurses specialising in RAS and existing personnel skilled in RAS with the Da Vinci platform, as well as for the management of surgical activities with the Versius and Hugo robotic platforms.

2.1 Catalogue of nursing skills in RAS

The Catalogue of Nursing Skills in RAS was defined using 15 nurses already on staff in the robotic operating room. This Catalogue was developed, according to the Association of periOperative Registered Nurses (AORN) (5), as a tool for identifying the activities and processes carried out by dedicated nurses in the context of RAS. This tool outlines the activities to be performed and the skills required by the organisation for this experienced professional role. The catalogue is organised by the stages of patient care within a macro-specialist area, such as urology, gynaecology, gastric surgery or general surgery: pre-operative phase (preparation for the surgical procedure; setting up of the operating room; sterile instrumentation); intra-operative phase (instrumentation during the surgical procedure); post-operative phase (closing of the surgical procedure; rearrangement of the operating room). The following additional relevant elements were also included in the context of the operating block: urgency/emergency conversion, problem management and coping with machine errors. Each specialist macro-area has been assigned a set of behaviours/activities. These indicate the reference standard for an activity “done well”. Table 1 describes all the behaviours to be followed in the Catalogue of skills to be mapped, which are divided by stages of patient care and by relevant elements. In accordance with level 3 of Kirkpatrick's model, each behaviour was given a score from 0 to 5 (0–1: novice; 2: advanced beginner; 3: competent; 4: proficient; 5: expert; according to Dreyfus five-stage model of adult skill acquisition) (6). A minimum score of 3 is required to achieve autonomy for each behaviour. Table 2 describes the definition of each skill acquisition stage and related score.

Table 1

Table 1. Catalogue of nursing skill to map the competencies of the RAS nurse in accordance with the stages of patient care and additional relevant elements.

Table 2

Table 2. Description of each skill acquisition stage and related score in the assessment of behaviour competences.

2.2 Training program

The training programme consisted of three phases. Phase 1 involved 9 h of theoretical and practical classroom lessons from January to March 2023. Phase 2 involved 12 h of dry lab simulation on robotic platforms (Da Vinci, Versius and Hugo) from April to June 2023. Phase 3 involved field training in the RAS operating theatre alongside experienced personnel from July to December 2023.

The theoretical and practical classroom lectures covered the following topics:

- The management of surgical conversions

- The point of view of the medical examiner and the jurist

- Overview of accessory materials and specialist instruments

- How to prevent patient positioning complications

- Correct mattress positioning

- Patient positioning systems

- Patient temperature management and Trendelenburg

- Overview of set-up preparation in the different surgical specialties

- Emergency management in RAS

- Troubleshooting, simulation and error management

- Scenario simulation and debriefing

The Kirkpatrick model (4) was used to evaluate the effectiveness of nursing education, considering the four levels:

1. Reaction: This level measures how nurses perceived the training course using a narrative approach. They were asked if they found the RAS training useful and interesting, and if they felt involved and motivated. In addition, participant satisfaction was measured using an institutional questionnaire containing three questions, scored using a 5-point Likert scale (see below).

2. Learning: This level assesses whether nurses have acquired the necessary skills and knowledge to use surgical robots correctly and handle procedures, through practical exercises.

3. Behaviour: This level concerns how nurses apply the new skills in the field to achieve an adequate degree of competence.

4. Results: The focus of this level is on evaluating the comprehensive effect on the hospital organisation as a whole, with a particular emphasis on the augmentation of active robotic operating theatres.

To assess trainees' reactions, we surveyed their satisfaction with a 5-point Likert global rating scale (1–5) of about 3 items: (1) How do you assess the relevance of the topics covered with respect to the subsequent need for updating?; (2) How do you assess the educational quality of this training programme? (3) How do you rate the usefulness of this training event for your training/upgrading?

To assess learning, a checklist was used in which the trainer indicated whether the learner had achieved the following learning objectives at the end of the first two phases of the training programme:- The learner learnt how to make connections and switch on the system;—The learner learnt how to configure the system according to the different types of surgical procedure;—The learner learnt how to handle and recognise standard and specialised poly-use and single-use instruments;—How to handle the main types of potentially occurring errors;—How to manage emergencies in RAS (conversion);—General rules of patient positioning in RAS.

For the evaluation of behaviour, at the end of phase 3 of the training programme, the competences were remapped as per the catalogue. The behaviour assessment was carried out by two experienced operating room nurses (E.V. and L.D.C.) from the research team. They had a Master's degree in nursing coordination and were appropriately trained in the use of Likert-type scales. The two assessors conducted the evaluations independently, and any disagreements were discussed to reach a consensus on a final, unambiguous assessment.

The final profile of autonomous nurse in RAS was defined as follows:

Competent:

- At least 20 robotic surgery cases assessed.

- Completion of the formal robotic surgery training programme.

- Passing the competency assessment with a minimum score of 3 for each behaviour.

Proficient:

- At least 30 robotic surgery cases assessed.

- Completion of the formal robotic surgery training programme.

- Passing the competency assessment with a minimum score of 4 for each behaviour.

Expert:

- At least 50 robotic surgery cases assessed.

- Completion of the formal robotic surgery training programme.

- Passing the competency assessment with a minimum score of 5 for each behaviour.

Kirpatrick Level 4 was evaluated in terms of the total number of active operating theatres and patients undergoing RAS in 2024 compared to 2023. All surgeons involved are experts in robotics and their numbers did not change during the study period.

2.3 Statistical analysis

Descriptive statistical methods were used to identify the basic characteristics and results of the general education survey of the study population. The overall reaction of the participants was expressed in mean and standard deviation according to each item. To assess learning, changes in participants' overall competency and their achievement of goals and objectives of each surgical skill from beginning to end were assessed with a paired t test. Behaviour change was expressed with mean and standard deviation. Spearman's Rank Correlation was used to measure inter-rater reliability (IRR) between assessors. All statistical analyses were performed using SPSS Statistics version 29.

3 Results

A total of 58 operating room (OR) nurses, including the 15 identified as expert nurses (both OR and first assistant nurse; 13 females and 2 males), were mapped to construct the competence catalogue. From January to December 2023, all nurses with a non-expert profile (43/58; 9 males and 34 females; mean score 1.22 ± 1.22) undertook the training programme, receiving theoretical and practical training in a dry lab (phases 1 and 2) and in the field (phase 3).

3.1 Level 1: reaction of trainees

At the end of the phase 1 and 2 training programme, a 5-point global rating scale survey was conducted among the 43 trainees. The overall satisfaction score was 4.79 ± 0.42 out of 5.00, which is between “good” and “very good”. Trainees rated the programme highly in terms of its relevance to their needs and interests (4.86 out of 5.00), its educational quality (4.72 out of 5.00) and its usefulness for personal development (4.81 out of 5.00). Only one trainee found the programme useful. None of the trainees thought that the training event was poor or neutral. Using a narrative approach, the following feedback was left by most of those who had not had previous experience of robotics: “The course was well structured, but some of the theoretical notions could be expanded upon in more surgical areas than the room. For those who have never used the robot or use it infrequently, it would be useful to have an introductory section on all the robotic components, the irons and the sterile dressing of the robot, as well as their specific names. The simulations that help you realise the difference between theory and practice were also very useful.”

3.2 Level 2: learning

All participants had achieved all the required learning objectives by the end of phases 1 and 2 of the programme.

3.3 Level 3: behaviour change

By the end of the entire training programme, 32 out of 43 nurses (mean score: 3.45 ± 0.95, p = 0.000) had demonstrated the ability to independently manage a robotic operating theatre from a nursing perspective (at a competent level). IRR between assessors was significantly high (0.992; p > 0.01). Figure 1 shows the distribution of competence levels before and after completing the training programme.

Figure 1

Figure 1. Distribution of competence levels before and after completing the full training programme.

3.4 Level 4: organisational change results

A total of 859 robotic operating theatres were activated in 2024 vs. 480 in 2023, with an overall increase of 154% (Figure 2). The number of operating theatres that were activated increased in all of the macro-specialty areas identified (Urology 384 in 2024 vs. 288 in 2023; Gynaecology 203 vs. 0; Gastric surgery 127 vs. 96; General surgery 145 vs. 96). This resulted in an increase in the total number of patients treated with RAS in the (559 in 2023 vs. 1004 in 2024) (Figure 3).

Figure 2

Figure 2. Trend in the monthly number of robotic operating theatres activated from December 2023 to December 2024.

Figure 3

Figure 3. Number of patients treated with robotic stratified by type of surgery, 2023 vs. 2024.

4 Discussion

Kirkpatrick's four-level model of evaluation and training assesses participants' reactions to the training, what they learn from the programme, how they apply the knowledge in real practice and transfer it to others, and the results of the training programmes with regard to clinical outcomes and organisational goals and objectives (4). This model is widely used to evaluate nursing education. However, the outcomes level (level IV) is the most challenging to achieve (7). The strength of our study is that, to our knowledge, it is the first to meet all four levels of the Kirkpatrick model in robotic nursing training. Indeed, few studies in the literature have directly investigated Kirkpatrick levels in the context of robotic nurse training. Instead, they tend to address related issues, such as the effectiveness of robotic surgery training and its potential impact on nurses' skills and clinical outcomes.

As far as Level I is concerned, most of the participants in our study were satisfied with the training programme they underwent. The feedback from those with no prior experience of RAS is consistent with findings in the literature. Indeed, a recent survey of 114 operating room nurses working in 12 institutions that have performed robotic surgery in Turkey demonstrated that nurses have a favourable opinion of robotic surgery; nurses with experience in robotic surgery exhibit higher levels of individual innovativeness; and nurses who have received training are significantly more adapted (8). A recent literature review on education and training of operating room nurses in RAS found that only 3 studies described the development of training programmes for operating room nurses working with RAS (2, 9, 10, 18). In 2001, Connor et al. described a learning laboratory for OR nurses working with a cardiothoracic surgical robot. They concluded that the approach to adapting this kind of technology should be stepwise and parallel to its development (9). A RAS programme to educate and assess OR nurses' thoracic RAS competencies for both technical and non-technical skills was described by Sarmanian et al. They described how an OR equipped with robots would be ideal for learning exercises, as learners could immerse themselves in the actual practice environment. They also emphasised the importance of leadership support for success (18). None of them focused on Levels 1 or 2 of the Kirkpatrick model.

A study evaluated medical and nursing students' knowledge of and interest in robotic surgery, analysing how this may change after a hands-on training course using new Hugo RAS and Versius System simulators (11). Before the hands-on course, 44.0% of nursing students were interested in surgical disciplines and robotic surgery. After the simulation, all students reported a high level of interest in robotic surgery and some required a dedicated internship (p < 0.001). The students also provided feedback on the ease of use of the robotic simulators (on a scale 0–10), with a median score of 8, and no differences between nursing and medical students. This study demonstrated a great interest in robotic surgery as part of medical and nursing education.

We observed that at the end of the training programme, 32 out of 43 nurses (with a mean score of 3.45 ± 0.95, p = 0.000) proved able to manage a robotic operating theatre independently from a nursing point of view. Collins et al. developed a new, stepwise robotic training programme to broaden and standardise training for bedside assistants (19). Thirteen participants were evaluated to determine if the training programme enhanced their confidence and comfort. A survey was conducted before and after the training programme and at different stages. All participants reported statistically significant increases in confidence and comfort after the training programme (p < 0.001) and at each stage (p < 0.001). Feedback regarding curriculum improvement was obtained, suggesting more training and practice in smaller groups. All participants felt RAST was beneficial and that it should be implemented as standard training during onboarding for all robotic bedside assistants.

Using Likert-type scales to assess behaviour may expose several limitations, such as the tendency to give a central response, subjective interpretation of options, the assumption of equal intervals and limited qualitative depth (12). Nevertheless, these scales are often used in literature, particularly in training, due to certain strengths (13, 14). For example, they can be used to evaluate attitudes and perceptions (15). In training, it is crucial to understand how involved, motivated or satisfied participants feel, and the Likert scale is perfect for this. They can also be used to measure perceived effectiveness. Participants can be asked to express their degree of agreement with statements. The Likert scale offers a common metric, making it easy to standardise and compare results. It is also easy to administer, as Likert-type scales work well both on paper and in digital format, and data interpretation analysis is straightforward.

Level IV of the Kirkpatrick model identifies the added value of a programme to society by evaluating intervention design application in health professionals' practice and the results of its use in a local context (16). The most common reason for excluding Level IV was the need for medium- and long-term follow-up periods, as well as the complexity of relating outcomes to the training programme and distinguishing them from other factors (17). Kirkpatrick's level IV is considered the most challenging as it assesses the impact of training on organisational practice. This requires additional instruments, information and stakeholder input, as well as more time. We overcame these issues by promoting the project to the hospital's strategic management. This enabled us to manage and coordinate scheduling, space, training and human resources, achieving the objectives for organisational change. There are no data in the literature for comparison.

The number of active robotic operating theatres has increased significantly as a result of a combination of detailed planning of training activities for existing operating theatre nurses, which has eliminated the need for additional recruitment, and periodic monitoring of newly acquired skills. This has had a substantial impact on the company's organisational model. During the study period the number of procedures per room per day did not increase. To assess the impact of our training programme also on the number of procedures per operating room per day, a longer follow-up period is required. Monitoring this parameter in future could certainly provide further data on procedure productivity and workflow efficiency, as well as information on the increase in the system's overall capacity (18).

Data availability statement

The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.

Author contributions

MC: Writing – original draft, Conceptualization, Formal analysis, Supervision, Methodology, Writing – review & editing, Validation. EV: Project administration, Data curation, Conceptualization, Supervision, Writing – original draft, Visualization. FM: Writing – review & editing, Validation. NM: Validation, Writing – review & editing. MA: Data curation, Investigation, Writing – review & editing. EP: Data curation, Writing – review & editing, Investigation. NV: Investigation, Writing – review & editing, Data curation. EV: Writing – review & editing, Data curation, Investigation. DG: Investigation, Writing – review & editing, Data curation. AB: Writing – review & editing, Data curation, Investigation. AV: Validation, Writing – review & editing. AP: Validation, Writing – review & editing. RB: Validation, Writing – review & editing. AA: Writing – review & editing, Validation. RL: Writing – review & editing, Data curation. LD: Writing – review & editing, Data curation. MZ: Methodology, Writing – original draft, Conceptualization, Validation, Supervision, Project administration, Writing – review & editing.

Funding

The author(s) declare that no financial support was received for the research and/or publication of this article.

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Generative AI statement

The author(s) declare that no Generative AI was used in the creation of this manuscript.

Any alternative text (alt text) provided alongside figures in this article has been generated by Frontiers with the support of artificial intelligence and reasonable efforts have been made to ensure accuracy, including review by the authors wherever possible. If you identify any issues, please contact us.

Publisher's note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

References

1. Soomro NA, Hashimoto DA, Porteous AJ, Ridley CJA, Marsh WJ, Ditto R, et al. Systematic review of learning curves in robot-assisted surgery. BJS Open. (2020) 4(1):27–44. doi: 10.1002/bjs5.50235 32011823

PubMed Abstract | Crossref Full Text | Google Scholar

2. Møller L, Olsen RG, Jørgensen L, Hertz P, Petersson J, Røder A, et al. Training and education of operating room nurses in robot-assisted surgery: a systematic review. Surg Endosc. (2024) 38:7024–36. doi: 10.1007/s00464-024-11335-3

PubMed Abstract | Crossref Full Text | Google Scholar

3. Moloney R, Coffey A, Coffey JC, Brien BO. Nurses’ perceptions and experiences of robotic assisted surgery (RAS): an integrative review. Nurse Educ Pract. (2023) 71:103724. doi: 10.1016/j.nepr.2023.103724

PubMed Abstract | Crossref Full Text | Google Scholar

4. Kirkpatrick JD, Kirkpatrick WK. Kirkpatrick’s Four Levels of Training Evaluation. Alexandria, VA: ATD Press (2016).

Google Scholar

5. Francis P. The evolution of robotics in surgery and implementing a perioperative robotics nurse specialist role. AORN J. (2006) 83:630–50. doi: 10.1016/S0001-2092(06)60191-9

PubMed Abstract | Crossref Full Text | Google Scholar

6. Dreyfus SE. The five- stage model of adult skill acquisition. Bull Sci Technol Soc. (2004) 24(3):177–81.

Google Scholar

7. Miranda FM, Santos BVD, Kristman VL, Mininel VA. Employing Kirkpatrick’s framework to evaluate nurse training: an integrative review. Rev Lat Am Enfermagem. (2025) 33:e4431. doi: 10.1590/1518-8345.7250.4431

PubMed Abstract | Crossref Full Text | Google Scholar

8. Silveira Thomas Porto C, Catal E. A comparative study of the opinions, experiences and individual innovativeness characteristics of operating room nurses on robotic surgery. J Adv Nurs. (2021) 77:4755–67. doi: 10.1111/jan.15020

PubMed Abstract | Crossref Full Text | Google Scholar

9. Connor MA, Reinbolt JA, Handley PJ. Perioperative nurse training in cardiothoracic surgical robotics. AORN J. (2001) 74:851–7. doi: 10.1016/s0001-2092(06)61503-2

PubMed Abstract | Crossref Full Text | Google Scholar

10. Zender J, Thell C. Developing a successful robotic surgery program in a rural hospital. AORN J. (2010) 92:72–6. doi: 10.1016/j.aorn.2009.10.024

PubMed Abstract | Crossref Full Text | Google Scholar

11. Sighinolfi MC, Terzoni S, Scanferla E, Bianchi PP, Formisano G, Gaia G, et al. Impact of hands-on practice with HugoRAS and versius system simulators on the attractiveness of robotic surgery among medical and nurse undergraduate students. J Robot Surg. (2023) 17(4):1471–6. doi: 10.1007/s11701-023-01543-9

PubMed Abstract | Crossref Full Text | Google Scholar

12. Bishop PA, Herron RL. Use and misuse of the Likert item responses and other ordinal measures. Int J Exerc Sci. (2015) 8(3):297–302. doi: 10.70252/LANZ1453

PubMed Abstract | Crossref Full Text | Google Scholar

13. Carifio L, Perla R. Resolving the 50 year debate around using and misusing Likert scales. Med Educ. (2008) 42:1150–2. doi: 10.1111/j.1365-2923.2008.03172.x

PubMed Abstract | Crossref Full Text | Google Scholar

14. Norman G. Likert scales, levels of measurement and the “laws” of statistics. Adv in Health Sci Educ. (2010) 15:625–32. doi: 10.1007/s10459-010-9222-y

Crossref Full Text | Google Scholar

15. Likert R. A technique for measurement of attitudes. Arch Psychol. (1932) 140:5–55.

Google Scholar

16. Castro JA, Filho Motta LB. Evaluation in distance learning: a case report of the UNASUS/UERJ postgraduate course in elderly health. Rev Bras Geriatr Gerontol. (2018) 21(5):513–22. doi: 10.1590/1981-22562018021.180021

Crossref Full Text | Google Scholar

17. LaMar RA. Using a virtual journal club for sharing evidence-based practice recommendations in critical care registered nurses. Teach Learn Nurs. (2017) 12(1):53–8.

Google Scholar

18. Sarmanian JD. Robot-assisted thoracic surgery (RATS): perioperative nursing professional development program. AORN J. (2015) 102:241–53. doi: 10.1016/j.aorn.2015.06.013

PubMed Abstract | Crossref Full Text | Google Scholar

19. Collins JM, Walsh DS, Hudson J, Henderson S, Thompson J, Zychowicz M. Implementation of a standardized robotic assistant surgical training curriculum. J Robot Surg. (2022) 16(4):789–97. doi: 10.1007/s11701-021-01291-8

PubMed Abstract | Crossref Full Text | Google Scholar