Implementation and assessment of a novel non-clinical skills curriculum for urology residents

Abstract

Background:

Urology is an increasingly competitive specialty that procures a highly selected and clinically excellent cohort of residents. However, other training needs such as leadership and professional development go underrecognized despite an identified need for formal training in these areas. The aim of this study was to implement, evaluate, and pilot a non-clinical skills curriculum, a novel individualized professional development workshop series, at a single institution.

Methods:

Eighteen urology residents (15/year, 3 graduates/year) participated in this study over the course of two academic years. A pre-curriculum needs assessment was completed by 15 residents in Year 1 for purposes of curriculum design. The curriculum itself was a series of 1-hour monthly workshops given by an expert speaker on topics relevant to healthcare delivery, leadership and career promotion across various contexts. Survey-based assessments tracked gains in subject knowledge and satisfaction via a pre-post test design.

Results:

The pre-curriculum needs assessment indicated that trainees desired additional instruction in non-clinical skills (ps>0.1) and endorsed formal teaching to ensure success in their future careers (p<0.001). Trainees reported pre- to post-curriculum gains across each individual learning topic (Mean=20%, p<0.001) with an aggregate increase in subject knowledge of 17% for senior residents and 21% for junior residents (p<0.001).

Conclusion:

A non-clinical skills curriculum implemented as a pilot ‘Hidden Curriculum’ for urology trainees was feasible and resulted in significant gains in non-clinical subject knowledge. Workshops were highly rated and trainees reported high satisfaction with the curriculum.

Introduction

With a 2022 Match rate of 66% (1), urology remains one of the most competitive residency programs and thus attracts some of the most qualified medical school applicants (2). Urology residents tend to excel clinically, with a 92% residency completion rate (3) and 97% pass rate in the qualifying examination over the past five years (4). While urology residency programs are frequently evaluated on clinical (5, 6), surgical (7, 8), and research-based training (9, 10), little work has been published addressing non-clinical skills (NCS), concepts not formally taught in a residency setting but deemed useful for the career of a developing urologist, despite an identified need for training in leadership (11, 12) and career development (13) in surgical specialties.

Given that participation of urology residents in patient care impacts patient satisfaction and outcomes (14) and residents with better leadership and directional skills are more efficient in the operating room (15), we must also arm our surgical trainees with solid NCS in communication, leadership, and administration. NCS in surgery have been correlated with higher patient satisfaction scores, lower rates of burnout, greater academic success, improved patient-physician relationships (16), lower rates of adverse iatrogenic events in surgery (17), and advanced technical skills (15). Furthermore, recent reports have called for formalized leadership and NCS training in surgical residency programs (11, 12, 18, 19). In one study, a majority of practicing urologists rated NCS training as “useful” but felt that residency training left them inadequately prepared for these aspects of practice (20). A more recent review identified deficiencies in NCS training in modern urological curricula and strategies to address them (21).

Thus, this study seeks to address a critical gap in trainees’ professional development by designing, implementing, and evaluating a NCS curriculum intended to hone skills in medical leadership and strengthen related competencies as a pilot model for a feasible ‘Hidden Curriculum’ for Urology residents.

Materials and methods

Study sample

Over the course of two academic years, 18 urology residents participated in this prospective curriculum at a large Midwestern public academic university. In Year 1, all 15 departmental urology residents completed a pre-curriculum needs assessment, participated in planned 1-hour learning workshops (where they completed a post-timepoint evaluation), and completed an identical post-curriculum assessment. Given the success of the curriculum pilot in Year 1, it was continued in Year 2. A pre/post-curriculum assessment was not given in Year 2, since this had already been collected in Year 1 and was primarily used for curriculum design. During Year 2, again all 15 departmental residents were enrolled in the sessions; 3 post graduate year (PGY) 5 residents completed training at the end of Year 1 and 3 new PGY-1 residents entered the program in Year 2. Junior residents were defined as PGY 2-3 while senior residents were defined as PGY 4-5. This study was considered exempt by the Institutional Review Board at our institution.

Curriculum design

In order to identify deficiencies in current training as well as to tailor the curriculum to our group of residents, a focused pre-curriculum needs assessment was completed by all 15 urology residents in our program at the start of the 2019-2020 academic year (Table 1). A copy of the pre-curriculum needs assessment can be found in Supplementary File 1.

In this assessment, residents were asked to rate their knowledge of specific topics proposed for learning sessions. This needs assessment was created internally by our department’s Education Specialist (PhD) with >11 years of expertise in survey creation and design. Evaluations also underwent iterative review by our Education Leaderhip (FB, DS, or CL), who also serve as members of our departmental Education Working Group, a 10-12 person internal committee of medical and surgical educators who review all proposed and ongoing curricula in the department on a quarterly basis. The Education Working Group and these faculty educators came up with the learning topics presented in our pre-curriculum needs assessment. Our goal was to create learning sessions for any topic not rated at a ‘high’ knowledge level, defined as 4 out of 5 on a 1-5 Likert-scale. Any topic with a mean value statistically significantly lower than 4 was created as a learning session in Year 1 or 2 (Table 1B).

The curriculum itself was initiated in August 2019 and consisted of a series of 1-hour monthly workshops given by an expert speaker (Table 2). Attendance by residents for each learning session was mandatory, barring patient care emergencies or pre-established vacation time. In Year 1, the last 2 sessions took place virtually via Zoom, given institutional restrictions on in-person gatherings due to the COVID-19 pandemic. Following each learning session, trainees filled out a short (<5 minute) timepoint assessment of the session (Supplementary File 1). At the end of Year 1 (2019–2020), a post-curriculum assessment identical to the pre-curriculum needs assessment, was administered electronically via REDCap (Research Electronic Data Capture) to all urology residents.

In Year 2 (2020-2021), an identical format was followed for the curriculum, with notable exceptions of all sessions taking place virtually via Zoom and no pre- or post-assessment administered (timepoint assessments only).

Learning sessions

Learning session topics came directly from resident feedback via the pre-curriculum needs assessment. Expert speaker identification and scheduling was performed by the department’s Education Specialist (PhD) with input from Education Leadership (FB, DS, or CL). Expert speakers were all internal to our College of Medicine and were identified by the Education Leadership as knowledgeable in that topic area. The format of each session was determined by the presenter and broadly categorized into (a) primarily lecture-based or (b) “interactive” (with time for learners to engage with institutional experts and leaders), with adjunct question-and-answer sessions for both sessions.

For example, the “Urology Administration and Governing Boards” session was a Powerpoint-based lecture delivered by the program director overviewing academic and professional hierarchies and memberships with time for questions at the end. On the other hand, the “Leadership Styles” session delivered by our institution’s chief medical officer was a loosely-structured interactive discussion between the trainees and the presenter. Other sessions such as “Teaching Skills/Feedback” had elements of lecture and discussion, but was ultimately categorized as “interactive” due to >50% time being spent in a discussion format.

Measures

Study measures consisted of surveys administered to subjects prior to participation in the curriculum in Year 1 (“pre-curriculum needs assessment”), following each learning session in Years 1 and 2, (“timepoint assessments”), and at the conclusion of the curriculum in Year 1 (“post-curriculum assessment”). We also report measures of study feasibility including cost and attendance.

Statistical analysis

Study data were collected and managed using REDCap electronic data capture tools hosted at our institution (22). REDCap was used for survey creation, distribution, and data storage.

All data analysis was performed using SPSS Statistics software (IBM SPSS Statistics for Windows, version 27.0. Armonk, NY: IBM Corp). Data are presented as means (standard deviations) or proportions (percentages). Per study objectives, analyses were performed comparing group differences (e.g. across timepoints; or junior vs. senior residents) via chi-square tests for categorical variables, paired t-tests for continuous variables (e.g. pre- vs. post-test), or independent t-tests for continuous variables (e.g., junior vs. senior residents). Single sample t-tests were used to compare single group means (e.g., gains = post – pre) to a predetermined standard (e.g. zero). All figure error bars indicate standard error of the mean.

Results

Demographics

Demographics information is listed in Table 3. Participants were 86% Caucasian, 60% married or living with a partner, and the mean age was 28.9 years (SD=2.73). Residents were 53% male and 47% female.

Pre-curriculum needs assessment

Urology residents (n=15) were asked to rate their satisfaction/dissatisfaction with their NCS training prior to participation in the NCS Curriculum and speculate on additional training needed to prepare them for career success (Tables 1A, B). Prior to participating in our new curriculum, residents reported that they were only “neutral” to “slightly satisfied” with their NCS and abilities [Mean (M)=3.36/5, p=0.055)], despite the fact that they thought a “moderate amount” to “significant amount” of non-clinical training was necessary to succeed in their current position and program (M=3.43/5), as well as in their future careers (M=3.64/5, ps<0.001). Residents rated their pre-curriculum NCS training on average between “slightly unsatisfied” and “neutral” (M=2.21/5, p=0.001), with slightly more education provided informally, although residents indicated this was still a “neutral” amount (M=3.07/5, p=0.720). Junior and senior residents did not differ in their opinions (ps>0.1), except that junior residents felt they received more informal training (M_Senior=2.83/5 vs. M_Junior=3.25/5, p<0.001, Table 1A).

With respect to individual topics, residents self-reported mostly “below average” or “average” knowledge (i.e., a mean statistically less than 4 out of 5 on a 5-point Likert scale) overall (M=2.98/3, p=0.902), as well as by topic (Table 1B).

The trainee cohort felt the NCS Curriculum would be most beneficial for junior residents (86.7%) and senior residents (80.0%), followed by chief residents (73.3%), interns (60.0%), fellows (46.7%), and medical students (33.3%). Trainees on average “agreed” that both junior (M=4.0/5, p<0.01) and senior (M=4.28/5, p<0.01) residents should be required to participate in the curriculum (Table 1A).

Learning assessments

Learning sessions spanned two academic years and both in-person and virtual formats (due to COVID-19 restrictions). We observed no difference in satisfaction with the curriculum or self-reported gains in content knowledge between the first and second academic year (ps>0.1), or between an in-person vs. virtual format (ps>0.1). Thus, learning sessions have been analyzed and reported together. Broadly, the curriculum was rated very highly in terms of key measures (Figure 1). Evaluation by individual learning session appear in Table 2.

Figure 1

Gains in Subject Matter Knowledge: Trainees were asked to rate their knowledge of the subject matter before vs. after participation in each learning session. Broadly, trainees demonstrated significant gains in content knowledge overall (Mean=20%, p<0.001) and across each learning session (Figure 2). The greatest gains across learning sessions were observed for junior residents (21%) followed by senior residents (17%).

Figure 2

Pre- to post-curriculum gains

Trainees indicated small but significant gains on the broad topics that emerged from the pre-curriculum needs assessment. Specifically, they indicated that, following participation in the first-year learning sessions, their level of satisfaction in the amount of NCS training they were currently receiving had increased (p=0.011) and that they were satisfied with this current offering (p=0.047). They continued to report that a high level of NCS training was necessary in both their current role and future careers, and this did not differ from pre-test (M=3.88/5, p=0.351; M=3.50/5, p=1.0). The increase in satisfaction was higher for junior residents (p=0.034) than senior residents.

We also assessed gains in knowledge for specific topics covered in Year 1. Broadly, trainees indicated significant pre- to post-curriculum gains (p=0.004), with the greatest gains seen for Leadership Styles (p=0.033), Urology Administration/Governing Boards (p=0.064), and Teaching/Feedback (p=0.095). Junior and senior residents generally did not differ in their knowledge gains (ps>0.2 except Gender: p=0.034).

Learner engagement

Although not part of initial study design, based on resident feedback, we decided to investigate degree of learner engagement for each session post-hoc. Though we did not objectively measure degree of trainee interaction for each session we were able to broadly categorize each session as lecture-based or non-lecture based (i.e., interactive in nature) post-hoc. While each session in the curriculum produced statistically significant gains in subject matter, there was consistently higher resident satisfaction for sessions that were more interactive and less didactic or lecture-based, namely Teaching/Feedback, Leadership Styles, Gender and Medicine/Surgery, Preparing for Fellowship, and Private and Community Practice Panel (Table 2).

Curriculum feasibility

Measures of curriculum feasibility included attendance and cost. Attendance ranged from 10 to 14 residents (66.7%-93.3%) for each timepoint curriculum, with 1-2 residents typically out on vacation for that week and 1-2 residents typically tending to patient care emergencies. We estimated the cost of our curriculum at $0 - $1,000 per year. This cost is estimated based on: room rental ($0 at our institution), expert speaker honorarium (we gave $50/speaker), cost of parking ($0 given trainees and speakers already paid for this as members of our institution), and administrative office supplies (<$100/year). Zoom was also provided for free from our institution (virtual sessions).

Discussion

Here we describe the development, implementation, and evaluation of a novel NCS-based curriculum (e.g., leadership, teaching, professional development, etc.) for urology trainees at our institution as piloted over a two-year timeframe via pre-post test design. In an era of continuous evaluation and feedback-driven implementation of changes to residency curricula, there is a scarcity of literature on this topic pertaining to urology residency programs.

Our novel NCS curriculum was generally well-received by trainees and resulted in statistically significant subjective knowledge gains. While these gains were most notable for junior trainees, there was evidence of improvement in NCS knowledge across all subgroups. The 1-hour sessions were rated highly, which prompted continuation of the curriculum at our institution for a second academic year.

While other leadership curricula have been implemented within surgical residencies (23–26), this study represents the first known reported to the authors designed and tailored specifically to a urology residency, though there are other examples being implemented to address this training gap. For example, the New York Section of the AUA recently ran an 8-part lecture series (available on YouTube) aimed at addressing topics not classically learned in residency, which included a multi-institutional array of guest speakers (27). This series was part of a larger 68-part educational lecture program initiated during the COVID-19 pandemic in an effort to improve resident distance learning, though the cohort was only comprised of 50% residents due to interest from attendings, fellows, and medical students (28). As the need for NCS training in urology is further elucidated, we anticipate the published landscape of this important topic within our specialty will continue to develop and buy-in continue to increase. Notably, there was strong resident buy-in at our institution resulting in a consistent sample group throughout our study.

One reviewer raised an interesting question, that is, whether our graduate medical education office provided any of the education as part of our series. None of our sessions were provided formally by our Office of Graduate Medical Education, however, several of the expert speakers work closely as medical educators with the Office of GME and give similar presentations to other interested departments or in other informal settings. Our group previously worked with the Office of GME at our institution to design, implement, and evaluate a leadership-specific curriculum via similar format. As this manuscript is still under review, we have not cited it here. This unpublished data does show that residents and fellows gained perceived leadership skills following participation in GME-designed content, and this was applicable across specialties. As we have continued to develop this curriculum past this initial implementation, we have continued to work with our institution’s Office of GME to integrate both their educators and available content as relevant (e.g. designing a QI study/fundaments of QI).

We have continued to implement this ‘Hidden Curriculum’ series for our residents since this pilot implementation. As learning has primarily returned to in-person for other conferences, so has this series, with virtual options as needed by speakers, although this is rare. Format and time-wise, the education has remained the same. Sessions continue to be scheduled and coordinated by the Department’s Education Specialist PhD and run by expert speakers in relevant topics, which are either suggested by or reviewed by the Department’s Education Working Group. Based on the feedback in our pilot implementation, some topics are reviewed annually, while some topics will be reviewed only every 2-3 years. As new topics are needed or available, those are added to the rotation and similar metrics are collected for evaluation and continued use.

Given our results, we propose that best practice for curriculum design and implementation – based on our pilot model and some ‘best practices’ developed by our education leadership team – should include a pre-curriculum needs-assessment, protected education time, expert speakers, dedicated support from the residency program coordinator, a pre-post design, and serial learning evaluations to permit continuous feedback and curriculum plasticity. Although these elements are certainly reproducible, the authors recognize that specific topics may vary from program to program and would like to reiterate that a pre-curriculum needs assessment is necessary to optimize the curriculum for individual residency programs.

An additional pre-curriculum tool for educators to assist in building a tailored leadership curriculum used by some studies is the Leadership Practices Inventory (LPI) developed by Posner and Kouzes (29). The results from this tool, which identifies the frequency of 30 leadership behaviors for each participant, may be pooled to identify deficiencies in the curriculum participant cohort and tailor learning sessions to focus on deficient areas.

Although junior and senior residents indicated similar desire for specific topics across the board, unsurprisingly the gaps in knowledge for each session were uniformly more pronounced for juniors. While each session in the curriculum produced statistically significant gains in subject matter, there was consistently higher resident satisfaction for sessions that were more interactive and less didactic or lecture-based. This finding is in agreement with prior published literature suggesting that interactive learning is superior to traditional didactic-based learning in medical education (30). Thus, further development of this curriculum may focus on establishing more interactive sessions.

Though participation was robust and two years of data are reported (i.e., multiple timepoints), knowledge assessment were self-reported and therefore subjective. Common limitations in assessing leadership competencies were thus likely at play in this study including the “honeymoon effect”, “horizon effect”, and “Hollywood effect” (31). Future studies could utilize alternative strategies in leadership assessment including multi-rater evaluations and post-program reflections.

We did not collect demographic data on pathway to medicine (traditional vs non-traditional), presence of medical professionals in the family, and alternate career experience, though these could be interesting variables to examine in future studies to determine whether these factors influence the usefulness/satisfaction of a NCS Curriculum.

Although we did not prospectively include medical students, as a function of our education and rotation schedule, we did have some advanced medical student rotators (n=13, 1-4 per session, single session attendance only) attend these timepoint sessions and asked them to fill out the respective timepoint evaluations, along with our residents. The medical student data is consistent with data collected from residents, indicating that the curriculum was generally felt as useful and well-received. Specifically, reported gains in subject matter knowledge ranged from 22% to 47% (Mean=32%). Medical students similarly reported that the curriculum was beneficial to their education and professional development (Mean=4.2/5), improved their teamwork abilities (Mean=4.1/5), that they would recommend it to a peer (Mean=4.5/5), that it was intellectually challenging (Mean=3.25/5), that it increased their subject matter knowledge (Mean=4.77/5), that it allowed them to practice skills taught in the course (Mean=3.79/5), increased their problem solving skills (Mean=3.88/5), and increased their leadership skills (Mean=3.82/5). Future work should prospectively examine this type of clinical skills education at the medical student level.

Lastly, one could argue that results from this single-institution study may lack generalizability and thus additional future directions include development of a multi-institutional curriculum as well as expanding curriculum topics. The authors do plan on continuing the NCS curriculum at our instution, with plans to cycle through topics every 2-3 years and continue to collect data for quality improvement and curriculum development.

In conclusion, we report the first results (to the authors knowledge) designing, implementing, and assessing an NCS-based curriculum for urology residents. The curriculum was developed to address a critical need within urology resident education and was broadly utilized, well-received, and resulted in measurable subjective gains to NCS and leadership competencies. The ongoing curriculum was affordable, feasible, easily organized, and individualized based on feedback elicited from our trainees. Given its success, the authors recommend similar formal training within urology and other surgery-based residency programs and offer resources for program leadership to institute similar curricula at other institutions.

References

• 1

  American Urological Association, Society of Academic Urology. Urology residency match statistics. 2022. (2022). Available at: https://www.auanet.org/meetings-and-education/for-residents/urology-and-specialty-matches

  • Google Scholar

• 2

  HalpernJADinermanBCosianoMFLeeRKSingerJSHuJC. National trends and characteristics of success in the urology match process. Urol Pract (2018) 5(4):311–6. doi: 10.1016/j.urpr.2017.04.005

  • CrossRef
  • Google Scholar

• 3

  ThompsonRHLohseCMHusmannDALeibovichBCGettmanMT. Predictors of a successful urology resident using medical student application materials. Urology (2017) 108:22–8. doi: 10.1016/j.urology.2017.06.046

  • CrossRef
  • Google Scholar

• 4

  pAmerican Board of Urology. National results of the qualifying (Part 1) examination (2021). Available at: https://www.abu.org/images/pdfs/National-Stats-QE.pdf.

  • Google Scholar

• 5

  WitherspoonLJalali STRobertsM. Resident-run urology clinics: a tool for use in competency-based medical education for teaching and assessing transition-to-practice skills. Can Urol Assoc J (2019) 13(9):E279–84. doi: 10.5489/cuaj.5710

  • CrossRef
  • Google Scholar

• 6

  OkhunovZSafiullahSPatelRJuncalSGarlandHKhajehNRet al. Evaluation of urology residency training and perceived resident abilities in the united states. J Surg Educ (2019) 76(4):936–48. doi: 10.1016/j.jsurg.2019.02.002

  • CrossRef
  • Google Scholar

• 7

  SchatzAKoganBFeustelP. Assessing resident surgical competency in urology using a global rating scale. J Surg Educ (2014) 71(6):790–7. doi: 10.1016/j.jsurg.2014.03.012

  • CrossRef
  • Google Scholar

• 8

  HarrimanDSinglaRNguanC. The resident report card: a tool for operative feedback and evaluation of technical skills. J Surg Res (2019) 239:261–8. doi: 10.1016/j.jss.2019.02.006

  • CrossRef
  • Google Scholar

• 9

  SolajaOSkinnerAAMcgregorTBSiemensDR. CanMEDS scholars: A national survey on urology residents’ attitudes towards research during training. Can Urol Assoc J (2017) 12(4):E191–6. doi: 10.5489/cuaj.4927

  • CrossRef
  • Google Scholar

• 10

  LarsenRGBowdinoCSVan LeeuwenBJLaGrangeCADeibertCM. The positive effect of monetary incentive on urology resident research. Urology (2020) 146:43–8. doi: 10.1016/j.urology.2020.07.077

  • CrossRef
  • Google Scholar

• 11

  BentJPFriedMPSmithRVHsuehWChoiK. Leadership training in otolaryngology residency. Otolaryngol Neck Surg (2017) 156(6):1078–9. doi: 10.1177/0194599817698441

  • CrossRef
  • Google Scholar

• 12

  ItaniKMFLiscumKBrunicardiFC. Physician leadership is a new mandate in surgical training. Am J Surg (2004) 187(3):328–31. doi: 10.1016/j.amjsurg.2003.12.004

  • CrossRef
  • Google Scholar

• 13

  BortnickEStockJSimma-ChiangV. Mentorship in urology residency programs in the united states. Urology (2020) 136:58–62. doi: 10.1016/j.urology.2019.09.053

  • CrossRef
  • Google Scholar

• 14

  HollandBCPatelNSulaverRStevensonBHealeyJSeverinoWet al. Esident impact on patient & surgeon satisfaction and outcomes: evidence for health system support for urology education. Urology (2019) 132:49–55. doi: 10.1016/j.urology.2019.04.043

  • CrossRef
  • Google Scholar

• 15

  GannonSJLawKERayRDNathwaniJNDiMarcoSMD’AngeloA-LDet al. Do resident’s leadership skills relate to ratings of technical skill? J Surg Res (2016) 206(2):466–71. doi: 10.1016/j.jss.2016.08.050

  • CrossRef
  • Google Scholar

• 16

  ThacoorASmithONikkhahD. The role of emotional intelligence in predicting a successful career for plastic surgeons. Plast Reconstr Surg - Glob Open (2020) 8(3):e2699. doi: 10.1097/GOX.0000000000002699

  • CrossRef
  • Google Scholar

• 17

  AllardMABlaniéABrouquetABenhamouD. Learning non-technical skills in surgery. J Visceral Surg (2020) 157:S131–6. doi: 10.1016/j.jviscsurg.2020.03.001

  • CrossRef
  • Google Scholar

• 18

  MaykelJA. Leadership in surgery. Clin Colon Rectal Surg (2013) 26(4):254–8. doi: 10.1055/s-0033-1356727

  • CrossRef
  • Google Scholar

• 19

  KiesauCDHeimKAParekhSG. Leadership and business education in orthopaedic residency training programs. J Surg Orthop Adv (2011) 20(2):117–21.

  • Google Scholar

• 20

  MorrisonKBMcLeanNJMacNeilyAE. Assessing the goals of urology residency training: perceptions of practicing urologists in British Columbia - PubMed. Can J Urol (2003) 10(4):1917–23.

  • Google Scholar

• 21

  BalbonaJPatelT. The hidden curriculum: strategies for preparing residents for practice. Curr Urol Rep (2020) 21(10):39. doi: 10.1007/s11934-020-00992-1

  • CrossRef
  • Google Scholar

• 22

  HarrisPATaylorRThielkeRPayneJGonzalezNCondeJG. Research electronic data capture (REDCap)–a metadata-driven methodology and workflow process for providing translational research informatics support. J BioMed Inform (2009) 42(2):377–81. doi: 10.1016/j.jbi.2008.08.010

  • CrossRef
  • Google Scholar

• 23

  HillDAJimenezJ-CCohnSMPriceMR. How to be a leader: a course for residents. Cureus (2018) 10(7). doi: 10.7759/cureus.3067

  • CrossRef
  • Google Scholar

• 24

  TakabeK. Leadership training of surgery in US. Keio J Med (2017) 66(3):54–4. doi: 10.2302/kjm.66-004-ABST

  • CrossRef
  • Google Scholar

• 25

  SchulzKPuscasLTucciDWoodardCWitsellDEsclamadoRMet al. Surgical training and education in promoting professionalism: a comparative assessment of virtue-based leadership development in otolaryngology-head and neck surgery residents. Med Educ Online (2013) 18(1):22440. doi: 10.3402/meo.v18i0.22440

  • CrossRef
  • Google Scholar

• 26

  AwadSSHayleyBFaganSPBergerDHBrunicardiFC. The impact of a novel resident leadership training curriculum. Am J Surg (2004) 188(5):481–4. doi: 10.1016/j.amjsurg.2004.07.024

  • CrossRef
  • Google Scholar

• 27

  American Urologic Association New York Section. EMPIRE: hidden curriculum lecture series (2021). Available at: https://nyaua.com/empire/empire-curriculum-series-presentations/.

  • Google Scholar

• 28

  SmigelskiMMovassaghiMSmallASajiAABadalatoGM. Virtual urology education in the epicenter of COVID-19: development and outcomes of the educational multi-institutional program for instructing residents (EMPIRE) lecture series. Urol Pract (2021) 8(3):417–24. doi: 10.1097/UPJ.0000000000000222

  • CrossRef
  • Google Scholar

• 29

  PosnerBZKouzesJM. Development and validation of the leadership practices inventory. Educ Psychol Meas (1988) 48(2):483–96. doi: 10.1177/0013164488482024

  • CrossRef
  • Google Scholar

• 30

  KhanKSCoomarasamyA. A hierarchy of effective teaching and learning to acquire competence in evidenced-based medicine. BMC Med Educ (2006) 6(1):59. doi: 10.1186/1472-6920-6-59

  • CrossRef
  • Google Scholar

• 31

  RoschDMPriestKL. Strategies in assessment of leadership competencies. New Dir Student Leadersh (2017) 2017(156):87–99. doi: 10.1002/yd.20273

  • CrossRef
  • Google Scholar