AUTHOR=Liu Pin , Chen Shou-Yen , Chang Yu-Che , Ng Chip-Jin , Chaou Chung-Hsien 

TITLE=Multimodal In-training Examination in an Emergency Medicine Residency Training Program: A Longitudinal Observational Study

JOURNAL=Frontiers in Medicine

VOLUME=Volume 9 - 2022

YEAR=2022

URL=https://www.frontiersin.org/journals/medicine/articles/10.3389/fmed.2022.840721

DOI=10.3389/fmed.2022.840721

ISSN=2296-858X

ABSTRACT=Background: In-training examinations (ITEs) has been widely adopted as an assessment tool to measure residents' competency. We incorporated different formats of assessments into the EM residency training program to form a multimodal, multistation ITE. This study examined the cost and effectiveness of the different testing formats on the ITE. 
Methods: We conducted a longitudinal study in a tertiary teaching hospital in Taiwan. Nine EM residents were enrolled and followed for four years, and the biannual ITE scores were recorded and analyzed. Each ITE consisted of 8-10 stations and was categorized into four formats: multiple-choice question (MCQ), question and answer (QAS), oral examination (OE), and HFS formats. The learner satisfaction, validity, reliability, and costs were analyzed. 
Results: 486 station scores were recorded during the four years. The numbers of MCQ, OE, QAS, and HFS stations were 45 (9.26%), 90 (18.5%), 198 (40.7%), and 135 (27.8%), respectively. The overall Cronbach's alpha reached 0.968, indicating good overall internal consistency. The correlation with EM board examination was highest for HFS (ρ= 0.657). The average costs of an MCQ station, an OE station, and an HFS station were approximately 3, 14, and 21 times that of a QA station.
Conclusions: Multi-dimensional assessment contributes to good reliability. High-Fidelity simulation correlates best with final training outcome but is also the most expensive format among ITEs. Increased testing domains with various formats improves ITE's overall reliability. Program directors must understand each test format's strengths and limitations to bring forth the best combination of exams under the local context.