AUTHOR=Puri Nitin , McCarthy Michael , Miller Bobby 

TITLE=Validity and Reliability of Pre-matriculation and Institutional Assessments in Predicting USMLE STEP 1 Success: Lessons From a Traditional 2 x 2 Curricular Model

JOURNAL=Frontiers in Medicine

VOLUME=Volume 8 - 2021

YEAR=2022

URL=https://www.frontiersin.org/journals/medicine/articles/10.3389/fmed.2021.798876

DOI=10.3389/fmed.2021.798876

ISSN=2296-858X

ABSTRACT=Purpose: We have observed that students' performance in our PreClerkship curriculum does not align well with their USMLE STEP1 scores. Students at-risk of failing or underperforming on STEP1 have often excelled in our institutional assessments. We sought to test the validity and reliability of our course assessments in predicting STEP1 scores, and in the process, generate and validate a more accurate prediction model for STEP1 performance.
Methods: Student pre-matriculation and course assessment data of the Class of 2020 (n = 76) is used to generate a stepwise STEP1 prediction model, which is tested with the students of the Class of 2021 (n = 71). Predictions are generated at the time of matriculation, and subsequently, at the end of each course in the programing language R. For the Class of 2021, the predicted STEP1 score is correlated with their actual STEP1 scores, and data agreement is tested with means-difference plots. A similar model is generated and tested for the Class of 2022.
Results: STEP1 predictions based on pre-matriculation data are unreliable and fail to identify at-risk students (R2 = 0.02). STEP1 predictions for most year one courses (anatomy, biochemistry, physiology) correlate poorly with students' actual STEP1 scores (R2 = 0.30). STEP1 predictions improve for year two courses (microbiology, pathology, and pharmacology). But integrated courses with customized NBMEs provide more reliable predictions (R2 = 0.66). Predictions based on these integrated courses are reproducible for the Class of 2022.
Conclusion: MCAT and undergraduate GPA are poor predictors of students' STEP1 scores. Partially integrated courses with biweekly assessments do not promote problem-solving skills and leave students' at-risk of failing STEP1. Only integrated courses with comprehensive assessments are reliable indicators of students' STEP1 preparation.