AUTHOR=Jentsch Armin , Senden Bas , Teig Nani , Nilsen Trude , König Johannes , Kaiser Gabriele 

TITLE=Using Generalizability Theory to examine the effects of presentation order on classroom observation scores

JOURNAL=Frontiers in Education

VOLUME=Volume 10 - 2025

YEAR=2025

URL=https://www.frontiersin.org/journals/education/articles/10.3389/feduc.2025.1547204

DOI=10.3389/feduc.2025.1547204

ISSN=2504-284X

ABSTRACT=BackgroundScores from classroom observations serve as important pieces of information for educational researchers, practitioners, and policymakers. However, scoring procedures typically remain untested. This study aims to investigate the effect of sequential and nonsequential scoring of lesson segments from videotaped Norwegian mathematics and science classrooms.MethodsUsing a cross-over design, we randomly assigned four raters to score 30 videotaped lessons in sequential (SEQ) or nonsequential (NON) 20-min segments. We explored means as well as correlations across conditions and conducted Generalizability Studies (G Studies) to estimate the precision of scores.ResultsWe found marginal fluctuations in scores across conditions, and correlation analysis indicated similar rankings of lessons and classrooms. However, G Studies reveal that nonsequential scoring captures a greater proportion of between-classroom variance, while sequential scoring captures more variance between lessons. This also implies that the precision of scores varies across conditions and the unit of analysis.ConclusionThe findings suggest that scoring procedures could affect decision-making drawing on scores from classroom observation. We argue, however, that results depend to some extent on the observation system used to generate the scores. Therefore, we encourage researchers to replicate our findings with other observation systems.