Using Generalizability Theory to examine the effects of presentation order on classroom observation scores

Armin Jentsch^1*Bas Senden¹Nani Teig¹Trude Nilsen¹Johannes König²Gabriele Kaiser^3,4

• ¹University of Oslo, Oslo, Norway
• ²University of Cologne, Cologne, North Rhine-Westphalia, Germany
• ³University of Hamburg, Hamburg, Hamburg, Germany
• ⁴Nord University, Bodø, Nordland, Norway

Scores from classroom observations serve as important pieces of information for educational researchers, practitioners, and policymakers. However, scoring procedures typically remain untested. This study aims to investigate the effect of sequential and nonsequential scoring of lesson segments from videotaped Norwegian mathematics and science classrooms.Using a cross-over design, we randomly assigned four raters to score 30 videotaped lessons in sequential (SEQ) or nonsequential (NON) 20-minute segments. We explored means as well as correlations across conditions and conducted Generalizability Studies (G Studies) to estimate the precision of scores.We found marginal fluctuations in scores across conditions, and correlation analysis indicated similar rankings of lessons and classrooms. However, G Studies reveal that nonsequential scoring captures a greater proportion of between-classroom variance, while sequential scoring captures more variance between lessons. This also implies that the precision of scores varies across conditions and the unit of analysis.The findings suggest that scoring procedures could affect decision-making drawing on scores from classroom observation. We argue, however, that results depend to some extent on the observation system used to generate the scores. Therefore, we encourage researchers to replicate our findings with other observation systems.