An Atomized Approach to Assessing Energy Problem Solving in Physics using Multidimensional Item Response Theory

André Meyer^*Gunnar Friege

• Physics Education Group - Institute for Mathematics and Physics Education, Leibniz University Hannover, Hanover, Germany

Problem solving is a central competence in STEM education. In particular, students in secondary education struggle with the multiple skills they have to coordinate in order to become successful problem solvers. Early assessment of problem-solving skills can facilitate individual feedback and support during this pivotal phase of school. Effective support is often achieved through immediate feedback during the learning process, e.g., using artificial intelligence. Current assessment methods focus mainly on analyses of complete problem solutions. While this allows detailed insights into the skills of students, it also requires a lot of effort and the test instruments cannot effectively measure skills of students that fail in an early phase of the problem-solving process. Therefore, the “atomized” problem-solving test (APST) was developed to automatically and digitally assess students’ skills separately for the four phases of problem-solving. The items of this new instrument were tested in two consecutive studies with a total of 800 German secondary school students. Multidimensional item response theory (MIRT) was used together with supplemental test instruments to validate the APST items and to analyze the dimensional structure of skills in the problem-solving sub-processes. The results indicate that the APST enables valid and reliable assessment, but shortcomings and room for improvements are also discussed. Dimensional analyses further suggest a distinctive role of Evaluation, which appears to integrate competencies from the other subprocesses alongside evaluation-specific skills.