A comparison of lessons planned by different publicly available Large Language Models in the context of physical education: an expert survey

Benedikt Meixner^1,2*Clara Tristram³Maritta Schranner⁴Alessandra Kenner³Esther Serwe-Pandrick⁵Billy Sperlich²Peter Düking^5*

• ¹Friedrich-Alexander-Universität Erlangen-Nürnberg, Erlangen, Germany
• ²Julius-Maximilians-Universitat Wurzburg, Würzburg, Germany
• ³Friedrich-Alexander-Universitat Erlangen-Nurnberg, Erlangen, Germany
• ⁴Janusz-Korczak Schule, Bayreuth, Germany
• ⁵Technische Universitat Braunschweig, Brunswick, Germany

Large language models (LLMs) have the potential to assist teachers, particularly in lesson planning. The quality of lessons generated by various LLMs remains unexplored. We investigated the quality of different LLMs for lesson planning, using the basketball layup as example and surveying experts in the field. A prompt was submitted to three LLMs (GPT-4o, Claude Sonnet, and Google Gemini). Twenty-eight quality criteria to evaluate lessons were predefined and employed. Teaching experts rated the plans on 5-point Likert scales and provided additional comments. A Friedman test was conducted to identify differences in quality among lesson plans. The most frequent median rating across all lesson plans was "acceptable" (3 on a 1-5 Likert scale), accounting for 64 out of 84 total ratings. For most criteria (26 out of 28), no group differences were observed between the lesson plans by Claude, Gemini, and GPT-4o. Free-text comments from raters highlighted that certain requirements such as allotted time or number of students were sometimes disregarded by specific LLMs despite being included in the prompt. LLMs are capable of generating basketball layup lessons of acceptable quality; however, these require review and refinement by experienced teachers. Herein investigated LLMs displayed no differences for most evaluated criteria. While LLMs can provide valuable starting points, teachers need to acknowledge their limitations and tailor the lessons accordingly. Key Points: Various publicly available LLMs are able to provide acceptable starting points for lesson planning in physical education, but need careful refinement by experienced teachers.