Heart rates, facial expressions and self-reports: A multimodal longitudinal approach of learners' emotions in the Foreign Language classroom

Delphine Guedat-Bittighoffer¹Abderrazzaq Moufidi²Jean-Marc Dewaele³Hugo Voyneau¹David Rousseau²Pejman Rasti^2*

• ¹UFR Lettres Langues et Sciences Humaines, Université d'Angers, Angers, France
• ²Laboratoire Angevin de Recherche en Ing´ enierie des Syst` emes (LARIS), UMR INRAe-IRHS,, Université d'Angers, Angers, France
• ³Institute of Education, University College London, London, England, United Kingdom

Emotions in educational settings are often studied through self-reports or lab experiments, limiting insights into their real-world dynamics. This study examines learner emotions in authentic foreign language classrooms using a multimodal longitudinal approach. Over 16 consecutive sessions, we collected heart rate (HR) signals, emotional facial expressions (EFE), classroom observations, and self-reports on enjoyment, anxiety, and boredom to capture both physiological and self-perceived emotional responses. Rather than aggregating data across students, we focused on individualized emotional patterns to understand variations in emotional experiences. Each dataset included extensive video recordings, continuous HR monitoring, detailed observational notes, and post-session questionnaires, providing a high-resolution picture of emotional dynamics. Using unsupervised clustering techniques, we identified key emotional episodes-peaks and drops in physiological arousal (heart rate variation) and facial expression-relative to individual emotional baselines. These moments were cross-referenced with classroom observations and self-reports for validation. Findings highlight moments of positive emotional contagion during peer interactions, emphasizing the social dimension of language learning. This multimodal approach captures the interplay of physiological, behavioral, and subjective responses, offering a scalable method for studying classroom emotions. Methodologically, it demonstrates how multimodal analytics can uncover transient emotional states in real-world settings, while practically informing adaptive teaching strategies, such as leveraging peer interactions to enhance engagement or reduce anxiety. By integrating physiological, behavioral, and subjective data, this study provides a comprehensive framework for understanding the affective dimensions of learning.