Event Abstract

Back to Event

The role of movement kinematics in facial emotion expression

Sophie L. Sowden1, Bianca A. Schuster1 and Jennifer L. Cook2*
  • 1 University of Birmingham, United Kingdom
  • 2 University of Birmingham, Institute of Cognitive Neuroscience, United Kingdom

Background Facial emotion expression and recognition play an important role in successful social interaction (Ekman and Friesen, 1975), providing cues about others’ affective states to guide behavior. The last decade has seen the use of facial emotion tracking software to dynamically adapt and personalize online learning platforms for education, in response to users’ spontaneous expressions (Saneiro et al., 2014). Such platforms use algorithms to detect the presence and intensity of key facial action units (actions of groups of muscles) typical for each emotion (Littlewort et al., 2011). However, this relies on the detection of spatial rather than temporal features. A recent body of evidence suggests that temporal features, such as whole-body movement kinematics, provide key information about emotional states: faster (high velocity) body movements are associated with anger and happiness, whilst low velocity movements are indicative of sadness (Ada et al., 2003; Edey et al., 2017; Michalak et al., 2009). Although studies have investigated the effect of manipulating the speed of video or expression-morphing playback on facial emotion recognition (Fayolle and Droit-Volet, 2014; Kamachi et al., 2001, Pollick et al., 2003), studies have not quantified whether, like whole-body movements, the velocity of face movements is indicative of emotional state. Although emotion tracking software typically aims to detect spontaneous expressions, much of our knowledge of movement kinematics comes from the posed expressions of professional actors. Preliminary evidence suggests that spontaneous and posed expressions differ with regards to timing and amplitude of facial movements (Schmidt et al., 2006; Valstar et al., 2006). Moreover, there are data to suggest that different facial regions may be more or less informative for recognizing different emotions (Bassili, 1979); movement kinematics may play a role in this. Consequently, the current study aims to investigate a) whether the velocity of movement of different parts of the face (i.e. face “actions”) differs as a function of emotion (happy, angry and sad), and b) whether this relationship differs for spontaneous and posed expressions. Method Forty-two healthy student volunteers (39 female) from the University of Birmingham gave written informed consent to take part and were reimbursed with a course credit or monetary incentive for their time. 7 further participants were excluded from analyses either due to poor registration with the facial tracking system (N = 4) or because of missing data (N = 3). Participants were seated with their head at the center of a polystyrene frame, positioned 80 cm from the computer monitor (21.5-inch iiyama display) and 1 m from a tripod-docked video camcorder (Sony Handycam HDR-CX240E). Participants’ facial movements were recorded during two conditions: 1) Spontaneous - wherein participants watched videos (selected from a pilot study) that prompted spontaneous expression of the target emotions happy, sad, and angry; 2) Posed - wherein participants posed the 3 target emotional expressions following the instruction to move from neutral, to peak expression, and back to neutral upon detecting a fixation cross along with a beep (9 seconds duration). The order of emotions was counterbalanced for posed and spontaneous conditions. Following each video, for the spontaneous condition, participants rated each target emotion plus surprise and disgust on a scale from 1-10. Recordings for the spontaneous condition were cropped to a 10-second scene rated, across all participants, as the most emotionally intense scene for each target emotion. Data analysis followed a novel pipeline (Figure 1), whereby recordings for each emotion, for posed and spontaneous (6 videos per participant), were fed into the open-source software OpenFace (Baltrušaitis et al., 2018) which identifies locations in pixels of 68 2D facial landmarks, sampled at a rate of 25 Hz. 9 facial ‘distances’ were calculated (following the procedure outlined in Zane et al. (2018)) by identifying key points on the face and calculating the distance between key points (i.e. the square root of the sum of the squared differentials of the x and y coordinates of each key point). These distances were then summed to create 5 face “actions”. For example, the distance between points 21 and 22 (Figure 1A), comprises the eyebrow widen action. Velocity was calculated as the differential of the action vector and represented as absolute values of each face distance collapsed across all movements within a given time window. Thus, this is not the onset or speed taken to reach peak expressions, which may be confounded with the difference in time taken to feel the emotion. Velocity vectors were low pass filtered at 10 Hz and absolute velocity was averaged, for each action, across each video. Results Emotion induction was successful: emotion rating discreteness (target emotion rating minus average rating of all non-target emotions) scores for each video were significantly greater than zero (ps < .001). A repeated-measures ANOVA (RM-ANOVA) with within-subjects factors of condition (posed, spontaneous), emotion (happy, angry, sad) and action (eyebrow widen, nose lengthen, lip raise, mouth open, mouth widen) revealed a significant main effect of emotion [F(2,82) = 6.16, p = .003, ηP2 = .69]. Happy expressions had the highest velocities (mean [pixels/frame] = 0.37, standard error of the mean [SEM] = 0.01), sad expressions were the slowest (mean = 0.34, SEM = 0.01) and angry expressions were intermediate (mean = 0.36, SEM = 0.01). A main effect of action was also observed [F(4,164) = 92.07, p < .001, ηP2 = .69]: lip raise actions were fastest (mean = 0.47, SEM = 0.02), mouth widen (mean = 0.27, SEM = 0.01) and eyebrow widen (mean = 0.26, SEM = 0.01) were slowest, and mouth open (mean = 0.39, SEM = 0.02) and nose lengthen (mean = 0.39, SEM = 0.02) were intermediate. There was no main effect of condition (p = .97). An action x emotion interaction indicated that the velocity of eyebrow widening [F(2,82) = 21.11, p < .001, ηP2 = .34], mouth widening [F(2,82) = 37.36, p < .001, ηP2 = .48], and mouth opening [F(2,82) = 18.67, p < .001, ηP2 = .31], but not lip raising (p = .84) or nose lengthening (p = .07), differed as a function of emotion. Velocity of eyebrow widening was highest for angry (mean = 0.29, SEM = 0.01), lowest for sad (mean = 0.25, SEM = 0.01) and happy (mean = 0.26, SEM = 0.01), whilst velocity for mouth movements were highest for happy (mouth widening: mean = 0.31, SEM = 0.01; mouth opening: mean = 0.45, SEM = 0.02), followed by angry (mouth widening: mean = 0.26, SEM = 0.01; mouth opening: mean = 0.40, SEM = 0.02) and lowest for sad (mouth widening: mean = 0.23, SEM = 0.01; mouth opening: mean = 0.34, SEM = 0.02). However, a significant interaction between condition, emotion and action was also observed [F(8,328) = 11.85, p < .001, ηP2 = .23]. Separate RM-ANOVAs, for each action, revealed condition x emotion interactions for mouth opening [F(2,82) = 15.01, p < .001, ηP2 = .27] and mouth widening [F(2,82) = 28.58, p < .001, ηP2 = .41]. Post-hoc t-tests revealed velocity of mouth movements during posed expression production to be highest for happy expressions when compared to both angry (mouth widening [t(41) = 11.54, p < .001]; mouth opening [t(41) = 4.51, p < .001]) and sad (mouth widening [t(41) = 11.62, p < .001]; mouth opening [t(41) = 9.63, p < .001]), whilst no such differences were observed for spontaneous expression production (ps > .05). See Figure 2 for a visual representation of the results. Discussion These data demonstrate that face movement kinematics provide important cues about emotional states. In line with the whole-body literature, anger and happiness were associated with high velocity face movements, from the eyebrows and mouth areas respectively, whereas sadness was characterized by low velocity movements. This has important implications for expression-centered adaptive learning platforms which employ facial emotion tracking, such algorithms currently overlook temporal features of facial expressions which carry useful information regarding emotional states. Emotion tracking algorithms may benefit from incorporating information about the kinematics of face movements, with a particular focus on the actions that were found to be significant differentiators of emotional state (eyebrow widening and mouth widening and opening). However, importantly, in the current study, mouth movements only differentiated posed, not spontaneous, expressions. Thus, any algorithm aiming to detect spontaneous emotions should not rely on posed expression datasets for training purposes.

Figure 1
Image
Figure 2
Image

Acknowledgements

The work in this paper was funded by a European Research Council Starting Grant held by J.C. We thank Alexandru-Andrei Moise, Maya Burns and Lucy D’Orsaneo for their help with data collection.

References

Ada, M. S., Suda, K., and Ishii, M. (2003). Expression of emotions in dance: Relation between arm movement characteristics and emotion. Percept. Motor Skills. 97, 697-708. Baltrušaitis, T., Zadeh, A., Lim, Y. C., and Morency, L. (2018). OpenFace 2.0: Facial behavior analysis toolkit, In IEEE International Conference on Automatic Face and Gesture Recognition. 59-66. https://github.com/TadasBaltrusaitis/OpenFace/wiki. Bassili, J. N. (1979). Emotion recognition: The role of facial movement and the relative importance of upper and lower areas of the face. J. Pers. Soc. Psychol. 37, 2049-2058. Edey, R., Yon, D., Cook, J., Dumontheil, I., and Press, C. (2017). Our own action kinematics predict the perceived affective states of others. J. Exp. Psychol. Human. 43, 1263-1268. Fayolle, S. L., and Droit-Volet, S. (2014). Time perception and dynamics of facial expressions of emotions. PLoS One. 9, e97944. Kamachi, M., Bruce, V., Mukaida, S., Gyoba, J., Yoshikawa, S., and Akamatsu, S. (2013). Dynamic Properties Influence the Perception of Facial Expressions. Perception. 42, 1266-1278 Littlewort, G., Whitehill, J., Wu, T., Fasel, I., Frank, M., Movellan, J., and Bartlett, M. (2011) The Computer Expression Recognition Toolbox (CERT). In Proc. IEEE International Conference on Automatic Face and Gesture Recognition. 298-305. Michalak, J., Troje, N. F., Fischer, J., Vollmar, P., Heidenreich, T., and Schulte, D. (2009). Embodiment of sadness and depression – gait patterns associated with dysphoric mood. Psychos. Med. 71, 580–587. Pollick, F. E., Hill, H., Calder, A., and Paterson, H. (2003). Recognising facial expression from spatially and temporally modified movements. Perception. 32, 813-826. Saneiro, M., Santos, O. C., Salmeron-Majadas, S., and Boticario, J. G. (2014). Towards emotion detection in educational scenarios from facial expressions and body movements through multimodal approaches. Sci. World J. 484873. Schmidt, K. L., Ambadar, Z., Cohn, J. F., and Reed, L. I. (2006). Movement differences between deliberate and spontaneous facial expressions: Zygomaticus major action in smiling. J. Nonverbal Behav. 30, 37-52. Valstar, M. F., Pantic, M., Ambadar, Z., and Cohn, J. F. (2006). Spontaneous vs. posed facial behavior: automatic analysis of brow actions. In Proceedings of the 8th international conference on multimodal interfaces. 162-170. Zane, E., Yang, Z., Pozzan, L., Guha, T., Narayanan, S., and Grossman, R. B. (2018). Motion-capture patterns of voluntarily mimicked dynamic facial expressions in children and adolescents with and without ASD. J. Autism Dev. Disord. 1-18.

Keywords: emotion, kinematics, facial emotion expression, Biological motion, happy, SAD, angry

Conference: 4th International Conference on Educational Neuroscience, Abu Dhabi, United Arab Emirates, 10 Mar - 11 Mar, 2019.

Presentation Type: Oral Presentation (invited speakers only)

Topic: Educational Neuroscience

Citation: Sowden SL, Schuster BA and Cook JL (2019). The role of movement kinematics in facial emotion expression. Conference Abstract: 4th International Conference on Educational Neuroscience. doi: 10.3389/conf.fnhum.2019.229.00018

Copyright: The abstracts in this collection have not been subject to any Frontiers peer review or checks, and are not endorsed by Frontiers. They are made available through the Frontiers publishing platform as a service to conference organizers and presenters.

The copyright in the individual abstracts is owned by the author of each abstract or his/her employer unless otherwise stated.

Each abstract, as well as the collection of abstracts, are published under a Creative Commons CC-BY 4.0 (attribution) licence (https://creativecommons.org/licenses/by/4.0/) and may thus be reproduced, translated, adapted and be the subject of derivative works provided the authors and Frontiers are attributed.

For Frontiers’ terms and conditions please see https://www.frontiersin.org/legal/terms-and-conditions.

Received: 10 Feb 2019; Published Online: 27 Sep 2019.

* Correspondence: Dr. Jennifer L Cook, University of Birmingham, Institute of Cognitive Neuroscience, Birmingham, WC1N 3AR, United Kingdom, j.l.cook@bham.ac.uk