Evaluation of Generative Models for Emotional 3D Animation Generation in VR

Kiran Chhatre^1,2*Renan Guarese^1,2Andrii Matviienko^2,3Christopher Edward Peters^1,2

• ¹Computational Science and Technology, School of Computer Science and Communication, KTH Royal Institute of Technology, Stockholm, Sweden
• ²Royal Institute of Technology, Stockholm, Stockholm, Sweden
• ³Department of Human Centered Technology, School of Electrical Engineering and Computer Science, Royal Institute of Technology, Stockholm, Stockholm, Sweden

Social interactions incorporate various nonverbal signals, including facial expressions and body gestures, to convey emotions alongside speech. Generative models have demonstrated promising results in creating full-body nonverbal animations synchronized with speech; however, evaluations using statistical metrics in 2D settings fail to fully capture user-perceived emotions, limiting our understanding of the effectiveness of these models. To address this, we evaluate emotional 3D animation generative models within an immersive VR environment, emphasizing user-centric metrics—emotional realism, naturalness, enjoyment, diversity, and interaction quality—in a real-time human–virtual character interaction scenario. Through a user study (N=48), we systematically examine perceived emotional quality for three state-of-the-art speech-driven 3D animation methods across two specific emotions: happiness (high arousal) and neutral (low arousal). Additionally, we compare these generative models against real human expressions obtained via a reconstruction-based method to assess both their strengths and limitations and how closely they replicate real human facial and body expressions. Our results demonstrate that methods explicitly modeling emotions lead to higher recognition accuracy compared to those focusing solely on speech-driven synchrony. Users rated happy animations significantly higher than neutral animations, highlighting limitations of current generative models in handling subtle emotional states. Generative models underperformed compared to reconstruction-based methods in facial expression quality, and all methods received relatively low ratings for animation enjoyment and interaction quality, emphasizing the importance of incorporating user-centric evaluations into generative model development. Finally, participants positively recognized animation diversity across all generative models.