EmoAsst: emotion recognition assistant via text-guided transfer learning on pre-trained visual and acoustic models - Wang, Minxiao; Yang, Ning (Frontiers in Computer Science, 2024)
Children diagnosed with Autism Spectrum Disorder (ASD) often struggle to grasp social conventions and promptly recognize others' emotions. Recent advancements in applying deep learning (DL) for emotion recognition solidify the role of AI-powered assistive technology in supporting autistic children. However, the cost of collecting and annotating large-scale high-quality human emotion data and the phenomena of imbalanced performance on different modalities of data challenge DL-based emotion recognition. In response to these challenges, this paper explores transfer learning, wherein large pre-trained models like CLIP (Contrastive Language-Image Pretraining) and wav2vec 2.0 are fine-tuned to improve audio and video-based emotion recognition with text-based guidance. In this work, we propose the EmoAsst framework, including a visual fusion module and emotion prompt fine-tuning for CLIP, as well as leveraging CLIP's text encoder and supervised contrastive learning for audio-based emotion recognition on the wav2vec 2.0 model. In addition, a joint few-shot emotion classifier enhances the accuracy and offers great adaptability for real-world applications. The evaluation results on the MELD dataset highlight the outstanding performance of our methods, surpassing the majority of existing video and audiobased approaches. Notably, our research demonstrates the promising potential of the suggested text-based guiding techniques in improving video and audio-based Emotion Recognition and Classification (ERC).