EmoShiftNet: A Shift-Aware Multi-Task Learning Framework with Fusion Strategies for Emotion Recognition in Multi-Party Conversations

Hinduja Nirujan^*Prasan Yapa

• Informatics Institute of Technology, Colombo, Sri Lanka

Emotion Recognition in Conversations (ERC) is vital for applications such as mental health monitoring, virtual assistants, and human-computer interaction. However, existing ERC models often neglect emotion shifts—transitions between emotional states that occur across dialogue turns in multi-party conversations (MPCs). These shifts are subtle, context-dependent, and further complicated by class imbalance in datasets like Multimodal EmotionLines Dataset (MELD). To address this, this study proposes EmoShiftNet, a shift-aware multi-task learning (MTL) framework that jointly performs emotion classification and emotion shift detection. The model fuses multimodal features, contextualized text embeddings from BERT, acoustic features such as Mel-Frequency Cepstral Coefficients (MFCCs), pitch, loudness, and temporal cues like pause duration, speaker overlap, utterance length to model both static and dynamic emotional signals. Emotion shift detection is included as an auxiliary task using a composite loss function combining focal loss, binary cross-entropy, and triplet margin loss. Evaluated on the MELD dataset, EmoShiftNet achieves higher emotion recognition F1-score compared to both traditional and graph-based ERC models. It also improves the recognition of minority emotions under imbalance. These results highlight the value of modeling emotional transitions and show that MTL enhances contextual awareness in ERC systems.