Complex Emotion Recognition System using Basic Emotions via Facial Expression, Electroencephalogram, and Electrocardiogram Signals: A Review

Javad Hassannataj Joloudari¹Mohammad Maftoun²Bahareh Nakisa³Roohallah Alizadehsani³Meisam Yadollahzadeh-Tabari²Silvia Gaftandzhieva^4*

• ¹University of Birjand Faculty of Electrical and Computer Engineering, Birjand, Iran
• ²Islamic Azad University, Tehran, Iran
• ³Deakin University, Melbourne, Australia
• ⁴Plovdivski universitet Paisij Hilendarski Fakultet po matematika i informatika, Plovdiv, Bulgaria

The Complex Emotion Recognition System (CERS) deciphers complex emotional states by examining combinations of basic emotions expressed, their interconnections, and their dynamic variations. Through the utilization of advanced algorithms, the system provides profound insights into emotional dynamics, facilitating a nuanced understanding and customized responses. Achieving this level of emotional recognition in machines necessitates knowledge distillation and the comprehension of novel concepts akin to human cognition. The development of artificial intelligence systems for discerning complex emotions poses substantial challenges with significant implications for affective computing. Also, obtaining a sizable dataset for such systems is daunting due to the intricacies involved in capturing subtle emotions, necessitating specialized methods for data collection and processing. Incorporating physiological signals, such as electrocardiograms (ECG) and electroencephalograms (EEG), notably enhances CERS by furnishing insights into users' emotional states, improving dataset quality, and fortifying system dependability. This study presents a comprehensive review assessing the efficacy of machine learning, deep learning, and meta-learning approaches in both basic and complex emotion recognition using facial expressions, EEG, and ECG signals. Selected research papers offer perspectives on potential applications, clinical implications, and results of such systems, intending to promote their acceptance and integration into clinical decision-making processes. Additionally, this study highlights research gaps and challenges in understanding emotion recognition systems, encouraging further investigation by relevant studies and organizations. Lastly, the significance of meta-learning approaches in improving system performance and guiding future research is underscored, with potential applications in universities for advancing educational research, monitoring student well-being, and developing intelligent tutoring systems.