DSP-MCF: Dual Stream Pre-training and Multi-view Consistency Fine-tuning for Cross-subject EEG Emotion Recognition

Jingjing Li¹Xinqi Liu¹Xia Wu²Ya Wang¹Xin Huang^1*

• ¹Nanyang Normal University, Nanyang, China
• ²Guizhou University of Traditional Chinese Medicine, Guiyang, China

Electroencephalogram (EEG) emotion recognition is attracting increasing attention in the field of brain-computer interface due to its strong objectivity and non-forgery. However, cross-subject emotion recognition is complicated by individual variability, limited availability of EEG data, and interference in certain channels during EEG acquisition. To address these issues, we propose a novel synergistic Dual Stream Pre-training and Multi-view Consistency Fine-tuning (DSP-MCF) framework. The DSP-MCF is based on a domain generalization architecture. The framework includes a dual stream pre-training stage, wherein the spatiotemporal encoder-decoder network extracts generalized spatiotemporal representations from masked channels and reconstructs EEG features from incomplete data. Then, a multi-view consistency loss function is proposed during the multi-view consistency fine-tuning. This loss function is essential for aligning the distribution of emotion predictions derived from various perspectives, specifically from actual and masked EEG data. To simulate and improve its capacity to handle actual data degradation, DSP-MCF implements a dynamic channel mask strategy during both training and testing. Experimentally, the proposed DSP-MCF outperforms the state-of-the-art in cross-subject EEG emotion recognition tasks, achieving an accuracy of 89.76 on the SEED dataset and 77.02 on the SEED-IV dataset.