AUTHOR=Wang Jing , Zhang Ci 

TITLE=Cross-modality fusion with EEG and text for enhanced emotion detection in English writing

JOURNAL=Frontiers in Neurorobotics

VOLUME=Volume 18 - 2024

YEAR=2025

URL=https://www.frontiersin.org/journals/neurorobotics/articles/10.3389/fnbot.2024.1529880

DOI=10.3389/fnbot.2024.1529880

ISSN=1662-5218

ABSTRACT=IntroductionEmotion detection in written text is critical for applications in human-computer interaction, affective computing, and personalized content recommendation. Traditional approaches to emotion detection primarily leverage textual features, using natural language processing techniques such as sentiment analysis, which, while effective, may miss subtle nuances of emotions. These methods often fall short in recognizing the complex, multimodal nature of human emotions, as they ignore physiological cues that could provide richer emotional insights.MethodsTo address these limitations, this paper proposes Emotion Fusion-Transformer, a cross-modality fusion model that integrates EEG signals and textual data to enhance emotion detection in English writing. By utilizing the Transformer architecture, our model effectively captures contextual relationships within the text while concurrently processing EEG signals to extract underlying emotional states. Specifically, the Emotion Fusion-Transformer first preprocesses EEG data through signal transformation and filtering, followed by feature extraction that complements the textual embeddings. These modalities are fused within a unified Transformer framework, allowing for a holistic view of both the cognitive and physiological dimensions of emotion.Results and discussionExperimental results demonstrate that the proposed model significantly outperforms text-only and EEG-only approaches, with improvements in both accuracy and F1-score across diverse emotional categories. This model shows promise for enhancing affective computing applications by bridging the gap between physiological and textual emotion detection, enabling more nuanced and accurate emotion analysis in English writing.