Multimodal MBC-ATT: Cross-Modality Attentional Fusion of EEG-fNIRS for Cognitive State Decoding

Yu Li¹Lei Zhu^1,2*Aiai Huang^1,2Jianhai Zhang¹Peng Yuan³

• ¹Hangzhou Dianzi University, Hangzhou, China
• ²Hangzhou Dianzi University School of Automation, Hangzhou, China
• ³Wuxi People's Hospital, Wuxi, China

With the rapid development of brain-computer interface (BCI) technology, the effective integration of multimodal biological signals to improve classification accuracy has become a research hotspot. However, existing methods often fail to fully exploit cross-modality correlations in complex cognitive tasks. To address this, this paper proposes a Multi-Branch Convolutional Neural Network with Attention (MBC-ATT) for BCI based cognitive tasks classification. MBC-ATT employs independent branch structures to process electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS) signals separately, thereby leveraging the advantages of each modality. To further enhance the fusion of multimodal features, we introduce a cross-modal attention mechanism to discriminate features, strengthening the model's ability to focus on relevant signals and thereby improving classification accuracy. We conducted experiments on the n-back and WG datasets. The results demonstrate that the proposed model outperforms traditional conventional approaches in classification performance, further validating the effectiveness of MBC-ATT in brain-computer interfaces. This study not only provides novel insights for multimodal BCI systems but also holds great potential for various applications.