Multimodal Fusion Technologies and Applications in the Context of Neuroscience

Pandemic-related sleep disorders affect human physical and mental health. The artificial intelligence (AI) based sleep staging with multimodal electrophysiological signals help people diagnose and treat sleep disorders. However, the existing AI-based methods could not capture more discriminative modalities and adaptively correlate these multimodal features. This paper introduces a multimodal attention network (MMASleepNet) to efficiently extract, perceive and fuse multimodal features of electrophysiological signals. The MMASleepNet has a multi-branch feature extraction (MBFE) module followed by an attention-based feature fusing (AFF) module. In the MBFE module, branches are designed to extract multimodal signals' temporal and spectral features. Each branch has two-stream convolutional networks with a unique kernel to perceive features of different time scales. The AFF module contains a modal-wise squeeze and excitation (SE) block to adjust the weights of modalities with more discriminative features and a Transformer encoder (TE) to generate attention matrices and extract the inter-dependencies among multimodal features. Our MMASleepNet outperforms state-of-the-art models in terms of different evaluation matrices on the datasets of Sleep-EDF and ISRUC-Sleep. The implementation code is available at: https://github.com/buptantEEG/MMASleepNet/.

Objective: Increased muscle co-contraction of the agonist and antagonist muscles during voluntary movement is commonly observed in the upper limbs of stroke survivors. Much remain to be understood about the underlying mechanism. The aim of the study is to investigate the correlation between increased muscle co-contraction and the function of the corticospinal tract (CST).

Methods: Nine stroke survivors and nine age-matched healthy individuals were recruited. All the participants were instructed to perform isometric maximal voluntary contraction (MVC) and horizontal task which consist of sponge grasp, horizontal transportation, and sponge release. We recorded electromyography (EMG) activities from four muscle groups during the MVC test and horizontal task in the upper limbs of stroke survivors. The muscle groups consist of extensor digitorum (ED), flexor digitorum (FD), triceps brachii (TRI), and biceps brachii (BIC). The root mean square (RMS) of EMG was applied to assess the muscle activation during horizontal task. We adopted a co-contraction index (CI) to evaluate the degree of muscle co-contraction. CST function was evaluated by the motor-evoked potential (MEP) parameters, including resting motor threshold, amplitude, latency, and central motor conduction time. We employed correlation analysis to probe the association between CI and MEP parameters.

Results: The RMS, CI, and MEP parameters on the affected side showed significant difference compared with the unaffected side of stroke survivors and the healthy group. The result of correlation analysis showed that CI was significantly correlated with MEP parameters in stroke survivors.

Conclusion: There existed increased muscle co-contraction and impairment in CST functionality on the affected side of stroke survivors. The increased muscle co-contraction was correlated with the impairment of the CST. Intervention that could improve the excitability of the CST may contribute to the recovery of muscle discoordination in the upper limbs of stroke survivors.