AUTHOR=Huang Wanwan , Zhang Han , Cheng Yu , Quan Xiongwen 

TITLE=DRCM: a disentangled representation network based on coordinate and multimodal attention for medical image fusion

JOURNAL=Frontiers in Physiology

VOLUME=Volume 14 - 2023

YEAR=2023

URL=https://www.frontiersin.org/journals/physiology/articles/10.3389/fphys.2023.1241370

DOI=10.3389/fphys.2023.1241370

ISSN=1664-042X

ABSTRACT=Recent studies on medical image fusion based on deep learning have made remarkable progress, but the common and exclusive features of different modalities, especially their subsequent feature enhancement are ignored. Since medical images of different modalities have unique information, special learning of exclusive features should be designed to express the unique information of different modalities, so as to obtain a medical fusion image with more information and details. Therefore, we propose an attention mechanism based disentangled representation network for medical image fusion, which design coordinate attention and multimodal attention to extract and strengthen the common and exclusive features. Firstly, the common and exclusive features of each modality were obtained by the cross mutual information and the adversarial objective methods, respectively. Then the coordinate attention is focused on the enhancement of the common and exclusive features of different modalities, and the exclusive features are weighted by multimodal attention. Finally, the two kinds of features are fused. The effectiveness of the three innovation modules is verified by ablation experiments. Furthermore, eight comparison methods are selected for qualitative analysis and four metrics are used for quantitative comparison. The values of four metrics demonstrate the effect of DRCM. Especially, DRCM achieved better results on SCD, Nabf and MS-SSIM metrics, which indicates that DRCM has achieved the goal of further improving the visual quality of the fused image with more information from source images and less noise. Through the comprehensive comparison and analysis of the experimental results, the DRCM outperforms the comparison method.