Adaptive Self-Attention for Enhanced Segmentation of Adult Gliomas in Multi-Modal MRI

Evan Savaria^*Jiangwen Sun

• Old Dominion University, Norfolk, United States

Every year there are an estimated 80,000 to 90,000 new glioma brain tumor cases, highlighting the need for reliable medical diagnostic techniques. Advancements in image segmentation techniques have significantly improved the ability to distinguish tumor sub-regions from surrounding tissues. However, current state-of-the-art models do not effectively capture the complementary information from T1, T1Gd, T2, and FLAIR modalities, resulting in suboptimal feature extraction and lower segmentation accuracy. Additionally, these models often function as “black boxes,” where internal feature representations and modality-specific feature prioritization are unknown, limiting physician trust when precise delineation of tumor sub-regions is essential. As a result, tumor boundaries may become blurred, and the risk of errors in surgical resection, radiation targeting, and treatment monitoring increases. To address these gaps, we propose AIMS: an Adaptive Integrated Multi-Modal Segmentation model that dynamically learns features from each modality and progressively integrates them. The AIMS framework employs adaptive self-attention within a hierarchical CNN-Transformer architecture to prioritize and fuse multi-modal MRI features, significantly improving segmentation accuracy. We achieve high Dice Similarity Coefficients (DSC) for three clinically relevant sub-regions: enhancing tumor (ET), tumor core (TC), and whole tumor (WT) on the BraTS 2019 adult glioma dataset, and show statistically significant improvements over strong hybrid baselines. We further evaluate AIMS on an independent BraTS 2021 cohort without fine-tuning to demonstrate robustness to scanner and protocol variability. Finally, we incorporate Grad-CAM-based explanations at adaptive fusion and attention layers, together with quantitative sanity checks, to provide modality-aware and spatially meaningful visualizations that support clinical interpretation.