Rectal Cancer Segmentation via HHF-SAM: A Hierarchical Hypercolumn-Guided Fusion Segment Anything Model

Wang Ye¹Yang Ying²Wu Xiaohong³Feng Zhoushan^3*Wang Congcong¹

• ¹Department of Pathology，the First Hospital of China Medical University, Shenyang, China
• ²Department of Nephrology, the Fourth People's Hospital of Shenyan, Shenyang, China
• ³Department of Neonatology, Guangzhou Key Laboratory of Neonatal Intestinal Diseases, The Third Affiliated Hospital of Guangzhou Medical University, Guangzhou, China

Rectal cancer is one of the most common cancers worldwide. Segmenting rectal cancer in abdominal CT images can assist doctors in making more accurate decisions in complex or uncertain cases, thereby improving overall diagnostic accuracy. Due to the low contrast between different tissues and the presence of significant noise in medical images, existing segmentation methods often struggle to accurately delineate the boundaries of lesion areas. To meet the growing demand for more refined segmentation techniques, this study proposes a novel framework, Hierarchical Hypercolumn-guided Fusion Segment Anything Model, called HHF-SAM. This framework fully leverages the SAM model’s strong understanding of natural images and enhances its adaptability to medical-specific features through a specially designed adapter module. Given the varying sizes and shapes of lesion areas, we introduced the Multi-scale Hypercolumn Processing Module, which generates multi-scale features to provide comprehensive guidance for the decoder. Additionally, the low contrast that blurs the boundaries between cancerous and healthy regions makes it challenging for the simple convolutional decoder in the original SAM model to accurately differentiate the complex details of lesions. To address this, we proposed the Progressive Hierarchical Fusion Decoder, which can aggregate multi-scale features and predict the final segmentation results. Finally, we introduced a multi-level supervision mechanism to further optimize network parameters. Performance tests on two large public abdominal CT datasets have demonstrated that our HHF-SAM framework outperforms other typical segmentation methods in terms of segmentation accuracy. The code will be available for model reproduction.