Improving Image-Retrieval Performance of Foundation Models in Gastrointestinal Endoscopic Images

Kim Kangsan¹Junseok Park²Sang Hyun Kim²Youngbae Hwang^1*

• ¹Chungbuk National University, Cheongju-si, Republic of Korea
• ²Soonchunhyang University, Asan, Republic of Korea

The quality of gastrointestinal endoscopy is verified by documenting specific required images. However, identifying these images from the numerous photographs captured during a procedure is tedious. Conventional deep-learning approaches to automate this process are often limited by subjective assessments and poor interpretability. We introduce a novel content-based image-retrieval framework that employs a dual-backbone architecture, integrating a general-purpose vision foundation model (DINOv2) and domain-specific endoscopic model (GastroNet). This system is trained using parameter-efficient metric learning and generates discriminative embeddings for efficient similarity searches. The framework is evaluated on 3,500 public endoscopic images (from the Kvasir and HyperKvasir datasets) and validated on entirely unseen real-world and synthetic data, wherein it achieved state-of-the-art performance (97.71% Recall@1, 99.14% Recall@5, and 96.74% mean average precision). These results are significantly superior to those of single-backbone baseline models. Ablation studies confirm that this improvement is primarily due to the two backbones capturing complementary features. This framework offers an accurate and automated tool for assessing the procedural quality of gastrointestinal endoscopy.