MedAlmighty: Enhancing disease diagnosis with Large Vision Model distillation

Yajing Ren^*Zheng GuWen Liu

• Artificial Intelligence and Smart Mine Engineering Technology Center, Xinjiang Institute of Engineering, Urumqi, China

Accurate disease diagnosis is crucial in the medical domain, but it is hindered by the limited, heterogeneous, and complex nature of medical data. These challenges are amplified in multimodal tasks, which require integrating diverse information. Lightweight models, though computationally efficient, often fail to capture the comprehensive medical knowledge needed for reliable predictions. To address this, we explore leveraging pre-trained large vision models, which offer robust features due to their extensive parameters and general-domain training, but lack specialized medical knowledge and rely on large-scale datasets often unavailable in the medical field. To overcome these limitations, we aim to combine the generalization strengths of large models with the domain-specific expertise of lightweight models trained on medical data, effectively bridging the gap between general and specialized performance. To this end, we propose MedAlmighty , a distillation-based framework that fuses the generalizable representations of large vision models with the domain-specific knowledge of compact medical models. Specifically, we employ DINOv2 as a frozen teacher model to guide a lightweight CNN student model via knowledge distillation. The student learns from both hard labels and soft targets provided by the teacher, balancing classification accuracy and generalization through a weighted combination of cross-entropy and KL divergence losses. This design enables the student to capture rich semantic features from DINOv2 while remaining efficient and tailored to the medical domain. Experimental results demonstrate that MedAlmighty adeptly leverages the general features of large vision models and the specialized knowledge of small models, thereby enhancing the accuracy and robustness of disease diagnosis, even in scenarios characterized by limited and diverse medical data.