ViTCNN: A Robust Hybrid CNN–Vision Transformer Based Deep Learning framework for Multi-Disease Diagnosis in Women's Healthcare

Sonam Juneja^1*Bhoopesh Singh Bhati^1*Ghanshyam Tejani²Seyed Jalaleddin Mousavirad^3*

• ¹Indian Institute of Information Technology Sonepat, Sonipat, India
• ²Yuan Ze University, Zhongli District, Taiwan
• ³Mittuniversitetet, Sundsvall, Sweden

Accurate and efficient detection of multiple diseases from diagnostic images remains a major challenge in today's world , especially in women's health conditions such as breast cancer, cervical cancer, and Polycystic Ovary Syndrome (PCOS). Each of these diseases presents its own unique imaging characteristics and visual patterns, making detection of these diseases all together through a single model is highly challenging. In this respect, in order to overcome this, we have proposed a hybrid deep learning framework that combines EfficientNetB0 and Vision Transformer for multiple multi-disease detection. This shared backbone and multi-head architecture of the proposed framework integrate the strong spatial feature extraction ability of EfficientNetB0 with the contextual reasoning ability of the Vision Transformer, ensuring that the model is able to capture both local and global features of diseases. Our framework was trained on a different dataset containing several thousand of annotated diagnostic images using a two-stage learning strategy: 70 epochs of initial training followed by 30 epochs of fine-tuning. Experimental results shows very impressive diagnostic performance, where our approach has achieved accuracies of 96.8% for breast cancer, 95% for cervical cancer, and 99.03% for PCOS. These numbers are improved to 98.5%, 96.5%, and 99.07%, respectively, after a fine-tuning stage. Future work on this study will focus on dataset expansion and clinical validation for real-world diagnostic deployment.