Two-Step Pipeline for Oral Diseases Detection and Classification: A Deep Learning Approach

Anna Luíza Damaceno Araújo^1*Arnaldo Vitor Barros da Silva²Ana Rita Marega Gonçalves³Cristina Saldivia Siracusa⁴Daniel Lobato Ferreira Ferraz⁴Camila Barcellos Calderipe⁴Ivan José Correia-Neto⁴Pablo Agustin Vargas⁴Marcio Ajudarte Lopes⁴André Carlos Ponce De Leon Ferreira De Carvalho³Marcos G. Quiles²Alan Roger Santos-Silva⁴Luiz Paulo Kowalski¹

• ¹Universidade de Sao Paulo Faculdade de Medicina, São Paulo, Brazil
• ²Universidade Federal de Sao Paulo, São Paulo, Brazil
• ³Universidade de Sao Paulo, São Paulo, Brazil
• ⁴Universidade Estadual de Campinas Faculdade de Odontologia de Piracicaba, Piracicaba, Brazil

Introduction: This study aimed to develop and evaluate an artificial intelligence pipeline combining object detection and classification models to assist in early identification and differentiation of oral diseases. Methods: This retrospective cross-sectional study utilized clinical images of oral potentially malignant disorders and oral squamous cell carcinoma, comprising a baseline dataset of 773 images from Faculdade de Odontologia de Piracicaba, Universidade Estadual de Campinas (FOP-UNICAMP) and an external validation dataset of 132 images from Federal University of Paraíba (UFPB). All images were obtained prior to biopsy, all with corresponding histopathological reports. For object detection, ten YOLOv11 models were developed with varying data augmentation strategies, trained for 200 epochs using pretrained COCO weights. For classification, three MobileNetV2 models were trained on images cropped according to the experts' reference bounding box annotations, each using different combinations of learning rates and data augmentation. After selecting the best detector–classifier combination, we integrated them into a two-step pipeline in which the images cropped by the detector were subsequently forwarded to the classifier. Results: The best YOLOv11 configuration achieved a mAP50 of 0.820, precision of 0.897, recall of 0.744, and F1-score of 0.813. For classification, the best MobileNetV2 configuration achieved an accuracy of 0.846, precision of 0.87,1 recall of 0.864, F1-score of 0.844, and AUC-ROC of 0.852. On external validation this same model reached an accuracy of 0.850, precision of 0.866, recall of 0.850, F1-score of 0.851, and an AUC-ROC of 0.935. The two-step approach, when applied to the test set from the baseline dataset, achieved an accuracy of 0.784, precision of 0.793, recall of 0.784, F1-score of 0.784, and an AUC-ROC of 0.811. When evaluated on the external validation dataset, it yielded an accuracy of 0.863, precision of 0.879, recall of 0.863, F1-score of 0.866, and an AUC-ROC of 0.934. The visual inspection of YOLO's inference outputs confirmed consistent lesion localization across diverse oral cavity images, with some missing (17.4%). The t-SNE visualization demonstrated partial separation between OPMD and OSCC feature embeddings, indicating the model captured discriminative patterns with some class overlap.