Implementation of a Net Benefit Parameter in ROC Curve Decision Thresholds for AI-Powered Mammography Screening

Anastasia Petrovna Pamova^1*Yuriy Aleksandrovich Vasilev^1,2Tatyana Mikhaylovna Bobrovskaya^1*Anton Vyacheslavovich Vladzimirskyy^1,3Olga Vasilyevna Omelyanskaya¹Kirill Mikhailovich Arzamasov^1,4

• ¹Research and Practical Clinical Center for Diagnostics and Telemedicine Technologies of the Moscow Health Care Department, Moscow, Russia
• ²FGBU Nacional'nyj mediko-hirurgiceskij Centr imeni N I Pirogova Ministerstva zdravoohranenia Rossijskoj Federacii, Moscow, Russia
• ³I.M. Sechenov First Moscow State Medical University of the Ministry of Health of the Russian Federation, Moccow, Russia
• ⁴MIREA Rossijskij tehnologiceskij universitet Institut informacionnyh tehnologij, Moscow, Russia

Background: The rapid integration of artificial intelligence (AI) into mammography necessitates robust quality control methods. The lack of standardized methods for establishing decision thresholds on the Receiver Operating Characteristic (ROC) curves makes it challenging to judge the AI performance. This study aims to develop a method for deter-mining the decision threshold for AI in screening mammography to ensure the widest possible population of women with a breast pathology is diagnosed.Methods: Three AI models were retrospectively evaluated using a dataset of digital mammograms. The dataset consisted of screening mammography examinations obtained from 663,606 patients over the age of 40. Our method estimates the decision threshold using a novel approach to net benefit (NB) analysis. Our approach to setting the cutoff threshold was compared with the threshold determined by Youden's index using McNemar's test.Results: Replacing the Youden index with our method across three AI models, resulted in a threefold reduction in false-positive rates, twofold reduction in false-negative rates, and twofold increase in true-positive rates. Thus, the sensitivity at the cutoff threshold determined by NB increased to 99% (maximum) compared to the sensitivity determined by Youden's index threshold (72% maximum). Correspondingly, the specificity when using our method decreased to 48% (minimum), compared to 75% (minimum) with the Youden's index method.Conclusions: We propose using AI as the initial reader together with our novel method for determining the decision threshold in screening with double reading. This approach enhances the AI sensitivity and improves timely breast cancer diagnosis. This study was registered in ClinicalTrials (NCT04489992) (https://clinicaltrials.gov/study/NCT04489992).