Deep Learning Analysis of MRI to assess Rectal Cancer Treatment

Heather M. Selby^1*Yewon A. Son¹Vipul R. Sheth²Todd Wagner^1,3Erqi Pollom⁴Arden M. Morris¹

• ¹S-SPIRE Center, Department of Surgery, Stanford University, Stanford, United States
• ²Department of Radiology, Stanford University, Stanford, United States
• ³VA Palo Alto Health Care System, Palo Alto, United States
• ⁴Department of Radiation Oncology, Stanford University, Stanford, United States

Introduction: Traditional neoadjuvant therapy for locally advanced rectal cancer (LARC) results in pathologic complete response (pCR) in approximately 15% of patients, supporting non-operative strategies for those with clinical complete response (cCR). The subjectivity and variability in MRI-based cCR assessments highlight the need for objective, quantitative tools. Objective: To develop deep learning models for automated rectal tumor segmentation on pre- and post-treatment MRIs, and to identify radiomic features differentiating cCR from non-cCR patients. Materials and Methods: We retrospectively analyzed pre- and post-treatment MRIs from 37 LARC patients enrolled in a Phase 2 TNT trial (NCT04380337). Rectal tumors were segmented on T2-weighted images by two data scientists, refined by a radiologist (reference standard), and independently segmented by a fellow. For pre-treatment segmentation, Model 1 (baseline; n=37) was trained on reference cases, then used to generate pseudo-labels for 81 additional cases. Model 2 (semi-supervised; n=118) was trained on the combined dataset. Model 3 (baseline; n=37) was trained on post-treatment cases. Radiomic features were extracted from post-treatment ADC maps, filtered by reproducibility (ICC ≥0.8) and redundancy (Spearman ρ≤0.95), then analyzed using unsupervised hierarchical clustering. Results: For pre-treatment segmentation, radiologist-fellow inter-rater agreement was DSC =0.748±0.092. Model 1 achieved mean DSC =0.682±0.254 versus the radiologist, significantly lower than inter-rater agreement. Model 2 improved performance to mean DSC =0.769±0.214 (mean gain =0.087; 12.8% relative improvement; p<0.001), slightly outperforming inter-rater agreement. For post-treatment segmentation, inter-rater agreement declined to mean DSC =0.362±0.256, while Model 3 achieved mean DSC =0.175±0.231 versus the radiologist, reflecting challenges from treatment-induced tissue changes affecting both automated models and human raters. Radiomic clustering revealed two distinct patient groups aligned with cCR and non-cCR status. Conclusion: This study demonstrates the feasibility of deep learning-based automated segmentation and radiomic profiling for differentiating treatment response in rectal cancer. Semi-supervised learning with pseudo-labeled data significantly improved segmentation performance, offering a practical approach to overcome limited annotations. Radiomic features warrant validation in larger multi-center studies for clinical translation.