Integrating Clinical Indications and Patient Demographics for Multi-label Abnormality Classification and Automated Report Generation in 3D Chest CT Scans

Theo Di Piazza^1,2,3*Carole Lazarus⁴Olivier Nempont⁵Loic Boussel^2,3

• ¹Univ Lyon, INSA-Lyon, INRAE, BF2I, UMR 203, 69621, Villeurbanne, France
• ²Centre de Recherche en Acquisition et Traitement de l'Image pour la Sante, Villeurbanne, France
• ³Hospices Civils de Lyon, Lyon, France
• ⁴Philips Healthcare Nederland, Eindhoven, Netherlands
• ⁵Philips Innovation Hub Eindhoven, Eindhoven, Netherlands

The increasing number of Computed Tomography (CT) scan examinations, combined with the time-intensive nature of manual analysis, necessitates efficient automated methods to assist radiologists to manage their growing workload. While deep learning approaches primarily classify abnormalities from three-dimensional (3D) CT images, radiologists also incorporate clinical indications and patient demographics, such as age and sex, for diagnosis. This study aims to enhance multi-label abnormality classification and automated report generation by integrating imaging and non-imaging data. We propose a multimodal deep learning model that combines 3D chest CT scans, clinical information reports, patient age, and sex to improve diagnostic accuracy. Our method extracts visual features from 3D volumes using a visual encoder, textual features from clinical indications via a pre-trained language model, and demographic features through a lightweight feedforward neural network. These extracted features are projected into a shared representation space, concatenated, and processed by a projection head to predict abnormalities. For the multi-label classification task, incorporating clinical indications and patient demographics into an existing visual encoder, called CT-Net, improves the F1-score to 51.58, representing a +∆6.13% increase over CT-Net alone. In the automated report generation task, we extend two existing methods, CT2Rep and CT-AGRG, by integrating clinical indications and demographic data. This integration enhances Clinical Efficacy metrics, yielding an F1-score improvement of +∆14.78% for the CT2Rep extension and +∆6.69% for the CT-AGRG extension. Our findings suggest that incorporating patient demographics and clinical information into deep learning frameworks can significantly improve automated CT scan analysis. This approach has the potential to enhance radiological workflows and facilitate more comprehensive and accurate abnormality detection in clinical practice.