Open LLM-based actionable incidental finding extraction from [18F]Fluorodeoxyglucose PET-CT radiology reports

Stephen Harry Barlow^1*Sugama Chicklore¹Yulan He^1,2,3Sebastien Ourselin¹Thomas Wagner^4,5Anna Barnes¹Gary Cook¹

• ¹King's College London, London, United Kingdom
• ²University of Warwick, Coventry, United Kingdom
• ³The Alan Turing Institute, London, United Kingdom
• ⁴Royal Free Hospital, London, United Kingdom
• ⁵University College London, London, United Kingdom

Introduction We developed an open large language model (LLM) based pipeline to extract actionable incidental findings (AIFs) from [18F]Fluorodeoxyglucose Positron Emission Tomography-Computed Tomography ([18F]FDG PET-CT) reports. This imaging modality often uncovers AIFs which can affect a patient's treatment. The pipeline classifies reports for the presence of AIFs, extracts the relevant sentences, and stores the results in structured JavaScript Object Notation format, enabling use in both short-and long-term applications. Methods Training, validation and test datasets of 1999, 248, and 250 lung cancer [18F]FDG PET-CT reports respectively, were annotated by a nuclear medicine physician. An external test dataset of 460 reports was annotated by two nuclear medicine physicians. The training dataset was used to fine-tune an LLM using QLoRA and chain-of-thought (CoT) prompting. This was evaluated quantitatively and qualitatively on both test datasets. Results The pipeline achieved document-level F1 scores of 0.917±0.016 and 0.79±0.025 on the internal and external test datasets. At sentence-level F1 scores of 0.754±0.011 and 0.522±0.012 were recorded and qualitative analysis demonstrated even higher practical utility. This qualitative analysis revealed how sentence-level performance is better in practice. Discussion Llama-3.1-8B-Instruct was the base LLM which provided best combination of performance and computational efficiency. The utilisation of CoT prompting improved performance further. Radiology reporting characteristics such as length and style affect model generalisation. Conclusion We find that a QLoRA adapted LLM utilising CoT prompting successfully extracts AIF information at both document-and sentence-level from both internal and external PET-CT reports. We believe this model can assist with short-term clinical challenges like clinical alerts and reminders, and long-term tasks like investigating comorbidities.