Local adaptation and validation of a transdiagnostic risk calculator for first episode psychosis using mental health patient records

Elizabeth Ford^1*James Stone¹Dominic Oliver²Ben Fell³Gloria Roque³Sam Robertson⁴Paolo Fusar-Poli²Kathryn Greenwood⁵

• ¹Brighton and Sussex Medical School, Brighton, United Kingdom
• ²King's College London, London, England, United Kingdom
• ³Akrivia Health Ltd, Oxford, United Kingdom
• ⁴Sussex Partnership NHS Foundation Trust, Worthing, England, United Kingdom
• ⁵University of Sussex, Brighton, West Sussex, United Kingdom

Few at-risk adults are identified by specialised services prior to the development of a first episode of psychosis. A transdiagnostic risk calculator, predicting psychosis using electronic health record (EHR) data, was developed in London, UK to identify patients at risk, using structured data and 14 natural language processing (NLP)-derived symptom and substance use concepts. We report the adaptation and internal validation of this risk calculator in a Southeast England region.In a retrospective cohort study using EHR patient notes we identified individuals accessing mental healthcare in Southeast England (Nov-1992 to Jan-2023) who received a primary diagnosis of a nonpsychotic or non-organic mental disorder. We developed new machine-learning NLP algorithms for diagnosis, symptom and substance use concepts by fine-tuning existing open-source transformer models. Baseline and outcome coded diagnoses were supplemented with NLP-derived diagnosis data.Cox regression was used to predict psychosis and prior weights were applied; discrimination (Harrell's C) was assessed.Nearly all NLP concepts achieved an F1-measure of accuracy above 0.8 following development. In an analysis sample of 63,922 patients with complete data, the risk calculator had acceptable but lower accuracy in Southeast England (Harrell's C 0.71) compared to the London benchmark (Harrell's C 0.85).The risk calculator performed similarly in Southeast England to in other external validation studies, discriminating acceptably, suggesting that this calculator may be adapted successfully for new patient populations, services and geographic areas. Differences in accuracy may be due to different cultures of data capture, different NLP approaches, or differences in the patient cohort.