Multimodal BEHRT: Transformers for Multimodal Electronic Health Records to predict breast cancer prognosis

Ndèye Maguette MBAYE^1,2,3*Michael Danziger⁴Aullène Toussaint^1,5Elise Dumas^2,3,5Julien Guerin⁶Anne-Sophie Hamy-Petit^1,5Fabien Reyal^5,7,8Michal Rosen-Zvi^4,9Chloé-Agathe AZENCOTT^1,2,3*

• ¹Institut Curie, Paris, Ile-de-France, France
• ²INSERM U900 Cancer Et Génome Bioinformatique, Biostatistiques Et Épidémiologie, Paris, Île-de-France, France
• ³Centre de Bioinformatique, ParisTech École Nationale Supérieure des Mines de Paris, Université de Sciences Lettres de Paris, Paris, France
• ⁴AI for Accelerated Healthcare and Life Sciences Discovery, IBM Research Lab - Israel, Haifa 3498825, Israel, Haifa, Israel
• ⁵Residual Tumor & Response to Treatment Laboratory, RT2Lab, Translational Research Department, INSERM, U932 Immunity and Cancer, Paris, France, Paris, France
• ⁶Data Office, Institut Curie, 26, rue Ulm 75248 PARIS, France, Paris, France
• ⁷Department of Surgical Oncology, Institut Curie, University of Paris, Paris, France, Paris, France
• ⁸Department of Surgery, Institut Jean Godinot, Reims, France, Reims, France
• ⁹The Hebrew University of Jerusalem, Ein Kerem Campus, Jerusalem, Israel, Jerusalem, Israel

Background Electronic Health Records (EHRs) contain a wealth of information about patients that could be useful towards improving treatment outcomes for breast cancer patients, but remain mostly unexploited. Recent methodological developments in deep learning, however, open the way to developing new methods to leverage this information to improve patient care.We propose M-BEHRT, a Multimodal BERT for EHR data based on BEHRT, itself an architecture based on the popular natural langugage architecture BERT (Bidirectional Encoder Representations from Transformers). M-BEHRT models multimodal patient trajectories as a sequence of medical visits, comprising a variety of information such as clinical features, results from biological lab tests, medical department and procedure, and the content of free-text medical reports. M-BEHRT uses a pretraining task analog to a masked language model to learn a representation of patient trajectories from data that includes patients that are unlabeled due to censoring, and is then fine-tuned to the classification task at hand. A gradient-based attribution method highlights which parts of the input patient trajectory were most relevant for the prediction.We applied M-BEHRT to a retrospective cohort of about 15 000 breast cancer patients treated with adjuvant chemotherapy, using patient trajectories for up to one year after surgery to predict 1 Mbaye et al.disease-free survival 3 years after surgery. M-BEHRT achieves an AUC-ROC of 0.77 [0.70-0.84] on a held-out data set, compared to 0.67 [0.58-0.75] for the Nottingham Prognostic Index (NPI) and random forests (p ¡ 0.05). In addition, we identified subsets of patients for which M-BEHRT performs particularly well such as older patients with at least one lymph node affected.Our work highlights both the potential of EHR data for improving our understanding of breast cancer and the ability of transformer-based architectures to learn from EHR data containing much fewer than the millions of records typically used in currently published studies.The representation of patient trajectories used by M-BEHRT captures their sequential aspect, and opens new research avenues for understanding complex diseases and improving patient care.