Machine learning-based predictor for neurologic outcomes in patients undergoing extracorporeal cardiopulmonary resuscitation

Background We investigated the predictors of poor neurological outcomes in extracorporeal cardiopulmonary resuscitation (ECPR) patients using machine learning (ML) approaches. Methods This study was a retrospective, single-center, observational study that included adult patients who underwent ECPR while hospitalized between January 2010 and December 2020. The primary outcome was neurologic status at hospital discharge as assessed by the Cerebral Performance Categories (CPC) score (scores range from 1 to 5). We trained and tested eight ML algorithms for a binary classification task involving the neurological outcomes of survivors after ECPR. Results During the study period, 330 patients were finally enrolled in this analysis; 143 (43.3%) had favorable neurological outcomes (CPC score 1 and 2) but 187 (56.7%) did not. From the eight ML algorithms initially considered, we refined our analysis to focus on the three algorithms, eXtreme Gradient Boosting, random forest, and Stochastic Gradient Boosting, that exhibited the highest accuracy. eXtreme Gradient Boosting models exhibited the highest accuracy among all the machine learning algorithms (accuracy: 0.739, area under the curve: 0.837, Kappa: 0.450, sensitivity: 0.700, specificity: 0.740). Across all three ML models, mean blood pressure emerged as the most influential variable, followed by initial serum lactate, and arrest to extracorporeal membrane oxygenation (ECMO) pump-on-time as important predictors in machine learning models for poor neurological outcomes following successful ECPR. Conclusions In conclusion, machine learning methods showcased outstanding predictive accuracy for poor neurological outcomes in patients who underwent ECPR.


Introduction
Neurological prognosis following cardiopulmonary resuscitation (CPR) remains an issue of critical importance for survivors (1,2).It is important to estimate the potential for normalization of cerebral function in patients after return of spontaneous circulation.The capacity to accurately forecast neurological outcomes can significantly impact subsequent medical management, enabling physicians to make informed decisions that optimize the balance between quality and quantity of intensive treatment (2,3).Recently, the application of extracorporeal membrane oxygenation (ECMO) as a supplementary measure to conventional CPR has experienced a marked increase (4,5).Concurrently, the estimation of neurological outcomes for patients subjected to extracorporeal cardiopulmonary resuscitation (ECPR) has become a critical aspect of patient management.However, the task of predicting neurological outcomes post-ECPR is intrinsically complex.It necessitates the comprehensive integration of a myriad of patient-specific factors along with unique circumstances associated with ECPR.
One of the strengths of machine learning (ML) approaches is their capacity to handle intricate nonlinear relationships between predictors, leading to more robust and consistent predictions (6).Harnessing the power of ML could offer a promising solution to the challenge of predicting neurological outcomes after ECPR.This approach can effectively analyze a myriad of patient-specific factors and ECPR-associated circumstances, possibly revealing new correlations and key variables.Consequently, it can enhance the accuracy of neurological prognosis predictions, and direct attention towards the most influential elements impacting patient outcomes in ECPR.While prior studies have identified associations between favorable neurological outcomes and predictors following successful ECPR (7)(8)(9), none have explored the potential of machine learning approaches to predict neurological outcomes in ECPR patients.In this study, we aim to utilize ML methodologies to identify critical factors that can influence neurological prognosis following ECPR.We postulate that this innovative approach will shed light on the hidden correlations and interactions among the variables and contribute to a more comprehensive and precise predictive model for neurological outcomes in ECPR patients.

Study population
This study was a retrospective, single-center, observational study that included adult patients who underwent ECPR while hospitalized between January 2010 and December 2020.The Institutional Review Board (IRB) of Samsung Medical Center approved this study (IRB No. 2020-09-082).Informed consent requirements were waived by the Institutional Review Board (IRB) of Samsung Medical Center, given the retrospective nature of the study.The study included all consecutive patients who underwent ECPR during the study period, resulting in a total of 389 patients.Of these patients who under the age of 18, those with inappropriate indications for ECPR, those with pre-existing severe neurological conditions such as traumatic brain injury, major stroke, malignant brain tumor, or severe dementia, those with insufficient medical records, and those who were transferred from another hospital after undergoing ECPR were excluded (Figure 1).

Definitions and outcomes
In this study, we retrospectively collected baseline characteristics, including comorbidities, behavioral risk factors, intensive care unit management, and laboratory data, utilizing our center's dedicated "Clinical Data Warehouse Darwin-C."This data warehouse has been specifically designed to facilitate investigators in searching and retrieving de-identified medical records from electronic archives.Mean blood pressure (MBP) was the mean of the values measured in the first 24 h, mainly based on arterial blood pressure, and patients without ABP used non-invasive blood pressure instead.Laboratory data was characterized by the most unfavorable value recorded within the 6 h window immediately preceding ECMO insertion, inclusive of the period during CPR.In this study, ECPR was defined as both a successful veno-arterial ECMO implantation and pump-on with cardiac massage during the index procedure in patients with cardiac arrest.Importantly, when ROSC occurs during ECMO cannulation, practitioners generally do not remove the alreadyinserted cannula nor do they halt the ECMO pump activation process, as referenced in studies (1,10).The term "ECMO pump-on" was characterized as the cessation of chest compressions following the successful implantation and activation of the ECMO device.In this study, ECPR was initiated under specific criteria: a witnessed arrest was confirmed; conventional CPR had been administered for a duration exceeding 10 min without success; and the etiological event causing the cardiac arrest was deemed reversible (4).Exceptions to ECPR initiation were cases with: anticipated life expectancy of less than 6 months; terminal malignancy; an unwitnessed collapse; limited physical activity; an unprotected airway; or instances where CPR had been performed for over 60 min at the time of initial contact.It should be noted that age alone was not considered a contraindication for the initiation of ECPR (4).ECPR was defined as use of venoarterial ECMO intended to treat cardiac arrest and arrest to ECMO pump-on time was defined as time from collapse to the point of ECMO setup and administration (11).In patients undergoing ECPR, the process of extracorporeal circulation, combined with external volume infusion, has the potential to decrease body temperature.This reduction in temperature could confer some degree of neuroprotection via induced hypothermia.It should be noted that aggressive therapeutic hypothermia might not always be pursued in cases where the patient exhibits hemodynamic instability or has complications such as bleeding during ECMO support.Consequently, in the context of ECPR, the initiation and extent of surface cooling, as well as the targeted temperature, are individually determined by the attending ICU intensivist.This decision-making process adheres to the therapeutic hypothermia protocol established by Samsung Medical Center (12).The primary outcome was neurologic status at hospital discharge as assessed by the Glasgow-Pittsburgh Cerebral Performance Categories (CPC) score (scores range from 1 to 5) (13).CPC scores of 1 and 2 were classified as favorable neurologic outcomes; CPC scores of 3, 4, and 5 were considered poor neurologic outcomes (14, 15).We thoroughly reviewed medical records and patients were assigned to the CPC scale upon agreement by two authors (JAR and TWK).

Machine learning (Ml) models
Utilizing Shapley Additive exPlanations, we first identified the critical variables, which were then incorporated into the ML analyses.Initially, we trained and tested eight ML algorithms for a binary classification task involving the neurological outcomes of survivors after ECPR.The algorithms included logistic regression (LR), random forest (RF), AdaBoost Classification Trees (AdaBoost), Bagged CART (Bagging), Stochastic Gradient Boosting (GBM), eXtreme Gradient Boosting (XGBoost), Multivariate Adaptive Regression Spline (MARS), and Support Vector Machines with Radial Basis Function Kernel (SVM).For the final analysis, we only utilized the top three algorithms with the highest accuracy out of the aforementioned eight ML algorithms.We divided the dataset into training and testing sets with an 8:2 ratio.The training set was used for statistical analysis, feature selection, and model training, while the independent testing set was employed to evaluate the trained models.Additionally, we detected a small number of missing values in the dataset.To address this issue, we utilized the k-Nearest Neighbors algorithm for imputation during the ML analysis (16)(17)(18).This technique involved estimating the missing values by considering the values of their nearest neighbors in the dataset.By applying this approach, we ensured that the dataset was complete and ready for further analysis and modeling.Furthermore, preprocessing procedure entailed scaling and one-hot encoding.Due to the limited sample size, we opted for the Leave-One-Out Cross-Validation (LOOCV) methodology.This approach could minimize bias by assessing the algorithm across the entire dataset, thereby ensuring more consistent and reproducible results.Afterwards, we trained each ML algorithm using the best hyperparameters until convergence was achieved on the training set.The cutoff threshold for each model was determined based on the receiver operating characteristic curve and Youden index (19, 20) obtained from the validation set, and this threshold was then applied to the test set.In order to identify which variables have the predictive performance, the importance of each variable in the ML model was evaluated by the permutation score of the test set.This score is defined as a decrease in model performance (area under the receiver operating characteristic curve) when all values of a given variable are randomly mixed (21).The magnitude of the model performance reduction reflects how dependent the model is on particular variable.The importance of variables is scaled so that the maximum value is 100.

Statistical analyses
For continuous variables, we first assessed their distribution for normality.Variables that followed a normal distribution were presented as means ± standard deviations, while those that did not were described using medians and interquartile ranges.Categorical variables are represented as numbers with subsequent percentages.Data

Baseline characteristics and clinical outcomes
During the study period, 330 patients were finally enrolled in this analysis; 143 (43.3%) had favorable neurological outcomes but 187 (56.7%) did not.The characteristics of the patients are shown in Table 1.There was no difference between the two groups in the age, sex, and comorbidities except for chronic kidney disease.Compared to the group with poor neurological outcomes, the group with favorable outcomes exhibited a higher prevalence of shockable rhythms, ECPR in the coronary catheterization laboratory, cardiac cause of arrest, and arrest by acute coronary syndrome.Hemoglobin was also higher in favorable group than poor group (10.6 ± 2.6 g/dl vs. 9.6 ± 3.0 g/dl,

ML-based predictive performance of poor neurologic outcome after ECPR
The predictive performances of all algorithms were depicted in Supplementary Figure S1.After initial analysis using ML models, LR, AdaBoost, Bagging, MARS, and SVM were excluded in final analysis because of relatively low predictive power.We only utilized the top three algorithms, XGBoost, RF, GBM, and with the highest accuracy from the eight ML algorithms.Predictive performance of each ML model for poor neurologic outcome was shown in Figure 2 and Table 2. Overall, all three models showcased excellent proficiency in predicting poor neurological outcomes, with mean accuracy scores ranging between 72.3% and 73.9%.Notably, XGBoost models exhibited the highest accuracy among all the machine learning algorithms (Table 3).Figure 3 illustrated the top 10 variables that contribute to the predictive performance of each ML model.Across all three ML models, MBP emerged as the most influential variable, followed by initial serum lactate, and arrest to ECMO pump-on-time as important predictors.Finally, we tested the XGBoost model using the testing dataset, and it exhibited excellent predictive performance for poor neurological outcomes (accuracy: 0.712, 95% CI: 0.609-0.809,Kappa: 0.213, sensitivity: 0.643, specificity: 0.771, positive predictive value: 0.667, negative predictive value: 0.546).Additionally, the performance of initial lactate level for prediction of poor neurologic outcomes was evaluated.The area under the receiver operating characteristic curve was 0.66 (95% CI: 0.598-0.724)and the cut-off value was 7.37 with 86.6% sensitivity and 46.4% specificity.

Discussion
In the present study, we investigated the predictors of poor neurological outcomes in ECPR patients using ML approaches.From the eight ML algorithms initially considered, we refined our analysis to focus on the three algorithms, XGBoost, RF and GBM, that exhibited the highest accuracy.XGBoost models exhibited the highest accuracy among all the machine learning algorithms.In addition, when we tested the XGBoost model a Chronic kidney disease is defined as either kidney damage or glomerular filtration rate less than 60 ml/min/1.73m 2 for 3 months or longer.
using the testing dataset, it demonstrated outstanding predictive accuracy for poor neurological outcomes.Across all three ML models, MBP emerged as the most influential variable, followed by initial serum lactate, and arrest to ECMO pump-on-time as important predictors.Generally, lactic acid serves as a valuable indicator of tissue hypoxia (24) and is a reliable predictor of patient outcomes in cases of circulatory shock (24)(25)(26).Previous studies showed that serum lactic acid levels are associated with neurological outcomes in survivors after cardiac arrest.Sawamoto et al. demonstrated a significant difference in serum lactic acid levels between patients with favorable and poor neurological outcomes who underwent ECPR (27).Moreover, Christian et al. demonstrated that absolute serum lactate levels might serve as pertinent markers for predicting mortality in ECPR patients.Furthermore, lactate clearance was associated with neurological outcomes in these patients (28).The findings from this study highlighted that the serum lactate level served as a prognostic indicator for poor neurological outcomes in patients treated with ECPR across all ML models.
Brain recovery hinges on the swift restoration of cerebral blood flow to meet the brain's metabolic demands, with MAP being a principal determinant of this flow (29).The current guideline recommends circumvention and immediate correction of MAP less than 65 mmHg in post-resuscitation care (23).However, the Predictive performance of random forest (RF), bagged CART (bagging), and eXtreme gradient boosting (XGBoost) machine learning model for poor neurologic outcome.Sens, sensitivity; Spec, specificity; ROC, receiver operating characteristic.The top variables contributing to the predictive performance of each model.The magnitude of the model performance reduction reflects how dependent the model is on particular variable.The importance of variables is scaled so that the maximum value is 100.(32).They suggest that maintaining an average MAP of approximately 75 mmHg could be pivotal for neurological recovery after ECPR.Our study also highlighted that the MAP served as a prognostic indicator for poor neurological outcomes in patients treated with ECPR across all ML models.Given that the brain is the organ most susceptible to hypoxia and insufficient perfusion, delays in initiating ECMO during ECPR can lead to significant neurological deficits (33).Several previous studies have demonstrated that duration of no flow or low-flow is one of the most important predictors of overall outcomes after ECPR along with factors such as age, initial shockable rhythm and lactate level (4,34,35).Recently, Matsuyama et al. analyzed 256 patients undergoing ECPR and found the probability of favorable neurological outcome decreased as low-flow duration increased.Similarly, low-flow time represented by arrest to pump-on time was associated with poor neurologic outcomes in the present study.Eventually, enhancing survival and neurological outcomes is more likely when patients are put on the ECMO pump-on promptly (4,34,36).
In our previous study, factors such as shockable rhythm, initial hemoglobin levels, cardiac cause of arrest and ECPR conducted at the cardiac catheterization lab were identified as significantly associated with poor neurological outcomes (2,4,37).Low pre-ECMO hemoglobin levels might correlate with adverse neurological results (4,28).Proactively addressing anemia either before or during ECMO deployment may improve oxygen delivery and offer neuroprotection (4).Shockable rhythm was associated with favorable neurological outcomes after ECPR (2,38).ECPR conducted in a cardiac catheterization lab resulted in a reduction of low flow and cannulation time (39).In this study, most of the patients with cardiac arrest in the catheterization lab underwent ECPR in the coronary catheterization laboratory.Reducing "arrest to ECMO pump-on time" would be crucial to improve clinical outcomes, including neurologic outcomes, regardless of the location of ECPR.
This study had several limitations.First, this was a nonrandomized cohort study.Therefore, confounding factors and selection bias might have affected the results.Second, CPC scale was retrospectively determined based on medical records.We excluded patients whose neurological status could not be assessed because of deterioration followed by death.However, we included patients who had a diagnosis of brain death.Third, most ECPR patients had low body temperature caused by extracorporeal circulation and external volume infusion.Therefore, ECMO itself could have some degree of neuroprotective effect through hypothermia.Finally, lactate clearance is associated with neurological outcomes in ECPR patients, but due to the nature of retrospective studies, it cannot be provided due to insufficient data after a specific time following the initial lactic acid test.

Conclusions
In conclusion, serum lactic acid levels and arrest to ECMO pump-on time emerged as the most potent predictors in machine learning models for poor neurological outcomes following successful ECPR.Furthermore, these machine learning methods showcased outstanding predictive accuracy for poor neurological outcomes in patients who underwent ECPR.

FIGURE 3
FIGURE 3 comparison was carried out using Student's t-test or Mann-Whitney U test for continuous variables, whereas the Chi-square test for categorical variables.Clinically relevant variables, including age, sex, comorbidities, habitual risk factors, variables associated with ECPR, classification of arrest subtypes, complications of ECMO, MBP and ICU management were subjected to multiple logistic regression analyses to obtain statistically meaningful predictors associated with poor neurological outcomes.All tests were twosided and p values of less than 0.05 were considered statistically significant.Statistical analyses were performed with R Statistical Software (version 4.2.0;R Foundation for Statistical Computing, Vienna, Austria).

TABLE 1
Baseline characteristics of patients.

TABLE 2
Clinical outcomes according to neurologic outcomes.

TABLE 3
Model performance in predicting poor neurologic outcome after extracorporeal cardiopulmonary resuscitation.