Evaluation of Severity of Illness Scores in the Pediatric ECMO Population

Though commonly used for adjustment of risk, severity of illness and mortality risk prediction scores, based on the first 24 h of intensive care unit (ICU) admission, have not been validated in the pediatric extracorporeal membrane oxygenation (ECMO) population. We aimed to determine the association of Pediatric Index of Mortality 2 (PIM2), Pediatric Risk of Mortality Score III (PRISM III) and Pediatric Logistic Organ Dysfunction (PELOD) scores with mortality in pediatric patients on ECMO. This was a retrospective cohort study of children ≤18 years of age included in the Pediatric ECMO Outcomes Registry (PEDECOR) from 2014 to 2018. Logistic regression and Receiver Operating Characteristics (ROC) curves were used to calculate the area under the curve (AUC) to evaluate association of mortality with the scores. Of the 655 cases, 289 (44.1%) did not survive until hospital discharge. AUCs for PIM2, PRISM III, and PELOD predicting mortality were 0.52, 0.52, and 0.51 respectively. PIM2, PRISM III, and PELOD scores are not associated with odds of mortality for pediatric patients receiving ECMO. These scores for a general pediatric ICU population should not be used for prognostication or risk stratification of a select population such as ECMO patients.


INTRODUCTION
Tools to gauge severity of illness (SOI) and predict outcomes for children treated with extracorporeal membrane oxygenation (ECMO) can facilitate medical decision making, counseling of patients' families, and prognostication. ECMO is a cardio-pulmonary support modality for patients in severe cardiac and/or respiratory failure. It can be a life-saving therapy for patients who might otherwise not survive but is associated with significant morbidity and mortality. Patients requiring ECMO have increased severity of illness than the average pediatric intensive care unit (PICU) patient and have higher mortality (1).
Various scores are commonly used in PICUs worldwide for standardization of risk of mortality for both clinical benchmarking and for stratification of patient groups for research purposes. It is not uncommon to include markers of SOI or predictors of mortality in multivariable analysis to adjust for illness severity in comparing patients.
These include Pediatric Index of Mortality 2 (PIM2), Pediatric Risk of Mortality Score III (PRISM III), and Pediatric Logistic Organ Dysfunction (PELOD) (2)(3)(4). While the validation set for these scores had an area under the curve (AUC) of >0.9 for outcome prediction, (2)(3)(4) they were calibrated for the general PICU population. They are calculated using data in the first 1 h (PIM2) or 24 h of PICU admission (PRISM III, PELOD) and significant variation in a patient's clinical course may exist from the time of score calculation on admission to when patients are cannulated onto ECMO. Illness severity scores validated for general PICU populations are sometimes used to adjust risk in specific populations, such as patients with hemophagocytic lymphohistiocytosis, those receiving renal replacement therapy, therapeutic plasma exchange, etc. (5)(6)(7). While these scores are also used in studies for pediatric patients on ECMO, there is however limited data on the ability of these scores to predict mortality in this select cohort. Knowing the validity of these illness severity scores in children receiving ECMO is needed to determine the appropriate of their use for clinical and research purposes in this cohort.
We sought to determine the association of PIM2, PRISM III, and PELOD scores calculated on admission to the PICU with mortality in pediatric patients treated with venovenous (VV) or venoarterial (VA) ECMO. We hypothesized that these scores would not be associated with outcomes for children receiving ECMO.

METHODS
We conducted a retrospective cohort study of patients in the Pediatric ECMO Outcomes Registry (PEDECOR) cannulated on ECMO from 2014 to 2018. PEDECOR is a web-based data platform, housed at Baylor College of Medicine's Dan L. Duncan Institute for Clinical and Translational Research. While initially built to support anticoagulation in ECMO research for BloodNet, a subgroup of PALISI (Pediatric Acute Lung Injury and Sepsis Investigators), it has expanded to include a robust clinical, laboratory and medication dataset for patients on ECMO. There were 10 U.S. children's hospitals participating in the registry. All participating institutions were trained on data entry; a data dictionary was provided, and two-person verification was undertaken at each site for interrater validation.
Baylor College of Medicine's Institutional Review Board approved the study protocol and waived the need for informed consent. We included all patients ≤ 18 years of age at the time of PICU admission. Children cannulated at an outside hospital and subsequently transferred to a PEDECOR site were excluded. We excluded patients in the registry prior to 2014 and those without mortality information. Patients that were missing all three scores were excluded from the study; we included patients with at least one of the three scores. For patients with multiple ECMO runs, only the first ECMO run was included in the analysis.
PIM2, PRISM III, and PELOD scores were directly added to the case report form if available to a study site, otherwise the individual variables comprising the score were directly entered into the online form from which the PIM2, PRISM III, and PELOD scores were then calculated. Other variables collected included demographic information such as sex and race, and clinical variables such as time from PICU admission to cannulation, weight, mode of ECMO cannulation, history of in-hospital cardiopulmonary arrest prior to ECMO, and ECPR which was defined as actively receiving CPR at the time of cannulation. The following variables were collected as markers of illness severity at the time of ECMO cannulation: urine output 8, 16, and 24 h prior to ECMO, highest lactate, total bilirubin, International Normalized Ratio (INR), platelet count, C-reactive protein (CRP), and creatinine in the 24 h prior to ECMO. OI was calculated within 6 h prior to cannulation, and VIS determined immediately prior to cannulation (8,9).

Statistical Analysis
Patient and clinical characteristics were summarized using mean with standard deviation, median with 25 and 75th percentiles, and frequency with percentage. The summary statistics were stratified by mortality and compared using two sample t-test, Wilcoxon rank sum test, Fisher's exact test, or Chi-square test, as appropriate. Logistic regression and Receiver Operating Characteristics (ROC) curves were used to calculate the area under the curve (AUC) for predicting mortality with PIM2, PRISM III, and PELOD to assess discrimination. The ROC curves were compared to each other. Calibration of the model based on the deciles of risk by the relevant score was performed using the Hosmer-Lemeshow test. Subgroup analyses were decided on a priori and included the VV, VA and ECPR cohorts, as well as early cannulation, defined as patients cannulated within the first 2 days of PICU admission. Statistical analyses were performed using Stata v 15.1 (Stata Corp, College Station, TX). For the calibration test a p-value >0.05 indicates a good model fit. For other analyses, a p-value of <0.05 was set as statistically significant a priori.

RESULTS
Seven hundred fifty six ECMO runs were evaluated in pediatric patients ≤18years. There were 655 patients in the registry that met inclusion and exclusion criteria (Figure 1). Prior to hospital discharge 289 (44.1%) patients died overall, with a mortality of 26.5% for VV cannulations and 49.1% for VA cannulations. Table 1 presents the summary statistics for patient and ECMO course characteristics by mortality. Table 2 presents the diagnostic indications for ECMO by survival and by cannulation type. Mortality was also higher for ECPR patients and those who received in-hospital CPR prior to ECMO.
The odds of mortality were not associated with PIM2, PRISM III, or PELOD scores ( Table 3 and Figure 2). The AUC for PIM2 was 0.52 (Figure 3), for PRISM III was 0.52 ( Figure 4) and for PELOD was 0.51 ( Figure 5). There was no difference in the performance of these scores for patients on VV or VA ECMO (Supplementary Figures 1, 2). Subgroup analysis of early cannulation, showed that of the three scores, PRISM III performed slightly better at predicting mortality in this cohort with an AUC of 0.60 (Supplementary Figure 3). Table 4 presents the AUC and 95% confidence intervals for PIM2, PRISM III, and PELOD, with mortality for all patients, by cannulation type, and by timing of ECMO start. AUC's range from 0.47 up to 0.60 indicating poor discrimination. Hosmer-Lemeshow goodness of fit test indicates adequate calibration by score deciles for the entire cohort, the VV ECMO group, the VA ECMO group, and the group cannulated after 2 days. Calibration was poor among those cannulated earlier than 2 days, and for PELOD (p = 0.0195) but was adequate for PIM2 and PRISM III (Supplementary Table 1).

DISCUSSION
Multiple scores have been developed for the general PICU population for use in clinical bench marking and in research stratification by adjusting the expected risk of mortality for degree of illness. PIM2, a revised version of Pediatric Index of Mortality, is a risk of mortality score published in 2003 (3). It was developed using patient data gathered during the first hour of ICU admission from 12 units in Australia, New Zealand and the United Kingdom. PRISM III, a revision of the PRISM score, is a pediatric physiology-based score for mortality risk published in 1996. It was a U.S. study, validated using 17 clinical variables from the first 24 h of admission to the study site ICU (4). PELOD is a score for the assessment of multi-organ dysfunction, developed in 1999 and validated in 2003. It includes 10 variables corresponding to 5 organ dysfunctions with data collected from the first 24 h of admission to ICU (2). These scores have been validated in the general PICU population with AUC >0.9 for mortality outcome prediction. They have been used for mortality outcome prediction in specific groups of patients, such as those supported with ECMO, even though they were not originally created for this purpose.
Until recently, there were no data available on whether these three scores were appropriate for discriminating mortality risk in children treated with ECMO. Barbaro et al. showed that the PIM2, PRISM III, and PELOD scores had poor ability to discriminate mortality (10). The study was limited to 178 pediatric patients with respiratory failure on veno-venous ECMO. The validity of these scores in pediatric patients treated with veno-arterial ECMO for respiratory and cardiac failure was not studied. Our results, using a different data set, validate the previous finding that PIM2, PRISM III, and PELOD scores do not discriminate for death in pediatric patients receiving VV ECMO for respiratory failure, in a more diverse cohort across several institutions. Our study also shows that, in addition to VV patients, these scores have poor discrimination with mortality for patients receiving VA ECMO, as well as the overall larger cohort of 655 patients PIM2, PRISM III, and PELOD scores are based on parameters collected within the first few hours of admission to the PICU. PICU lengths of stay can range from days to even months, and a patient's clinical course can be highly variable influenced in part by the various therapies administered as well as individual patient responses to treatment. Cannulation for ECMO is often remote from admission, and scores calculated based on variables at the time of admission less accurately reflect the clinical state at the time of cannulation. We sought to determine if the scores performed better when applied to patients cannulated closer to the time the score was calculated. For the subgroup of patients cannulated within 2 days of PICU admission, 50% of study cohort, PRISM III had the highest AUC of 0.6 among the scores, which is far from ideal for a prognostic tool, and vastly inferior to the AUC of 0.9 for the overall PICU population. These SOI scores are not adequate for discrimination of mortality even when applied to patients who go on to receive ECMO cannulation closer to when the scores are calculated.
Severity of illness scores developed for the general PICU population are often used in research studies to compare patients based on illness severity. They have also been used to describe illness severity as well as risk of mortality in pediatric patients on ECMO. The PRISM score was used in the description of patients with meningococcal disease that were supported on ECMO (11) and in pediatric ECMO patients with acute respiratory failure (12). Even more recent studies such as a study on delirium in pediatric patients on VA ECMO in a cardiac ICU have used the PRISM III score to assess risk of mortality of patients in their cohort (13). PIM2, PELOD and PRISM scores were used in the baseline comparison of groups of patients who did not undergo tracheostomy vs. those that underwent an early or late tracheostomy after ECMO cannulation (14). Our study found that these scores cannot discriminate survivors from nonsurvivors after ECMO, hence they should not be used to compare baseline illness severity or to correlate outcomes. These general ICU scores used for risk stratification in focused populations are not valid for patients on ECMO.

LIMITATIONS
Limitations of this study are variation of clinical practice among different centers, missing data, and limitation of analysis to the variables collected in the registry. The PIM2, PRISM III, and PELOD scores were contemporary at the start of the PEDECOR

CONCLUSION
Commonly used SOI scores-PIM2, PRISM III, and PELOD are not associated with mortality for pediatric patients receiving Frontiers in Pediatrics | www.frontiersin.org  ECMO. This lack of discrimination persists for patients receiving VA, VV or those cannulated within 2 days of ICU admission. These scores should not be used by clinicians for medical decision making or prognostication for ECMO patients. They should not be used in research studies involving patients that receive ECMO to compare baseline illness severity between groups of patients.

DATA AVAILABILITY STATEMENT
The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.

ETHICS STATEMENT
Baylor College of Medicine's Institutional Review Board reviewed and approved the studies involving human participants and waived the need for informed consent.  Supplemental Figure 1C | Receiver operating characteristic (ROC) curve and Logistic regression for PELOD predicting mortality for Patients receiving Venovenous ECMO.
Supplemental Figure 2A | Receiver operating characteristic (ROC) curve and Logistic regression for PIM 2 predicting mortality for Patients receiving Venoarterial ECMO.
Supplemental Figure 2B | Receiver operating characteristic (ROC) curve and Logistic regression for PRISM III predicting mortality for Patients receiving Venoarterial ECMO.
Supplemental Figure 2C | Receiver operating characteristic (ROC) curve and Logistic regression for PELOD predicting mortality for Patients receiving Venoarterial ECMO.
Supplemental Figure 3A | Receiver operating characteristic (ROC) curve and Logistic regression for PIM2 predicting mortality for Patients receiving early ECMO.
Supplemental Figure 3B | Receiver operating characteristic (ROC) curve and Logistic regression for PRISM III predicting mortality for Patients receiving early ECMO.
Supplemental Figure 3C | Receiver operating characteristic (ROC) curve and Logistic regression for PELOD predicting mortality for Patients receiving early ECMO.