Ordinal Prediction Model of 90-Day Modified Rankin Scale in Ischemic Stroke

Background and Purpose: Prediction models for functional outcomes after ischemic stroke are useful for statistical analyses in clinical trials and guiding patient expectations. While there are models predicting dichotomous functional outcomes after ischemic stroke, there are no models that predict ordinal mRS outcomes. We aimed to create a model that predicts, at the time of hospital discharge, a patient's modified Rankin Scale (mRS) score on day 90 after ischemic stroke. Methods: We used data from three multi-center prospective studies: CRISP, DEFUSE 2, and DEFUSE 3 to derive and validate an ordinal logistic regression model that predicts the 90-day mRS score based on variables available during the stroke hospitalization. Forward selection was used to retain independent significant variables in the multivariable model. Results: The prediction model was derived using data on 297 stroke patients from the CRISP and DEFUSE 2 studies. National Institutes of Health Stroke Scale (NIHSS) at discharge and age were retained as significant (p < 0.001) independent predictors of the 90-day mRS score. When applied to the external validation set (DEFUSE 3, n = 160), the model accurately predicted the 90-day mRS score within one point for 78% of the patients in the validation cohort. Conclusions: A simple model using age and NIHSS score at time of discharge can predict 90-day mRS scores in patients with ischemic stroke. This model can be useful for prognostication in routine clinical care and to impute missing data in clinical trials.


INTRODUCTION
Prediction models of functional outcome after ischemic stroke can aid clinical decision making for providers, patients, and families by guiding rehabilitation goals, discharge planning, and patient expectations (1)(2)(3). They can also be useful for imputing missing data in clinical trials. These models stroke have generally focused on predicting a dichotomization of the modified Rankin Scale (mRS) such as functional independence (mRS 0-2) vs. functional dependency or death (mRS 3-6), or alive (mRS 0-5) vs. dead (mRS 6) (4)(5)(6). While these dichotomizations are meaningful, a model that could predict outcome across the entire spectrum of the mRS would be more informative.
For example, for patients who have less severe strokes, a model predicting mortality may be less useful than a model that predicts the exact score on the mRS (7). Such a model could also be used to impute missing data in clinical trials when patients are lost to follow-up or when outcome data is not yet available.

Aims
To address this need, we aimed to develop an ordinal logistic regression model that predicts the 90-day mRS score based on variables available at the time of hospital discharge.

METHODS
The data that support the findings of this study are available from the corresponding author upon reasonable request.

Study Patients
This study used de-identified patient data from three prior studies: CRISP, DEFUSE 2, and DEFUSE 3 (8)(9)(10). CRISP and DEFUSE 2 were multi-center prospective cohort studies and DEFUSE 3 was a prospective randomized open-label multicenter trial of endovascular therapy. Patients were older than 18, and only DEFUSE3 had an upper age limit of 90 years. Other inclusion criteria included diagnosis of ischemic stroke, eligibility for endovascular therapy, baseline NIHSS ≥ 5 (CRISP, DEFUSE 2) or ≥6 (DEFUSE 3). All studies received approval from local institutional review boards, and patients or their proxies provided written informed consent.

Variables
The dependent variable was the mRS score obtained 90 days after the index event. For patients with missing 90-day mRS scores, 30-day mRS scores were carried forward. Patients missing both outcome measures were excluded from the study. Patients who died during the initial hospitalization were also excluded.
Predictor variables included baseline characteristics [sex, age, history of atrial fibrillation, history of diabetes, history of hypertension, history of stroke or transient ischemic stroke (TIA)], imaging measures (24-h follow-up infarct volume, hemorrhagic transformation per the European Cooperative Acute Stroke Study criteria), and clinical measures (NIHSS score at discharge or day 5 of hospitalization, whichever occurred earlier).

Model Derivation
The CRISP and DEFUSE 2 datasets were used as the derivation set. The ordinal outcome measure was the mRS score at 90 days. We used a proportional odds model, which estimates intercepts for each level, but assumes a common coefficient across ordered response categories. Validity of the proportional odds assumption was verified by trending univariate odds ratios for each cutoff and plotting partial residuals. The derivation dataset was "upsampled" to account for the relatively small number of participants with an mRS score of five at day 90. The upsampling method augments the minority class by sampling random observations from this class with replacement (i.e., bootstrapping). We implemented upsampling using the upsample function in the package R-splitters. After upsampling, variables that were associated with outcome at a p < 0.2 in univariate analysis were entered in a multivariable model and were retained if they reduced the AIC by seven or more points.

Model Validation
The model was internally and externally validated to assess model performance on unseen data, thereby mitigating possible overfitting of the model. The model was internally validated within the derivation set using five-fold cross-validation. Univariate screening and forward selection were repeated to derive a model for each fold. The model was externally validated using the DEFUSE 3 dataset.
Measures used to evaluate for model performance included R 2 (coefficient of determination), mean absolute error (MAE), and the percentage of predicted outcomes that fell within one point of the observed outcomes. As an additional measure of model performance, we ran the primary efficacy analysis of the DEFUSE 3 trial based on imputed 90-day mRS scores and compared it to the same analysis using observed 90-day mRS scores.

Statistical Analyses
Proportions were compared using Fisher's exact test, and distributions of continuous and ordinal variables were compared using the t-test or Wilcoxon rank-sum test. We report twosided results and used a p < 0.05 as a threshold for statistical significance.
All statistical analyses were performed using R software (version 3.6.2) and SAS software (version 9.4).

RESULTS
There were 201 patients enrolled in the CRISP study, 130 in DEFUSE 2, and 182 in DEFUSE 3. In the derivation set (CRISP and DEFUSE 2), we excluded two patients with missing 30 and 90-day mRS outcome data, and 32 patients who died during their initial hospitalization. In the validation set (DEFUSE 3), we excluded 22 patients who died during their initial hospitalization. (Supplementary Figure I). The patient characteristics are presented in Table 1. Age (65.6 vs. 68.9, p = 0.02), prevalence of hypertension (66.7 vs. 78.8%, p = 0.01), hemorrhagic transformation score (p = 0.01), NIHSS score at discharge (6 vs. 8, p = 0.03), and infarct volume at early follow up (26.2 vs. 37.7 ml, p < 0.001) were different between the derivation and validation groups.
In the derivation set, predictors that were associated (p < 0.2) with the 90-day mRS score in univariate analyses, included age (p < 0.001), hypertension (p < 0.001), diabetes mellitus (p < 0.001), history of stroke (p = 0.12), hemorrhagic transformation (p < 0.001), NIHSS score at discharge (p < 0.001), infarct volume at early follow-up (p < 0.001). After forward selection, age and the NIHSS score at discharge remained as independent predictors of the 90-day mRS score in the multivariable ordinal regression model (p < 0.001 for each, Table 2).
With five-fold internal cross-validation, age and NIHSS were retained in each model and no other variable was retained in a  majority of models. The mean R 2 of the five validation models was 0.60, the average mean absolute error was 0.88 (95% CI 0.82-0.94), and a mean of 80% of predictions were within one of the observed value. Additional results of the five-fold internal crossvalidation are shown in Table 3 and Supplementary Figure II.
In external validation, using the DEFUSE 3 dataset, the model had an R 2 of 0.60, and a mean absolute error of 0.94 (95% CI 0.80-1.07). The model predicted 34% of the 90-day mRS scores correctly, 78% of the scores within one point of the observed value, 96% within two points of the observed value, and 99% within three points of the observed value. Additional performance metrics of the model in external validation are presented in Table 3 and illustrated in The primary analysis of the DEFUSE 3 trial, a comparison of the distribution of 90-day mRS scores between patients in the endovascular treatment and control groups, was rerun with 90day mRS scores predicted by the final model. The odds ratio for benefit from endovascular therapy using predicted mRS scores was 5.01 (95% CI 2.77-9.05), which is similar to the results of the same analysis performed using observed mRS scores (OR 3.45, 95% 1.94-6.13).

DISCUSSION
A prediction model based on two easily obtainable clinical variables, age and NIHSS score assessed at the time of hospital discharge, had moderately high accuracy for predicting a patient's 90-day modified Rankin Scale score. It accurately predicted the 90-day mRS within one point in ∼ 80% of patients. The model's independent variables -age and NIHSS score -are corroborated by prior studies that have demonstrated that these factors are correlated with functional outcome after ischemic stroke (11)(12)(13)(14). This model can be used for prognostication in clinical practice and for imputation of 90-day outcome data in clinical trials.
Several prior studies have focused on predicting survival or other dichotomized functional outcomes after ischemic stroke (5,6). A recent meta-analysis by Fahey et al. identified over 60 models for outcomes following ischemic stroke (4). Variables such as sex, age, disease characteristics, and comorbidities were the best predictors for mortality and functional outcomes after ischemic stroke. The meta-analysis noted that a limitation of  current models is that few models are externally validated (4). In addition, because the current models were derived with dichotomous outcome data, none predict a patient's exact score on the mRS at long-term follow-up. In contrast, our externally validated ordinal logistic regression model predicts a patient's 90-day mRS score with moderately high accuracy. One of the relative advantages of an ordinal regression is that there is less information loss as compared to dichotomous outcome models. Multiple studies have shown that ordinal analyses increase statistical power and efficiency, and suggest that further clinical research could benefit from increased utilization of ordinal analyses where relevant (15)(16)(17). Because of this, most recent acute stroke studies use an ordinal model for their primary outcome analysis. Unlike dichotomous prediction models, our model could be used to impute missing outcomes if 30-and 90-day mRS data are missing. Moreover, an ordinal prognostic model could help optimize future randomized trials of stroke rehabilitation interventions, by excluding patients who are likely to be non-responders because they have either a high chance of spontaneous recovery or a high chance of mortality (18). In addition, our prediction model could be used in clinical practice as a tool to assist with the assessment of a patient's prognosis. This could improve stroke rehabilitation by personalizing rehabilitation plans, reducing variation in therapy, and increasing equity of services. A recent review of prediction tools for stroke rehabilitation found that models were most helpful if they were available at the time of rehabilitation or discharge planning and predicted functional status beyond binary outcomes. The authors were concerned that a general prognosis of good or poor is not sufficiently detailed to be useful (6). Thus, 90-day ordinal predictions could be a helpful reference in discussions with the patient and family and provide additional context when discussing rehabilitation and future residence destination (19). Even though our model is only able to predict well within one point, we believe this presents an advantage over dichotomous models which predict a range without identifying the most likely outcome within that range.
Some researchers have cited concern about the use of ordinal regression models and the need to test for proportionality of the odds. To address this, we confirmed proportionality by visual inspection of the univariate odds ratios and partial residuals. In addition, we empirically demonstrated the model's performance by validating its accuracy in an external dataset.
There are limitations to our study. First, our dataset is limited to patients from three endovascular stroke therapy studies and is therefore not representative of all patients with ischemic strokes. Specifically, the model may not apply to patients with mild strokes or strokes in the posterior circulation who were not eligible for the trials. The model may also not apply well to patients with pre-existing severe disability who were excluded from the studies. The model did perform similarly in patients who underwent endovascular therapy and those who did not, likely because the effect of endovascular therapy is captured in the NIHSS score at discharge, which is one of the prediction variables in the model (20,21) Future studies validating this model in a larger cohort could provide additional information about the generalizability of this model. Second, while the simplicity of our model, which only includes two predictor variables, is a relative strength, a model derived from an even larger dataset could identify additional independent variables that might further improve the model's performance.

CONCLUSIONS
In summary, our internally and externally validated model predicts the ordinal mRS score at 90 days after ischemic stroke with moderately good accuracy and could be used for prognostication in clinical practice and to impute missing data in clinical trials.

DATA AVAILABILITY STATEMENT
The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.

ETHICS STATEMENT
The studies involving human participants were reviewed and approved by the Stanford IRB (DEFUSE 2 and CRISP) and the University of Cincinnati IRB serving as the Central IRB for StrokeNet (DEFUSE 3). The patients/participants provided their written informed consent to participate in this study.

AUTHOR CONTRIBUTIONS
ML and MM organized the database. MZ wrote the first draft of the manuscript. All authors contributed to conception and design of the study, statistical analysis, manuscript revision, read, and approved the submitted version.

FUNDING
The study was funded by grants from the National Institute for Neurological Disorders and Stroke (principal investigators, ML and GA; 1TR01ActNS075209Project01A1Year, 5TR01ActNS075209Project02Year, 5TR01ActNS075209Project 03Year, 5TR01ActNS075209Project04Year, 5TR01ActNS075209