Characterizing Risk of In-Hospital Mortality Following Subarachnoid Hemorrhage Using Machine Learning: A Retrospective Study

Abstract

Background:

Subarachnoid hemorrhage has a high rate of disability and mortality, and the ability to use existing disease severity scores to estimate the risk of adverse outcomes is limited. Collect relevant information of patients during hospitalization to develop more accurate risk prediction models, using logistic regression (LR) and machine learning (ML) technologies, combined with biochemical information.

Methods:

Patient-level data were extracted from MIMIC-IV data. The primary outcome was in-hospital mortality. The models were trained and tested on a data set (ratio 70:30) including age and key past medical history. The recursive feature elimination (RFE) algorithm was used to screen the characteristic variables; then, the ML algorithm was used to analyze and establish the prediction model, and the validation set was used to further verify the effectiveness of the model.

Result:

Of the 1,787 patients included in the mimic database, a total of 379 died during hospitalization. Recursive feature abstraction (RFE) selected 20 variables. After simplification, we determined 10 features, including the Glasgow coma score (GCS), glucose, sodium, chloride, SPO₂, bicarbonate, temperature, white blood cell (WBC), heparin use, and sepsis-related organ failure assessment (SOFA) score. The validation set and Delong test showed that the simplified RF model has a high AUC of 0.949, which is not significantly different from the best model. Furthermore, in the DCA curve, the simplified GBM model has relatively higher net benefits. In the subgroup analysis of non-traumatic subarachnoid hemorrhage, the simplified GBM model has a high AUC of 0.955 and relatively higher net benefits.

Conclusions:

ML approaches significantly enhance predictive discrimination for mortality following subarachnoid hemorrhage compared to existing illness severity scores and LR. The discriminative ability of these ML models requires validation in external cohorts to establish generalizability.

Introduction

Subarachnoid hemorrhage (SAH) is a type of hemorrhagic stroke that accounts for 3% of all stroke types. With the development of medicine, the global case fatality rate has decreased from 50% to 17%, but the mortality rate of subarachnoid hemorrhage remains high (1–3). In addition, survivors are often left with a permanent disability, cognitive deficits (particularly in executive function and short-term memory), and mental health symptoms (depression, anxiety), leading to significant reductions in health-related quality of life. In recent years, machine learning (ML), as an area of artificial intelligence, has been able to learn from data based on computational modeling. Similarly, ML can fit higher-order relationships between covariates and outcomes in data-rich environments (4–6).

The purpose of this study was to determine whether ML algorithms using demographics, comorbidities, laboratory tests, and other variables can predict the prognosis of SAH fairly accurately and to identify factors that contribute to predictive ability.

Patient Selection and Methods

Data Source

This study was a retrospective study based on the Medical Information Mart for Intensive Care IV (7) (MIMIC-IV version 1.0) database. An individual who has finished the Collaborative Institutional Training Initiative examination (Certification number 43357625 for author Deng) can access the database.

Participant Selection

Inclusion criteria are as follows: (1) patients with subarachnoid hemorrhage confirmed by ICD-9 or ICD-10; (2) people with an age of more than 16 years old; and (3) admission to ICU with the Glasgow coma score (GCS). Moreover, for patients with ICU admissions more than once, only data of the first ICU admission of the first hospitalization were included in the analysis.

Predictors

In this study, the data were extracted from MIMIC-IV, including age, gender, race, language, GCS, sepsis-related organ failure assessment (SOFA) score, and history of trauma. Then, we extracted data containing vital signs, laboratory findings, treatment history of heparin, and antibiotics during hospitalization. Besides, we collected the Charlson comorbidity index (CCI) composed of myocardial infarction, congestive heart failure, peripheral vascular disease, cerebrovascular disease, dementia, chronic pulmonary disease, rheumatic disease, peptic ulcer disease, diabetes, paraplegia, renal disease, malignant cancer, severe liver disease, metastatic solid tumor, and acquired immunodeficiency syndrome (AIDS).

Outcomes

Patients diagnosed with subarachnoid hemorrhage died during hospitalization.

Statistical Analysis

Categorical variables were presented as numbers and percentages that were analyzed using the χ² test or the Fisher exact test, while continuous variables were expressed as mean ± SD or median with interquartile range (IQR), which were analyzed by an independent t-test or Mann–Whitney U test.

Each feature has different importance or coef attributes in the model, and these data determine the importance of the feature in the model. Recursive feature elimination (RFE) returns the importance of each feature through the learner (8, 9). Then, the least important feature is removed from the current feature set. This step of recursion on the feature set is repeated until the required number of features is finally reached. Then, features are then considered in groups of 5–60; they are organized according to the grade obtained by the feature selection method. In order to find the best hyperparameters, 10-fold cross-validation is used as a resampling method. In each iteration, every nine folds are used as a training subset, and the remaining one is processed to adjust the hyperparameters. In this way, each sample will participate in the training model and test the model, so that all data can be used to the greatest extent.

In this study, we divided the data set (ratio 70:30), trained the model, and verified it. We calculated the median and 95% confidence interval of the area under the curve (AUC), where the AUC value of 1.0 indicated complete discrimination and 0.5 indicated no discrimination. Finally, the accuracy, sensitivity, specificity, negative predictive value, and positive predictive value of external data verification were calculated. Additionally, we conducted the decision curve analysis (DCA) to determine the clinical usefulness of the included variables by quantifying the net benefit at different threshold probabilities.

All analyses were performed by the statistical software package R version 4.1.3 (http://www.R-project.org, The R Foundation). In our study, we used the “Caret” R packages to achieve the process. P values less than 0.05 (two-sided test) were considered statistically significant.

Results

Baseline Characteristics

Variable values of the SAH patients in MIMIC-IV were analyzed. A total of 1,787 cases were included in the study, of which 349 died during hospitalization. It is found from the data in the table that the infection indexes of the dead patients are significantly increased, and the coagulation system has an abnormal function, thrombocytopenia, electrolyte disorder, and so on. At the same time, the temperature and oxygen saturation of these patients fluctuate more widely and are more likely to be accompanied by other diseases (Table 1 and Figure 1).

Figure 1

Variable Importance

Through feature screening by the RFE algorithm, we find that it has the highest accuracy when 20 features are included (Figure 2). In order to further simplify the model, we choose the models with an accuracy similar to the best feature number to verify the analysis. Therefore, we establish the prediction model with the characteristic numbers of 10 and 20. Model1 includes GCS, glucose, sodium, chloride, SPO₂, bicarbonate, temperature, white blood cell (WBC), heparin use, and SOFA score, while Model2 include GCS, glucose, sodium, chloride, SPO₂, bicarbonate, temperature, white blood cell (WBC), heparin use, SOFA score, creatinine, bun, platelet, age, marital, trauma, lymphocytes, calcium, race, and cerebrovascular disease. Then, these variables were used in all the subsequent analyses for all models in both training and testing sets.

Figure 2

Prediction performance of different models

We use 10 features and 20 features to establish the traditional regression and ML models, respectively. In simplified Model 1, the logistic regression (LR), random forest (RF), gradient boosting machine (GBM), artificial neural network (NNET), support vector machine (SVM), eXtreme gradient boosting (XGB), adapting boosting (ADA), and naïve Bayes (NB) models obtained AUCs of 0.883, 0.949, 0.945, 0.888, 0.926, 0.925, 0.936, and 0.920, respectively (Table 2 and Figure 3). In Model 2, the LR, RF, GBM, NNET, SVM, XGB, ADA, and NB models obtained AUCs of 0.921, 0.958, 0.959, 0.801, 0.942, 0.950, 0.951, and 0.927, respectively (Table 3 and Figure 4). Through the Delong test, different models constructed by LR, NNET, and XGB algorithms are different (Table 4). Comparatively, RF-Model 1 had the highest predictive performance among these models. The decision curve is suitable for comparing the net benefits of the best model and alternative methods of clinical decision-making. Among the two different models, the net benefit of the model composed of the GBM algorithm is higher than that of other models, indicating that the model has a better effect in predicting the in-hospital mortality of SAH (Figures 5, 6).

Figure 3

Figure 4

Figure 5

Figure 6

Through the importance ranking of the ML algorithm, the first 10 important characteristics of two different models of RF are consistent (Figure 7). Moreover, the importance of the GCS accounted for the highest proportion.

Figure 7

Performance of Models in Subgroup (Non-Traumatic Subarachnoid Hemorrhage) Analysis

In order to verify the prediction ability of the model in non-traumatic subarachnoid hemorrhage, we took the cases without definite trauma as a new research subgroup (Table 5) and divided them into a training set and a test set (ratio 70:30). After establishing the model with the simplified characteristic variables in the training set, the prediction ability was verified with the test set. Within the training set, the LR, RF, GBM, NNET, SVM, XGB, Ada, and NB models were established, and the testing set obtained AUCs of 0.909, 0.951, 0.955, 0.891, 0.929, 0.956, 0.947, and 0.921 (Table 6 and Figure 8). Among the eight models, GBM has the highest prediction performance and NNET has the worst generalization ability. As shown in Figure 9, the net benefit of the GBM model exceeded that of other ML models and LR regression models, indicating that the model has better performance in predicting the queue.

Figure 8

Figure 9

Discussion

Subarachnoid hemorrhage (SAH) has a high mortality and disability rate, and many complications may occur after the onset, while most of the current studies have used a single feature for prognosis research, ignoring the adverse outcomes caused by other factors. Recently, a large number of studies have reported that peripheral blood, biochemical, and other conventional indicators are associated with the prognosis of subarachnoid hemorrhage, so we used the indicators commonly found in the mimic database for model building.

In this study, we use RFE to screen important features. After simplifying the model, we use the traditional logistic regression and ML algorithm for modeling. There is basically no significant difference in the prediction ability between these simplified models and the best models. At the same time, the simplified models can reduce the phenomenon of overfitting and are more suitable for clinical use to reduce unnecessary workloads. In subgroup analysis, the model established with the same characteristics has higher AUCs, which also proves that the model has a better ability to predict the prognosis of patients with non-traumatic subarachnoid hemorrhage.

From the study, we found a larger association of mortality with patients’ electrolyte levels, glucose levels, and whether they used heparin in addition to the traditional GCS. In addition, the SOFA score also history a significant mortality factor, and this score mainly describes indicators of impairment in multiple organ functions (10) (respiratory, neurological, cardiovascular, hepatic, coagulation, and renal). The underlying mechanism may be caused by the patient’s past medical history leading to organ failure or by coagulopathy due to bleeding.

Impaired consciousness occurs in some patients after SAH. GCS is assessed by the ability to eye opening, best verbal response, and best motor response, can easily and rapidly assess the state of consciousness of a patient, and to identify development of complications and the potential degree of ultimate recovery (11). Meanwhile, in our study, glucose level served as an important factor in the prediction of death. Pppacena et al. found that higher blood glucose was associated with higher mortality after SAH (12). Meanwhile, a higher rate of glycemic variability was also associated with prognosis after SAH (13).

Recently, the neutrophil to lymphocyte ratio (NLR) was reported by most literature studies to have a correlation with the prognosis of SAH (14), so we also calculated NLR as a feature. In univariate analysis, there was a clear difference between the two groups, and after filtering by ML algorithms, NLR failed to be included in the model as a better feature, perhaps because of inconsistent outcomes across studies. The higher importance of leukocytes at the same time is consistent with the finding by Srinivasan et al. and Chamling et al. that early elevation of peripheral leukocytes is associated with the occurrence of DCI and poor functional outcomes (15).

Sodium and chloride are important components of electrolytes in humans, and 36% of SAH patients present with hyponatremia after the onset, mainly as a result of cerebral salt-wasting syndrome (CSWS) and syndrome of inappropriate antidiuretic hormone secretion (SIADH). Vrsajkov et al. and Saramma et al. found better outcomes in patients who did not develop hyponatremia during ICU treatment (16, 17). Hyponatremia has also been reported to be associated with an increased risk of vasospasm. This may be the main reason for the poor prognosis of patients (18).

Low bicarbonate concentrations occur in patients with severe acute illness. Although the current mechanism is unknown, increased systemic vascular resistance can occur after SAH, leading to transient lactic acidosis with the formation of neurogenic pulmonary edema, resulting in poor patient outcomes reported in a case study (19), Satoh et al. found that patients presenting with neurogenic pulmonary edema had lower bicarbonate concentrations (20). In addition, Stephan et al. found that one in five patients had abnormally low bicarbonate levels on admission and a poor prognosis (21).

Our study found that the use of heparin in SAH patients was able to improve outcomes, which was consistent with the findings of Post et al. (22) that the use of heparin was able to reduce mortality after SAH. The concomitant use of low-dose heparin may reduce the risk of thrombosis and reduce the poor prognosis resulting from thrombus shedding (23, 24).

In summary, the characteristic factors screened by RFE in our study were all investigated in SAH; meanwhile, they were all correlated with prognosis. The strength of this study is that the method of ML was used to combine the relevant factors to predict the mortality of SAH, while feature acquisition was simple and able to be acquired within a smaller hospital. Patients with SAH are sicker, and early and accurate prediction of mortality is able to provide clinicians with more time to adjust the corresponding treatment options, while, at the same time, in clinical work, further treatment should be given to the related diseases. In addition, the validation set was adopted in this study to verify the reliability of the model so that it had better reliability. Finally, most of the data in this study come from publicly available databases, and their data have good reliability.

Our study has limitations, which are similar to most studies related to public databases. First, the MIMIC database cannot provide the relevant imaging examination of cases. Therefore, we cannot perform an M-Fisher score on patients to establish a model nor can we evaluate whether patients have obvious trauma information and the nature of aneurysms. Second, as a public database, the MIMIC database may cause data errors due to the errors of researchers or the database itself when extracting data. In addition, there is the possibility of SAH error classification. In order to reduce the deviation caused by inaccurate code, we adopt the extensively used ICD-9 and -10 codes. Third, as with all potential retrospective studies, there are unmeasured confounding factors that lead to selection bias. Finally, although our study explored the mortality of SAH in the intensive care unit, other results, such as long-term prognosis and complications, also need further study.

Conclusion

This study suggests that some important features may be related to the prognosis after SAH. The ML model deals with a large number of variables and then distinguishes patients who die in hospitals to promote the implementation of timely and effective treatment. In the future, further verification of its clinical application value will be necessary.

References

• 1.

  GoASMozaffarianDRogerVLBenjaminEJBerryJDBlahaMJet alHeart disease and stroke statistics—2014 update: a report from the American Heart Association. Circulation. (2014) 129(3):e28–292. 10.1161/01.cir.0000441139.02102.80

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 2.

  LovelockCERinkelGJERothwellPM. Time trends in outcome of subarachnoid hemorrhage: population-based study and systematic review. Neurology. (2010) 74(19):1494–501. 10.1212/WNL.0b013e3181dd42b3

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 3.

  MuehlschlegelS. Subarachnoid hemorrhage. Contin Lifelong Learn Neurol. (2018) 24(6):1623–57. 10.1212/CON.0000000000000679

  • CrossRef
  • Google Scholar

• 4.

  BeamALKohaneIS. Big data and machine learning in health care. JAMA. (2018) 319(13):1317. 10.1001/jama.2017.18391

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 5.

  ZhangZHoKMHongY. Machine learning for the prediction of volume responsiveness in patients with oliguric acute kidney injury in critical care. Crit Care. (2019) 23(1):112. 10.1186/s13054-019-2411-z

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 6.

  ZhangZ. Predictive analytics in the era of big data: opportunities and challenges. Ann Transl Med. (2020) 8(4):68–8. 10.21037/atm.2019.10.97

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 7.

  GoldbergerALAmaralLAGlassLHausdorffJMIvanovPCMarkRGet alPhysioBank, PhysioToolkit, and PhysioNet: components of a new research resource for complex physiologic signals. Circulation. (2000) 101:e215-20. 10.1161/01.CIR.101.23.e215

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 8.

  SanzHValimCVegasEOllerJMReverterF. SVM-RFE: selection and visualization of the most relevant features through non-linear kernels. BMC Bioinformatics. (2018) 19(1):432. 10.1186/s12859-018-2451-4

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 9.

  ChenQMengZLiuXJinQSuR. Decision variants for the automatic determination of optimal feature subset in RF-RFE. Genes. (2018) 9(6):301. 10.3390/genes9060301

  • CrossRef
  • Google Scholar

• 10.

  LambdenSLaterrePFLevyMMFrancoisB. The SOFA score—development, utility and challenges of accurate assessment in clinical trials. Crit Care. (2019) 23(1):374. 10.1186/s13054-019-2663-7

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 11.

  MiddletonPM. Practical use of the Glasgow Coma Scale: a comprehensive narrative review of GCS methodology. Australas Emerg Nurs J. (2012) 15(3):170–83. 10.1016/j.aenj.2012.06.002

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 12.

  PappacenaSBaileyMCabriniLLandoniGUdyAPilcherDVet alEarly dysglycemia and mortality in traumatic brain injury and subarachnoid hemorrhage. Minerva Anestesiol. (2019) 85(8):830–9. 10.23736/S0375-9393.19.13307-X

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 13.

  OkazakiTHifumiTKawakitaKShishidoHOgawaDOkauchiMet alBlood glucose variability: a strong independent predictor of neurological outcomes in aneurysmal subarachnoid hemorrhage. J Intensive Care Med. (2018) 33(3):189–95. 10.1177/0885066616669328

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 14.

  Al-MuftiFAmuluruKDamodaraNDodsonVRohDAgarwalSet alAdmission neutrophil–lymphocyte ratio predicts delayed cerebral ischemia following aneurysmal subarachnoid hemorrhage. J NeuroInterventional Surg. (2019) 11(11):1135–40. 10.1136/neurintsurg-2019-014759

  • CrossRef
  • Google Scholar

• 15.

  SrinivasanAAggarwalAGaudihalliSMohantyMDhandapaniMSinghHet alImpact of early leukocytosis and elevated high-sensitivity C-reactive protein on delayed cerebral ischemia and neurologic outcome after subarachnoid hemorrhage. World Neurosurg. (2016) 90:91–5. 10.1016/j.wneu.2016.02.049

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 16.

  VladimirVGordanaJSnezanaSArsenUPanticVJ. Clinical and predictive significance of hyponatremia after aneurysmal subarachnoid hemorrhage. Balk Med J. (2012). 10.5152/balkanmedj.2012.037

  • CrossRef
  • Google Scholar

• 17.

  SarammaPPGirish MenonPSrivastavaASankara SarmaP. Hyponatremia after aneurysmal subarachnoid hemorrhage: implications and outcomes. J Neurosci Rural Pract. (2013) 04(01):24–8. 10.4103/0976-3147.105605

  • CrossRef
  • Google Scholar

• 18.

  MaimaitiliAMaimaitiliMRexidanALuJAjimuKChengXet alPituitary hormone level changes and hypxonatremia in aneurysmal subarachnoid hemorrhage. Exp Ther Med. (2013) 5(6):1657–62. 10.3892/etm.2013.1068

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 19.

  MayerSAFinkMEHommaSShermanDLiMandriGLennihanLet alCardiac injury associated with neurogenic pulmonary edema following subarachnoid hemorrhage. Neurology. (1994) 44(5):815–5. 10.1212/WNL.44.5.815

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 20.

  SatohETagamiTWatanabeAMatsumotoGSuzukiGOndaHet alAssociation between serum lactate levels and early neurogenic pulmonary edema after nontraumatic subarachnoid hemorrhage. J Nippon Med Sch. (2014) 81(5):305–12. 10.1272/jnms.81.305

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 21.

  ClaassenJVuAKreiterKTKowalskiRGDuEYOstapkovichNet alEffect of acute physiologic derangements on outcome after subarachnoid hemorrhage*. Crit Care Med. (2004) 32(3):832–8. 10.1097/01.CCM.0000114830.48833.8A

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 22.

  PostRZijlstraIBergRvdCoertBAVerbaanDVandertopWP. High-dose nadroparin following endovascular aneurysm treatment benefits outcome after aneurysmal subarachnoid hemorrhage. Neurosurgery. (2018) 83(2):281–7. 10.1093/neuros/nyx381

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 23.

  HantscheAWilhelmyFKasperJWendeTHamerlaGRascheSet alEarly prophylactic anticoagulation after subarachnoid hemorrhage decreases systemic ischemia and improves outcome. Clin Neurol Neurosurg. (2021) 207:106809. 10.1016/j.clineuro.2021.106809

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 24.

  KunzMSillerSNellCSchnieppRDornFHugeVet alLow-dose versus therapeutic range intravenous unfractionated heparin prophylaxis in the treatment of patients with severe aneurysmal subarachnoid hemorrhage after aneurysm occlusion. World Neurosurg. (2018) 117:e705–11. 10.1016/j.wneu.2018.06.118

  • Pubmed Abstract
  • CrossRef
  • Google Scholar