An Externally Validated Dynamic Nomogram for Predicting Unfavorable Prognosis in Patients With Aneurysmal Subarachnoid Hemorrhage

Abstract

Background: Aneurysmal subarachnoid hemorrhage (aSAH) leads to severe disability and functional dependence. However, no reliable method exists to predict the clinical prognosis after aSAH. Thus, this study aimed to develop a web-based dynamic nomogram to precisely evaluate the risk of poor outcomes in patients with aSAH.

Methods: Clinical patient data were retrospectively analyzed at two medical centers. One center with 126 patients was used to develop the model. Least absolute shrinkage and selection operator (LASSO) analysis was used to select the optimal variables. Multivariable logistic regression was applied to identify independent prognostic factors and construct a nomogram based on the selected variables. The C-index and Hosmer–Lemeshow p-value and Brier score was used to reflect the discrimination and calibration capacities of the model. Receiver operating characteristic curve and calibration curve (1,000 bootstrap resamples) were generated for internal validation, while another center with 84 patients was used to validate the model externally. Decision curve analysis (DCA) and clinical impact curves (CICs) were used to evaluate the clinical usefulness of the nomogram.

Results: Unfavorable prognosis was observed in 46 (37%) patients in the training cohort and 24 (29%) patients in the external validation cohort. The independent prognostic factors of the nomogram, including neutrophil-to-lymphocyte ratio (NLR) (p = 0.005), World Federation of Neurosurgical Societies (WFNS) grade (p = 0.002), and delayed cerebral ischemia (DCI) (p = 0.0003), were identified using LASSO and multivariable logistic regression. A dynamic nomogram (https://hu-ping.shinyapps.io/DynNomapp/) was developed. The nomogram model demonstrated excellent discrimination, with a bias-corrected C-index of 0.85, and calibration capacities (Hosmer–Lemeshow p-value, 0.412; Brier score, 0.12) in the training cohort. Application of the model to the external validation cohort yielded a C-index of 0.84 and a Brier score of 0.13. Both DCA and CIC showed a superior overall net benefit over the entire range of threshold probabilities.

Conclusion: This study identified that NLR on admission, WFNS grade, and DCI independently predicted unfavorable prognosis in patients with aSAH. These factors were used to develop a web-based dynamic nomogram application to calculate the precise probability of a poor patient outcome. This tool will benefit personalized treatment and patient management and help neurosurgeons make better clinical decisions.

Introduction

Aneurysmal subarachnoid hemorrhage (aSAH) is an acute cerebrovascular injury with high morbidity and mortality worldwide (1). Roughly one-third of survivors have severe disability and functional dependence, some of which may be predicted and treated early (2, 3). Therefore, a practical prediction model for treating and evaluating these patients with aSAH is urgently needed.

Several studies have shown that a series of serum biochemical and immunological indices could predict the prognosis of patients with aSAH after surgery, including white blood cell (WBC) count, neutrophil-to-lymphocyte ratio (NLR), C-reactive protein level, and blood glucose level on admission (4–6). Meanwhile, predictive models combining multiple independent predictors have been used to predict prognosis. However, these predictive models lack calibration tests, model external validation, or evaluation of their clinical usefulness; therefore, the accuracy and practicability of the models are difficult to assess (7, 8). High-quality predictive models are necessary to guide clinical decision-making, personal care, and patient management (9). Therefore, a more practical and accurate predictive model is urgently needed in this field. A web-based dynamic nomogram that precisely calculates the probability of the disease is a more precise and practical tool than standard nomograms and some predictive models (10, 11). However, no study has reported a dynamic nomogram to predict the clinical prognosis after aSAH.

Thus, this study aimed to develop a dynamic web-based nomogram that incorporated blood laboratory tests, clinical state at admission, and postoperative complications for the prognosis of patients with aSAH. The predictive model was validated using internal and independent external validation cohorts.

Materials and Methods

Study Design and Patient Enrollment

This retrospective observational cohort study was conducted in two centers. A training cohort of 126 patients with aSAH treated in the Department of Neurosurgery, Renmin Hospital of Wuhan University, from September 1, 2019 to September 1, 2020, was used to develop the model. A total of 84 patients admitted to the Department of Neurosurgery, the Affiliated Hospital of Panzhihua University, from September 1, 2019 to September 1, 2020, were used to validate the model. The diagnosis of aSAH was assessed by head CT, CT angiography, or digital subtraction angiography according to the guidelines (12, 13).

The exclusion criteria were as follows: (1) non-aneurysmal or traumatic SAH; (2) clinical history of aSAH for more than 48 h; (3) complicated with vascular anomalies and malformations; (4) complicated with intracerebral hemorrhage; (5) taking anticoagulants or corticosteroids within 1 month of hospitalization; (6) postoperative state on admission; (7) no surgical treatment performed within 2 days of onset; (8) acute infection on admission; (9) other organs with extreme dysfunction; (10) bilateral mydriasis or other permanent brain injuries on admission; (11) missing data.

Data Collection

Clinical information including demographic data (admission number, name, sex, age, hypertension, smoking, and alcohol consumption), clinical state, radiological findings, serum laboratory tests (glucose, D-dimer, WBCs, neutrophils, lymphocytes, and monocytes), and CT Hounsfield unit (HU) values on admission were collected using an electronic medical record system. The number, location length, and neck size of the aneurysms were also recorded. Hunt and Hess classification and the World Federation of Neurosurgical Societies (WFNS) were used to measure the clinical characteristics and neurological status at admission. The radiological features at admission were classified according to the modified Fisher grade of the CT scan (14–16).

All patients were treated with either a surgical clipping or coil embolization as soon as possible. The postoperative routine therapies included hemostasis, analgesics, anti-inflammatory drugs, and nimodipine for anti-vasospasm. An immediate postoperative head CT scan was performed to identify intracranial rebleeding or cerebral infarction after the operation.

Admission CT Values

The mean CT values of the blood clots in the subarachnoid space were measured. The regions of interest were manually drawn on representative slices by two neurosurgeons who were blinded to the patients' clinical characteristics. The density of blood clotting in the subarachnoid space on CT slices was assessed using the mean HU value. The following subarachnoid cisterns/fissures were used to measure the mean HU: lateral Sylvian fissures, anterior interhemispheric fissures, medial Sylvian fissures, suprasellar cistern, ambient cistern, and quadrigeminal cistern, as previously described (17, 18).

The Definition of Delayed Cerebral Ischemia

Delayed cerebral ischemia (DCI) was identified as follows: (1) CT scan omitting other causes of focal neurological decline such as aphasia, apraxia, hemianopia, or neglect, either permanent or temporary, within 4 to 14 days after SAH; (2) a Glasgow Coma Scale decrease of at least 2 points lasting for at least 1 h and not immediately evident after surgery; (3) head CT scan revealing a new cerebral infarction within 4–30 days after aSAH, which was not noticeable on admission or immediately after the operation, and no other explanation except for vasospasm (17, 19).

Outcome Assessment

Three months after the initial bleeding, the functional outcome was dichotomized as excellent or poor according to the modified Rankin Scale (mRS) (20–22). A favorable outcome was defined as a score of 0–2 (perform activities without assistance) on the mRS, while a score of 3–6 (moderate to severe disability or death) was defined as a poor outcome. The results were evaluated via telephone by a neurosurgeon who was blinded to the clinical and imaging data.

Sample Size

In the current literature, the events per variable (EPV) criterion, notably an EPV of 10, is widely applied as the lower limit for developing logistic regression models that predict a binary outcome (23, 24). Thus, an EPV of 10 was used to estimate the effective sample size in this study. A total of three variables, which were incorporated into the multivariate logistic analysis, were selected by least absolute shrinkage and selection operator (LASSO) regression analysis. Thus, the effective sample size was at least 30 in the training cohort.

Statistical Analysis

Statistical analysis was performed using IBM SPSS Statistics for Windows, version 26.0 (IBM Corp., Armonk, NY, USA) and R software (https://www.r-project.org/). Continuous variables were analyzed using independent t-test or Mann–Whitney U test and are presented as means ± SD or medians with interquartile range. Categorical variables are expressed as numbers (percentages) and were analyzed using χ² or Fisher exact tests. Based on the limited sample size in this study, LASSO regression, which is suitable for analyzing high-dimensional data, was used to select the most informative prognostic variables from the data cohort (25–27). The variables were then entered into a multivariable logistic regression model; a nomogram model was then constructed by integrating independent factors.

The discrimination and calibration capacities evaluated the performance of the nomogram model. Receiver operating characteristic (ROC) curve and area under the curve (AUC) analyses were performed to assess the discriminatory capabilities, while the bias-corrected C-index was calculated using 1,000 bootstraps. The Hosmer–Lemeshow test, Brier score, and calibration curve were used to assess the calibration ability (28). In addition to internal validation, the model performance was also assessed using an external validation cohort. The clinical effectiveness of the nomogram was evaluated by decision curve analysis (DCA) and clinical impact curve (CIC) (29, 30).

We used the “glmnet, corrplot, caret package” in R software to generate the LASSO regression results and the “rms package” to establish the nomogram. The “pROC,” “rmda,” “MASS,” “survival,” “ggplot2,” “ggridges,” “plotROC,” and “riskRegression” packages were applied to generate the C-index, ROC, DCA, CIC, and calibration curve. The “shinyapps.io” and “DynNom packages” were used to develop a web-based dynamic nomogram application to predict unfavorable prognosis, which could precisely calculate the risk probability for unfavorable prognosis at 3 months after aSAH. All tests were two-tailed, and p-values < 0.05 were considered statistically significant.

Results

Baseline Cohort Characteristics

This study included a total of 210 patients with aSAH, with 126 and 84 patients in the training and external validation cohorts, respectively. The baseline characteristics of the two cohorts were shown in Table 1. Women comprised 137 (65%) patients in the two cohorts, and the mean age was 56.50 years (interquartile range: 53.00, 63.00). We observed no significant differences between the training and external validation cohorts, except for higher admission glucose level, percentages of DCI occurrence and WFNS grade IV–V, and lower aneurysm neck size in the training cohort. The numbers of patients with poor prognoses in the two cohorts, 46 (37%) and 24 patients (29%), respectively, did not differ significantly. Table 2 showed the baseline characteristics of the model development cohort.

Variable Selection

LASSO regression was used to select the most useful variables. As shown in Figure 1A, when using the minimum error criterion, 21 variables were decreased to 11 variables and further reduced to three variables after applying the one standard error (1-SE) criterion. Thus, an optimal λ of 0.1356, with log(λ) = −1.997, was adopted. Three factors with non-zero coefficients were finally selected by 5-fold cross-validation to prevent overfitting (Figure 1B). After adjusting by multivariable logistic regression, NLR [adjusted odds ratio (aOR), 1.143; 95% CI 1.045–1.263, p = 0.005], WFNS grade (aOR: 5.025, 95% CI 1.777–14.75, p = 0.002), and DCI (aOR: 6.143, 95% CI 2.374–18.13, p = 0.0003) were independent prognostic factors (Table 3).

Figure 1

Prognostic Model Development

All independent factors were used to construct a nomogram to predict unfavorable prognosis 3 months after aSAH (Figure 2). According to the patient's information on admission, each of the three prognostic factors in the nomogram was projected upward to a point. NLR had continuous values ranging from 0 to 100. The WFNS grade was divided into two levels (I–III and IV–V), while DCI was divided into non-DCI and DCI, each of which were assigned points. The total sum of the points from the three variables was converted into an individual poor prognosis risk, in which the higher was the total score, the higher was the risk of an unfavorable prognosis. Based on the ordinary nomogram, we developed a dynamic nomogram web-based application (https://hu-ping.shinyapps.io/DynNomapp/) to precisely calculate the risk probability for unfavorable prognosis at 3 months after aSAH.

Figure 2

Nomogram Validation

The original AUC was 0.88 (95% CI 0.82–0.95), while the bias-corrected C-index with 1,000 bootstraps was 0.82, suggesting that the model had excellent discrimination. Compared to a single independent prognostic factor, the nomogram model better predicted the occurrence of DCI (Figure 3A). The calibration capacity was also internally validated; the Hosmer–Lemeshow p-value of 0.41 in the training cohort suggested an excellent fitting of the nomogram. By contrast, a Brier score of 0.12 and a calibration curve with 1,000 bootstrap resamples showed that the model has a good calibration ability (Figure 4A), demonstrating no significant deviation between the actual and predicted probabilities.

Figure 3

Figure 4

A total of 84 patients retrospectively collected from September 2019 to September 2020 at another center were used for external validation of the model. The validation AUC value of 0.84 (95% CI 0.74–0.94) was consistent with the original AUC value (0.88) in the training cohort (Figure 3B). A Brier score of 0.13 and the calibration curve plotted for external validation showed good calibration of the model in the external validation cohort (Figure 4B). Furthermore, DCA and CIC were used to evaluate the clinical usefulness of our prognostic nomogram. DCA showed a superior overall net benefit for a threshold probability of 0–1 (Figure 5A). CIC also demonstrated good performance over the entire range of threshold probabilities (Figure 5B).

Figure 5

Discussion

This study constructed and validated a dynamic nomogram model to predict unfavorable prognosis at 3 months after aSAH. The nomogram incorporated the NLR, admission WFNS grade, and DCI. To our knowledge, this study is the first to develop a web-based dynamic nomogram application to calculate the risk of poor prognosis (mRS, 3–6 points). In addition, external validation using data from another center was conducted for the first time. Precise calculation of the probability of the disease makes the dynamic nomogram a more accurate and practical tool compared to ordinary predictive models.

In a previous study, Zhang et al. developed an ordinary nomogram combined with a systemic inflammation response index and other conventional factors to predict the prognostic outcome in patients with aSAH (8). These factors were selected using univariate and multivariate logistic analyses. Lai et al. conducted a predicting model for aSAH prognosis, which included the GCS score, surgical clipping, and NLR (7). These independent prognostic variables were also selected by multivariate logistic regression. Compared with the study by Zhang et al., the model by Lai et al. was only internally validated rather than externally validated. Although a large-scale retrospective study has been conducted on poor recovery in patients with aSAH, the study only contained an internal validation (31). Our study included a total of 84 patients retrospectively collected from September 2019 to September 2020 from another center for external validation of the model. The validation AUC value was nearly identical to the original AUC value in the training cohort. The calibration curve showed good calibration of the model in the external validation cohort.

Contrary to the conventional multivariate logistic method, this study applied the LASSO regression analysis, which performed well in reducing the data dimensionality and decreasing multicollinearity between variables, and it was used to select variables for multivariable logistic regression by minimizing coefficients and reducing variance (32). As a result, an optimal λ of 0.1356, with log(λ) = −1.997, was adapted. Three factors with non-zero coefficients were finally selected by 5-fold cross-validation to prevent overfitting. A logistic regression model was then used to establish a nomogram. The original AUC was 0.88, while the bias-corrected C-index with 1,000 bootstraps was 0.82, suggesting that the model had excellent discrimination. Compared with a single independent prognostic factor, the nomogram model better predicted the occurrence of DCI. The calibration capacity was also internally validated; the Hosmer–Lemeshow p-value in the training cohort suggested an excellent fit of the nomogram. By contrast, a Brier score and calibration curve with 1,000 bootstrap resamples showed that the model had a good calibration ability, demonstrating no significant deviation between the actual and predicted probabilities. In addition, DCA showed a superior overall net benefit for a threshold probability of 0–1. The CIC also demonstrated a good performance over the entire range of threshold probabilities.

We observed no significant difference in NLR between the training and external validation cohorts. However, the NLR in patients with an unfavorable prognosis was higher than that in patients with a favorable prognosis in the training cohort (p = 0.005). Our study identified NLR on admission as an independent risk factor for unfavorable prognosis, similar to previous findings (4, 7, 33, 34). Lai et al. reported that a high NLR on admission could increase the risk of poor prognosis 3 months postoperatively. Giede-Jeppe et al. suggested that higher NLR was associated with more significant neurological injury, as reflected by higher subarachnoid and intraventricular blood. Therefore, increased NLR resulted in worse functional outcomes beyond these associations. The following possible mechanisms may explain why high NLR on admission increased the risk of unfavorable prognosis at 3 months after aSAH. First, a high NLR could promote the synthesis and secretion of matrix metalloproteinase, which could degrade components of the extracellular matrix and play an essential role in the blood–brain barrier, resulting in secondary brain injury (35, 36). Second, increased NLR at admission may include excessive neuroinflammation caused by elevated neutrophil granulocyte counts and lymphocyte depletion leading to immunodepression. Physiological stress responses lead to an increased postictal release of endogenous catecholamines and cortisol, resulting in secondary brain injury and possibly inflammatory complications (37, 38).

The WFNS grade or GCS have gradually become essential validated scales to evaluate clinical characteristics and neurological status on admission. A good grade was defined as I–III on WFNS, while IV–V on the WFNS was considered poor (39, 40). In our study, 40 (32%) and 11 (13%) patients in the training and external validation cohorts, respectively, had WFNS grades of IV–V. WFNS grade (p = 0.002) on admission independently predicted unfavorable prognosis and patients with WFNS grades IV–V had a higher risk of developing a poor prognosis compared with patients with WFNS grades I–III. This finding is similar to those of previous studies showing that WFNS grades IV–V was correlated with a poor outcome and was a good predictor of unfavorable prognosis after aSAH (41, 42).

Furthermore, the DCI occurrence in the two cohorts was 35 and 15%, respectively. The onset of aSAH was concentrated during the COVID-19 period, which resulted in patients not receiving timely anti-vasospasm. It may be interpreted that the training cohort had a larger percentage of DCI. DCI was identified as another crucial prognostic factor in this study. The results of a large-scale retrospective study also suggested that DCI plays an important role in short-term and long-term prognosis (43). Cerebrovascular disorders, microthrombosis, cortical diffuse depolarization, and neuroinflammation all reportedly play potential roles in DCI pathogenesis (17, 44–46).

Contrary to nomograms or other models calculated as an approximation in previous studies, the web-based dynamic nomogram application developed in the present study (https://hu-ping.shinyapps.io/DynNomapp/) can be used to calculate an exact value for developing poor outcomes at 3 months after aSAH. This tool will help in the individualized treatment and management of patients, including neuro-intensive care, blood pressure control, nimodipine anti-vasospasm, anti-inflammatory drugs, and rehabilitation treatment, and help neurosurgeons make better clinical decisions.

However, this study has several limitations. First, this was a retrospective study; therefore, prospective studies are needed to verify our results. Second, owing to the low effective sample size in the training cohort, to reduce the data dimensionality and decrease multicollinearity between variables, we used LASSO regression analysis to select predictors. Third, future studies should collect complete clinical treatment information.

Conclusion

This study identified independent predictors for an unfavorable prognosis in patients with aSAH, including NLR on admission, the WFNS grade, and DCI. On this basis, a web-based dynamic nomogram application, after externally validated, was developed to calculate the precise probability of a poor outcome in patients 3 months after aSAH. This will benefit personalized treatment and patient management and help neurosurgeons make better clinical decisions.

Publisher's Note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

References

• 1.

  ConnollyESJrRabinsteinAACarhuapomaJRDerdeynCPDionJHigashidaRTet al. Guidelines for the management of aneurysmal subarachnoid hemorrhage: a guideline for healthcare professionals from the American Heart Association/american Stroke Association. Stroke. (2012) 43:1711–37. 10.1161/STR.0b013e3182587839

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 2.

  RinkelGJAlgraA. Long-term outcomes of patients with aneurysmal subarachnoid haemorrhage. Lancet Neurol. (2011) 10:349–56. 10.1016/S1474-4422(11)70017-5

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 3.

  MackeyJKhouryJCAlwellKMoomawCJKisselaBMFlahertyMLet al. Stable incidence but declining case-fatality rates of subarachnoid hemorrhage in a population. Neurology. (2016) 87:2192–7. 10.1212/WNL.0000000000003353

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 4.

  Giede-JeppeAReichlJSprügelMILückingHHoelterPEyüpogluIYet al. Neutrophil-to-lymphocyte ratio as an independent predictor for unfavorable functional outcome in aneurysmal subarachnoid hemorrhage. J Neurosurg. (2019) 132:400–7. 10.3171/2018.9.JNS181975

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 5.

  ZhangDZhuangZWeiYLiuXLiWGaoYet al. Association of admission serum glucose-phosphate ratio with severity and prognosis of aneurysmal subarachnoid hemorrhage. World Neurosurg. (2019) 127:e1145–51. 10.1016/j.wneu.2019.04.071

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 6.

  MaXLanFZhangY. Associations between C-reactive protein and white blood cell count, occurrence of delayed cerebral ischemia and poor outcome following aneurysmal subarachnoid hemorrhage: a systematic review and meta-analysis. Acta Neurol Belg. (2021) 9:1–14. 10.1007/s13760-020-01496-y

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 7.

  LaiXZhangWYeMLiuXLuoX. Development and validation of a predictive model for the prognosis in aneurysmal subarachnoid hemorrhage. J Clin Lab Anal. (2020) 34:e23542. 10.1002/jcla.23542

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 8.

  ZhangPLiYZhangHWangXDongLYanZet al. Prognostic value of the systemic inflammation response index in patients with aneurismal subarachnoid hemorrhage and a Nomogram model construction. Br J Neurosurg. (2020) 12:1–7. 10.1080/02688697.2020.1831438

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 9.

  JajaBNCusimanoMDEtminanNHanggiDHasanDIlodigweDet al. Clinical prediction models for aneurysmal subarachnoid hemorrhage: a systematic review. Neurocrit Care. (2013) 18:143–53. 10.1007/s12028-012-9792-z

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 10.

  ChenSLiXLvHWenXDingQXueNet al. Prognostic dynamic nomogram integrated with inflammation-based factors for non-small cell lung cancer patients with chronic hepatitis B viral infection. Int J Biol Sci. (2018) 14:1813–21. 10.7150/ijbs.27260

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 11.

  ChenLQianJLinLLinJChenQZhuangZet al. Prognostic value of preoperative lymphocyte-to-monocyte ratio in oral cancer patients and establishment of a dynamic nomogram. Oral Dis. (2021) 27:1127–36. 10.1111/odi.13629

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 12.

  BedersonJBConnollyESJrBatjerHHDaceyRGDionJEDiringerMNet al. Guidelines for the management of aneurysmal subarachnoid hemorrhage: a statement for healthcare professionals from a special writing group of the Stroke Council, American Heart Association. Stroke. (2009) 40:994–1025. 10.1161/STROKEAHA.108.191395

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 13.

  AjiboyeNChalouhiNStarkeRMZanatyMBellR. Unruptured cerebral aneurysms: evaluation and management. Sci World J. (2015) 2015:954954. 10.1155/2015/954954

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 14.

  FronteraJAClaassenJSchmidtJMWartenbergKETemesRConnollyESJret al. Prediction of symptomatic vasospasm after subarachnoid hemorrhage: the modified fisher scale. Neurosurgery. (2006) 59:21–7; discussion 21–7. 10.1227/01.NEU.0000218821.34014.1B

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 15.

  SanoHSatohAMurayamaYKatoYOrigasaHInamasuJet al. Modified World Federation of Neurosurgical Societies subarachnoid hemorrhage grading system. World Neurosurg. (2015) 83:801–7. 10.1016/j.wneu.2014.12.032

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 16.

  ZhangYZhuXHouKZhaoJGaoXSunYet al. Clinical outcomes of surgical clipping for intracranial aneurysms in patients with a Hunt and Hess grade 4 or 5. Arq Neuropsiquiatr. (2016) 74:478–81. 10.1590/0004-282x20160064

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 17.

  VergouwenMDVermeulenMCoertBAStroesESRoosYB. Microthrombosis after aneurysmal subarachnoid hemorrhage: an additional explanation for delayed cerebral ischemia. J Cereb Blood Flow Metab. (2008) 28:1761–70. 10.1038/jcbfm.2008.74

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 18.

  WooPYMTseTPKChanRSKLeungLNYLiuSKKLeungAYTet al. Computed tomography interobserver agreement in the assessment of aneurysmal subarachnoid hemorrhage and predictors for clinical outcome. J Neurointerv Surg. (2017) 9:1118–24. 10.1136/neurintsurg-2016-012576

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 19.

  VergouwenMDVermeulenMVan GijnJRinkelGJWijdicksEFMuizelaarJPet al. Definition of delayed cerebral ischemia after aneurysmal subarachnoid hemorrhage as an outcome event in clinical trials and observational studies: proposal of a multidisciplinary research group. Stroke. (2010) 41:2391–5. 10.1161/STROKEAHA.110.589275

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 20.

  NewcommonNJGreenTLHaleyECookeTHillMD. Improving the assessment of outcomes in stroke: use of a structured interview to assign grades on the modified Rankin Scale. Stroke. (2003) 34:377–8; author reply 377–8. 10.1161/01.STR.0000055766.99908.58

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 21.

  BanksJLMarottaCA. Outcomes validity and reliability of the modified Rankin scale: implications for stroke clinical trials: a literature review and synthesis. Stroke. (2007) 38:1091–6. 10.1161/01.STR.0000258355.23810.c6

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 22.

  JanssenPMVisserNADorhout MeesSMKlijnCJAlgraARinkelGJ. Comparison of telephone and face-to-face assessment of the modified Rankin Scale. Cerebrovasc Dis. (2010) 29:137–9. 10.1159/000262309

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 23.

  MoonsKGDe GrootJABouwmeesterWVergouweYMallettSAltmanDGet al. Critical appraisal and data extraction for systematic reviews of prediction modelling studies: the CHARMS checklist. PLoS Med. (2014) 11:e1001744. 10.1371/journal.pmed.1001744

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 24.

  PavlouMAmblerGSeamanSDe IorioMOmarRZ. Review and evaluation of penalised regression methods for risk prediction in low-dimensional data with few events. Stat Med. (2016) 35:1159–77. 10.1002/sim.6782

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 25.

  SteyerbergEWEijkemansMJHarrellFEJrHabbemaJD. Prognostic modeling with logistic regression analysis: in search of a sensible strategy in small data sets. Med Decis Making. (2001) 21:45–56. 10.1177/0272989X0102100106

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 26.

  SauerbreiWRoystonPBinderH. Selection of important variables and determination of functional form for continuous predictors in multivariable model building. Stat Med. (2007) 26:5512–28. 10.1002/sim.3148

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 27.

  RileyRDSnellKIEMartinGPWhittleRArcherLSperrinMet al. Penalization and shrinkage methods produced unreliable clinical prediction models especially when sample size was small. J Clin Epidemiol. (2020) 132:88–96. 10.1016/j.jclinepi.2020.12.005

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 28.

  HanleyJAMcneilBJ. The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology. (1982) 143:29–36. 10.1148/radiology.143.1.7063747

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 29.

  VickersAJElkinEB. Decision curve analysis: a novel method for evaluating prediction models. Med Decis Making. (2006) 26:565–74. 10.1177/0272989X06295361

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 30.

  Van CalsterBWynantsLVerbeekJFMVerbakelJYChristodoulouEVickersAJet al. Reporting and interpreting decision curve analysis: a guide for investigators. Eur Urol. (2018) 74:796–804. 10.1016/j.eururo.2018.08.038

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 31.

  YanYHuJFangXZhenYFengLZhangXet al. Predicting the poor recovery risk of aneurysmal subarachnoid hemorrhage: clinical evaluation and management based on a new predictive nomogram. Clin Neurol Neurosurg. (2021) 200:106302. 10.1016/j.clineuro.2020.106302

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 32.

  AkkolS. The prediction of live weight of hair goats through penalized regression methods: LASSO and adaptive LASSO. Arch Anim Breed. (2018) 61:451–8. 10.5194/aab-61-451-2018

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 33.

  GuoRWuYChenRYuZYouCMaLet al. Clinical value of neutrophil-to-lymphocyte ratio in primary intraventricular hemorrhage. World Neurosurg. (2019) 127:e1051–6. 10.1016/j.wneu.2019.04.040

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 34.

  WangJYZhangXTWangJQWangCYZhengWLPanZMet al. Admission neutrophil-lymphocyte ratio predicts rebleeding following aneurismal subarachnoid hemorrhage. World Neurosurg. (2020) 138:e317–22. 10.1016/j.wneu.2020.02.112

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 35.

  LazaridisCRusinCGRobertsonCS. Secondary brain injury: predicting and preventing insults. Neuropharmacology. (2019) 145:145–52. 10.1016/j.neuropharm.2018.06.005

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 36.

  LattanziSDi NapoliMRicciSDivaniAA. Matrix metalloproteinases in acute intracerebral hemorrhage. Neurotherapeutics. (2020) 17:484–96. 10.1007/s13311-020-00839-0

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 37.

  ZhouYWangYWangJAnne StetlerRYangQW. Inflammation in intracerebral hemorrhage: from mechanisms to clinical translation. Prog Neurobiol. (2014) 115:25–44. 10.1016/j.pneurobio.2013.11.003

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 38.

  LattanziSBrigoFTrinkaECagnettiCDi NapoliMSilvestriniM. Neutrophil-to-lymphocyte ratio in acute cerebral hemorrhage: a system review. Transl Stroke Res. (2019) 10:137–45. 10.1007/s12975-018-0649-4

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 39.

  OshiroEMWalterKAPiantadosiSWithamTFTamargoRJ. A new subarachnoid hemorrhage grading system based on the Glasgow Coma Scale: a comparison with the Hunt and Hess and World Federation of Neurological Surgeons Scales in a clinical series. Neurosurgery. (1997) 41:140–7; discussion 147–8. 10.1097/00006123-199707000-00029

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 40.

  JainSIversonLM. Glasgow Coma Scale. In: BabakAet al. StatPearls. Treasure Island, FL: StatPearls Publishing Copyright, ©, 2020. StatPearls Publishing LLC. (2020).

  • Google Scholar

• 41.

  ZhaoBCaoYTanXZhaoYWuJZhongMet al. Complications and outcomes after early surgical treatment for poor-grade ruptured intracranial aneurysms: A multicenter retrospective cohort. Int J Surg. (2015) 23:57–61. 10.1016/j.ijsu.2015.09.008

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 42.

  ShenJHuangKShenJZhuYJiangHPanJet al. Clinical efficacy between microsurgical clipping and endovascular coiling in the treatment of ruptured poor-grade anterior circulation aneurysms. World Neurosurg. (2019) 127:e321–9. 10.1016/j.wneu.2019.02.248

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 43.

  OlsenMHOrreMLeisnerACWRasmussenRBacheSWellingKLet al. Delayed cerebral ischaemia in patients with aneurysmal subarachnoid haemorrhage: functional outcome and long-term mortality. Acta Anaesthesiol Scand. (2019) 63:1191–9. 10.1111/aas.13412

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 44.

  BudohoskiKPGuilfoyleMHelmyAHuuskonenTCzosnykaMKirollosRet al. The pathophysiology and treatment of delayed cerebral ischaemia following subarachnoid haemorrhage. J Neurol Neurosurg Psychiatry. (2014) 85:1343–53. 10.1136/jnnp-2014-307711

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 45.

  EngelhardtBVajkoczyPWellerRO. The movers and shapers in immune privilege of the CNS. Nat Immunol. (2017) 18:123–31. 10.1038/ni.3666

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 46.

  MeyerCMartin-BlondelGLiblauRS. Endothelial cells and lymphatics at the interface between the immune and central nervous systems: implications for multiple sclerosis. Curr Opin Neurol. (2017) 30:222–30. 10.1097/WCO.0000000000000454

  • Pubmed Abstract
  • CrossRef
  • Google Scholar