An Externally Validated Dynamic Nomogram for Predicting Unfavorable Prognosis in Patients With Aneurysmal Subarachnoid Hemorrhage

Background: Aneurysmal subarachnoid hemorrhage (aSAH) leads to severe disability and functional dependence. However, no reliable method exists to predict the clinical prognosis after aSAH. Thus, this study aimed to develop a web-based dynamic nomogram to precisely evaluate the risk of poor outcomes in patients with aSAH. Methods: Clinical patient data were retrospectively analyzed at two medical centers. One center with 126 patients was used to develop the model. Least absolute shrinkage and selection operator (LASSO) analysis was used to select the optimal variables. Multivariable logistic regression was applied to identify independent prognostic factors and construct a nomogram based on the selected variables. The C-index and Hosmer–Lemeshow p-value and Brier score was used to reflect the discrimination and calibration capacities of the model. Receiver operating characteristic curve and calibration curve (1,000 bootstrap resamples) were generated for internal validation, while another center with 84 patients was used to validate the model externally. Decision curve analysis (DCA) and clinical impact curves (CICs) were used to evaluate the clinical usefulness of the nomogram. Results: Unfavorable prognosis was observed in 46 (37%) patients in the training cohort and 24 (29%) patients in the external validation cohort. The independent prognostic factors of the nomogram, including neutrophil-to-lymphocyte ratio (NLR) (p = 0.005), World Federation of Neurosurgical Societies (WFNS) grade (p = 0.002), and delayed cerebral ischemia (DCI) (p = 0.0003), were identified using LASSO and multivariable logistic regression. A dynamic nomogram (https://hu-ping.shinyapps.io/DynNomapp/) was developed. The nomogram model demonstrated excellent discrimination, with a bias-corrected C-index of 0.85, and calibration capacities (Hosmer–Lemeshow p-value, 0.412; Brier score, 0.12) in the training cohort. Application of the model to the external validation cohort yielded a C-index of 0.84 and a Brier score of 0.13. Both DCA and CIC showed a superior overall net benefit over the entire range of threshold probabilities. Conclusion: This study identified that NLR on admission, WFNS grade, and DCI independently predicted unfavorable prognosis in patients with aSAH. These factors were used to develop a web-based dynamic nomogram application to calculate the precise probability of a poor patient outcome. This tool will benefit personalized treatment and patient management and help neurosurgeons make better clinical decisions.

Methods: Clinical patient data were retrospectively analyzed at two medical centers. One center with 126 patients was used to develop the model. Least absolute shrinkage and selection operator (LASSO) analysis was used to select the optimal variables. Multivariable logistic regression was applied to identify independent prognostic factors and construct a nomogram based on the selected variables. The C-index and Hosmer-Lemeshow p-value and Brier score was used to reflect the discrimination and calibration capacities of the model. Receiver operating characteristic curve and calibration curve (1,000 bootstrap resamples) were generated for internal validation, while another center with 84 patients was used to validate the model externally. Decision curve analysis (DCA) and clinical impact curves (CICs) were used to evaluate the clinical usefulness of the nomogram.
Results: Unfavorable prognosis was observed in 46 (37%) patients in the training cohort and 24 (29%) patients in the external validation cohort. The independent prognostic factors of the nomogram, including neutrophil-to-lymphocyte ratio (NLR) (p = 0.005), World Federation of Neurosurgical Societies (WFNS) grade (p = 0.002), and delayed cerebral ischemia (DCI) (p = 0.0003), were identified using LASSO and multivariable logistic regression. A dynamic nomogram (https://hu-ping.shinyapps.io/DynNomapp/) was developed. The nomogram model demonstrated excellent discrimination, with a bias-corrected C-index of 0.85, and calibration capacities (Hosmer-Lemeshow p-value, 0.412; Brier score, 0.12) in the training cohort. Application of the model to the external validation cohort yielded a C-index of 0.84 and a Brier score of 0.13. Both DCA and CIC showed a superior overall net benefit over the entire range of threshold probabilities.

INTRODUCTION
Aneurysmal subarachnoid hemorrhage (aSAH) is an acute cerebrovascular injury with high morbidity and mortality worldwide (1). Roughly one-third of survivors have severe disability and functional dependence, some of which may be predicted and treated early (2,3). Therefore, a practical prediction model for treating and evaluating these patients with aSAH is urgently needed.
Several studies have shown that a series of serum biochemical and immunological indices could predict the prognosis of patients with aSAH after surgery, including white blood cell (WBC) count, neutrophil-to-lymphocyte ratio (NLR), C-reactive protein level, and blood glucose level on admission (4)(5)(6). Meanwhile, predictive models combining multiple independent predictors have been used to predict prognosis. However, these predictive models lack calibration tests, model external validation, or evaluation of their clinical usefulness; therefore, the accuracy and practicability of the models are difficult to assess (7,8). High-quality predictive models are necessary to guide clinical decision-making, personal care, and patient management (9). Therefore, a more practical and accurate predictive model is urgently needed in this field. A web-based dynamic nomogram that precisely calculates the probability of the disease is a more precise and practical tool than standard nomograms and some predictive models (10,11). However, no study has reported a dynamic nomogram to predict the clinical prognosis after aSAH.
Thus, this study aimed to develop a dynamic web-based nomogram that incorporated blood laboratory tests, clinical state at admission, and postoperative complications for the prognosis of patients with aSAH. The predictive model was validated using internal and independent external validation cohorts.

Study Design and Patient Enrollment
This retrospective observational cohort study was conducted in two centers. A training cohort of 126 patients with aSAH treated in the Department of Neurosurgery, Renmin Hospital of Wuhan University, from September 1, 2019 to September 1, 2020, was used to develop the model. A total of 84 patients admitted to the Department of Neurosurgery, the Affiliated Hospital of Panzhihua University, from September 1, 2019 to September 1, 2020, were used to validate the model. The diagnosis of aSAH was assessed by head CT, CT angiography, or digital subtraction angiography according to the guidelines (12,13).

Data Collection
Clinical information including demographic data (admission number, name, sex, age, hypertension, smoking, and alcohol consumption), clinical state, radiological findings, serum laboratory tests (glucose, D-dimer, WBCs, neutrophils, lymphocytes, and monocytes), and CT Hounsfield unit (HU) values on admission were collected using an electronic medical record system. The number, location length, and neck size of the aneurysms were also recorded. Hunt and Hess classification and the World Federation of Neurosurgical Societies (WFNS) were used to measure the clinical characteristics and neurological status at admission. The radiological features at admission were classified according to the modified Fisher grade of the CT scan (14)(15)(16).
All patients were treated with either a surgical clipping or coil embolization as soon as possible. The postoperative routine therapies included hemostasis, analgesics, antiinflammatory drugs, and nimodipine for anti-vasospasm. An immediate postoperative head CT scan was performed to identify intracranial rebleeding or cerebral infarction after the operation.

Admission CT Values
The mean CT values of the blood clots in the subarachnoid space were measured. The regions of interest were manually drawn on representative slices by two neurosurgeons who were blinded to the patients' clinical characteristics. The density of blood clotting in the subarachnoid space on CT slices was assessed using the mean HU value. The following subarachnoid cisterns/fissures were used to measure the mean HU: lateral Sylvian fissures, anterior interhemispheric fissures, medial Sylvian fissures, suprasellar cistern, ambient cistern, and quadrigeminal cistern, as previously described (17,18).

The Definition of Delayed Cerebral Ischemia
Delayed cerebral ischemia (DCI) was identified as follows: (1) CT scan omitting other causes of focal neurological decline such as aphasia, apraxia, hemianopia, or neglect, either permanent or temporary, within 4 to 14 days after SAH; (2) a Glasgow Coma Scale decrease of at least 2 points lasting for at least 1 h and not immediately evident after surgery; (3) head CT scan revealing a new cerebral infarction within 4-30 days after aSAH, which was not noticeable on admission or immediately after the operation, and no other explanation except for vasospasm (17,19).

Outcome Assessment
Three months after the initial bleeding, the functional outcome was dichotomized as excellent or poor according to the modified Rankin Scale (mRS) (20)(21)(22). A favorable outcome was defined as a score of 0-2 (perform activities without assistance) on the mRS, while a score of 3-6 (moderate to severe disability or death) was defined as a poor outcome. The results were evaluated via telephone by a neurosurgeon who was blinded to the clinical and imaging data.

Sample Size
In the current literature, the events per variable (EPV) criterion, notably an EPV of 10, is widely applied as the lower limit for developing logistic regression models that predict a binary outcome (23,24). Thus, an EPV of 10 was used to estimate the effective sample size in this study. A total of three variables, which were incorporated into the multivariate logistic analysis, were selected by least absolute shrinkage and selection operator (LASSO) regression analysis. Thus, the effective sample size was at least 30 in the training cohort.

Statistical Analysis
Statistical analysis was performed using IBM SPSS Statistics for Windows, version 26.0 (IBM Corp., Armonk, NY, USA) and R software (https://www.r-project.org/). Continuous variables were analyzed using independent t-test or Mann-Whitney U test and are presented as means ± SD or medians with interquartile range. Categorical variables are expressed as numbers (percentages) and were analyzed using χ 2 or Fisher exact tests. Based on the limited sample size in this study, LASSO regression, which is suitable for analyzing high-dimensional data, was used to select the most informative prognostic variables from the data cohort (25)(26)(27). The variables were then entered into a multivariable logistic regression model; a nomogram model was then constructed by integrating independent factors.
The discrimination and calibration capacities evaluated the performance of the nomogram model. Receiver operating characteristic (ROC) curve and area under the curve (AUC) analyses were performed to assess the discriminatory capabilities, while the bias-corrected C-index was calculated using 1,000 bootstraps. The Hosmer-Lemeshow test, Brier score, and calibration curve were used to assess the calibration ability (28). In addition to internal validation, the model performance was also assessed using an external validation cohort. The clinical effectiveness of the nomogram was evaluated by decision curve analysis (DCA) and clinical impact curve (CIC) (29,30).
We used the "glmnet, corrplot, caret package" in R software to generate the LASSO regression results and the "rms package" to establish the nomogram. The "pROC, " "rmda, " "MASS, " "survival, " "ggplot2, " "ggridges, " "plotROC, " and "riskRegression" packages were applied to generate the C-index, ROC, DCA, CIC, and calibration curve. The "shinyapps.io" and "DynNom packages" were used to develop a web-based dynamic nomogram application to predict unfavorable prognosis, which could precisely calculate the risk probability for unfavorable prognosis at 3 months after aSAH. All tests were two-tailed, and p-values < 0.05 were considered statistically significant.

Baseline Cohort Characteristics
This study included a total of 210 patients with aSAH, with 126 and 84 patients in the training and external validation cohorts, respectively. The baseline characteristics of the two cohorts were shown in Table 1. Women comprised 137 (65%) patients in the two cohorts, and the mean age was 56.50 years (interquartile range: 53.00, 63.00). We observed no significant differences between the training and external validation cohorts, except for higher admission glucose level, percentages of DCI occurrence and WFNS grade IV-V, and lower aneurysm neck size in the training cohort. The numbers of patients with poor prognoses in the two cohorts, 46 (37%) and 24 patients (29%), respectively, did not differ significantly. Table 2 showed the baseline characteristics of the model development cohort.

Variable Selection
LASSO regression was used to select the most useful variables. As shown in Figure 1A, when using the minimum error criterion, 21 variables were decreased to 11 variables and further reduced to three variables after applying the one standard error (1-SE) criterion. Thus, an optimal λ of 0.1356, with log(λ) = −1.997, was adopted. Three factors with non-zero coefficients were finally selected by 5-fold cross-validation to prevent overfitting ( Figure 1B)

Prognostic Model Development
All independent factors were used to construct a nomogram to predict unfavorable prognosis 3 months after aSAH (Figure 2). According to the patient's information on admission, each of the three prognostic factors in the nomogram was projected upward to a point. NLR had continuous values ranging from 0 to 100. The WFNS grade was divided into two levels (I-III and IV-V), while DCI was divided into non-DCI and DCI, each of which were assigned points. The total sum of the points from the three variables was converted into an individual poor prognosis risk, in which the higher was the total score, the higher was the risk of an unfavorable prognosis. Based on the ordinary nomogram, we developed a dynamic nomogram web-based application (https://hu-ping.shinyapps.io/ DynNomapp/) to precisely calculate the risk probability for unfavorable prognosis at 3 months after aSAH.

Nomogram Validation
The original AUC was 0.88 (95% CI 0.82-0.95), while the biascorrected C-index with 1,000 bootstraps was 0.82, suggesting that the model had excellent discrimination. Compared to a single independent prognostic factor, the nomogram model better predicted the occurrence of DCI (Figure 3A). The calibration capacity was also internally validated; the Hosmer-Lemeshow p-value of 0.41 in the training cohort suggested an excellent fitting of the nomogram. By contrast, a Brier score of 0.12 and a calibration curve with 1,000 bootstrap resamples showed that the model has a good calibration ability (Figure 4A), demonstrating no significant deviation between the actual and predicted probabilities. A total of 84 patients retrospectively collected from September 2019 to September 2020 at another center were used for external validation of the model. The validation AUC value of 0.84 (95% CI 0.74-0.94) was consistent with the original AUC value (0.88) in the training cohort ( Figure 3B). A Brier score of 0.13 and the calibration curve plotted for external validation showed good calibration of the model in the external validation cohort ( Figure 4B). Furthermore, DCA and CIC were used to evaluate the clinical usefulness of our prognostic nomogram. DCA showed a superior overall net benefit for a threshold probability of 0-1 ( Figure 5A). CIC also demonstrated good performance over the entire range of threshold probabilities ( Figure 5B).

DISCUSSION
This study constructed and validated a dynamic nomogram model to predict unfavorable prognosis at 3 months after aSAH. The nomogram incorporated the NLR, admission WFNS grade, and DCI. To our knowledge, this study is the first to develop a web-based dynamic nomogram application to calculate the risk of poor prognosis (mRS, 3-6 points). In addition, external validation using data from another center was conducted for the first time. Precise calculation of the probability of the disease makes the dynamic nomogram a more accurate and practical tool compared to ordinary predictive models.
In a previous study, Zhang et al. developed an ordinary nomogram combined with a systemic inflammation response index and other conventional factors to predict the prognostic outcome in patients with aSAH (8). These factors were selected using univariate and multivariate logistic analyses. Lai et al. conducted a predicting model for aSAH prognosis, which included the GCS score, surgical clipping, and NLR (7). These independent prognostic variables were also selected by multivariate logistic regression. Compared with the study by Zhang et al., the model by Lai et al. was only internally validated rather than externally validated. Although a largescale retrospective study has been conducted on poor recovery in patients with aSAH, the study only contained an internal    Contrary to the conventional multivariate logistic method, this study applied the LASSO regression analysis, which performed well in reducing the data dimensionality and decreasing multicollinearity between variables, and it was used to select variables for multivariable logistic regression by minimizing coefficients and reducing variance (32). As a result, an optimal λ of 0.1356, with log(λ) = −1.997, was adapted. Three factors with non-zero coefficients were finally selected by 5fold cross-validation to prevent overfitting. A logistic regression model was then used to establish a nomogram. The original AUC was 0.88, while the bias-corrected C-index with 1,000 bootstraps was 0.82, suggesting that the model had excellent discrimination. Compared with a single independent prognostic factor, the nomogram model better predicted the occurrence of DCI. The calibration capacity was also internally validated; the Hosmer-Lemeshow p-value in the training cohort suggested an excellent fit of the nomogram. By contrast, a Brier score and calibration curve with 1,000 bootstrap resamples showed that the model had a good calibration ability, demonstrating no significant deviation between the actual and predicted probabilities. In addition, DCA showed a superior overall net benefit for a threshold probability of 0-1. The CIC also demonstrated a good performance over the entire range of threshold probabilities.
We observed no significant difference in NLR between the training and external validation cohorts. However, the NLR in patients with an unfavorable prognosis was higher than that in patients with a favorable prognosis in the training cohort (p = 0.005). Our study identified NLR on admission as an independent risk factor for unfavorable prognosis, similar to previous findings (4,7,33,34). Lai et al. reported that a high NLR on admission could increase the risk of poor prognosis 3 months postoperatively. Giede-Jeppe et al. suggested that higher NLR was associated with more significant neurological injury, as reflected by higher subarachnoid and intraventricular blood. Therefore, increased NLR resulted in worse functional outcomes beyond these associations. The following possible mechanisms may explain why high NLR on admission increased the risk of unfavorable prognosis at 3 months after aSAH. First, a high NLR could promote the synthesis and secretion of matrix metalloproteinase, which could degrade components of the extracellular matrix and play an essential role in the blood-brain barrier, resulting in secondary brain injury (35,36). Second, increased NLR at admission may include excessive neuroinflammation caused by elevated neutrophil granulocyte counts and lymphocyte depletion leading to immunodepression. Physiological stress responses lead to an increased postictal release of endogenous catecholamines and cortisol, resulting in secondary brain injury and possibly inflammatory complications (37,38).
The WFNS grade or GCS have gradually become essential validated scales to evaluate clinical characteristics and neurological status on admission. A good grade was defined as I-III on WFNS, while IV-V on the WFNS was considered poor (39,40). In our study, 40 (32%) and 11 (13%) patients in the training and external validation cohorts, respectively, had WFNS grades of IV-V. WFNS grade (p = 0.002) on admission  independently predicted unfavorable prognosis and patients with WFNS grades IV-V had a higher risk of developing a poor prognosis compared with patients with WFNS grades I-III. This finding is similar to those of previous studies showing that WFNS grades IV-V was correlated with a poor outcome and was a good predictor of unfavorable prognosis after aSAH (41,42).
Furthermore, the DCI occurrence in the two cohorts was 35 and 15%, respectively. The onset of aSAH was concentrated during the COVID-19 period, which resulted in patients not receiving timely anti-vasospasm. It may be interpreted that the training cohort had a larger percentage of DCI. DCI was identified as another crucial prognostic factor in this study.
The results of a large-scale retrospective study also suggested that DCI plays an important role in short-term and long-term prognosis (43). Cerebrovascular disorders, microthrombosis, cortical diffuse depolarization, and neuroinflammation all reportedly play potential roles in DCI pathogenesis (17,(44)(45)(46).
Contrary to nomograms or other models calculated as an approximation in previous studies, the web-based dynamic nomogram application developed in the present study (https:// hu-ping.shinyapps.io/DynNomapp/) can be used to calculate an exact value for developing poor outcomes at 3 months after aSAH. This tool will help in the individualized treatment and management of patients, including neuro-intensive care, blood  pressure control, nimodipine anti-vasospasm, anti-inflammatory drugs, and rehabilitation treatment, and help neurosurgeons make better clinical decisions.
However, this study has several limitations. First, this was a retrospective study; therefore, prospective studies are needed to verify our results. Second, owing to the low effective sample size in the training cohort, to reduce the data dimensionality and decrease multicollinearity between variables, we used LASSO regression analysis to select predictors. Third, future studies should collect complete clinical treatment information.

CONCLUSION
This study identified independent predictors for an unfavorable prognosis in patients with aSAH, including NLR on admission, the WFNS grade, and DCI. On this basis, a web-based dynamic nomogram application, after externally validated, was developed to calculate the precise probability of a poor outcome in patients 3 months after aSAH. This will benefit personalized treatment and patient management and help neurosurgeons make better clinical decisions.

DATA AVAILABILITY STATEMENT
The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.

ETHICS STATEMENT
The studies involving human participants were reviewed and approved by the Medical Ethics Committee of Renmin Hospital of Wuhan University. (approval number WDRM2021-K022). Written informed consent from the patients/participants or patients/participants legal guardian/next of kin was not required to participate in this study in accordance with the national legislation and the institutional requirements.

AUTHOR CONTRIBUTIONS
PH, YX, GD, and QC: Study design. PH, SZ, XZ, QS, and YW: Literature search. PH, YLiu, YLi, and LY: Data acquisition, data analysis, and statistical analysis. PH, YX, YQ, HZ, GD, and QC: Article preparation, editing, and review. All authors read and approved the final article.

FUNDING
This work was supported by the National Natural Science Foundation of China (No. 82001311).