Establishment of Routine Clinical Indicators-Based Nomograms for Predicting the Mortality in Patients With COVID-19

This study aimed to establish and validate the nomograms to predict the mortality risk of patients with coronavirus disease 2019 (COVID-19) using routine clinical indicators. This retrospective study included a development cohort enrolled 2,119 hospitalized patients with COVID-19 and a validation cohort included 1,504 patients with COVID-19. The demographics, clinical manifestations, vital signs, and laboratory tests of the patients at admission and outcome of in-hospital death were recorded. The independent factors associated with death were identified by a forward stepwise multivariate logistic regression analysis and used to construct the two prognostic nomograms. The nomogram 1 was a full model to include nine factors identified in the multivariate logistic regression and nomogram 2 was built by selecting four factors from nine to perform as a reduced model. The nomogram 1 and nomogram 2 showed better performance in discrimination and calibration than the Multilobular infiltration, hypo-Lymphocytosis, Bacterial coinfection, Smoking history, hyper-Tension and Age (MuLBSTA) score in training. In validation, nomogram 1 performed better than nomogram 2 for calibration. We recommend the application of nomogram 1 in general hospitals which provide robust prognostic performance though more cumbersome; nomogram 2 in the out-patient, emergency department, and mobile cabin hospitals, which depend on less laboratory examinations to make the assessment more convenient. Both the nomograms can help the clinicians to identify the patients at risk of death with routine clinical indicators at admission, which may reduce the overall mortality of COVID-19.


INTRODUCTION
With the continuing pandemic of severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) infection, there have been more than 200 million patients with coronavirus disease 2019 (COVID- 19) globally and more that 4 million deaths as of August 8, 2021. The clinical manifestations and outcomes of COVID-19 have been delineated in several studies, with 81% of patients presenting with subtle or minor symptoms, and 19% severe or critical cases (1). The mortality of COVID-19 was 11% as first reported (2), dropped to 2.3% (1,023 of 44,672 confirmed cases) on February 11, 2020, in China (3) and was reported to be 2.1% according to the data provided by WHO as of August 8, 2021. The previous studies have reported that the risk factors, such as age, pre-existing comorbidities, and neutrophil-to-lymphocyte ratio (NLR) were associated with higher mortality risk in COVID-19 (1,(4)(5)(6)(7). During the initial outbreak of COVID-19 in Wuhan, the MuLBSTA score (8) was reported to be associated with the outcome of COVID-19 in a few observational studies (9). However, the MuLBSTA score was developed to assess the outcome of influenza A, rhinovirus, and other respiratory virus pneumonia and has not been rigorously tested in predicting the risk of death in patients with COVID-19. A nomogram, widely used output of the prognostic model, generate an individual probability of a clinical event with a visualized interface. There are few studies of nomograms in predicting the death risk of COVID-19 (10)(11)(12)(13). The nomograms developed from those studies provided useful tools for the researchers and clinicians in stratifying patients with COVID-19. However, the size of participants enrolled was limited in all the above studies, and a lack of independent validation was noted in one study (10).
In this study, we aimed to (1) develop a full model (nomogram 1 designated as Nomo1) and a reduced model (nomogram 2 designated as Nomo2) with routine clinical indicators to predict the risk of death using 2,119 cases of confirmed COVID-19; (2) compare the predictive efficacy of Nomo1 and Nomo2 with the MuLBSTA score; (3) assess Nomo1 and Nomo2 in an external validation cohort comprising 1,507 cases.

Study Design and Participants
This retrospective study included a training cohort before being tested in a validation cohort. Within 10 days, from January 22 to February 2, 2020, an emergency hospital with 1,000 beds named Huo-Shen-Shan (HSS) hospital was built in Wuhan (Hubei Province, China) by the Chinese government to admit confirmed patients with COVID-19. In the training cohort, the patients admitted to HSS Hospital from February 4 to March 31, 2020, were retrospectively screened and were followed up to April 15, 2020, when the HSS Hospital closed. A validation cohort included the patients with COVID-19 admitted to Jin Yin-tan Hospital (Hubei Province, China) from January 26 to February 1, 2020 and the patients with COVID-19 admitted to Taikang Tongji Hospital (Hubei Province, China) from February 19 to April 2, 2020. In this way, we almost covered the whole spectrum of time from the COVID-19 outbreak to remission in Wuhan in the validation cohort to ensure the data are representative. The study followed the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) checklist, and was approved by the ethics committee of Xinqiao Hospital (2020-yd073-01), Chongqing, China with written informed consent waived due to the retrospective nature of the study.

Inclusion Criteria
The patients diagnosed according to the WHO interim guidance for COVID-19 were included in the study (14).

Exclusion Criteria
(1) Receiving cardiopulmonary resuscitation on admission and fail to survive; (2) the cases with missing data of some important demographics and clinical indicators; (3) duplications due to readmission or in-hospital transfer between wards.

Procedure and Data Collection
The eligible patients were enrolled and categorized into the two groups according to the outcome of in-hospital death. The enrollment flowchart is shown in Figure 1. The electronic medical records, nursing records, and laboratory tests of included patients were reviewed by a team consisting of experienced clinicians and statisticians. The dates of admission, discharge, and death were recorded and cross-checked. We collected data on age, sex, pre-existing comorbidities (hypertension, diabetes, cardiovascular diseases, chronic lung diseases, and liver diseases), symptoms from onset to hospital admission (fever, cough, sputum, dyspnea, chest tightness, hemoptysis, fatigue, nausea, abdominalgia, diarrhea, and anorexia), the duration time for initial symptoms, vital signs at hospital admission [body temperature, breathing rate, heart rate, blood pressure [the worst of the first 24

Statistical Analysis
The variables with missing data over 20% (IL-6, ERS, PCT, etc.) were not included for further statistical analysis. The remaining items were the routine clinical indexes, and the proportion of observation with missing data was <12%. We employed mean substitution for imputation and completed some of the missing data by follow-up with a phone call. The detailed information about missing data was reported in Supplementary Table 1 in the supplement. The continuous variables were expressed as the median [interquartile range (IQR)] or mean ± SD, and categorical variables were presented as n (%). The differences in the demographical, clinical characteristics, and laboratory values between the survivors and non-survivors were compared using Mann-Whitney Utest, t-test, chi-squared test, or Fisher's exact test as appropriate. The predictors with P-values <0.05 were fed to a forward stepwise multivariate logistic regression model to identify the independent candidate variables associated with COVID-19 fatality. The factors finally included constructing the nomograms were determined by both the statistical significance and clinical values. To assess the discrimination of established nomograms, the area under the receiver operating characteristic curve (AUC) was measured. A calibration curve was generated for the evaluation of calibration and judged with Hosmer-Lemeshow test (15). The clinical usefulness of the established score was evaluated with the decision curve analysis (DCA) by assessing the net benefits at various threshold probabilities (16). In addition, the performance of the MuLBSTA score was compared with the established nomograms using the same methods described above, and the optimal cut-off value for the MuLBSTA score was adjusted according to Youden's index. The statistical analyses were performed using the IBM SPSS 23 statistics software (SPSS Inc., Chicago, IL, USA) and R software (version 3.4, R Foundation, Vienna, Austria. www.R-project.Org). All the P-values were two-sided, and P < 0.05 was considered statistically significant.

Characteristics of the Training and Validation Cohorts
In total, 2,271 COVID-19 cases were screened from the HSS Hospital, among which 2,119 cases were included as a training set according to the inclusion and exclusion criteria. Total 92 COVID-19 cases from Jin Yin-tan hospital and 1,412 cases from Taikang Tongji hospital were included as a validation cohort (Figure 1). The characteristics of the training and validation cohort were listed in the supplement (Supplementary Table 2). In comparing the training and validation cohort, the majorities of symptoms and part of underlying diseases were similar; whereas some of the laboratory tests were different. The in-hospital mortality rate was 3.1% in the training cohort and 2.0% in the validation cohort.

Building Nomogram Prognostic Models
In univariate analysis, the variables between the survivors and non-survivors in the training cohort were compared, and the P-value <0.05 was chosen as the potential factors associated with in-hospital death of COVID-19 ( Table 1). The variables chosen above were put in a forward stepwise multivariate logistic regression analysis to explore the independent risk factors. We identified age, dyspnea, anorexia, NLR, PLT, AST, ALB, and CRP as independent risk factors associated with in-hospital mortality of COVID-19 ( Table 2). We first built up a full model (Nomo1) with all the indicators above. Considering the clinical practicability of applying a model including nine variables is cumbersome, we then constructed a reduced model (Nomo2) regarding the weight value, the previous reports of significance as well as accessibility in a clinic setting. The nomograms can be used to generate an individual probability of in-hospital death, to fulfill our needs of quick stratifying COVID-19 at the risk of death. To use the nomograms, a ruler ranging from 0 to 100 points was scaled on top, with independent prognostic factors array on the relevant axis below. First, age of a subject was converted to a score by drawing a straight line upward to the ruler on the top and get the score related to age, the procedure was carried out for every covariate, and the final risk score was calculated by adding up the score of each item to estimate the probability of in-hospital death referring to the risk axis at the bottom (Figure 2 and Supplementary Figure 1).

The Performance of Nomograms and the Comparison With MuLBSTA Score in the Training Cohort
We first used AUC to compare the discrimination of the established nomograms and MuLBSTA score in predicting the mortality of COVID-19. Both Nomo1 and Nomo2 performed better discrimination than MuLBSTA score (AUC Nomo1 = 0.920,  Table 3). The calibration curves of Nomo1 and Nomo2 showed high consistency between predicted survival probability and actual survival proportion, better than the calibration curve of MuLBSTA score (Figures 3C-E). Furthermore, the generated curve of DCA indicated that employing the Nomo1 and Nomo2 to identify the patients with a high risk of death would be advantageous over the MuLBSTA score (Supplementary Figure 2).

External Validation of the Nomograms
The predictive values of Nomo1 and Nomo2 were further validated in the external datasets. The Nomo1 showed an AUC of 0.92 (95% CI 0.86-0.98) and Nomo2 showed an AUC of 0.89 (95% CI 0.83-0.96) in the validation cohort (Figures 4A,B and Table 3). For calibration, Nomo1 showed a better agreement of observed proportion of death with predicted one than Nomo2 presented. Overall, the DCA also showed that Nomo1 is more beneficial in predicting the mortality of the patients with COVID-19 than Nomo2 in a validation cohort (Figures 4C,D).

DISCUSSION
Coronavirus disease 2019 pandemic has emerged as one of the greatest challenges of our time and has caused more the 4 million deaths up to now. To control the mortality of COVID-19 is of greatest importance to reduce the public crisis and health budget. Considering no effective anti-viral medicine and breakthrough infections after vaccination, how to recognize the patients who are at risk of death became an urgent issue. In consideration of those current predictive models that were built by a relatively  small number of patients, and some indicators included were not routinely detected in the out-patient department, emergency room, or underdeveloped regions or countries, we explored the routine clinical predictive factors and developed nomograms to predict the risk of death for the patients with COVID-19 using a total of 3,623 cases. In this study, we have identified nine factors, including age, dyspnea, anorexia, WBC, NLR, PLT, AST, ALB, and CRP, that were the independent risk factors for COVID-19 fatality. Age is an acknowledged risk factor associated with disease severity and prognosis of COVID-19 (17). The mortality increased sharply to 7.8% in the aged patients over 80, while the overall death rate was estimated to be 0.66% (18). In our study, the median ages in the survivors and non-survivors were 60.0 (49.3∼68.0) and 69.0 (62.0∼78.0). Zhou (20). Another study in New York showed the median age of non-survivors was 68 (60.0-75.0) (21).
Dyspnea can reflect the severity of the disease as it was associated with an increased risk of developing acute respiratory distress syndrome (ARDS) in another study (3). We observed that 18 (25.35%) non-survivors were common type when admitted, who reported symptoms of dyspnea but no respiratory distress, ahead of their subsequent disease progression into severe type. The involvement of dyspnea in the score compensates for underestimating death in patients in the early stage before disease progression. Another symptom, anorexia, is an independent predictor of death in our study. Consistently, gastrointestinal involvement has been observed in the patients with COVID-19 (22,23) and anorexia was associated with intensive care unit (ICU) admission for the patients with COVID-19 (24).
For laboratory parameters, a higher neutrophil count and lymphocytopenia were noted in the non-survivors. Lymphocytopenia is a characteristic of severe patients with COVID-19 since the lymphocyte count in the ICU patients was 0.4 (0.2-0.8) compared with 1.0 (0.7-1.1) in non-ICU patients (25). The lymphocyte count was integrated into a predictive model for COVID-19 fatality (26). The net effect of elevated neutrophils and decreased lymphocytes resulted in raised NLR. The value of NLR ≥ 2.22 had been used to recognize COVID-19, and NLR ≥ 4.06 was an indicator of severe disease (27).
In our study, the PLT count was lower in non-survivors compared with the survivors [190 (87∼265) vs. 227 (184∼280), p < 0.01], which is in line with a previous report (7). However, PLT was reported to be significantly higher in the ICU patients than in the non-ICU patients (25), while the study carried out in Jin Yin-tan Hospital that enrolled 52 critically ill patients showed that the non-survivors had elevated PLT (28). We speculate that the difference between these studies might be influenced by selection bias and the number of patients enrolled.
Regarding the liver function, AST and TB were higher while ALB was lower in the non-survivors compared with the survivors, while ALT was comparable between these two groups. A meta-analysis including 35 studies of 6,686 patients with COVID-19 showed that the pooled prevalence of abnormal liver function was 19%. ALT, AST, and TB predicted severe cases with pooled odds ratios (ORs) of 1.89 (1.30-2.76), 3.08 (2.14-4.42), and 1.39 (0.78-2.47), respectively (3).
Here, based on these predictive factors, the two nomograms to evaluate the mortality risk of COVID-19 were developed, validated, and compared with the MulBSTA score. The MuLBSTA score was previously used to predict the mortality risk of viral pneumonia. The model was established mainly by the patients with influenza pneumonia and other viral pneumonia. Seven parameters, including multilobular infiltrates, lymphocytes, bacterial coinfection, acute smoker, former smoker, hypertension, and age ≥60 years were included. It has been reported that the death with COVID-19 had high MuLBSTA scores (2). In our study, the non-survivors had higher MuLBSTA scores than the survivors [11 (7∼13) vs. 7 (5∼9), P < 0.001]. However, the AUC of the MuLBSTA score was 0.814 [95% CI 0.76-0.868], with a sensitivity of 40.91% (28.79-53.03%). The poor sensitivity of the MuLBSTA score made it unsuitable for predicting the mortality risk of COVID-19. By adjusting the optimal cut-off value from the reported 12-10.5 according to the Youden Index, the sensitivity of the MuLBSTA score increased to 66.67% (95% CI 54.55-77.27%).
The medical nomograms, using biological and clinical variables to determine a probability of a clinical event, are a pictorial representation of a predictive model. In the present study, we constructed two nomograms, Nomo1 and Nomo2, that predicted the mortality of COVID-19 with AUC nomo1 of 0.92 (95% CI 0.86-0.98) and AUC nomo2 of 0.89 (95% CI 0.83-0.96) in validation. Nomo1 consisted of nine indexes, the procedure to calculate the total score by adding the nine scores together is slightly cumbersome and time-consuming. To get a more userfriendly score, we constructed a reduced model Nomo2 regarding the weight value, the previous reports of significance as well as accessibility in a clinic setting. Age, dyspnea, NLR, and CRP were included to construct Nomo2 and showed AUC nomo2 of 0.89 (95% CI 0.83-0.96) in validation. In an out-patient or emergency department, the risk of COVID-19 mortality can be acquired easily using Nomo2 while we recommended the use of Nomo1 in general hospitals.
A number of similar studies before us have constructed nomograms to predict the outcome of COVID-19. Nguyen constructed a nomogram using 279 hospitalized patients with COVID-19 to predict the 14-day probability of an unfavorable outcome defined as the need for artificial ventilation and/or death. Age, male gender, BMI, respiratory rate, body temperature, lymphocyte count, CRP, and TNI were used to construct the predictive nomogram with a C-statistics of 0.75 (10). However, it was a single-center study without external validation of the model, and the definition of endpoint consisted of the patients receiving artificial ventilation. Furthermore, the factors, such as TNI (10,11), D-dimer (12), NT-proBNP (13) included were not routine laboratory tests, and the application of the nomograms was hindered in out-patient department, mobile cabin hospitals, or emergency healthcare centers. Compared with the above nomograms, our models included a large number of patients with COVID-19; all the variables included in the nomogram are routine clinical indexes, making it applicable in most medical institutions worldwide; the two nomograms were built for different purposes, Nomo1 is more robust and Nomo2 is more convenient, a clinician may choose either one appropriate according to the situation.
As with any retrospective study, there are several limitations in our study. First, as a retrospective study, bias is inevitable, and the results should be interpreted carefully as an exploratory study. Second, since the study was carried out in a single city (Wuhan, China), the results are not fully representative. Third, from the beginning of 2021, the new virus strains, such as B.1.617.2 (Delta) variant spread worldwide, showed increased ability to transmit, pathogenicity, and lethality, the predictive potency of the nomogram needs to be verified in countries and areas with new virus strains. However, despite these limitations, we have successfully identified the in-hospital mortality risk factors of COVID-19 and have constructed predictive nomograms to estimate the inhospital mortality risk of COVID-19 based on the routine clinical indicators.

DATA AVAILABILITY STATEMENT
The original contributions presented in the study are included in the article/Supplementary Material, further inquiries can be directed to the corresponding author/s.

ETHICS STATEMENT
The studies involving human participants were reviewed and approved by the Ethics Committee of Xinqiao Hospital (2020-yd073-01). Written informed consent from the participants' legal guardian/next of kin was not required to participate in this study in accordance with the national legislation and the institutional requirements.

AUTHOR CONTRIBUTIONS
SY and YX: had full access to all the data in the study and are responsible for the authenticity, accuracy of the data manipulation, and analysis. SY, MH, and YX: design and organization. JH, XL, YL, XR, CZ, WZ, WD, YT, and PL: data collection, extraction, and cleansing. EL and HL: statistical analysis. JH and YX: drafting of the manuscript.
QL and ZX: technical support. HW and CS: administrative support. All authors contributed to the article and approved the submitted version.