Unveiling the future of COVID-19 patient care: groundbreaking prediction models for severe outcomes or mortality in hospitalized cases

Background Previous studies have identified COVID-19 risk factors, such as age and chronic health conditions, linked to severe outcomes and mortality. However, accurately predicting severe illness in COVID-19 patients remains challenging, lacking precise methods. Objective This study aimed to leverage clinical real-world data and multiple machine-learning algorithms to formulate innovative predictive models for assessing the risk of severe outcomes or mortality in hospitalized patients with COVID-19. Methods Data were obtained from the Taipei Medical University Clinical Research Database (TMUCRD) including electronic health records from three Taiwanese hospitals in Taiwan. This study included patients admitted to the hospitals who received an initial diagnosis of COVID-19 between January 1, 2021, and May 31, 2022. The primary outcome was defined as the composite of severe infection, including ventilator use, intubation, ICU admission, and mortality. Secondary outcomes consisted of individual indicators. The dataset encompassed demographic data, health status, COVID-19 specifics, comorbidities, medications, and laboratory results. Two modes (full mode and simplified mode) are used; the former includes all features, and the latter only includes the 30 most important features selected based on the algorithm used by the best model in full mode. Seven machine learning was employed algorithms the performance of the models was evaluated using metrics such as the area under the receiver operating characteristic curve (AUROC), accuracy, sensitivity, and specificity. Results The study encompassed 22,192 eligible in-patients diagnosed with COVID-19. In the full mode, the model using the light gradient boosting machine algorithm achieved the highest AUROC value (0.939), with an accuracy of 85.5%, a sensitivity of 0.897, and a specificity of 0.853. Age, vaccination status, neutrophil count, sodium levels, and platelet count were significant features. In the simplified mode, the extreme gradient boosting algorithm yielded an AUROC of 0.935, an accuracy of 89.9%, a sensitivity of 0.843, and a specificity of 0.902. Conclusion This study illustrates the feasibility of constructing precise predictive models for severe outcomes or mortality in COVID-19 patients by leveraging significant predictors and advanced machine learning. These findings can aid healthcare practitioners in proactively predicting and monitoring severe outcomes or mortality among hospitalized COVID-19 patients, improving treatment and resource allocation.


Introduction
The emergence of the coronavirus disease 2019 (COVID- 19) outbreak in China during late 2019 has escalated into a worldwide health apprehension, primarily due to its rapid transmission and deleterious health implications (1).Its prevalent symptoms encompass fever, dry cough, and dyspnea (2).According to prior investigations, a distinct subset of afflicted individuals faces a heightened susceptibility to severe infection, with respiratory impairments such as dyspnea, elevated respiratory rate, and diminished oxygen saturation dominating the symptomatology.Individuals with advanced disease may also manifest respiratory failure, septic shock, or multi-organ dysfunction (3).
The swift propagation and extensive ramifications of this worldwide pandemic have imposed a significant strain on healthcare systems across diverse nations.This strain is particularly evident in the realms of clinical resource allocation and decision-making protocols.Numerous medical institutions have encountered unparalleled scarcities of essential supplies, among them mechanical ventilators, primarily stemming from the rapid surge in critically ill COVID-19 patients necessitating both airway assistance and mechanical ventilatory support.This predicament, confronting healthcare delivery systems, underscores the urgency of employing innovative and pioneering technologies to navigate acute and systemic challenges in healthcare provisioning.With the overarching aims of mitigating mortality and sustaining healthcare infrastructure, the primary objective entails averting severe outcomes and fatalities among patients.
The incorporation of artificial intelligence (AI) and machine learning (ML) within the healthcare domain, spanning tasks such as image analysis, clinical decision-making, and prognosis prediction, constitutes a burgeoning discipline with broad applications across diverse maladies (4).Within the context of COVID-19, artificial intelligence has demonstrated its pivotal role in both diagnostic and prognostic domains, encompassing prediction, detection, classification, screening, and diagnosis of COVID-19 infections (5,6).Scoping reviews have underscored the potential of artificial intelligence as a weapon in the fight against COVID-19; nonetheless, many proposed methodologies are yet to secure clinical acceptance (7).Predictive models stand as extensively investigated tools within biotechnology, enriching clinical comprehension of the diagnostic and prognostic dimensions of various illnesses.
According to the Taiwan Centers for Disease Control, during the initial phase of the COVID-19 outbreak, a substantial proportion (42%) of the cases were primarily located in the northern region of Taiwan, probably due to the presence of the International airports in that area and May 2022 marked the onset of the first wave of the pandemic (8).The Taipei Medical University Clinical Research Database (TMUCRD) gathers data from multiple centers and sources of various data types.It systematically collects both structured and unstructured data from three affiliated hospitals: Taipei Medical University Hospital, Wanfang Hospital, and Shuangho Hospital (9)(10)(11).The National Health Insurance database in Taiwan has a gap of 2 years in the dissemination of data for research purposes.Therefore, in terms of finding recent breakthroughs in the field of COVID-19, TMUCRD could help enhance the understanding of factors influencing COVID-19 outcomes.
Based on the most accurate information available, no prediction model study of COVID-19 severe symptoms in Taiwan.This study aimed to predict severe outcomes, including the use of ventilators, intubation, admission to the intensive care unit (ICU), and mortality, among COVID-19 patients hospitalized in Taiwan.The primary objective of this study is to develop predictive models that can assist clinicians in identifying individuals who are most vulnerable to severe outcomes, including mortality.This focused identification provides healthcare practitioners with the tools to carry out prompt interventions.

Study design and data source
To create the dataset, this study utilized clinical data obtained from the Taipei Medical University Clinical Research Database (TMUCRD).TMUCRD consolidates extensive clinical data derived from three associated hospitals: Taipei Medical University Hospital, Wanfang Hospital, and Shuang-Ho Hospital.The database comprises structured and unstructured information.This study obtained approval from the Taipei Medical University Joint Institutional Review Board (TMU-JIRB) with grant number N202302020.

Population selection
This study included patients who were hospitalized and confirmed to have contracted COVID-19 within the period spanning from January 1, 2021, to May 31, 2022.The diagnosis of COVID-19 was established either through a positive outcome from a real-time reverse transcription polymerase chain reaction (RT-PCR) test or a positive outcome from a rapid antigen test.
The exclusion criteria encompassed newly registered patients who had not previously sought medical care at the three hospitals due to the lack of complete medical background information records, individuals under the age of 20, and patients with undisclosed gender information.As a result, a total of 22,192 patients were retained for inclusion in this study.The selection process for the study population is visually depicted in Figure 1.

Outcome measurement
The index date is defined as the date of the first COVID diagnosis.The primary outcome was defined as a serious event, encompassing occurrences such as ventilator use, intubation, intensive care unit (ICU) admission, and mortality within 3 months of confirmed COVID-19 infection.Additionally, each of the aforementioned specific indicators was considered as a secondary outcome in this study.Data censoring occurred either at the date of death, loss to follow-up, or at the end of the study (May 31, 2022).

Features
Based on a literature review and consultation with clinicians, this study identified features associated with the above outcomes based on demographic information, health status, COVID-19related details, comorbidities, long-term medication records, and  The Charlson Comorbidity Index (CCI) score was computed, and comorbidity was determined using disease codes sourced from the ICD-9 or ICD-10 classification systems found in the medical records.Among the cohort members, individuals were categorized as having comorbidities if they had undergone a minimum of two outpatient visits or one hospitalization related to the specific disease before the index date.Evaluation of the COVID-19 vaccine status is based on the vaccination records within the year preceding the index date.Assessment of COVID-19 medications is grounded in the medication status during the 3 months following the index date.Long-term medication users in the cohort were characterized as patients who had received a prescription for one or more of the aforementioned drugs for a period of 28 days or longer in the year (365 days) prior to the index date.In cases where multiple test results were obtainable, priority was given to the latest laboratory test value within a one-year period before the index date.The technique of Multiple Imputation by Chained Equations (MICE) was employed to address the presence of missing continuous features (12).

Statistical analysis
In the realm of descriptive statistics, continuous data are elucidated through the utilization of the mean (standard deviation, S.D.) and median (minimum and maximum values).Conversely, categorical data are expounded upon by presenting the count of cases along with their corresponding percentages.Additionally, the count and proportion of missing values were computed.Statistical analyses were conducted employing R version 4.1.3(R Project for Statistical Computing).

Algorithms used in this study
Seven machine learning algorithms were utilized to formulate personalized prediction models.The machine learning algorithms encompass Linear Discriminant Analysis (LDA), Logistic Regression (LR), Support Vector Machine (SVM), Random Forest (RF), Gradient Boosting Machine (GBM), Light GBM, and Extreme Gradient Boosting (XGBoost) (refer to Supplementary Appendix 1).Prediction models were developed in this study based on two modes and employing diverse algorithms: (1) Full mode: encompassing all selected features' data; (2) Simplified mode: incorporating 30 crucial features chosen based on the algorithm used by the best model in full mode.

Model training and testing
The participant cohort was divided into training and testing datasets, with 80% of participants assigned to the training subset, and the remaining portion constituting the testing dataset.The crossvalidation technique was also performed to access the over-fitting (13,14).

Evaluation of model performance and interpretation
Performance assessment and comparison of all prediction models involved the calculation of metrics including the area under the receiver operating characteristic curve (AUROC), accuracy, sensitivity (recall), specificity, positive predictive value (PPV or precision), negative predictive value (NPV), and F1-score.The optimal model was determined by identifying the one with the highest AUROC through a comparative analysis of various models using testing results.Data processing was executed using MSSQL Server 2017, while model training and testing were carried out utilizing the Python programming language version 3.9 (15).The SHapley Additive exPlanations (SHAP) values were used to assess feature's contribution (also known as its importance) to the most optimal model when interpreting the models (16).

Full mode
Table 2 presents the performance evaluation of prediction models for overall severe outcome prediction, encompassing mortality, in the full mode.Upon analyzing the test outcomes, the Light GBM model exhibited the highest AUROC (0.939), surpassing other models including XGBoost (AUROC = 0.938), GBM (AUROC = 0.937), RF (AUROC = 0.936), LR (AUROC = 0.869), SVM (AUROC = 0.852), and LDA (AUROC = 0.852).The best-performing model (Light GBM) demonstrated accuracy, sensitivity, and specificity of 85.5%, 0.897, and 0.853, respectively.The cross-validation performance is provided in the Supplementary Appendices 6 and 8.In the cross-validation performance, the Light GBM had the consistent result with the external AUC at 0.924.Figure 2 illustrates the AUROC values of different models in the context of the full mode.The ROC curve delineating the performance of the prediction models for each specific outcome is provided in Supplementary Appendix 3(A). Figure 3 presents the feature importance for predicting severe outcomes or mortality using the optimal model within the full mode.The most significant features were age, vaccination before having PCR test, neutrophil count result, levels of sodium test and platelet count result.

Simplified mode
The LGBM algorithm selected the 30 most crucial features from the entire set, which encompassed: sex type, age, BMI, CCI score, vaccination before having PCR test, COVID-19 medications, comorbidities including cardiovascular disease, COPD, renal disease, depression or anxiety, long-term medication such as NSAID, drugs for hypertension, drugs for GORD, aspirin, statin, antihyperuricemic, laboratory test results contain AST (GOT), ALT (GPT), creatinine, RBC, hemoglobin, MCH.MCHC, WBC, Neutrophil, PLT, HCT, NA and K. Table 3 displays the performance evaluation of prediction models for overall severe outcome prediction, inclusive of mortality, in the simplified mode.Based on the results of the tests, the XGBoost model achieved the highest AUROC (0.935) among the other models, namely RF (AUROC = 0.934), Light GBM (AUROC = 0.934), GBM (AUROC = 0.933), LR (AUROC = 0.863), SVM (AUROC = 0.846), and LDA (AUROC = 0.841).The optimal model (XGBoost) achieved accuracy, sensitivity, and specificity of 89.9%, 0.843, and 0.902, respectively.The XGBoost model demonstrates consistent performance when using the cross-validation strategy, with an external AUC of 0.934 The cross-validation performance of the prediction of individual indicators in the simple mode is shown in Supplementary Appendices 7 and 8. Figure 4 illustrates the AUROC values of different models within the context of the simplified mode.The ROC curve delineating the performance of the prediction models for each specific outcome is provided in Supplementary Appendix 3(B).
The calibration plot showcasing the performance of prediction models for severe outcomes or mortality can be found in Supplementary Appendix 4. Additionally, the calibration plots illustrating the performance of prediction models for specific outcomes are furnished in Supplementary Appendix 5.

Discussion
Precise and personalized assessment of individuals at risk of developing severe COVID-19 outcomes holds the potential to enhance both the efficacy of clinical interventions and the judicious utilization of medical resources (17,18).Several pivotal factors contribute to the heightened predictive capacity of machine learning (ML) models compared to conventional techniques.The considerable advantage of ML models lies in their capacity to generate predictions from vastly expanded datasets, a facet not to be understated.Moreover, ML models remain impervious to human emotions and subjective perspectives, thereby ensuring the objectivity and impartiality of the predictive process.Simultaneously, the innate adaptability inherent to ML models empowers them to swiftly acclimate and assimilate alterations, thereby amplifying their responsiveness to dynamic environments.Ultimately, ML models exhibit an aptitude for discerning intricate patterns of great complexity, often surpassing the capabilities of conventional methodologies.The choice of seven unique machine learning algorithms in this study is based on a comprehensive approach to developing personalized prediction models (19).The algorithms were chosen based on careful evaluation of their attributes and capabilities, ensuring they were in line with the project's goals and the specific peculiarities of the dataset.The prediction models were developed by employing a range of algorithms, including traditional ones like LDA and LR, as well as basic methods like SVM.Additionally, this study utilize ensemble techniques that involve tree-based algorithms such as RF, GBM, Light GBM, and XGBoost (20,21).
While prior investigations have constructed and validated predictive models with the goal of forecasting COVID-19 outcomes  algorithms, this study also employed advanced algorithms, a measure that facilitated the attainment of heightened precision in predictive models.Lastly, through a meticulous analysis of feature significance, this study procured a collection of the most pivotal predictors profoundly impacting model performance (6,26,27).The meticulous and personalized appraisal of patients susceptible to severe COVID-19 would undoubtedly amplify the efficacy of clinical interventions and streamline the judicious allocation of medical resources.This study elucidates that the age of COVID-19 patients stands as the foremost predictor of severe outcome risk, aligning harmoniously with the conclusions drawn from diverse antecedent observational studies, which consistently affirm that elderly COVID-19 patients exhibit a heightened vulnerability to severe outcomes (27)(28)(29).Furthermore, this study's findings expound upon the notion that pre-infection vaccination of COVID-19 patients equally serves as a pivotal predictor of serious events' risk (including ventilator utilization, intubation, and mortality), as its primary function lies in averting the manifestation of numerous severe outcome risks.This alignment with prior research findings attests to the study's robustness (30)(31)(32).
Presently, numerous national health authorities have issued declarations stipulating the utilization of antiviral agents against COVID-19, notably paxlovid (for individuals aged ≥12 years and weighing ≥40 kg) and molnupiravir (for individuals aged ≥18 years), as a crucial treatment avenue for at-risk patients (33) The findings further highlight the significance of prolonged utilization of specific medications (such as benzodiazepines) as a salient affirmative predictor of severe outcome risk, a trend congruent with precedent observational investigations.This discovery bears noteworthy implications within clinical contexts (29).Benzodiazepines, encompassing medications frequently employed to address insomnia, anxiety, seizures, and alcohol withdrawal syndromes, interface with gamma-aminobutyric acid (GABA) receptors within the central nervous system, engendering a tranquilizing and pacifying impact upon the physiological framework.Notably, alongside the potential for immunosuppressive reactions entailing benzodiazepine administration, protracted usage might entail diminished respiratory function, exacerbating complexities among COVID-19 patients (37,38).
Moreover, study's investigation unveiled the substantial predictive potency of laboratory test outcomes, encompassing neutrophil count, white blood cell count, platelet count, MCH, and GOT, GPT, NA, and K levels.These variables assumed pivotal roles Nonetheless, this study does encompass certain limitations.Primarily, it hinges upon electronic health records culled from diverse hospitals, constituting the primary wellspring of data.While these records amass a wealth of clinical intricacies, such as demographic particulars, disease management particulars, comprehensive medical histories incorporating comorbidities, prolonged medication use, and pivotal diagnostic outcomes, they regrettably omit several other data categories of import.Absent from this compilation are diverse facets of an individual's lifestyle, spanning dietary habits, physical activity, tobacco and alcohol consumption, as well as socioeconomic indicators.In prospective endeavors, incorporation of this omitted information might yield alternative predictive models.In clinical practices, hospitals can adopt similar models to assist physicians in the prognostic process.However, a major obstacle is the limited availability and quality of data.The selection of these features was meticulously made, taking into account the available literature.While multiple features were employed in the study, the ones the study utilized are highly accessible and easily obtainable in the electronic health record (EHR) system.Therefore, our findings can be readily applied in future research.The issue of model interpretability is of utmost importance, as healthcare practitioners may struggle to comprehend complex machine learning algorithms.To improve the model's interpretability, SHAP value ranking was additionally conducted in the findings.Secondarily, it merits mention that the hospital-held electronic health records solely chronicle the specifics of a patient's clinical visits, bypassing documentation of medical procedures and interventions executed within other healthcare institutions.Consequently, the clinical insights accessible for each patient might not have attained a truly all-encompassing status, potentially culminating in inaccuracies within the predictions of the predictive model.
Finally, a veritable acknowledgement is that the data origination in this study emanates solely from clinical archives of three hospitals within a singular Taiwanese system.While these hospitals have the largest number of COVID-19 patients in Taiwan, the study may not fully represent the entire population of Taiwan.Therefore, these models, which rely exclusively on hospital cases specific to Northern Taiwan, may have limitations in terms of the generalizability of their findings.Hence, for forthcoming research, it is prudent to foster inter-hospital collaboration and international partnership.Standardized case selection, research blueprinting, data structuring, processing methodologies, and analytical toolswhen conjoined with predictive models engendered through multi-center federated learning-will furnish the substratum for the impending research trajectory.

Conclusion
This study has successfully developed an innovative and precise computer-aided risk prediction model designed to anticipate severe outcomes (including ventilator use, intubation, and intensive care unit admission) or mortality among COVID-19 patients.The outcomes of this research reveal that both the comprehensive and simplified models achieved an area under the curve (AUC) exceeding 0.9, accompanied by an accuracy rate surpassing 85%.The potential to apply timely medical interventions tailored to high-risk patients holds promise for preventing adverse outcomes and thereby ameliorating the disease's impact on a substantial patient cohort.Although prediction model in this study performed well in the test set, one limitation of this study is the need to take into account the dataset's representation.The future focus will be on externally validating the model.Collaboration with both domestic hospitals in Taiwan and hospitals in other countries, along with the utilization of the international database, is imperative.There is an expectation that further hospitals in southern Taiwan will be used to validate and enhance this model.

FIGURE 1 Flowchart
FIGURE 1Flowchart of cohort selection.
Zheng et al. conducted a meta-analysis, revealing Paxlovid's efficacy and safety in managing high-risk COVID-19 patients (34).Debbiny et al. 's outcomes further underscored Paxlovid's heightened efficacy within vulnerable demographics, encompassing elderly patients, those under immunosuppression, and individuals contending with underlying neurological or cardiovascular conditions (35).Concurrently, Benaicha et al. 's meta-analysis showcased the substantial reduction in all-cause mortality and hospitalization risk attributed to molnupiravir (36).Remarkably, this study's findings reinforce the pivotal role of COVID-19 antiviral agents in predicting severe outcome risks.Post-COVID-19 infection, individuals incorporating COVID-19 antiviral medications within their treatment regimens evinced a substantial decline in the necessity for ventilator assistance within a three-month timeframe, vis-à-vis counterparts devoid of such treatment.The alignment of the predictive model with antecedent research outcomes underscores its congruence with established clinical practice and the prudent integration of prior findings.

FIGURE 2 ROC
FIGURE 2ROC curve of performance of prediction models of severe outcomes or mortality under the full mode.

FIGURE 3
FIGURE 3Shapley additive explanations chart of the feature importance for predicting severe outcomes or mortality by the best model under the full mode.

TABLE 1
Baseline of patient characteristics.

TABLE 2
Performance of prediction models under full mode.

TABLE 3
Performance of prediction models under simplified mode.