Calibration and validation of the Pneumonia Shock Score in critically ill patients with SARS-CoV-2 infection, a multicenter prospective cohort study
Thomas A. Carmo1,2,3† 
Isabella B. B. Ferreira2,3† 
Rodrigo C. Menezes2,3,4,5† Márcio L. T. Pina3† 
Roberto S. Oliveira3† Gabriel P. Telles3 Antônio F. A. Machado1 Tércio C. Aguiar1 Juliana R. Caldas6 
María B. Arriaga2,4,5‡ 
Kevan M. Akrami2,4,5,7‡ Nivaldo M. Filgueiras Filho1,8‡ 
Bruno B. Andrade1,2,3,5*‡- 1Universidade Salvador (UNIFACS), Salvador, Bahia, Brazil
- 2Multinational Organization Network Sponsoring Translational and Epidemiological Research Initiative, Salvador, Brazil
- 3Escola Bahiana de Medicina e Saúde Pública (EBMSP), Salvador, Bahia, Brazil
- 4Faculdade de Medicina da Bahia, Universidade Federal da Bahia, Salvador, Brazil
- 5Laboratório de Inflamação e Biomarcadores, Instituto Gonçalo Moniz, Fundação Oswaldo Cruz, Salvador, Bahia, Brazil
- 6Hospital São Rafael, Rede Dor, Salvador, Bahia, Brazil
- 7Divisions of Infectious Diseases and Pulmonary, Critical Care and Sleep Medicine, Department of Medicine, University of California, San Diego, La Jolla, CA, United States
- 8Núcleo de Pesquisa, Ensino e Comunicação, Hospital de Cidade, Salvador, Bahia, Brazil
Background: Prognostic tools developed to stratify critically ill patients in Intensive Care Units (ICUs), are critical to predict those with higher risk of mortality in the first hours of admission. This study aims to evaluate the performance of the pShock score in critically ill patients admitted to the ICU with SARS-CoV-2 infection.
Methods: Prospective observational analytical cohort study conducted between January 2020 and March 2021 in four general ICUs in Salvador, Brazil. Descriptive statistics were used to characterize the cohort and a logistic regression, followed by cross-validation, were performed to calibrate the score. A ROC curve analysis was used to assess accuracy of the models analyzed.
Results: Six hundred five adult ICU patients were included in the study. The median age was 63 (IQR: 49–74) years with a mortality rate of 33.2% (201 patients). The calibrated pShock-CoV score performed well in prediction of ICU mortality (AUC of 0.80 [95% Confidence Interval (CI): 0.77–0.83; p-value < 0.0001]).
Conclusions: The pShock-CoV score demonstrated robust discriminatory capacity and may assist in targeting scarce ICU resources during the COVID-19 pandemic to those critically ill patients most likely to benefit.
Introduction
The COVID-19 outbreak created a worldwide emergency in the face of rapid dissemination throughout the world (1). To date, the pandemic has more than 240 million cases worldwide and over 4.9 million deaths spread over 220 countries (2). While most infected individuals develop mild forms of the disease, those who develop life threatening infections requiring intensive care units (ICU) care may succumb to their infection with mortality rates up to 49% (3, 4). Scarcity of healthcare resources has profoundly impacted low-middle-income countries, with significant strain on pre-existing limited ICU capacity (2, 5). In Brazil, significant viral transmissibility, associated with excess mortality rates in the elderly and those with a high burden of disease, rapidly overwhelmed health services in the country (3, 6). Existing prognostic tools to triage resources to those most likely to benefit from critical care, such as SAPS3, SOFA and APACHE IV, lack sufficient accuracy in those hospitalized with COVID-19 (7–9). Despite several novel prognostic models emerged during the pandemic, many have been found to have a high risk of bias, and not sufficient attempt has been made to develop a simple routinely applicable scoring system to early predict higher risk of mortality for patients admitted in ICUs (7). Recently, our group developed and externally validated a prognostic score for mortality risk stratification of patients admitted to the ICU with pneumonia, the Pneumonia Shock Score (pShock) (10). This tool demonstrated excellent discriminate function, outperforming other prognostic scores evaluated in our derivation and external validation cohorts. Given the severity of pneumonia in those with COVID-19, this study seeks to calibrate and evaluate the performance of the pShock score in critically ill patients admitted to the ICU with SARS-CoV-2 infection.
Materials and methods
Study design
This was a prospective observational analytical cohort study conducted between January 2020 and March 2021 in four general ICUs in Salvador, Bahia, Brazil. All patients older than 18 years of age with confirmed SARS-CoV-2 infection by reverse transcriptase polymerase chain reaction analysis were included. The primary outcome assessed was ICU mortality. During the study period the assistance provided at all centers was in accordance with the guidelines and protocols for COVID-19 management. Clinical and laboratory data were prospectively collected in the medical records and registered in an encrypted database stored on the RedCap system (11). Study variables included age, weight, height, sex, length of ICU and hospital stay, and physiological and laboratory data within the first 6 h of admission. Complications including need for mechanical ventilation, vasopressors, and other supportive therapy in the ICU were noted. In addition, the score derivation dataset was used to compare the performance of the original score against a calibration of the pShock score (pShock-CoV).
Statistical analysis
Categorical variables were expressed as frequency and percentages, and continuous variables were expressed as medians with inter-quartile ranges (IQR). The proportion of categorical variables between groups were compared using Fisher's exact test. The median of continuous variables was compared using Mann-Whitney U test when analyzing the outcome groups. All tests were two-tailed and considered statistically significant for p ≤ 0.05. Variables that demonstrated possible statistical associations in univariate analysis (p ≤ 0.05) were transformed from continuous variables into categorical variables whose cutoff values were based on the Youden Index J on AUROC analysis. Additionally, a stepwise multivariate logistic regression was used to identify characteristics independently associated with ICU mortality. Data were categorized, then a ROC curve analysis was performed to assess accuracy and discrimination of the scores. Hosmer-Lemeshow tests for goodness of fit was used to assess the calibration of the model by comparing both the observed and expected mortality. The study was conducted accordingly with the Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD) guidelines (12). Extending our analyses, an internal validation using the K (10) Fold Cross Validation was performed (13). Resampling was used to evaluate the models on the data sample, using a parameter called “k” that refers to the number of groups the data sample was split into. One proportion of the data was used to discover the classification and the rest to validate and measure the prediction power of a limited data set. Probability of ICU survival during distinct timepoints since admission, was calculated by Kaplan-Meier analysis. Data analysis was carried out using GraphPad Prism version 6.01, SPSS, version 25.0 software and R statistical software.
Ethics approval and consent to participate
The study was approved directly by the National Committee of Ethics in Research (CONEP) from Brazil in accordance with local guidelines during COVID-19 pandemic (14), Certificate of Presentation of Ethical Appreciation (CAAE) Number: 30660720.0.0000.0008, and by the Ethical Committee of the Centro de Pesquisas Gonçalo Moniz, Fundação Oswaldo Cruz (FIOCRUZ) under CAAE number 39059320.8.1001.0040. The need for informed consent was waived in both committees and the anonymity of the study subjects was preserved.
Results
During the study period, 650 patients were admitted to the four study ICUs, of whom 605 met inclusion criteria (Figure 1). The median age was 63 (IQR: 49–74) years with a mortality rate of 33.2% (201 patients). Overall, non-survivors were significantly older when compared with survivors [70 (IQR: 62–80) years vs. 57 (IQR: 44–70) years; p ≤ 0.001]. No mortality differences were observed according to gender, nor objective clinical parameters such as heart rate, lowest systolic blood pressure, and sodium levels (Table 1). Importantly, the following factors were distinct in non-survivors compared to survivors: increased respiratory rate, elevated leukocyte count and urea, increased FiO2 within the first 6 h of admission, need for mechanical ventilation and vasopressors, and a lower Glasgow Coma Scale score and hematocrit in non survivors (Figure 2A). Description of prognostic scores analyzed are detailed in Table 2. Regarding each center characteristics, no significant discrepancies were observed concerning age distribution and vasopressors use meanwhile gender and vital signs exhibited some differences between cohorts.
Figure 1. Flow chart of study enrollment and analyzed population.
Table 1. General population description and comparison between survivors and non survivors.
Figure 2. General study population description and Calibration of pShock-CoV score. (A) Scatter plots depicting the distribution of age, hematocrit, leukocytes, urea, lowest Glasgow coma score, highest respiratory rate and highest FiO2 in non-survivors and survivors. Lines represent median and interquartile range values. The Mann-Whitney U test was employed to compare the values detected between the study groups. Use of vasopressors and use of mechanical ventilation variables are shown as frequency (%) and compared using the Fisher's exact test. (B) Adjusted and unadjusted binary regression model for ICU mortality. Multivariable regression adjusted for differences in baseline characteristics (variables of p ≤ 0.05 identified in univariable analysis).
Table 2. Prognostic scores in cohort stratified by mortality.
pShock score development and calibration of the pShock-CoV
The original pShock score was developed in a derivation cohort of critically ill patients admitted with pneumonia in the ICU, with an external validation cohort derived from the Community-Acquired Pneumonia Organization (CAPO). The primary outcome evaluated was ICU mortality, and independent risk factors identified by a binary logistic regression were included in the composite score. Of note results were remarkable by a good prediction performance of the pShock score, with an AUC of 0.80 [95% Confidence Interval (CI): 0.73–0.86; p-value <0.0001] and better discriminate function than other models analyzed (SAPS 3, qSOFA, CURB-65, and CRB-65) (10). Further, in this study, pShock variables and clinically important parameters routinely available in the first hours of admission were assessed over multiple analyses and a stepwise multivariate logistic regression model yielded 6 variables associated with ICU mortality: age ≥ 65 years, hematocrit ≤ 35%, white blood cell count ≥ 12 × 109/L, FiO2 ≥ 50%, urea ≥ 40 mg/dL and use of vasopressors (Figure 2B). The calibrated pShock-CoV score system was determined based on variability in the odds ratio for a confidence interval (CI) of 95%. Similar to the original derivation cohort for pShock, age and vasopressor use were weighted 2 points while other variables were given 1 point in the score calculation, with total score values ranging from a minimum of 0 to a maximum of 8. Notably, goodness of fit test exhibited good calibration of the model (Hosmer-Lemeshow statistics, p = 0.65).
pShock-CoV score discrimination and validation performance
In COVID-19 infected patients admitted to the ICU, the pShock-CoV score demonstrated robust performance accuracy with an AUC of 0.80 [95% Confidence Interval (CI): 0.77–0.83; p-value < 0.0001] for mortality prediction without a notable loss in discriminative capacity compared with the derivation cohort for the original pShock score (p-value = 0.9410, Figure 3A). The pShock-CoV score demonstrated superior discriminate function compared with CURB-65 (p = 0.0003) and qSOFA (p < 0.0001) (Figure 3B). Internal validation conducted by K (10) Fold Cross Validation analysis confirmed consistent discriminative capacity of the score compared with the original sample, with an AUC of 0.78 [95% Confidence Interval (CI): 0.71–0.83; p-value < 0.0001] (Supplementary Figure 1). Score performance was consistent in 30-day mortality similar to overall ICU mortality (p-value = 0.9759, Figure 4A), and with the others scores analyzed (Figure 4B). Temporal analysis from admission demonstrated decreased survival probability in those with higher scores of pShock-CoV over time (Supplementary Figure 2).
Figure 3. Discrimation of pShock-CoV in critically ill patients with SARS-CoV-2 infection and comparison with other severity models. (A) Receiver operating characteristic (ROC) curve analysis of pShock-CoV for prediction of ICU mortality in the ICU original sample and comparison of area under the ROC curve (Δ AUC) with pShock in the derivation cohort. (B) Overlap between ROC curves showing pShock-CoV performance and comparing with CURB-65 and qSOFA in COVID patients. Differences between AUC-ROCS were accessed by the DeLong test.
Figure 4. Discrimation of pShock-CoV over prediction of 30-day mortality for patients with COVID in ICU and comparison with other severity models. (A) Perfomance of the pShock-CoV score in predicting 30-day mortality in the intensive care unit, and comparison with discrimination capacity for overall mortality. (B) Comparison pShock-CoV with CURB-65 and qSOFA for prediction of ICU 30-day mortality in COVID patients. Differences between AUC-ROCS were accessed by the DeLong test.
Discussion
The persistence of high ICU mortality rates associated with COVID-19 infection may reflect delayed early recognition of those at highest risk of death resulting in missed opportunities to targeted interventions over the first hours of ICU admission. While ICU specific severity scores have been refined and new scores designed, a robust systematic model to predict mortality risk in a complex and diverse ICU population is lacking. Though vaccines and improved support measures have led to decreased morbidity and mortality, uncertainties remain in how best to stratify who is most likely to survive and target limited ICU resources to these patients (15). While recent studies have sought to develop prognostic tools to predict in-hospital COVID-19 mortality, these tools were not designed to evaluate risk for ICU mortality (16, 17). Other COVID-specific scores focused on triage evaluation to predict ICU admissions, which may inaccurately determine risk of deterioration and mortality in patients already admitted in these units (7, 8, 18, 19). Existing disease severity models including SOFA and SAPS3 lack adequate discriminant function, hindering accurate screening of critically ill patients in areas with supply shortages (20, 21). Furthermore, conclusions from clinical trial of novel therapeutics may be confounded as these severity scores are inaccurate in identification of the most critically ill subset of hospitalized patients with COVID-19 infection. Alternatively, to other models, pShock-CoV is a simple straightforward tool that doesn't uses radiographic images or complex variables to be obtained in the first hours of admission. In addition, some of the selected parameters are compatible with earlier described prognostic factors for COVID-19 patients, aiding the applicability of the model in routine clinical practice. While the pShock-CoV Score demonstrated significant discriminatory capacity and sustained performance in ICU and 30-day mortality including cross validation, certain study limitations must be acknowledged. First, the modest number of individuals included in the analysis may have underestimated the performance of the score in a larger ICU cohort of individuals with COVID-19 infection. Secondly, interventions including steroids, remdesivir and possibly a more experienced COVID treatment team could have impacted the performance of the score. Analysis over various time points through the pandemic demonstrated stable score performance reflecting ongoing excess mortality in those admitted to the ICU with COVID-19 independent of new treatment approaches (Supplementary Figure 3).
Conclusions
Our calibrated pShock-CoV score is a robust bedside tool that may better define severity of disease at time of trial enrollment and ensure that results reflect the studied interventions rather than unbalanced study groups.
Data availability statement
The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.
Ethics statement
The studies involving human participants were reviewed and approved by National Committee of Ethics in Research (CONEP) (CAAE Number: 30660720.0.0000.0008), and by the Ethical Committee of the Centro de Pesquisas Gonçalo Moniz, Fundação Oswaldo Cruz (FIOCRUZ) (CAAE Number: 39059320.8.1001.0040). Written informed consent for participation was not required for this study in accordance with the national legislation and the institutional requirements.
Author contributions
Conceptualization, design of study, and manuscript draft: TC, IF, RM, KA, and BA. Investigation and visualization: TC, MP, RO, GT, AM, TA, JC, and NF. Data acquisition: MP, RO, GT, AM, TA, JC, and NF. Data analysis and interpretation: TC, IF, RM, MA, KA, and BA. Supervision and critical revision: TC, IF, RM, JC, MA, KA, and BA. Editing and final approval of the manuscript: TC, RM, MA, KA, and BA. All authors read and approved the final manuscript.
Funding
The work of BA was supported by the Intramural Research Program of the Oswaldo Cruz Foundation, Brazil. RM received a fellowship from the Programa Nacional de Pós-Doutorado/Coordenação de Aperfeiçoamento de Pessoal de Nível Superior (PNPD/CAPES). Fogarty International Center and National Institute of Child Health and Human Development of the National Institutes of Health under (Award Number D43 TW009763 through a research scholarship awarded to MA), MA and TC (scientific initiation with Award Number: PPSUS/BA-FAPESB003/2017/SESAB/CNPq/MS 5125/2017) received a fellowship from the Fundação de Amparo à Pesquisa do Estado da Bahia (FAPESB).
Acknowledgments
The authors acknowledge the research groups Grupo de Estudo em Medicina Intensiva and Multinational Organization Network Sponsoring Translational and Epidemiological Research Initiative, linked to Fundação Oswaldo Cruz.
Conflict of interest
Authors TC, IF, RM, MA, KA, and BA were employed by fellows from Multinational Organization Network Sponsoring Translational and Epidemiological Research Initiative.
The remaining authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
Publisher's note
All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.
Supplementary material
The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fmed.2022.958291/full#supplementary-material
References
1. Coronaviridae Study Group of the International Committee on Taxonomy of Viruses. The species Severe acute respiratory syndrome-related coronavirus: classifying 2019-nCoV and naming it SARS-CoV-2. Nat Microbiol. (2020) 5:536–544. doi: 10.1038/s41564-020-0695-z
2. World Health Organization. Coronavirus Disease (COVID-19) Situation Reports. (2020). Available online at: https://www.who.int/emergencies/diseases/novel-coronavirus-2019/situation-reports (accessed October 28, 2021).
3. Armstrong RA, Kane AD, Cook TM. Outcomes from intensive care in patients with COVID-19: a systematic review and meta-analysis of observational studies. Anaesthesia. (2020) 75:1340–9. doi: 10.1111/anae.15201
4. Elhadi M, Alsoufi A, Abusalama A, Alkaseek A, Abdeewi S, Yahya M, et al. Epidemiology, outcomes, and utilization of intensive care unit resources for critically ill COVID-19 patients in Libya: a prospective multi-center cohort study. PLoS One. (2021) 16:e0251085. doi: 10.1371/journal.pone.0251085
5. Marinho PRD, Cordeiro GM, Coelho HFC, Brandão SCS. Covid-19 in Brazil: A sad scenario. Cytokine Growth Factor Rev. (2021) 58:51–4. doi: 10.1016/j.cytogfr.2020.10.010
6. Rubinson L. Intensive Care Unit Strain and Mortality Risk Among Critically Ill Patients With COVID-19—There Is No “Me” in COVID. JAMA Netw Open. (2021) 4:e2035041. doi: 10.1001/jamanetworkopen.2020.35041
7. Wynants L, Van Calster B, Collins GS, Riley RD, Heinze G, Schuit E, et al. Prediction models for diagnosis and prognosis of covid-19: systematic review and critical appraisal. BMJ. (2020) 369:m1328. doi: 10.1136/bmj.m1328
8. Gupta RK, Marks M, Samuels THA, Luintel A, Rampling T, Chowdhury H, et al. Systematic evaluation and external validation of 22 prognostic models among hospitalised adults with COVID-19: an observational cohort study. Eur Respir J. (2020) 56:2003498. doi: 10.1101/2020.07.24.20149815
9. Antommaria AHM, Gibb TS, McGuire AL, Wolpe PR, Wynia MK, Applewhite MK, et al. Ventilator triage policies during the COVID-19 pandemic at US hospitals associated with members of the association of bioethics program directors. Ann Intern Med. (2020) 173:188–94. doi: 10.7326/M20-1738
10. Carmo TA, Ferreira IB, Menezes RC, Telles GP, Otero ML, Arriaga MB, et al. Derivation and validation of a novel severity scoring system for pneumonia at intensive care unit admission. Clin Infect Dis. (2021) 72:942–9. doi: 10.1093/cid/ciaa183
11. Harris PA, Taylor R, Minor BL, Elliott V, Fernandez M, O'Neal L, et al. The REDCap consortium: building an international community of software platform partners. J Biomed Inform. (2019) 95:103208. doi: 10.1016/j.jbi.2019.103208
12. Collins GS, Reitsma JB, Altman DG, Moons KGM. Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD): The TRIPOD Statement. Ann Intern Med. (2015) 162:55–63. doi: 10.7326/M14-0697
13. Kuhn M. Building predictive models in R using the caret package. J Stat Softw. (2008) 28:1–26. doi: 10.18637/jss.v028.i05
14. Ministry of Health. Guidelines for Conducting Research and REC Activities During the Coronavirus SARS-CoV-2 (COVID-19) Pandemic. (2020). p. 1–6. Availabel online at: http://abracro.org.br/pdfs/covid19/CONEP_Orientacoes-para-pesquisa-Covid19_09Maio2020_EN.pdf (accessed May 9, 2020).
15. Rubin R. COVID-19 vaccines vs variants-determining how much immunity is enough. JAMA. (2021) 325:1241–3. doi: 10.1001/jama.2021.3370
16. Gue YX, Tennyson M, Gao J, Ren S, Kanji R, Gorog DA. Development of a novel risk score to predict mortality in patients admitted to hospital with COVID-19. Sci Rep. (2020) 10:21379. doi: 10.1038/s41598-020-78505-w
17. Knight SR, Ho A, Pius R, Buchan I, Carson G, Drake TM, et al. Risk stratification of patients admitted to hospital with covid-19 using the ISARIC WHO clinical characterisation protocol: development and validation of the 4C mortality score. BMJ. (2020) 370:m3339. doi: 10.1136/bmj.m3339
18. Gidari A, De Socio GV, Sabbatini S, Francisci D. Predictive value of National Early Warning Score 2 (NEWS2) for intensive care unit admission in patients with SARS-CoV-2 infection. Infect Dis. (2020) 52:698–704. doi: 10.1080/23744235.2020.1784457
19. Galloway JB, Norton S, Barker RD, Brookes A, Carey I, Clarke BD, et al. A clinical risk score to identify patients with COVID-19 at high risk of critical care admission or death: an observational cohort study. J Infect. (2020) 81:282–8. doi: 10.1016/j.jinf.2020.05.064
20. Dao B, Savulescu J, Suen JY, Fraser JF, Wilkinson DJC. Ethical factors determining ECMO allocation during the COVID-19 pandemic. BMC Med Ethics. (2021) 22:70. doi: 10.1186/s12910-021-00638-y
Keywords: critical care, prognosis, COVID-19, mortality, risk factors
Citation: Carmo TA, Ferreira IBB, Menezes RC, Pina MLT, Oliveira RS, Telles GP, Machado AFA, Aguiar TC, Caldas JR, Arriaga MB, Akrami KM, Filgueiras Filho NM and Andrade BB (2022) Calibration and validation of the Pneumonia Shock Score in critically ill patients with SARS-CoV-2 infection, a multicenter prospective cohort study. Front. Med. 9:958291. doi: 10.3389/fmed.2022.958291
Received: 31 May 2022; Accepted: 28 July 2022;
Published: 15 August 2022.
Edited by:
Yuetian Yu, Shanghai Jiao Tong University, ChinaReviewed by:
Etrusca Brogi, University of Pisa, ItalyVera Clérigo, Hospital de São Bernardo, Portugal
Copyright © 2022 Carmo, Ferreira, Menezes, Pina, Oliveira, Telles, Machado, Aguiar, Caldas, Arriaga, Akrami, Filgueiras Filho and Andrade. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
*Correspondence: Bruno B. Andrade, bruno.andrade@fiocruz.br
†These authors have contributed equally to this work
‡These authors have contributed equally to this work and share senior authorship