Development and validation of a prediction score for failure to casirivimab/imdevimab in hospitalized patients with COVID-19 pneumonia

Abstract

Introduction:

Casirivimab and imdevimab (CAS/IMV) are two non-competing, high-affinity human IgG1 anti-SARS-CoV-2 monoclonal antibodies, that showed a survival benefit in seronegative hospitalized patients with COVID-19. This study aimed to estimate the day-28 risk of mechanical ventilation (MV) and death in individuals hospitalized for severe COVID-19 pneumonia and receiving CAS/IMV. Additionally, it aimed to identify variables measured at the time of hospital admission that could predict these outcomes and derive a prediction algorithm.

Methods:

This is a retrospective, observational cohort study conducted in 12 hospitals in Italy. Adult patients who were consecutively hospitalized from November 2021 to February 2022 receiving CAS/IMV were included. A multivariable logistic regression model was used to identify predictors of MV or death by day 28 from treatment initiation, and β-coefficients from the model were used to develop a risk score that was derived by means of leave-one-out internal cross-validation (CV), external CV, and calibration. Secondary outcome was mortality.

Results:

A total of 480 hospitalized patients in the training set and 157 patients in the test set were included. By day 28, 36 participants (8%) underwent MV and 28 died (6%) for a total of 58 participants (12%) experiencing the composite primary endpoint. In multivariable analysis, four factors [age, PaO²/FiO² ratio, lactate dehydrogenase (LDH), and platelets] were independently associated with the risk of MV/death and were used to generate the proposed risk score. The accuracy of the score in the area under the curve (AUC) was 0.80 and 0.77 in internal validation and test for the composite endpoint and 0.87 and 0.86 for death, respectively. The model also appeared to be well calibrated with the raw data.

Conclusion:

The mortality risk reported in our study was lower than that previously reported. Although CAS/IMV is no longer used, our score might help in identifying which patients are not likely to benefit from monoclonal antibodies and may require alternative interventions.

Introduction

The treatment of hospitalized patients with COVID-19 has undergone profound changes since March 2020, and interventions varied over time as a result of accumulating evidence from both observational and randomized studies conducted across different waves of the pandemic (1–3). This is particularly true for monoclonal antibodies. Since the beginning of the pandemic, it was understood that they could play a role in treating COVID-19 disease especially during the viral phase of the infection, as proved to be effective in animal models and in vitro studies on rapid culture conversion of SARS-CoV-2 infection (4, 5). Indeed, several monoclonal antibodies, when tailored against the specific predominant circulating viral variant, were shown to be effective in preventing SARS-CoV-2 infection (especially in immunocompromised individuals) and hospitalization if administered within the first 5–7 days after the infection (6–10).

Concerning hospitalized patients, since patients in this setting are typically in the inflammatory phase of COVID-19, the role of these compounds is more unclear. Indeed, two placebo-controlled randomized trials, one with bamlanivimab and one with either sotrovimab or the combination of BRII-196 and BRII-198, were terminated earlier due to futility (11–13).

Casirivimab and imdevimab (CAS/IMV) are two non-competing, high-affinity human IgG1 anti-SARS-CoV-2 monoclonal antibodies, which bind specifically to the receptor-binding domain of the spike glycoprotein of SARS-CoV-2, thereby blocking viral entry into host cells (14). Recently, two randomized trials were conducted among hospitalized patients with CAS/IMV: the RECOVERY platform trial and a smaller trial conducted in the USA on 1,336 patients (6, 15). The RECOVERY trial showed a clear benefit in terms of survival, particularly when restricting the analysis to patients with a negative antibody status at hospital admission (RR 0.79, 95% CI 0.69–0.91; P = 0.0009). A similar result in terms of survival, still when restricting the analysis to seronegative patients, was obtained in another recently published study (15). Based on these results, all international guidelines suggested monoclonal antibodies for hospitalized subjects seronegative for SARS-CoV-2 in the pre-Omicron era.

To our knowledge, no real-life data on CAS/IMV outcomes in hospitalized patients with severe COVID-19 pneumonia have been published until now. With these premises, the aim of our analysis was twofold: first, we aimed to estimate the in-hospital day-28 risk of mechanical ventilation (MV) and death in a large cohort of patients admitted to hospital with SARS-CoV-2 pre-omicron variants of concern (VoCs) and treated with CAS/IMV. Second, we aimed at identifying variables measured at the time of hospital admission that could predict these poor outcomes as well as derive a prediction algorithm for clinical use (16–18).

Methods

This multi-center observational study analyzed individuals who confirmed positive for SARS-CoV-2 through nasal swab tests and hospitalized patients with severe COVID-19 who were treated with CAS/IMV in 12 tertiary care hospitals in Italy. Among the 12 hospitals, nine hospitals provided data for the training set and three hospitals provided for the test set (Supplementary Table S3). All adult patients consecutively hospitalized with severe COVID-19 and treated with CAS/IMV from November 2021 to March 2022 were included. Severe COVID-19 was defined in accordance with the NIH (National Institute of Health) guidelines. We excluded patients for whom there was no confirmation of the SARS-CoV-2 positive test, those who were not eligible for treatment with CAS/IMV, and those who started other treatments. The retrospective data were fully anonymized and were last assessed on February 2023. CAS/IMV were administered intravenously at the dose of 4 g for each antibody.

IRB approval and data information

The study was approved by the ethics committee of each participating site.

Data collection

The participants' demographics (age, sex, calendar month and year of hospitalization, vaccination history, and SARS CoV-2 serology test results), comorbidities, and laboratory and respiratory function parameters were collected at the time of hospital admission. Data were collected from both paper and electronic clinical records using a standardized operating procedure across clinics.

Study outcomes

The primary outcome was the composite event of initiation of MV or death by day 28 from the date of starting CAS/IMV while the secondary outcome was death. Participants who experienced the event (MV or death) during the follow-up period and up to day 28 from starting the drug were labeled as events and the remaining were labeled as event-free.

Statistical analysis

Participants' characteristics were described and compared based on their event/event-free status. For continuous variables (such as age and laboratory and respiratory function parameters), the ranks of the distributions were compared between events and event-free using the Mann–Whitney test. Chi-squared and Fisher exact tests were used to compare proportions as appropriate.

Factors that were most strongly associated with the primary outcome in univariable analysis were selected as candidates to be included in a prediction score. Univariable and multivariable associations between these factors and the risk of the outcome were estimated using the logistic regression method, and the magnitude of the association were expressed by means of odds ratios (OR) with 95% confidence interval.

For constructing the score for the continuous variables (16–18), participants were grouped according to the median value of these variables, and ORs of the primary endpoint associated with a value below/above the median were shown.

To evaluate the ability of the model to predict the outcomes, the area under the receiver operating characteristic (ROC) curve was calculated. This was first calculated in the training set and compared to the classifier with area under the curve (AUC) = 0.5 using a Mann-Whitney statistics and chi-square test. To control for extra-sample variation, a leave-one-out cross-validation (CV) was implemented. This means that N separate times, the function approximator was trained on all the data after excluding one participant and a prediction was made for this excluded person. The AUC in CV was computed and used to evaluate the predictive ability of the model by comparing it with the value obtained on the training set. In addition to evaluating the predictive ability, we also investigated the model calibration by means of a calibration plot in the test dataset.

Furthermore, we used an independent sample of patients obtained from other three Italian hospitals (one for each region of the country: North, Central, and South) with identical inclusion criteria and definition of outcomes (Supplementary Table S3, the test set).

Symmetrical analyses were performed for the primary and secondary endpoint of death. For the purpose of simplicity and to broadly categorize the risk of failing CAS/IMV, the study population was divided into those with low (0–5%), moderate (6–19%), and high (≥20%) risk of day-28 mechanical ventilation/death. These groups were matched to actual risk ranges calculated from the propensity score formula below Equation (1). Exact individual risk can be calculated as per the formula given below by entering the individual's own demographics and biomarkers values.

X1, X2, etc. are the patients' characteristic values at hospital admission, and β0, β1, β2, etc. are the parameter estimates from the logistic regression model.

The results were reported according to the TRIPOD guidelines (17, 18).

Results

Training set

The training set included a total of 480 hospitalized patients. By day 28 since the date of CAS/IMV initiation, 36 participants (8%) underwent MV and 28 died (6%) for a total of 58 participants (12%) who experienced the composite primary endpoint.

Table 1 shows the main demographic characteristics and average marker values recorded at hospital admission stratified by events/event-free status. As expected, the level of PaO²/FiO² ratio and platelets were lower in events compared to event-free. In contrast, lactate dehydrogenase (LDH) levels were higher in events who were also more than 10 years older than event-free (72 years vs. 60 years on average). Table 2 shows the prevalence of comorbidities in events and event-free. Unexpectedly, the only condition strongly associated with the outcome was cardiac insufficiency (21% in events vs. 8% in event-free, P < 0.0001) which remained associated after controlling for age (Table 3).

Table 3 shows factors significantly associated with the risk of the outcome in the univariable analysis. These included the following four factors: age, PaO²/FiO² ratio, LDH, and platelets, which were considered to build the predictive score. Notably, PaO²/FiO² ratio, LDH, and platelets remained significantly associated after mutual adjustment as well as after controlling for age. PaO²/FiO² ratio that indicated the level of respiratory function at admission was by large the factor associated with the bigger effect (OR).

IL-6 was also strongly associated with the risk of MV/death but was left out of the score because too many participants had a missing value for this marker. In contrast, there was little evidence for an association between the extent of total comorbidities (summarized by the Charlson Index), year of hospital admission (2021 vs. 2022), vaccination history, and SARS-CoV-2 serology test results. Consequently, these factors were also left out of the score.

Calibration of the model in the test set was not perfect but lied close to the “perfect calibration” dotted line for the slope and within its 95% confidence limits (Supplementary Figure S1). The analysis for the secondary endpoint of death identified the same predictors, although the magnitude of the association was smaller for PaO²/FiO² and LDH (Supplementary Table S2).

Internal validation

The AUC of the ROC curve using the training dataset was 0.81, indicating a good trade-off between sensitivity and specificity (Figure 1). The AUC was significantly better than a random classification of the participants (Mann–Whitney test vs. AUC = 0.5, P < 0.001). Although the performance of the model is likely to be overestimated in training, the internal validation, by means of CV, provided only a slightly lower value for AUC = 0.80 (Figure 1). For the secondary endpoint of death, the AUCs were even bigger with 0.87 in the training set and 0.85 in cross-validation (Supplementary Figure S2).

Figure 1

External validation

The test set included 157 participants with 24 events. In the test set, 44% were female participants, with the mean age of 69 years, PaO²/FiO² ratio of 319 mmHg, LDH of 275.8 mg/dL, and platelets 189.4 mg/dL (Supplementary Table S1). The AUC in the test set was 0.77 for the composite endpoint (Figure 2) and slightly better for the endpoint death, indicating a smaller decrease in accuracy from 0.87 to 0.86 (Supplementary Figure S3).

Figure 2

Derivation of the score for the primary endpoint MV/death

Based on the estimates of the logistic regression analysis, a simplified prediction score was developed from fitting the factors as binary variables (using the median as the threshold for continuous variables) and allocating weights to each of the four components of the score. Thus, for example, a virtual person aged 62+ with LDH above the median at hospital admission and PaO²/FiO² ratio and platelets below the median has all four risk factors (age, LDH, PaO²/FiO² ratio, and platelets, Table 4). Thus, for our virtual participant who is older than 62 years (+6), with a PaO²/FiO² ratio < 306 mmHg (+8), LDH >287 (+2), and platelets < 177 (+4), his/her simplified sum risk score would be 6 + 7 + 3 + 5 = 21, placing him/her in the high-risk (≥20) category. The exact propensity risk for this person (i.e., his/her probability of experiencing MV/death given his/her covariate profile) estimated from the logistic regression model is 37.4% (it can be calculated from the Equation 1 described in the Methods and shown again at the bottom of Table 4).

Discussion

To our knowledge, this study can be considered the largest real-world analysis evaluating the effectiveness of CAS/IMV on mortality and MV in subjects with severe COVID-19 at hospital admission. The main result is that, in our setting, where 80% tested SARS CoV-2 negative and approximately 20% of the study population received vaccination for SARS-CoV-2, the risk of day 28 MV/death remained high. Indeed, our cohort exhibited a mortality rate of 6%, with 12% of the patients experiencing the primary end-point MV/death. However, these estimates are much lower than those observed in the RECOVERY trial (24% by day 28 in seronegative patients receiving CAS/IMV vs. 30% of the controls receiving usual care). In contrast, our estimates are similar to those of a recently published study conducted in the USA, which reported a mortality rate of 6% among seronegative patients treated with CAS/IMV compared to 15% of those receiving placebo (15). This difference in the mortality rate across studies is not unexpected and might be explained by the period in which these studies were conducted. Indeed, the randomized studies were conducted between June 2020 and May 2021, before the start of the COVID-19 vaccination campaign. Given the rapid evolution of SARS-CoV-2, changes in virus pathogenicity, and variations in the severity of the conditions among patients admitted to the hospital, even a difference of just 1 month could determine a significant difference in mortality rates. Furthermore, patients enrolled in our study had a lower disease severity at entry and were hospitalized, possibly benefiting from a larger number of drugs recommended in the guidelines. Finally, we cannot rule out that our cohort represents a selected population of individuals who survived long enough after hospital admission to be treated with CAS/IMV.

Contrary to what was previously described, only cardiac insufficiency [and not diabetes or hypertension, for example (19–22)] showed an association with the risk of outcomes after controlling for sex and age. However, this association did not persist after further controlling for other factors included in the score.

The patients who underwent MV or died and the controls differed significantly in age, PaO²/FiO² ratio, LDH, and platelet count, all factors that characterized severe COVID-19 since the beginning of the pandemic and were included in similar predictive algorithms. However, it was important to confirm that they are still strong predictors of outcomes in our specific setting.

Prediction accuracy of the derived score was high (84% in the toughest external validation) and did not vary by the endpoint chosen. If anything, the score derived on the composite endpoint was even more predictive when applied to the secondary endpoint of death alone. The score also showed good calibration, a metric which is often neglected when evaluating the performance of a score in addition to discrimination (23). A well-discriminating model similar to our model (i.e., with a c-statistic >0.8) may be useless if the decision threshold for clinical decisions is outside the range of predictions provided by the model. Importantly, good calibration was retained in the test set.

Our study has several limitations; first, CAS/IMV are not currently used in clinical practice as they are considered to be not active against the new omicron VoC and therefore it is unlikely that the score could be used in clinical practice for this particular combination. However, monoclonal antibodies are still in development and are tailored to the fast-evolving virus. Some of these new compounds might be re-introduced for the treatment of hospitalized patients as well. Second, it is a retrospective cohort; therefore, data collection was not complete for some biomarkers, and we performed a complete case analysis excluding those with missing laboratory data. Third, we constructed a simple model based on a linear predictor. We cannot rule out that a full machine learning approach with a larger number of variables as well as their interactions could have led to better values for the AUC in validation and test. For example, the score does not account for time-varying data including medical interventions received post hospitalization. Fourth, the sample size of the test cohort was relatively small including approximately 150 participants. However, the AUC in external validation test for both endpoints remained >80% which is considerably above the minimum for a good discrimination.

The main strength of our analysis is that it is the first score derived for hospitalized patients treated with CAS/IMV. In addition, it is very easy to use in practice, uses a small number of routinely collected features, and is calculable without having to input patients' data into a website or a similar graphic user interface (24).

In conclusion, although CAS/IMV is no longer routinely used in clinical practice, our newly developed score might help in identifying at hospital admission which patients are not likely to benefit from monoclonal antibodies and may require alternative interventions.

References

• 1.

  Mussini C Cozzi-Lepri A . Another piece in the COVID-19 treatment puzzle. Lancet. (2022) 399:609–10. 10.1016/S0140-6736(22)00154-4

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 2.

  Salvarani C Dolci G Massari M Merlo DF Cavuto S Savoldi L et al . Effect of tocilizumab vs. standard care on clinical worsening in patients hospitalized with COVID-19 pneumonia: a randomized clinical trial. JAMA Intern Med. (2021) 181:24–31. 10.1001/jamainternmed.2020.6615

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 3.

  RECOVERY Collaborative Group . Tocilizumab in patients admitted to hospital with COVID-19 (RECOVERY): a randomised, controlled, open-label, platform trial. Lancet. (2021) 397:1637–45. 10.1016/S0140-6736(21)00676-0

  • CrossRef
  • Google Scholar

• 4.

  Rogers TF Zhao F Huang D Beutler N Burns A He WT et al . Isolation of potent SARS-CoV-2 neutralizing antibodies and protection from disease in a small animal model. Science. (2020) 369:956–63. 10.1126/science.abc7520

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 5.

  Boucau J Chew KW Choudhary MC Deo R Regan J Flynn JP et al . Monoclonal antibody treatment drives rapid culture conversion in SARS-CoV-2 infection. Cell Rep Med. (2022) 3:100678. 10.1016/j.xcrm.2022.100678

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 6.

  RECOVERY Collaborative Group . Casirivimab and imdevimab in patients admitted to hospital with COVID-19 (RECOVERY): a randomised, controlled, open-label, platform trial. Lancet. (2022) 399:665–76. 10.1016/S0140-6736(22)00163-5

  • CrossRef
  • Google Scholar

• 7.

  Weinreich DM Sivapalasingam S Norton T Ali S Gao H Bhore R et al . REGEN-COV antibody combination and outcomes in outpatients with Covid-19. N Engl J Med. (2021) 385:e81. 10.1056/NEJMoa2108163

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 8.

  Dougan M Nirula A Azizad M Mocherla B Gottlieb RL Chen P et al . Bamlanivimab plus Etesevimab in Mild or Moderate Covid-19. N Engl J Med. (2021) 385:1382–92. 10.1056/NEJMoa2102685

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 9.

  Gupta A Gonzalez-Rojas Y Juarez E Crespo Casal M Moya J Falci DR et al . Early treatment for COVID-19 with SARS-CoV-2 neutralizing antibody sotrovimab. N Engl J Med. (2021) 385:1941–50. 10.1056/NEJMoa2107934

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 10.

  Levin MJ Ustianowski A De Wit S Launay O Avila M Templeton A et al . Intramuscular AZD7442 (Tixagevimab-Cilgavimab) for Prevention of Covid-19. N Engl J Med. (2022) 386:2188–200. 10.1056/NEJMoa2116620

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 11.

  ACTIV-3/TICO Bamlanivimab Study Group . Responses to a neutralizing monoclonal antibody for hospitalized patients with COVID-19 according to baseline antibody and antigen levels: a randomized controlled trial. Ann Intern Med. (2022) 175:234–43. 10.7326/M21-3507

  • CrossRef
  • Google Scholar

• 12.

  ACTIV-3/Therapeutics for Inpatients with COVID-19 (TICO) Study Group . Efficacy and safety of two neutralising monoclonal antibody therapies, sotrovimab and BRII-196 plus BRII-198, for adults hospitalised with COVID-19 (TICO): a randomised controlled trial. Lancet Infect Dis. (2022) 22:622–35. 10.1016/S1473-3099(21)00751-9

  • CrossRef
  • Google Scholar

• 13.

  ACTIV-3–Therapeutics for Inpatients with COVID-19 (TICO) Study Group . Tixagevimab-cilgavimab for treatment of patients hospitalised with COVID-19: a randomised, double-blind, phase 3 trial. Lancet Respir Med. (2022) 10:972–84. 10.1016/S2213-2600(22)00215-6

  • CrossRef
  • Google Scholar

• 14.

  Hansen J Baum A Pascal KE Russo V Giordano S Wloga E et al . Studies in humanized mice and convalescent humans yield a SARS-CoV-2 antibody cocktail. Science. (2020) 369:1010–14. 10.1126/science.abd0827

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 15.

  Somersan-Karakaya S Mylonakis E Menon VP Wells JC Ali S Sivapalasingam S et al . Casirivimab and imdevimab for the treatment of hospitalized patients with COVID-19. J Infect Dis. (2022) 227:23–34. 10.1093/infdis/jiac320

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 16.

  Leisman DE Harhay MO Lederer DJ Abramson M Adjei AA Bakker J et al . Development and reporting of prediction models: guidance for authors from editors of respiratory, sleep, and critical care journals. Crit Care Med. (2020) 48:623–33. 10.1097/CCM.0000000000004246

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 17.

  Collins GS Reitsma JB Altman DG Moons KG . Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD). Ann Intern Med. (2015) 162:735–6. 10.7326/L15-5093-2

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 18.

  Moons KG Altman DG Reitsma JB Ioannidis JP Macaskill P Steyerberg EW et al . Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): explanation and elaboration. Ann Intern Med. (2015) 162:W1–73. 10.7326/M14-0698

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 19.

  Williamson EJ Walker AJ Bhaskaran K Bacon S Bates C Morton CE et al . Factors associated with COVID-19-related death using OpenSAFELY. Nature. (2020) 584:430–6. 10.1038/s41586-020-2521-4

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 20.

  Bartoletti M Giannella M Scudeller L Tedeschi S Rinaldi M Bussini L et al . Development and validation of a prediction model for severe respiratory failure in hospitalized patients with SARS-CoV-2 infection: a multicentre cohort study (PREDI-CO study). Clin Microbiol Infect. (2020) 26:1545–53. 10.1016/j.cmi.2020.08.003

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 21.

  Mussini C Cozzi-Lepri A Menozzi M Meschiari M Franceschini E Milic J et al . Development and validation of a prediction model for tocilizumab failure in hospitalized patients with SARS-CoV-2 infection. PLoS ONE. (2021) 16:e0247275. 10.1371/journal.pone.0247275

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 22.

  Tiseo G Margalit I Ripa M Borghi V Green H Prendki V et al . Predictors of survival in elderly patients with coronavirus disease 2019 admitted to the hospital: derivation and validation of the FLAMINCOV score. Clin Microbiol Infect. (2023) 29:379–85. 10.1016/j.cmi.2022.09.019

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 23.

  Gerry S Bonnici T Birks J Kirtley S Virdee PS Watkinson PJ et al . Early warning scores for detecting deterioration in adult hospital patients: systematic review and critical appraisal of methodology. BMJ. (2020) 369:m1501. 10.1136/bmj.m1501

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 24.

  Watts NB . The fracture risk assessment tool (FRAX^®): applications in clinical practice. J Womens Health. (2011) 20:525–31 10.1089/jwh.2010.2294

  • Pubmed Abstract
  • CrossRef
  • Google Scholar