A COVID-19 Test Triage Tool, Predicting Negative Results and Reducing the Testing Burden on Healthcare Systems During a Pandemic

Introduction

Detecting and isolating cases of COVID-19 are amongst the key elements listed by the WHO to reduce transmission (1). While everybody is at risk of infection, the majority of the population has not been tested. It has been reported that blanket testing and isolation of positive cases in a village in Italy with a population of ~3,000 people, saw the number of people with COVID-19 symptoms fall by over 90% in 10 days (2).

While initially there were testing capacity constraints in the USA, Government officials reported that there would be a much greater supply and availability of testing (3). However, now over a month since such assurances, there are still reports of shortages of testing for SARS-CoV-2 (4–6). While a PCR test of a nasopharyngeal swab or serum antibody test are now standards for COVID-19 diagnosis, providing this on a population level is neither practical nor necessary. The decision of whom to test is further complicated by the fact that this pandemic overlaps with seasonal flu; a syndrome which presents with many similar features to COVID-19.

However, a freely available and rapid risk assessment, which provides the likelihood of a positive test, may help provide the assurances many need, and encourage many more to continue to isolate themselves in their individual effort to reduce further SARS-CoV-2 transmission.

Risk calculators have been widely used in clinical practice for many years, particularly in urological cancers to help identify those at risk of a cancer diagnosis and also to correctly identify those at risk of a high risk cancer diagnosis (7, 8). Given the range of symptoms that patients with COVID-19 present with to hospitals in the US, we postulated that a novel risk calculator may aid in the risk assessment and triaging of these patients to help identify those at risk of being diagnosed with COVID-19 to help tailor their medical care. As this pandemic continues, all US hospitals will be under considerable pressure to maintain a high standard of care and a novel risk calculator may aide in the initial triaging of these patients.

Methods

Patient Data

De-identified patient data was obtained from the Mount Sinai Healthcare System (MSHS) Data Warehouse (https://msdw.mountsinai.org/) for all patients who present for testing prior to April 4th 2020 (n = 21,790). This includes data from 10 institutes and facilities across four boroughs of New York City. Of these, 9,516 having been assessed as meeting state and institutional criteria, underwent testing for SARS-CoV-2 as they were suspected of having COVID-19 (9). There were 4,640 who tested positive following either an oro or naso-pharyneal swab and test with Cobas COVID-19 (Roche Diagnostics, NJ, USA), or CDC COVID-19 assay.

Of this dataset, 4,187 had complete information available on the following variables: age, gender, race, BMI, reported maximum temperature, temperature at time of review, smoking status and comorbidities such as asthma, COPD, hypertension and diabetes.

The MSHS Ethics Committee approved a waiver of documentation of informed consent for use of the; de-identified patient data was obtained from the MSHS Data Warehouse (https://msdw.mountsinai.org/).

Statistical Analysis

Data Analysis

All analyses were performed using R software (10). Continuous data were presented as median [interquartile range (IQR)]. Categorical data were presented as number (percentage). The χ²-test was used to compare differences in clinical outcomes between SARS-CoV-2 test Positive and SARS-CoV-2 test negative groups. The dataset was randomly divided with 80% used to develop the prediction model, and 20% maintained as a holdout dataset. Correlation of available demographic and clinical covariates was performed. Using the available demographic and clinical covariates, and a proprietary prediction platform which also captures non-linear interactions, we developed a model to predict a positive SARS-CoV-2 test in the training set, which also used an iterative strategy to select a subset of the available predictors. This produced multiple models which were then validated in the holdout data set. The model which required the least covariates and which was not statistically significant from the model with the greatest discriminative ability in the holdout dataset was chosen.

Classification metrics of the model in the holdout dataset were calculated and include model sensitivity (recall), specificity, accuracy, positive predictive value (precision), and negative predictive value. We assessed the discrimination of the models with the area under the receiver operating characteristic curve [AUC (95% CI)]. AUC values for the various models were compared using U-statistics (11). Calibration curves were computed by comparing observed proportions of a positive SARS-CoV-2 test to mean calculated risks from the model in the holdout cohort. Decision curve analysis was performed to assess for the gain derived from using this model in the holdout cohort over the corresponding net benefit curves of testing all of these patients, or none of these patients (12).

Performance metrics of the model in the hold out set were calculated and tabularized, based on a cut points of 0.7, derived from the training dataset; the point at which negative predictive value was optimal.

Results

Of the 4,187 patients with complete data, 2,022 (48%) had tested positive for SARS-CoV-2 and 2,165 (52%) had tested negative. The training dataset had 3,349 patients with complete data; of which 1,612 (48%) had a positive SARS-CoV-2 test. Of 838 patients in the holdout dataset, 410 (49%) had a positive SARS-CoV-2 test, and 428 (51%) had a negative SARS-CoV-2 test (Table 1).

TABLE 1

Table 1. Characteristics of all patients; subdivided by the result of Covid-19 testing.

The AUC value for prediction of a positive SARS-CoV-2 test was 0.77 for this model in the holdout dataset (Figure 1A). A cut-off point of calculated risk was chosen so as to optimize the negative predictive rate and the reduction in SARS-CoV-2 tests. At this cut-off point, the negative predictive value is 96%, and positive predictive value is 66%. Using a risk of 70% in the hold-out dataset would have reduced testing volume by 76%; or 636 tests out of 838, and 91 patients with a positive SARS-CoV-2 test would have been misclassified as negative by this model (Table 2).

FIGURE 1

Figure 1. (A) Receiver operating characteristic curves and AUC values for the model in the holdout dataset demonstrating the discriminative ability of this model (AUC = 0.766; 95% CI 0.733–0.798). (B) Calibration curves of the model in the holdout dataset demonstrating the agreement between predicted and observed probabilities of a positive Covid-19 test. (C) Decision curve analysis for positive Covid-19 test prediction in the holdout dataset. (D) Correlation matrix demonstrating the correlation between risk factors and COVID-19 diagnosis: the larger the size, and the stronger the color, the higher the correlation.

TABLE 2

Table 2. Model performance characteristics for the holdout dataset.

Calibration plots provide a visual representation of how reliable the predicted risk estimate is; the accuracy of risk estimates relating to the agreement between estimated and observed events.⁴ A curve close to the diagonal indicates that predicted risks correspond well to observed proportions; Figure 1B demonstrates the model has excellent overall calibration. Some overestimation of the risk of a positive test for those with a positive SARS-CoV-2 test is seen for those with calculated risks ~<50% and some underestimation of risk for those with higher calculated risks.

Decision curve analysis was performed and is plotted in Figure 1C. The straight black line at y = 0 represents the net benefit derived from employing a strategy of testing nobody and the gray line represents the net benefit if a strategy of testing everybody was employed. The model was superior to both of these strategies across the entire range of clinically useful threshold risks. The relationship between COVID-19 diagnosis and each risk factor included in this analysis is described in the correlation matrix Figure 1D.

The model was deployed using online using Shiny, an interactive web based application framework for R statistical software (see Figure 2) (13).

FIGURE 2

Figure 2. Screenshot of the deployed web-app which is freely available for use online at https://darasriskcalculators.shinyapps.io/Covid_app_x4/.

Discussion

To our knowledge, we are the first in the world to develop and describe such a risk tool to predict the probability of a positive test for SARS-CoV-2. As this pandemic continues to progress and our healthcare system is put under increasing pressure, this novel, world-first risk tool has the potential to aide clinicians, patients and healthcare systems in the response to the COVID-19 pandemic.

The Value of This Risk Tool and the Significance of Negative Predictive Value Being ~96%

The value of this model, is that not every patient that presents with symptoms suggestive of COVID-19 will test positive; neither will every patient that tests positive require inpatient hospital care (14). While up to 18% of patients infected with SARS-CoV-2 remain asymptomatic, most who do become symptomatic and test positive, will be discharged to their home (15), and only certain patients will need admission (16).

This risk tool could be used as an aide for triage, initial assessment or the first telehealth visit; patients who receive a result of >0.7 (70%) are most likely to test positive for SARS-CoV-2. If they are otherwise well, they can continue to be managed at home, and observe protocols to reduce transmission to others. Follow up telehealth visits can be arranged as necessary.

If a patient receives a result of <0.7 (<70%), they are most likely to test negative for SARS-CoV-2; our results demonstrate that 96% patients in the holdout dataset did in fact test negative (see NPV in Table 2). They can be reassured that they do not need to increase their risk for getting infected by leaving their home unnecessarily. As patients in this cohort did present with minimal criteria for a suspected diagnosis, it is unlikely that their symptoms are due to a COVID-19 diagnosis. Their symptoms therefore are more likely due to another cause such as an underlying ailment, seasonal allergies or influenza. Amidst this pandemic, it is important to remember that influenza and other infections and conditions are still claiming lives. Influenza has a mortality rate of 2/100,000 (17), and the management of patients who are otherwise not in respiratory distress and hemodynamically stable, is, similar to COVID-19: symptomatic management at the patient's home, liaising with their primary care physician and healthcare team.

The negative predictive value of this model is 96% when a threshold risk of 70% is applied. This compares favorably with other widely used medical diagnostics such as standardized extended pattern template biopsy of the prostate, which has a NPV of 25–31% (18), and ultrasound and CT for suspected appendicitis which have respective NPVs of 76 and 95% (19).

The Significance of the Positive Predictive Value Being 66%

At a threshold risk of 70%, this risk tool has a positive predictive value of 66%: meaning that for every 100 people told that they are likely to have a positive test, 34 will not test positive for Covid-19, while 66 will test positive. The management for many in this group, will be at home; where they should reduce social contacts, and continue transmission mitigation efforts while closely monitoring symptoms and signs. The 47 who will not test positive, will benefit from these measures and help to further flatten the curve. The PPV of this risk tool is greater than the PPV of the criteria applied; the clinical suspicion which led to the COVID-19 test was 49% in the holdout cohort (see 0% risk threshold in Table 2). Placing this figure into context, the PPV of a PSA of 4 ng/ml to predict prostate cancer is ~14–37%, and mammography has a PPV of between 3.3 and 28.6% (20, 21).

The net impact of this is best demonstrated on the decision curve analysis (Figure 1C): Where you can see the net benefit of using this score is greater than the strategies of testing everybody, and testing nobody. It is an informed decision aide, and offers an alternative to the strategy of testing everybody in person, or the strategy of testing nobody at all (Figure 1C).

Reduce the Burden on Healthcare System, and Reduce Unnecessary Exposures to Patients and Healthcare Workers

As the extent of SARS-CoV-2 continues to spread, other regions and healthcare facilities will come under increased strain on available resources including assessing patients to ensure they warrant testing, administering, transporting, performing and reporting these tests; all of which place patients and healthcare workers at an increased risk of infecting others or being infected. The effectiveness of PPE and prophylaxis was estimated to be ~79% during a previous outbreak (22). We have demonstrated that at a risk cut-off of 70%, testing volume can be reduced by 75%. Using such as solution as this as a triage aide, cannot just reduce the testing burden but in doing so, reduce the exposure risk to patients by reducing unnecessary visits to testing sites.

Algorithms Are Becoming More Used in Clinical Practice

The use of such a risk tool would not be unprecedented. Numerous algorithms have recently received regulatory approval for broad clinical use (23). The WAVE Clinical Platform is an early warning system integrating real-time vital sign data to identify hospital inpatients who are at risk of vital sign instability and was approved by the FDA in 2018. Since then the FDA has granted clearance for similar algorithms in a number of fields including diagnostics. The successful implementation of such tools has also been documented in other healthcare systems such as the National Health Service in the UK (24). Risk tools to identify those most likely to require ventilatory support or intensive care unit admission would also be of support to clinical decision making.

Does Patient Behavior Change When They Know They Test Positive for an Infectious Disease?

Behavior changes associated with a positive test for other infectious diseases has been well-documented (25–28). Risk perception of infection has been demonstrated to be a predictor of a range of preventative behaviors. SARS-CoV-2 is unlike influenza in that there are no mechanisms currently available for SARS-CoV-2 prophylaxis. However, in a study on perceptions related to Avian Influenza, Lau et al. identified that those who perceived they had symptoms similar to influenza were more than four times as likely to wear a face mask, than those who did not (29).

Rudisill et al. documented behavioral changes during the course of the H5N1 avian influenza pandemic; noting that residence in a nation in which H5N1 had been found in humans had a significant and positive relationship with being less likely to consume poultry, eggs and egg-related products, whereas in nations where H5N1 was present, but not in humans, there was limited influence on these behaviors (30).

In the absence of PCR and antibody testing for SARS-CoV-2, an approach such as a freely available online risk tool, could be used to help reinforce compliance with positive behaviors which are associated with reduced transmission of SARS-CoV-2.

Limitations

This dataset is comprised of a cohort of patients with an encounter at a single healthcare system's facility in NYC, who were screened for SARS-CoV-2 infection, and only cases where complete data was available were included (44% of cohort). Therefore, there is an inherent selection bias in this dataset; and in effect it should be used in a similar cohort: those cohorts with a similar social and economic demographic composition to the greater New York City catchment area, who are symptomatic or high risk for COVID-19 infection and presenting to a US healthcare institution. The molecular based assays used in this cohort to diagnose SARS-CoV-2 infection while used widely, are not used universally; and have sensitivity and specificities of >95%; it should be noted that rapid antigen based tests are also used in this setting, and while highly specific, on average they have a lower sensitivity (56%) (31). It is important to note that such a risk tool is for use in symptomatic patients, and does not address those patients who would test positive for COVID-19 but remain asymptomatic. It has been designed as an aide to clinical decision making, and not a replacement for it.

As the pandemic continues to overwhelm healthcare systems, risk tools such as the one described in this paper have the potential to aide clinicians and healthcare institutions. They can more accurately risk stratify patients, identifying those most likely to test positive, and those most likely to test negative for SARS-CoV-2 and aide in the strategic allocation of testing resources and the response of health systems to this pandemic.

Conclusion

We have developed a risk model which can predict the outcome of SARS-CoV-2 testing which is accurate and reliable. It offers an instant quantification of risk, and is available online for free. Such a tool could be used to help improve compliance with transmission mitigation strategies, reduce the workload and burden on health systems and help reduce exposures associated with unnecessary visits to testing sites.

Data Availability Statement

The data analyzed in this study is subject to the following licenses/restrictions: De-identified dataset was made available by the Mount Sinai Scientific Computing. Requests to access these datasets should be directed to Ashutosh Tewari, ash.tewari@mountsinai.org.

Ethics Statement

The studies involving human participants were reviewed and approved by Mount Sinai Hospital Ethics Committee. Written informed consent for participation was not required for this study in accordance with the national legislation and the institutional requirements.

Author Contributions

DL and AT: concept, literature search, study design, data analysis, data interpretation, writing, and figures. BK: concept, literature search, study design, data analysis, data interpretation, and writing. SN: literature search and data interpretation. DB: data interpretation and writing. GP and DR: literature search, data interpretation, and writing. All authors contributed to the article and approved the submitted version.

Conflict of Interest

DR is President and Chief Operating Officer of The Mount Sinai Hospital and President of Mount Sinai Queens.

The remaining authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Acknowledgments

We would like to acknowledge the contributions of the following, without whom this work would not have been possible: The staff of Mount Sinai Hospitals for their dedication and care in documenting the care of and caring for patients; the Department of Scientific Computing at the Icahn School of Medicine at Mount Sinai for the collation of this data and the Department of Pathology, Molecular and Cell-Based Medicine and Department of Microbiology for their work on developing a quantitative assay for measuring viral load of SARS-CoV-2.

References

1. World Health Organisation Coronavirus disease (2019). (COVID-19) Situation Report −72. Available online at: www.who.int (2020).

Google Scholar

2. Day M. Covid-19: identifying and isolating asymptomatic people helped eliminate virus in Italian village. BMJ. (2020) 368:m1165. doi: 10.1136/bmj.m1165

PubMed Abstract | CrossRef Full Text | Google Scholar

3. Adalja AA, Toner E, Inglesby TV. Priorities for the US Health Community Responding to COVID-19. JAMA. (2020) 323:1343–4. doi: 10.1001/jama.2020.3413

PubMed Abstract | CrossRef Full Text | Google Scholar

4. Siegler K. Many Who Need Testing For COVID-19 Fail To Get Access. Washington, DC: National Public Radio. NPR.org (2020).

Google Scholar

5. Laura Strickler AK. Private Labs do 85 Percent of U.S. COVID-19 Tests But Still Struggle With Backlogs, Shortages. nbcnews.com. New York, NY (2020).

Google Scholar

6. Ellis B, Hicken M, Fantz A. Coroners worry Covid-19 test shortages could lead to uncounted deaths, CNN.com, (2020).

Google Scholar

7. Roobol MJ, van Vugt HA, Loeb S, Zhu X, Bul M, Bangma CH, et al. Prediction of prostate cancer risk: the role of prostate volume and digital rectal examination in the ERSPC risk calculators. Eur Urol. (2012) 61:577–83. doi: 10.1016/j.eururo.2011.11.012

PubMed Abstract | CrossRef Full Text | Google Scholar

8. Lundon DJ, Kelly BD, Foley R, Loeb S, Fitzpatrick JM, Watson RW, et al. Prostate cancer risk assessment tools in an unscreened population. World J Urol. (2015) 33:827–32. doi: 10.1007/s00345-014-1365-7

PubMed Abstract | CrossRef Full Text | Google Scholar

9. Cuomo AM, Zucker HA, Dreslin S. 2019 Novel coronavirus (COVID-19) interim guidance: protocol for COVID-19 testing applicable to all health care providers and local health departments (LHD). New York State Health Do. Albany, NY. (2020). Available online at: health.ny.gov.

Google Scholar

10. R Core Team R: A Language and Environment for Statistical Computing. Vienna: R Foundation for Statistical Computing.

Google Scholar

11. DeLong ER, DeLong DM, Clarke-Pearson DL. Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics. (1988) 44:837–45. doi: 10.2307/2531595

PubMed Abstract | CrossRef Full Text | Google Scholar

12. Vickers AJ, Elkin EB. Decision curve analysis: a novel method for evaluating prediction models. Med Decis Making. (2006) 26:565–74. doi: 10.1177/0272989X06295361

PubMed Abstract | CrossRef Full Text | Google Scholar

13. Chang W, Cheng J, Allaire J, Xie Y, McPherson J. Shiny: Web Application Framework for R. R package version 0.13 (2016).

Google Scholar

14. World health Organization. Novel Coronavirus (2019-nCoV) Technical Guidance: Patient Management. Stockholm: WHO.

Google Scholar

15. Mizumoto K, Kagaya K, Zarebski A, Chowell G. Estimating the asymptomatic proportion of coronavirus disease 2019 (COVID-19) cases on board the Diamond Princess cruise ship, Yokohama, Japan, 2020. Euro Surveill. (2020) 25. doi: 10.2807/1560-7917.ES.2020.25.10.2000180

PubMed Abstract | CrossRef Full Text | Google Scholar

16. Centers for Disease Control and Prevention. Interim Clinical Guidance for Management of Patients with Confirmed 2019 Novel Coronavirus (2019-nCoV) Infection. CDC (2020). Available online at: https://www.cdc.gov/coronavirus/2019-ncov/hcp/clinical-guidance-management-patients.html (accessed April 18, 2020).

Google Scholar

17. Centers for Disease Control and Prevention. Influenza. cdc.gov. Atlanta, GA (2015).

Google Scholar

18. Schulte RT, Wood DP, Daignault S, Shah RB, Wei JT. Utility of extended pattern prostate biopsies for tumor localization: pathologic correlations after radical prostatectomy. Cancer. (2008) 113:1559–65. doi: 10.1002/cncr.23781

PubMed Abstract | CrossRef Full Text | Google Scholar

19. Old JL, Dusing RW, Yap W, Dirks J. Imaging for suspected appendicitis. Am Family Phys. (2005) 71:71–8. Available online at: https://www.aafp.org/afp/2005/0101/p71.html

Google Scholar

20. Vollmer RT. Predictive probability of serum prostate-specific antigen for prostate cancer. Am J Clin Pathol. (2006) 125:336–42. doi: 10.1309/R5H6VUQ32KGJW448

PubMed Abstract | CrossRef Full Text | Google Scholar

21. Lehman CD, Arao RF, Sprague BL, Lee JM, Buist DS, Kerlikowske K, et al. National performance benchmarks for modern screening digital mammography: update from the breast cancer surveillance consortium. Radiology. (2017) 283:49–58. doi: 10.1148/radiol.2016161174

PubMed Abstract | CrossRef Full Text | Google Scholar

22. te Beest DE, van Boven M, Bos ME, Stegeman A, Koopmans MP. Effectiveness of personal protective equipment and oseltamivir prophylaxis during avian influenza A (H7N7) epidemic, the Netherlands 2003. Emerg Infect Dis. (2010) 16:1562–8. doi: 10.3201/eid1610.091412

PubMed Abstract | CrossRef Full Text | Google Scholar

23. Parikh RB, Obermeyer Z, Navathe AS. Regulation of predictive analytics in medicine. Science. (2019) 363:810–2. doi: 10.1126/science.aaw0029

CrossRef Full Text | Google Scholar

24. Fullerton JN, Price CL, Silvey NE, Brace SJ, Perkins GD. Is the Modified Early Warning Score (MEWS) superior to clinician judgement in detecting critical illness in the pre-hospital environment? Resuscitation. (2012) 83:557–62. doi: 10.1016/j.resuscitation.2012.01.004

PubMed Abstract | CrossRef Full Text | Google Scholar

25. Carlos S, Nzakimuena F, Reina G, Lopez-Del Burgo C, Burgueno E, Ndarabu A, et al. Factors that lead to changes in sexual behaviours after a negative HIV test: protocol for a prospective cohort study in Kinshasa. BMC Public Health. (2016) 16:606. doi: 10.1186/s12889-016-3285-5

PubMed Abstract | CrossRef Full Text | Google Scholar

26. Cleary PD, Van Devanter N, Rogers TF, Singer E, Shipton-Levy R, Steilen M, et al. Behavior changes after notification of HIV infection. Am J Public Health. (1991) 81:1586–90. doi: 10.2105/AJPH.81.12.1586

PubMed Abstract | CrossRef Full Text | Google Scholar

27. Goodwin R, Gaines SO Jr, Myers L, Neto F. Initial psychological responses to swine flu. Int J Behav Med. (2011) 18:88–92. doi: 10.1007/s12529-010-9083-z

PubMed Abstract | CrossRef Full Text | Google Scholar

28. Kalichman SC, Cain D, Simbayi LC. Behavioral changes associated with testing HIV-positive among sexually transmitted infection clinic patients in Cape Town, South Africa. Am J Public Health. (2010) 100:714–9. doi: 10.2105/AJPH.2009.162602

PubMed Abstract | CrossRef Full Text | Google Scholar

29. Lau JT, Kim JH, Tsui HY, Griffiths S. Perceptions related to bird-to-human avian influenza, influenza vaccination, and use of face mask. Infection. (2008) 36:434–43. doi: 10.1007/s15010-008-7277-y

PubMed Abstract | CrossRef Full Text | Google Scholar

30. Rudisill C, Costa-Font J, Mossialos E. Behavioral adjustment to avian flu in Europe during spring 2006: the roles of knowledge and proximity to risk. Soc Sci Med. (2012) 75:1362–71. doi: 10.1016/j.socscimed.2012.06.005

PubMed Abstract | CrossRef Full Text | Google Scholar

31. Dinnes J, Deeks JJ, Adriano A, Berhane S, Davenport C, Dittrich S, et al. Rapid, point-of-care antigen and molecular-based tests for diagnosis of SARS-CoV-2 infection. Cochrane Database Syst Rev. (2020) 8:CD013705. doi: 10.1002/14651858.CD013705

PubMed Abstract | CrossRef Full Text | Google Scholar