Diagnostic Performance of Rapid Antigen Testing for SARS-CoV-2: The COVid-19 AntiGen (COVAG) study

Introduction

On December 31, 2019, the Municipal Health Commission of Wuhan in Hubei, China, reported a series of cases of pneumonia with unknown etiology (1). The Chinese Centre for Disease Control and Prevention (CCDC) described severe acute respiratory syndrome-related coronavirus-2 (SARS-CoV-2) as the causative agent (2–4), which then quickly spread worldwide. Severe cases of SARS-CoV-2 infection are associated with a substantial risk of prolonged critical illness and death (5).

Because the virus can be spread by asymptomatic, pre-symptomatic, and symptomatic carriers, public health experts have recommended fast and accurate testing, followed by the identification and monitoring of positive cases and subsequent self-isolation and contact tracing to contain the spread of SARS-CoV-2. Direct detection of SARS-CoV-2 is achieved by identifying viral RNA in specimens from the respiratory tract of patients utilizing nucleic acid amplification tests (NAATs) or by recognizing viral proteins through antigen-detecting rapid diagnostic tests (Ag-RDTs). The NAAT-based assays, such as real-time reverse transcription-polymerase chain reaction (rRT-PCR) have become the “gold standard” for establishing the presence of SARS-CoV-2 in patients (6). However, rRT-PCR-based testing takes several hours and is conducted in specialized laboratories, which are usually located away from sample collection sites. This may produce long turnaround times, resulting in delayed self-isolation, a risk of more contacts, and further potential transmission. Therefore, rapid antigen tests for SARS-CoV-2 detection have been made commercially available. Ag-RDTs can be conducted at the point of care and the results visualized after 15-30 min (7). There is common consensus that positive Ag-RDT results must be verified by rRT-PCR testing. Studies have indicated that the antigen tests' analytical sensitivities vary between 25 and 50% for rRT-PCR-positive samples, which may increase to more than 80% for samples with a higher viral load (8–10). We set out to comprehensively examine two of the most sensitive (11) and widely used commercial RDTs in a real-world, prospective, head-to-head study, placing specific emphasis on clinical characteristics, COVID-19-associated symptoms, and the presence of SARS-CoV-2 variant genotypes.

Methods

Setting and Participants

This prospective study was conducted at the Corona Test Centre Cannstatter Wasen in Stuttgart, Germany. Individuals scheduled for rRT-PCR testing of nasopharyngeal swabs were advised of the study orally and in writing. Participants had to be aged ≥ 18 years and capable of understanding the nature, significance, and implications of the study. Children and adolescents <18 years of age and patients obviously suffering from clinical conditions requiring emergency hospitalization were excluded. All participants provided written and informed consent. The study was approved by Ethics Committee II (Mannheim) of the University of Heidelberg (reference number 2020-417MF) and the German Institute for Drugs and Medical Devices.

We recorded demographic characteristics, reasons for testing, medical history, major risk factors, acute symptoms, and vital signs, including heart rate, blood pressure, body temperature, and oxygen saturation and we divided the reasons for testing into three major categories: participants referred by their primary care physicians, by the Health Department and participants who were tested for other reasons. We also framed our data collection and the emergence of variants into the course of the second and third wave of the COVID-19 pandemic in Germany (Figure 1).

FIGURE 1

Figure 1. Framing of the COVAG study (February 1-March 31) into the time course of the COVID-19 pandemic in Germany. Abszissa: calendar week within 2021; ordinate: Germany-wide weekly incidence rate of SARS-Cov-2 infections per 100,000 inhabitants; bars: weekly incidence rates of infections with SARS-CoV-2 variant B.1.1.7 (alpha, blue), variant B.1.617.2 (delta, red), and wild-type or other variants (NIM-type, gray). Blue solid line: proportion of variant B.1.1.7 (alpha) in the COVAG study extrapolated to Germany.

Sociodemographic characteristics of the study participants were not recorded in detail upon enrollment. As a proxy, we created a sociodemographic map. For this purpose, postal codes of the study participants at baseline were merged with freely accessible data of the Robert Koch Institute (RKI, Berlin, Germany) for the German Index of Socio-Economic Deprivation (GISD) (12–14), which describes the regional socio-economic deprivation in Germany. It summarizes the extent of the socio-economic disadvantages of regions in the dimensions of education, employment, and income. The higher the GISD, the higher the deprivation. For geographic data (shape files) and regional representation, freely accessible geodata from the German Federal Office for Cartography and Geodesy (14, 15) were used. Postal codes and GISD scores were merged with the shape files with the software R version 4.1.1. and the R packages “leaflet” and “sp”; the spatial representation of a choropleth map was created to assign a GISD score to each study participant. The GISD score was represented at the district level by different colors in the choropleth map (Supplementary Figure 1).

In addition to collecting the oro- and nasopharyngeal swab for rRT-PCR testing, we collected two completely independent nasopharyngeal swab specimens to run two commercially available and widely used Ag-RDTs. The swabs were collected by medically educated personnel of the test center in changing teams with strict adherence to the instructions issued by the manufacturer. We used the Abbott Panbio^TM COVID-19 Ag Rapid Test (Abbott Rapid Diagnostics Jena GmbH, Jena, Germany, www.abbott.com/poct) and the Roche-SD Biosensor SARS-CoV-2 Rapid Antigen Test (identical to SD BIOSENSOR Standard Q COVID-19 Ag, www.sdbiosensor.com; Roche Diagnostics; Mannheim, Germany, www.roche.com). We chose the two tests because they were widely available and used in Germany, and because smaller studies suggested acceptable sensitivity (16–20) compared to others. While this study was ongoing, a Cochrane analysis was published which identified the SD Biosensor STANDARD Q and the Abbott Panbio test as the most sensitive amongst many others (11).

Hereafter, we refer to the tests as Abbott-RDT and Roche-RDT, respectively. We randomly assigned the participants to three sampling groups according to the time sequence of collecting the nasopharyngeal swabs (group 1: rRT-PCR, RDT-Roche, RDT-Abbott; group 2: RDT-Roche, RDT-Abbott, rRT-PCR; and group 3: RDT-Abbott, rRT-PCR, RDT-Roche).

Analytical Procedures

Both the Abbott-RDT and the Roche-RDT were carried out by medically educated staff according to the manufacturers' instructions on-site at the Corona Test Centre Cannstatter Wasen, Stuttgart, Germany, immediately after sampling the nasopharyngeal swabs. The nasopharyngeal swabs for rRT-PCR testing were placed in 2 ml of a phosphate-buffered saline solution (ISOTON™ II Diluent, Becton Dickinson, Galway, Ireland) and delivered to the SYNLAB Medical Care Centre Leinfelden-Echterdingen. This ensured that the performers of the RDTs were unaware of the rRT-PCR-results.

SARS-CoV-2 RNA was extracted from the nasopharyngeal swab samples and purified using the PurePrep Pathogens kit and a PurePrep 96 instrument (Molgen, Veenendaal, the Netherlands) within 6 h after sampling to limit degradation. The integrity of the RNA was monitored by co-amplification of a control RNA included in the solution for the lysis of the swabs. In cases in which neither SARS-CoV-2 RNA nor the control RNA were detected, the RNA preparation was repeated. The rRT-PCR assay was performed using the RIDA®GENE SARS-CoV-2 test kit (R-Biopharm, Darmstadt, Germany) on the CFX96 Touch Real-Time PCR detection device (Bio-Rad, Feldkirchen, Germany) according to the manufacturers' instructions. This test kit targets the SARS-CoV-2 2 envelope (E) gene; samples producing a cycle threshold (Ct) ≤ 35 were considered positive by rRT-PCR.

We screened rRT-PCR-positive samples for SARS-CoV-2 variants of concern (VOC) B.1.1.7 (Alpha, United Kingdom), B.1.351 (Beta, South Africa), and P.1 (Gamma, Brazil) using VirSNiP SARS-CoV-2 Spike N501Y and VirSNiP SARS-CoV-2 Spike del H69/V70 from TIB Molbiol (Berlin, Germany) according to the supplier's instructions. Genotyping was restricted to samples with a Ct ≤ 30 in rRT-PCR (260 of 338 samples). Samples with positive results for both the N501Y substitution and H69/V70 deletion were assigned to the Alpha variant. The presence of N501Y and absence of H69/V70 was considered the Beta or Gamma variant. Samples in which both N501Y and H69/V70 were absent may have contained variants other than Alpha, Beta, or Gamma or the wild-type. For the purpose of this article, these samples were termed negative for investigated mutations (NIM).

Statistical Analysis

Continuous data are presented as the mean, standard deviation (SD), median, and 25th and 75th percentiles. Categorical data are presented as absolute numbers and percentages. The risk of having COVID-19 according to baseline anthropometric and anamnestic characteristics was expressed in terms of crude odds ratios (ORs) and ORs adjusted for age and sex as calculated by logistic regression (Supplementary Table 1). Sensitivity, specificity, positive predictive values (PPVs), negative predictive values (NPVs), and diagnostic efficacy [(number of true positive plus true negative tests) divided by the total number of tests] of the two RDTs compared to rRT-PCR were calculated (Table 2). These performance indicators were compared between the Abbott-RDT and Roche-RDT. The P-value refers to two-sided testing of the null hypothesis, that the difference between the respective performance indicators is equal to zero and is based on 5,000 bootstrap iterations and subsequent percentile analysis. We also visualized the sensitivities of both RDTs relative to the rRT-PCR-derived Ct values (Figure 2) and the PPVs and NPVs according to hypothetical disease prevalence rates in the range of 0-0.05 (Figure 3). To compare the PPV and NPV of the RDTs with standardized criteria on performance, we also used the following hypothetical sensitivity and specificity levels (tiers 1-3) recommended by Kost (21): tier 1, 90%, 95%; tier 2, 95%, 97.5%; and tier 3, 100%, ≥99% (Figure 3).

FIGURE 2

Figure 2. Relationships between the sensitivities of RDTs vs. rRT-PCR cycle threshold (Ct) values. The solid lines indicate sensitivities, the dotted lines represent the upper, and the lower bounds the corresponding 95% confidence intervals. (A) left: Roche-RDT; right: Abbott-RDT. (B) Sensitivities according to SARS-CoV-2 genotypes. left: Roche-RDT; right: Abbott-RDT; red: NIM genotype; blue: SARS-CoV-2 Alpha variant.

FIGURE 3

Figure 3. Predictive values of positive tests (PPV, left) and predictive values of negative tests (NPV, right) of two commercial RDTs for SARS-CoV-2-associated antigens in relation to disease prevalence rates up to 0.05. Black dotted line: hypothetical sensitivity and specificity of 90 and 95% (tier 1); magenta dotted line: hypothetical sensitivity and specificity of 95 and 97.5% (tier 2); green dotted line: hypothetical sensitivity and specificity of 100 and 99% (tier 3). (A) Red solid line: Abbott-RDT; blue solid line: Roche-RDT; (B) stratified according symptomatic and asymptomatic study participants. Red solid line: symptomatic participants, Abbott-RDT; blue solid line: symptomatic participants, Roche-RDT; magenta solid line: asymptomatic participants, Abbott-RDT; light blue solid line: asymptomatic participants, Roche-RDT.

Finally, we investigated whether the sensitivities of the two RDTs were related to the reason for testing, comorbidities, clinical symptoms, vital signs, or SARS-CoV-2 genotypes using univariate (Table 2) and multivariate logistic regression (Table 3).

The statistical tests were two-sided and P < 0.05 was considered significant. The analyses were carried out using R v4.0.2 (http://www.r-project.org).

Results

Clinical Characteristics of Participants

The study was conducted between February 1, 2021, and March 31, 2021. During this period, nearly 17,000 adult persons attended the test centre Cannstatter Wasen to receive an rRT-PCR test. A total of 2,222 persons agreed to participate in the study. Seven of them were disregarded for further evaluation because at least one of the three tests was not available. This resulted in 2,215 persons with valid data (Table 1). Adverse events from performing any of the tests were not experienced.

TABLE 1

Table 1. Clinical characteristics of the 2215 participants in the COVAG study.

Figure 1 shows our data collection period within the frame of the Germany-wide weekly incidence rates and the proportion of the NIM-type, alpha (B.1.1.7) and delta (B.1.617.2) variants. During calendar weeks 9 through 13 the proportions of the B.1.1.7 in the COVAG study and in the whole of Germany completely coincided.

707 (32%) participants were referred by their primary care physicians, 962 (43%) sought testing following the advice of the Health Department and were mostly incriminated contact persons of COVID-19 patients, and 546 (25%) participants were tested for miscellaneous reasons (264 following a warning message from the German Corona App, 140 out-of-pocket payers, 82 kindergartners or teachers, and 60 for another reason, including 2 for confirmation of a positive RDT and 2 cluster students in quarantine).

When we attempted to correlate the German Index of Socio-Economic Deprivation (GISD) which is a proxy for the socioeconomic status with the risk of COVID-19, as shown in Supplementary Figure 1, 1,879 out of the 2,215 (85%) participants had a GISD between 0.55 and 0.60. Evidently, the variance of the GISD was too low to afford further analyses.

Hypertension, dyslipoproteinaemia, and diabetes mellitus were self-reported at rates of 12, 5, and 2%, respectively. Chronic obstructive lung disease and ischaemic heart disease were comparatively low in frequency. Overall, comorbidities occurred more often in men than in women (Table 1).

The most often reported clinical symptoms were malaise, headache, and musculoskeletal pain (Table 1). Symptoms were significantly more frequent in women than in men. Systolic and diastolic blood pressures were markedly and significantly higher in men than in women, but body temperature was slightly higher in women than in men (Table 1).

Risk of rRT-PCR-Proven SARS-CoV-2 Infection According to Baseline Characteristics

Among the 2,215 participants, 338 carried SARS-CoV-2 based on rRT-PCR. Age and sex were not related to the likelihood of testing positive for SARS-CoV-2 (Supplementary Table 1). Participants referred by treating physicians and health departments were positive significantly more often than participants with miscellaneous reasons for testing [OR 0.22, 95% confidence interval (CI) 0.14-0.34, adjusted for age and sex and compared to referrals from physicians].

Persons with at least one comorbidity more frequently tested positive than those without comorbidity (OR 2.94, 95% CI 2.25-3.83, adjusted for age and sex). Among the individual comorbidities, dyslipoproteinaemia, diabetes mellitus, and ischaemic heart disease significantly increased the probability of a positive rRT-PCR test.

The presence of at least one clinical symptom at presentation resulted in a higher frequency of positive rRT-PCR (OR 2.91, 95% CI 2.28-3.71, adjusted for age and sex). In addition, each of the individual symptoms was positively and significantly related to the rate of SARS-CoV-2 detected by rRT-PCR. the objectively measured vital signs (blood pressure, body temperature, and oxygen saturation), only elevated body temperature was associated with the probability of COVID-19.

Diagnostic Performance of RDTs

Sensitivity. The Roche-RDT and Abbott-RDT had overall sensitivities of 60.4 and 56.8%, respectively (P < 0.0001, Table 2). Figure 2A shows that the sensitivities of both RDTs were strongly related to the Ct values derived from rRT-PCR. Only at Ct values <20 did both RDTs reach a sensitivity of 100%.

TABLE 2A

Table 2A. Diagnostic performance of two commercial RDTs for SARS-Cov-2 antigen (part 1).

TABLE 2B

Table 2B. Diagnostic performance of two commercial RDTs for SARS CoV-2 antigen (part 2).

We further examined whether the sensitivities of the two RDTs were significantly different in subgroups (Table 2). The overall difference in sensitivity between the Roche-RDT and Abbott-RDT may be due to participants below the median age (P < 0.0001) rather than those above the median age (P = 0.199, Table 2). Among participants referred by physicians, the sensitivities of both RDTs were substantially higher than in the total study population (79.5 and 78.7%, Roche-RDT and Abbott-RDT, respectively), but did not differ significantly. In contrast, they were <50% in persons referred by the Health Department and those tested for other reasons, whereby the Roche-RDT appeared to perform better than the Abbott-RDT (Table 2).

Sensitivities were markedly lower in persons with at least one comorbidity (38.2 and 34.4%, Roche-RDT and Abbott-RDT, respectively, P = 0.005) than in persons without comorbidities (74.4 and 71.0%, Roche-RDT and Abbott-RDT, respectively, P = 0.004, Table 2). This also applied to the individual comorbidities, with the one exception that the sensitivities of both the Roche-RDT and Abbott-RDT were 86% in the small number of participants reporting ischaemic heart disease (for detailed information refer to Supplementary Table 2). This unexpected finding is consistent with the Ct values being markedly higher in individuals with comorbidities (Table 2).

In persons with at least one clinical symptom, the sensitivities of both RDTs were higher (75.2 and 74.3%, Roche-RDT and Abbott-RDT, respectively, not significant) than in persons without clinical symptoms (31.9 and 23.3%, Roche-RDT and Abbott-RDT, respectively, P < 0.0001). The presence of any of single symptom augmented the sensitivities of both RDTs, with the exception of shortness of breath and diarrhea, which were not related to sensitivity (for detailed information refer to Supplementary Table 2). This finding is also in line with the Ct values, which were lower in cases with symptoms than in those without (Table 2).

We also analyzed whether the SARS-CoV-2 genotype affects the sensitivities of the RDTs. Only samples with Ct values ≤ 30 (n = 286) were included in this evaluation. The NIM SARS-CoV-2 and Alpha variant (B.1.1.7) were present in 81 and 166 samples with Ct values ≤ 30, respectively. The NIM genotype was detected at sensitivities of 87.7 and 84.0% (Roche-RDT vs. Abbott-RDT, respectively, not significant). In carriers of the Alpha variant, sensitivities were 77.1 and 72.3% (Roche-RDT vs. Abbott-RDT, respectively, P < 0.002). At any given Ct value, the sensitivities of both RDTs were lower for the Alpha variant than for the NIM genotype (Figure 2B).

To firmly establish independent predictors of sensitivity, we calculated ORs for having a positive RDT according to subgroups in multivariate logistic regression. Covariables were age, gender, reason for testing, presence or absence of any comorbidity, presence of absence of any clinical symptom, and the SARS-CoV-2 genotype. As expected, Ct values were associated with sensitivities of both tests. The sensitivities of the Abbott-RDT and Roche-RDT were higher in symptomatic than in asymptomatic individuals. Remarkably, the sensitivities of both tests were significantly lower for the Alpha variant than for the NIM genotypes (Table 3).

TABLE 3

Table 3. Predictors of positive RDTs amongst SARS-CoV-2-rRT-PCR-positive samples in multivariate logistic regression models^*.

Specificity. The rate of false-positive RDTs was low. With both RDTs, specificity exceeded 99% overall and in mostly all participant strata (Table 2, Supplementary Table 2).

PPV, NPV, and diagnostic performance. At a prevalence rate of 15% in the study population, the PPVs of the two RDTs were 98 and 99% and within the range of 90-100% in all subgroups examined. The NPVs of the RDTs were approximately 93%. Diagnostic efficiency also ranged between 90 and 100% (Table 2, Supplementary Table 2).

Because patients with SARS-CoV-2 infections were enriched in our study population compared to the general population, we examined the PPVs and NPVs at assumed prevalence rates up to 0.05 (Figure 3A). At this prevalence rate, our results suggest a PPV and NPV of 96.6 and 97.8% for Abbott-RDT, and 92.3 and 98.0% for the Roche-RDT, the Abbott-RDT displaying a higher PPV than the Roche-RDT and both scoring higher than the hypothetical tiers 1 through 3, reflecting increases in sensitivity in the order of Abbott-RDT < Roche-RDT < tier 1 < tier 2 < tier 3. The NPVs ranged in the order of tier 3 > tier 2 > tier 1 > Roche-RDT > Abbott-RDT.

In symptomatic persons, the PPVs and NPVs at the actual prevalence within the study were 98.2 and 93.2% for the Roche-RDT and 98.8 and 92.9 % for the Abbott-RDT (Table 2). In asymptomatic persons, the respective figures were 94.9 and 93.4% for the Roche-RDT and 98.2 and 92.7% for Abbott-RDT (Table 2). Figure 3B shows PPVs and NPVs as functions of the disease prevalence and stratified according to the presence or absence of clinical symptoms. As expected, PPVs were highest and NPV lowest in asymptomatic patients.

Discussion

We completed one of the largest prospective evaluations of RDTs for SARS-CoV-2-associated antigens in a real-world environment to date. We evaluated two of the most sensitive (11) contemporary lateral-flow devices provided by Roche Diagnostics and Abbott Diagnostics. We found that the Abbott-RDT and Roche-RDT had significantly different sensitivities, but they were inversely and strongly related to the rRT-PCR-derived Ct values. In unadjusted examinations, the RDTs had higher sensitivity in individuals referred from treating physicians and health departments than in those tested for other reasons, in individuals without comorbidities compared to those with comorbidities, in individuals presenting with clinical symptoms and fever, and in carriers of the SARS-CoV-2 NIM genotype compared to carriers of the Alpha variant. Of note, the significant negative association between positivity of RDT results and presence of the alpha variant remains statistically significant when we considered age, sex, Ct-value, reason for testing, presence of comorbidities and presence of clinical symptoms in the same logistic regression model. This implies that the RDTs are less sensitive toward the alpha variant even adjusting for other conditions affecting sensitivity.

Prevalence of SARS-CoV-2 Infections in the Study Cohort

Among the attendees of our corona test center, the rate of individuals testing positive for SARS-CoV-2 infection by rRT-PCR was 15%, which markedly exceeds the prevalence rate in the German population during the study (~0.3% estimated on the basis of 7-day incidence rates during the study period). The probability of testing positive by rRT-PCR was not related to age and sex. This is in contrast to the expected over-representation of older people and may reflect pre-selection for persons having an evident clinical indication for testing. The probability of testing positive by rRT-PCR was strongly linked to individual reasons for testing, to the presence of comorbidities, clinical complaints, and elevated body temperature. For example, the persons referred by physicians due to suspected COVID-19 or those coming from the Health Department were more often positive than the group with miscellaneous reasons (kindergartners/teachers, out-of-pocket payers). Yet, in the latter group (prevalence rate 0.044), the proportion of SARS-CoV-2 carriers was still 10-fold higher than in the general population (assumed prevalence 0.003).

Sensitivity of the RDTs

Expectedly, both RDTs had higher sensitivities in subgroups with high viral loads (referral by physicians and health departments, clinical symptoms). However, both sensitivities and viral loads were low in patients presenting with comorbidities. This is unexpected and may reflect a referral bias in the sense that the indication for testing is more frequent and earlier in patients at high risk for severe COVID-19.

The relationship between the RDTs' analytical sensitivity and viral load is in line with reports of sensitivities between 24.3 and 50% for RT-PCR-positive samples, which increased up to 81.8 and 100% for samples with high viral loads (>6 log₁₀ RNA copies/ml) (8–10). In specimens from the upper respiratory tract, SARS-CoV-2 RNA peaks with the beginning of symptoms around day 4, decreases steadily during the first 10 days after illness onset, and can be detected up to 20 days after the onset of symptoms (22–25). However, viral loads are low during the earlier stage of infection and in the second week after the onset of symptoms (26, 27). Thus, the time window for the detection of SARS-CoV-2 by RDTs appears to be narrower than with rRT-PCR and may be confined to the acute phase of infection. Consistently, we found a lower sensitivity of RDTs in asymptomatic individuals, suggesting limited usefulness of RDTs for screening such individuals, even if it is repeated on a regular basis.

We want to emphasize that the RDTs were carried out by medically educated personnel with strict adherence to the instructions issued by the manufacturers, perhaps explaining the low rate of false-positive results. Yet, RDTs have been widely recommended for self-testing or for testing by lay persons. Indeed, when Ag-RDT self-testing results were evaluated in a comparative study among symptomatic outpatients, self-testing (including self-read-out) yielded a sensitivity of 82.5% compared to professional nasopharyngeal sampling and testing (28). The same study noted variations in the sensitivity of Ag-RDT self-testing depending on the viral load of the sample. High viral loads ≥ 7.0 log10 SARS-CoV-2 RNA copies/ml led to a sensitivity of 96.6%, whereas low viral loads <7.0 log10 SARS-CoV-2 RNA copies/ml had decreased sensitivity (45.6% for Ag-RDT self-use and 54.5% for Ag-RDT professional use) (28). Thus, it appears that, at low viral loads, as encountered in asymptomatic persons, self-administered RDT testing may be even less effective in reality than it was in the current study.

Sensitivity for the SARS-CoV-2 Alpha Variant

During the conduct of our study, the SARS-CoV-2 Alpha variant became the prevailing genotype in Southern Germany. Consequently, the study included 81 carriers of SARS-CoV-2 NIM, 166 carriers of the Alpha variant, and 4 patients with other viral genotypes (251 of 338 samples with Ct values ≤ 30). Remarkably, the sensitivities of both RDTs were lower in Alpha variant carriers, and this finding was robust against adjustments for the viral load expressed in terms of Ct values (Table 3). The Alpha variant may be differentiated from the wild-type by two key mutations in the Spike protein: the N501Y substitution within the receptor-binding domain and the H69/V70 deletion. It may be ~80% more transmissible than the wild-type (29) due to conformational changes increasing the Spike protein's affinity for the angiotensin-converting enzyme 2 (ACE2) receptor (30–32). SARS-CoV-2 variants may also confer decreased binding of therapeutic antibodies and protection by vaccination, whereby the Alpha variant may display the smallest variation in antigenicity compared to other circulating variants. This raises the attractive possibility that the lower reactivity of the current RDTs for the Alpha variant was related to structural alterations in the epitope(s) recognized by the detecting antibodies. However, the antibodies incorporated in both RDTs recognize the nucleocapsid protein (N-protein) (33). In a laboratory-based investigation (virus suspended in cell culture medium and saliva), the performance of the Roche-RDT and Abbott-RDT was not affected by variants. This is in contrast to the current results, which were collected in a real-world setting. A potential speculative explanation for the discrepancies may be that both the Spike protein and the N-protein tightly interact, and that conformational changes in the Spike protein may affect the three-dimensional structure and accessibility for antibodies against the N-protein (34). Finally, we cannot rule out that the structural changes in other variants may be even greater than in the Alpha variant (35), so that their effect on sensitivity may be even stronger. Thus, any validation of RDTs for SARS-CoV-2 would also have to be extended to known and future variants rather than limited to the wild-type SARS-CoV-2.

Implications for Screening

A recent meta-analysis issued by the Cochrane Collaboration reviewed 48 studies including 58 commercial RDTs. It reported sensitivities between 34.1% (95% CI 29.7-38.8%) and 88.1% (95% CI 84.2-91.1%) for RDTs in symptomatic persons, whereas in asymptomatic persons, the sensitivities varied between 28.6%(8.4-58.1%) and 69.2% (38.6-90.9%) (11).

The Cochrane analysis identified three studies of the Abbott-RDT with 1,094 symptomatic participants, including 252 SARS-CoV-2 cases, and one study with 474 asymptomatic persons and 47 cases. It also identified three studies with 1,948 symptomatic participants and 336 cases and one study with 127 asymptomatic persons and 13 cases for the Roche-RDT (listed as Biosensor Standard Q). The number of asymptomatic persons in the current single study thus the number in the study in the Cochrane meta-analysis (11).

For the Abbott-RDT, the Cochrane analysis reported sensitivities of 75.1% (57.3-87.1%) and 48.9% (35.1-62.9%), and specificities of 99.5% (98.7-99.8%) and 98.1% (96.3-99.1%) in symptomatic and asymptomatic persons, respectively. The sensitivity of the Roche-RDT was reported to be 88.1% (84.2-91.1%) and 69.2% (38.6-90.9%) in symptomatic and asymptomatic patients, respectively, with specificities similar to the Abbott-RDT. The current findings therefore almost exactly coincide with the Cochrane analysis and significantly extends the available evidence.

Analogous with the viewpoint of the Cochrane analysis (11), at a prevalence rate of 0.05, sensitivities of 60.4 and 56.8% (symptomatic: 75.23 and 74.32%, asymptomatic: 31.9 and 23.28%) and specificities of 99.7 and 99.9% (symptomatic: 99.6 and 99.73%, asymptomatic: 99.82 and 99.96%) with the Roche-RDT and Abbott-RDT, respectively, in symptomatic patients with SARS-CoV-2 infections, our data would translate as follows.

If 10,000 patients with expected 500 (0.05) true positives were examined, 414 and 397 persons would have tested positive, of which 38 and 25 would have been false-positives and 124 and 128 persons with negative test results would be falsely negative for Roche-RDT and Abbott-RDT, respectively (Table 4). Assuming a prevalence rate of 0.005 with expected 50 true positives in 10,000 asymptomatic patients, there would be 34 and 16 persons testing positive, of which 18 and 4 would have been false-positives for Roche-RDT and Abbott-RDT, respectively, and 34 and 38 persons with a negative test would be falsely negative (Table 4). This is crucially important, as RDTs have specifically been recommended for screening asymptomatic persons. Furthermore, as shown in Table 4, these numbers would be affected if the SARS-CoV-2 Alpha variant is predominant.

TABLE 4A

Table 4A. Number of false-positive and false-negative results in a hypothetical cohort of 10,000 people tested with the Roche-RDT.

TABLE 4B

Table 4B. Number of false-positive and false-negative results in a hypothetical cohort of 10 000 people tested with the Abbott –RDT.

Limitations

We applied rRT-PCR as the reference method to detect SARS-CoV-2 infection. Despite being considered the gold standard, this technique has the limitation that the detection of SARS-CoV-2 RNA in patient samples does not indicate the presence or shedding of viable virus with replicative capacity or whether the tested individual is contagious at the time of the test (22, 36). Although the presence of viral RNA proven by rRT-PCR does not automatically equate to infectiousness, a significant correlation between the Ct value (reflecting viral load) and subsequent virus cultivation has been observed. Samples with Ct values between 13 and 17 have a culture positivity rate of 100%, which declines gradually to 12% when Ct values of 33 are analyzed. No viral growth occurs at Ct values ≥ 34, suggesting that patients with these values do not excrete infectious viral particles (27). Therefore, if infectivity rather than a positive rRT-PCR test were considered the reference for RDTs, our results would stand more in favor of RDT testing. However, a direct conversion of Ct values or a positive RDT to contagiousness has not yet been established. Ct values can hardly be compared across studies, and the correlation between viral load and the risk of transmission from a positive case is still not entirely clear (23, 37), with a variety of circumstances, such as the individual's behavior, the type and duration of contact, the environment, and the implementation of transmission-reducing measures (e.g., filter masks) affect infectiveness (37, 38).

The sensitivity of RDTs may relate to the time elapsed since infection. It is a limitation of the current study that the time point of infection in rRT-PCR-positive samples was not available. However, we consider the Ct values from rRT-PCR as a good proxy for the changes in viral load during the course of the study.

Finally, we demonstrated that the SARS-CoV-2 Alpha variant markedly diminishes the sensitivity of both RDTs. We cannot explain this finding. Other variants were not encountered in sufficient numbers, and we were not able to infer the sensitivities of the RDTs for other variants.

Directions for Future Research

This evaluation of two of the most sensitive RDTs currently available for SARS-CoV-2 suggests that screening asymptomatic persons with this approach may fail to identify a substantial proportion of viral carriers. Thus, further methodical refinements are needed, such as attempts to determine the viral load at least semi-quantitatively. Alternatively, rapid, on-site, direct detection of SARS-CoV-2 RNA RDTs could be pursued. The lower sensitivity of the RDTs for the Alpha variant indicates that their performance may be substantially influenced by the virus genotype, and strategies need to be developed to ensure that any of the circulating variants are captured. Finally, further head-to-head research is needed into how current screening strategies, RDT or laboratory-based, directly translate into controlling virus transmission and spread in the population. This will show whether the obvious practical advantage of RDTs is offset by their limitations.

Data Availability Statement

Data will be made available to researchers upon justified request and formal agreement to make sure that rules of good scientific practice are obeyed and that credit is given to the people who have been in charge of the design and the organization of the study. Interested researchers are invited to address their request or proposal to WM (winfried.maerz@synlab.com). The authors confirm that they accessed and validated these data and that all other researchers can access the data in the same manner the authors did.

Ethics Statement

The studies involving human participants were reviewed and approved by Ethics Committee II (Mannheim) of the University of Heidelberg (reference number 2020-417MF). The patients/participants provided their written informed consent to participate in this study.

Author Contributions

WM, CMa, and CMu designed the study. CW, GB, UH, and H-JW collected the data. AD performed the statistical analysis. EW and GB surveyed the laboratory analyses. AM and SL examined the sociodemographic characteristics of study participants. CW and WM wrote the manuscript. All authors validated, reviewed, and edited the manuscript.

Funding

The costs of the study were defrayed by SYNLAB Holding Deutschland GmbH. The management had no role in writing of the report or the decision to submit for publication. There was no financial support to SYNLAB Holding Deutschland GmbH from the manufacturers of the assays used in this evaluation and there has been no other financial support for this work that could have influenced its outcome.

Conflict of Interest

CW, GB, CP, UH, EW, CMa, MR, and WM were employed by SYNLAB Holding Germany GmbH or its regional subsidiaries. AD is the owner of Company Dr. Dressel Consulting. CMu was employed by company SGS Analytics Germany GmbH.

The remaining authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher's Note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Acknowledgments

We thank the participants for joining in free of remuneration. Katja Pöhl, CEO, SYNLAB MVZ Leinfelden-Echterdingen and Christoph Mahnke, CEO SYNLAB Holding Deutschland GmbH, supported the study. We thank Madeline Beckers, Alexander Ignatenko, Anna Süßmuth, Annika Saile, Berfin Cicek, Brigitte Schwandt, Carolin Meixner, Christian Lang, Felix Eisenmann, Jacques Pfander, Katerina Triantafillidi, Lars Banzhaf, Leo Rahmig, Miriam Berner, Robin Bohn, Susanne Kunz, Tim Fröschle, Verena Lebherz, and Jacob Göhring for carrying out the RDTs and documenting the participants at the Corona Test Center Cannstatter Wasen, Stuttgart. The content of this manuscript has previously appeared online on the preprint-server medRxiv (39).

Supplementary Material

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fmed.2022.774550/full#supplementary-material

Abbreviations

NIM, negative for investigated mutations. Samples in which both N501Y and H69/V70 were absent may have contained variants other than Alpha, Beta, or Gamma or the wild-type. For the purpose of this article these samples were termed negative for investigated mutations (NIM).

References

1. Zhu N, Zhang D, Wang W, Li X, Yang B, Song J, et al. A novel coronavirus from patients with pneumonia in China, 2019. N Engl J Med. (2020) 382:727-33. doi: 10.1056/NEJMoa2001017

PubMed Abstract | CrossRef Full Text | Google Scholar

2. World Health Organization. WHO Statement Regarding Cluster of Pneumonia Cases in Wuhan, China. Beijing: World Health Organization (2020).

Google Scholar

3. Li JY, You Z, Wang Q, Zhou ZJ, Qiu Y, Luo R, et al. The epidemic of 2019-novel-coronavirus (2019-nCoV) pneumonia and insights for emerging infectious diseases in the future. Microbes Infect. (2020) 22:80-5. doi: 10.1016/j.micinf.2020.02.002

PubMed Abstract | CrossRef Full Text | Google Scholar

4. Zhou P, Yang XL, Wang XG, Hu B, Zhang L, Zhang W, et al. A pneumonia outbreak associated with a new coronavirus of probable bat origin. Nature. (2020) 579:270-3. doi: 10.1038/s41586-020-2951-z

PubMed Abstract | CrossRef Full Text | Google Scholar

5. Wölfel R, Corman VM, Guggemos W, Seilmaier M, Zange S, Müller MA, et al. Virological assessment of hospitalized patients with COVID-2019. Nature. (2020) 581:465-9. doi: 10.1038/s41586-020-2196-x

PubMed Abstract | CrossRef Full Text | Google Scholar

6. Carter LJ, Garner LV, Smoot JW, Li Y, Zhou Q, Saveson CJ, et al. Assay techniques and test development for COVID-19 diagnosis. ACS Cent Sci. (2020) 6:591-605. doi: 10.1021/acscentsci.0c00501

PubMed Abstract | CrossRef Full Text | Google Scholar

7. Schildgen V, Demuth S, Lüsebrink J, Schildgen O. Limits and opportunities of SARS-CoV-2 antigen rapid tests: an experienced-based perspective. Pathogens. (2021) 10:38. doi: 10.3390/pathogens10010038

PubMed Abstract | CrossRef Full Text | Google Scholar

8. Prince-Guerra JL, Almendares O, Nolen LD, Gunn JKL, Dale AP, Buono SA, et al. Evaluation of Abbott BinaxNOW Rapid Antigen Test for SARS-CoV-2 infection at two community-based testing sites - Pima County, Arizona, November 3-17, 2020. MMWR Morb Mortal Wkly Rep. (2021) 70:100-5. doi: 10.15585/mmwr.mm7003e3

PubMed Abstract | CrossRef Full Text | Google Scholar

9. Kohmer N, Toptan T, Pallas C, Karaca O, Pfeiffer A, Westhaus S, et al. The comparative clinical performance of four SARS-CoV-2 rapid antigen tests and their correlation to infectivity in vitro. J Clin Med. (2021) 10:328. doi: 10.3390/jcm10020328

PubMed Abstract | CrossRef Full Text | Google Scholar

10. Mak GC, Lau SS, Wong KK, Chow NL, Lau CS, Lam ET, et al. Analytical sensitivity and clinical sensitivity of the three rapid antigen detection kits for detection of SARS-CoV-2 virus. J Clin Virol. (2020) 133:104684. doi: 10.1016/j.jcv.2020.104684

PubMed Abstract | CrossRef Full Text | Google Scholar

11. Dinnes J, Deeks JJ, Berhane S, Taylor M, Adriano A, Davenport C, et al. Rapid, point-of-care antigen and molecular-based tests for diagnosis of SARS-CoV-2 infection. Cochrane Database Syst Rev. (2021) 3:Cd013705. doi: 10.1002/14651858.CD013705

PubMed Abstract | CrossRef Full Text | Google Scholar

12. Kroll LE, Robert-Koch-Institut. German Index of Socioeconomic Deprivation (GISD). Berlin: Robert-Koch-Institut (2021).

Google Scholar

13. Kroll LE, Schumann M, Hoebel J, Lampert T. Regional health differences – developing a socioeconomic deprivation index for Germany. J Health Monitor. (2017) 2:98–114. doi: 10.17886/RKI-GBE-2017-048.2

CrossRef Full Text | Google Scholar

14. Data, Licence Germany Version 2,.0. Available online at: https://www.govdata.de/dl-de/by-2-0 (accessed December 29, 2021).

Google Scholar

15. Geodata: Federal Agency for Cartography Geodesy Frankfurt December 2021. Available online at: https://gdz.bkg.bund.de/index.php/default/digitale- geodaten/verwaltungsgebiete/nuts-gebiete-1-250-000-stand-31-12-nuts250-31-12.html. Data licence Germany version 2.0. (accessed December 29, 2021).

Google Scholar

16. FIND. Evaluation of Abbott Panbio COVID-19 Ag Rapid Test Device - External Report. Version 2.1. (2020). Available online at: https://www.finddx.org/wp-content/uploads/2020/12/Panbio_Ag-Public-Report_v2.1.pdf (accessed October 18, 2021).

Google Scholar

17. FIND. Evaluation of SD Biosensor, Inc. STANDARD Q COVID-19 Ag Test - External Report. Version 1.0. (2020). Available online at: https://www.finddx.org/wp-content/uploads/2020/09/SDQ-Ag-Public-Report_20200918.pdf (accessed October 18, 2021).

Google Scholar

18. Cerutti F, Burdino E, Milia MG, Allice T, Gregori G, Bruzzone B, et al. Urgent need of rapid tests for SARS CoV-2 antigen detection: Evaluation of the SD-Biosensor antigen test for SARS-CoV-2. J Clin Virol. (2020) 132:104654. doi: 10.1016/j.jcv.2020.104654

PubMed Abstract | CrossRef Full Text | Google Scholar

19. Linares M, Pérez-Tanoira R, Carrero A, Romanyk J, Pérez-García F, Gómez-Herruz P, et al. Panbio antigen rapid test is reliable to diagnose SARS-CoV-2 infection in the first 7 days after the onset of symptoms. J Clin Virol. (2020) 133:104659. doi: 10.1016/j.jcv.2020.104659

PubMed Abstract | CrossRef Full Text | Google Scholar

20. Albert E, Torres I, Bueno F, Huntley D, Molla E, Fernández-Fuentes M, et al. Field evaluation of a rapid antigen test (Panbio™ COVID-19 Ag Rapid Test Device) for COVID-19 diagnosis in primary healthcare centres. Clin Microbiol Infect. (2021) 27:472.e7-10. doi: 10.1016/j.cmi.2020.11.004

PubMed Abstract | CrossRef Full Text | Google Scholar

21. Kost GJ. The impact of increasing disease prevalence, false omissions, and diagnostic uncertainty on coronavirus disease 2019 (COVID-19) test performance. Arch Pathol Lab Med. (2021) 145:797-813. doi: 10.5858/arpa.2020-0716-SA

PubMed Abstract | CrossRef Full Text | Google Scholar

22. Romero-Gómez MP, Gómez-Sebastian S, Cendejas-Bueno E, Montero-Vega MD, Mingorance J, García-Rodríguez J. Ct value is not enough to discriminate patients harbouring infective virus. J Infect. (2021) 82:e35-7. doi: 10.1016/j.jinf.2020.11.025

PubMed Abstract | CrossRef Full Text | Google Scholar

23. Singanayagam A, Patel M, Charlett A, Lopez Bernal J, Saliba V, Ellis J, et al. Duration of infectiousness and correlation with RT-PCR cycle threshold values in cases of COVID-19, England, January to May 2020. Euro Surveill. (2020) 25. doi: 10.2807/1560-7917.ES.2020.25.32.2001483 [Epub ahead of print].

PubMed Abstract | CrossRef Full Text | Google Scholar

24. Cevik M, Tate M, Lloyd O, Maraolo AE, Schafers J, Ho A. SARS-CoV-2, SARS-CoV, and MERS-CoV viral load dynamics, duration of viral shedding, and infectiousness: a systematic review and meta-analysis. Lancet Microbe. (2021) 2:e13-22. doi: 10.1016/S2666-5247(20)30172-5

PubMed Abstract | CrossRef Full Text | Google Scholar

25. Zou L, Ruan F, Huang M, Liang L, Huang H, Hong Z, et al. SARS-CoV-2 viral load in upper respiratory specimens of infected patients. N Engl J Med. (2020) 382:1177-9. doi: 10.1056/NEJMc2001737

PubMed Abstract | CrossRef Full Text | Google Scholar

26. Walsh KA, Jordan K, Clyne B, Rohde D, Drummond L, Byrne P, et al. SARS-CoV-2 detection, viral load and infectivity over the course of an infection. J Infect. (2020) 81:357-71. doi: 10.1016/j.jinf.2020.06.067

PubMed Abstract | CrossRef Full Text | Google Scholar

27. La Scola B, Le Bideau M, Andreani J, Hoang VT, Grimaldier C, Colson P, et al. Viral RNA load as determined by cell culture as a management tool for discharge of SARS-CoV-2 patients from infectious disease wards. Eur J Clin Microbiol Infect Dis. (2020) 39:1059-61. doi: 10.1007/s10096-020-03913-9

PubMed Abstract | CrossRef Full Text | Google Scholar

28. Lindner AK, Nikolai O, Rohardt C, Kausch F, Wintel M, Gertler M, et al. Diagnostic accuracy and feasibility of patient self-testing with a SARS-CoV-2 antigen-detecting rapid test. J Clin Virol. (2021) 141:104874. doi: 10.1016/j.jcv.2021.104874

PubMed Abstract | CrossRef Full Text | Google Scholar

29. Leung K, Shum MH, Leung GM, Lam TT, Wu JT. Early transmissibility assessment of the N501Y mutant strains of SARS-CoV-2 in the United Kingdom, October to November 2020. Euro Surveill. (2021) 26. doi: 10.2807/1560-7917.ES.2020.26.1.2002106 [Epub ahead of print].

PubMed Abstract | CrossRef Full Text | Google Scholar

30. Spratt AN, Kannan SR, Woods LT, Weisman GA, Quinn TP, Lorson CL, et al. Evolution, correlation, structural impact and dynamics of emerging SARS-CoV-2 variants. Comput Struct Biotechnol J. (2021) 19:3799-809. doi: 10.1016/j.csbj.2021.06.037

PubMed Abstract | CrossRef Full Text | Google Scholar

31. Gobeil SM, Janowska K, McDowell S, Mansouri K, Parks R, Stalls V, et al. Effect of natural mutations of SARS-CoV-2 on spike structure, conformation, and antigenicity. Science. (2021) 373:eabi6226. doi: 10.1126/science.abi6226

PubMed Abstract | CrossRef Full Text | Google Scholar

32. Cai Y, Zhang J, Xiao T, Lavine CL, Rawson S, Peng H, et al. Structural basis for enhanced infectivity and immune evasion of SARS-CoV-2 variants. Science. (2021) 373:642-8. doi: 10.1126/science.abi9745

PubMed Abstract | CrossRef Full Text | Google Scholar

33. Jungnick S, Hobmaier B, Mautner L, Hoyos M, Haase M, Baiker A, et al. Detection of the new SARS-CoV-2 variants of concern B.1.1.7 and B.1.351 in five SARS-CoV-2 rapid antigen tests (RATs), Germany, March 2021. Euro Surveill. (2021) 26. doi: 10.2807/1560-7917.ES.2021.26.16.2100413 [Epub ahead of print].

PubMed Abstract | CrossRef Full Text | Google Scholar

34. Park BK, Kim J, Park S, Kim D, Kim M, Baek K, et al. MERS-CoV and SARS-CoV-2 replication can be inhibited by targeting the interaction between the viral spike protein and the nucleocapsid protein. Theranostics. (2021) 11:3853-67. doi: 10.7150/thno.55647

PubMed Abstract | CrossRef Full Text | Google Scholar

35. Wang R, Zhang Q, Ge J, Ren W, Zhang R, Lan J, et al. Analysis of SARS-CoV-2 variant mutations reveals neutralization escape mechanisms and the ability to use ACE2 receptors from additional species. Immunity. (2021) 54:1611-21.e5. doi: 10.1016/j.immuni.2021.06.003

PubMed Abstract | CrossRef Full Text | Google Scholar

36. Basile K, McPhie K, Carter I, Alderson S, Rahman H, Donovan L, et al. Cell-based culture informs infectivity and safe de-isolation assessments in patients with coronavirus disease 2019. Clin Infect Dis. (2021) 73:e2952-9. doi: 10.1093/cid/ciaa1579

PubMed Abstract | CrossRef Full Text | Google Scholar

37. Marks M, Millat-Martinez P, Ouchi D, Roberts CH, Alemany A, Corbacho-Monné M, et al. Transmission of COVID-19 in 282 clusters in Catalonia, Spain: a cohort study. Lancet Infect Dis. (2021) 21:629-36. doi: 10.1016/S1473-3099(20)30985-3

PubMed Abstract | CrossRef Full Text | Google Scholar

38. Ghinai I, McPherson TD, Hunter JC, Kirking HL, Christiansen D, Joshi K, et al. First known person-to-person transmission of severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) in the USA. Lancet. (2020) 395:1137-44. doi: 10.1016/S0140-6736(20)30607-3

PubMed Abstract | CrossRef Full Text | Google Scholar

39. Wertenauer C, Michael GB, Dressel A, Pfeifer C, Hauser U, Wieland E, et al. Diagnostic efficacy of rapid antigen testing for SARS-CoV-2: the COVid-19 AntiGen (COVAG) study. medRxiv. (2021). doi: 10.1101/2021.08.04.21261609

CrossRef Full Text | Google Scholar