Quality Assessment of the Chinese Clinical Trial Protocols Regarding Treatments for Coronavirus Disease 2019

Background With the global spread of coronavirus disease 2019 (COVID-19), an increasing number of clinical trials are being designed and executed to evaluate the efficacy and safety of various therapies for COVID-19. We conducted this survey to assess the methodological quality of registry protocols on potential treatments for COVID-19. Methods Clinical trial protocols were identified on the ClinicalTrials.gov and the Chinese Clinical Trial Registry. Protocols were screened by two investigators independently against pre-defined eligibility criteria. Quality of the included protocols was assessed according to the modified 14-item SPIRIT (Standard Protocol Items: Recommendations for Interventional Trials) 2013 Statement. Results We included 82 randomized controlled trial (RCT) protocols investigating treatment modalities for COVID-19. These ongoing trials are being conducted in 16 provinces, autonomous regions, and municipalities of China, and study interventions were either Western medicines (n = 56) or traditional Chinese medicine (n = 26). Findings of our quality assessment indicated that the existing trial protocols could be further improved on several aspects, including selection and definition of outcome measures, descriptions of study interventions and comparators, study subject recruitment time, definition of study inclusion and exclusion criteria, and allocation concealment methods. Descriptions of random sequence generation methodologies were accurate for the majority of included trial protocols (n = 64; 78.05%); however, reporting of allocation concealment remained unclear in 63 (76.83%) protocols. Therefore, the overall risk of selection bias across these RCTs was judged to be unclear. A total of 52 (63.41%) included RCT protocols were open-label trials and are thus associated with a high risk of performance bias and detection bias. Conclusion Quality of currently available RCT protocols on the treatments for COVID-19 could be further improved. For transparency and effective knowledge translation in real-world clinically settings, it is important for trial investigators to standardize baseline treatments for patients with COVID-19 and assess clinically important core outcome measures. Despite eager anticipation from the public on the results of effectiveness trials in COVID-19, robust design, execution, and reporting of these trials should be regarded as high priority.


INTRODUCTION
In December 2019, coronavirus disease 2019 , an infectious disease caused by severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2), outbroke in Wuhan, Hubei province, China (Hui et al., 2020;Lu et al., 2020). As of March 4, 2020, a total of 80,409 cases have been confirmed and over 3,000 deaths were reported in China alone (National Health Commission of the People's Republic of China, 2020). The total numbers of confirmed cases and deaths in other countries were 12,668 and 214, respectively (World Health Organization, 2020a). The World Health Organization (WHO) declared COVID-19 a public health emergency of international concern (World Health Organization, 2020b).
There is a paucity of evidence regarding the therapeutic options for COVID-19. Four case series involving 41, 99, 138, and 1099 patients with laboratory-confirmed COVID-19, respectively, have been published (Chen N. et al., 2020;Guan et al., 2020;Huang et al., 2020;; and a wide array of antiviral therapies such as oseltamivir, ganciclovir, and lopinavir/ritonavir were used. However, the efficacy of these drugs was not evaluated. Two preclinical studies showed that remdesivir, chloroquine, arbidol, and darunavir could effectively inhibit SARS-CoV-2 (Huang, 2020;Wang M. et al., 2020); and two clinical studies investigated the effects of remdesivir, arbidol, lopinavir/ritonavir, and Shufeng Jiedu capsules in treating COVID-19 (Holshue et al., 2020;Wang Z. et al., 2020). Driven by the effectiveness of lopinavir/ritonavir in the early treatment of patients with Severe Acute Respiratory Syndrome (SARS), a systematic review suggested that it could also serve as an experimental antiviral therapy for CoVID-19, in particular for newly diagnosed patients (Jiang et al., 2020). Nevertheless, a retrospective cohort study of 134 patients did not find any effects of lopinavir/ritonavir and arbidol on relieving symptoms or accelerating virus clearance among patients with COVID-19 . The latest Guidance for Corona Virus Disease 2019 Prevention, Control, Diagnosis and Management (GCVD2019PCDM) guidelines proposed alpha-interferon nebulization, lopinavir/ritonavir, ribavirin, chloroquine phosphate, and arbidol as antiviral treatments (National Health Commission of PRC and National Administration of Traditional Chinese Medicine of the RPC, 2020).
Randomized controlled trial (RCT) by design is the gold standard for evaluating the effects of interventions. Up to now, more than 80 clinical trials exploring potential treatment options for COVID-19 are registered/ongoing in China (Maxmen, 2020). For research transparency and validity, clinical trials should be pre-registered in a validated study register where study plans and protocols are available in the public domain (Chhapola et al., 2018;Hendarto et al., 2019). To our knowledge, there is currently no attempt to assess the methodological quality of existing trial protocols in the field of COVID-19 and we thus conducted this cross-sectional analysis to evaluate the quality of clinical trial protocols on potential COVID-19 treatments.

Study Design
This is a cross-sectional analysis of clinical trial protocols on treatment modalities of COVID-19.

Eligibility Criteria
We included intervention trial protocols meeting the following criteria: (1) RCT by design; (2) study participants with laboratoryconfirmed CoVID-19; (3) involving Western medicine (WM) or traditional Chinese medicine (TCM) as a treatment intervention. Study protocols enrolling patients treated in the recovery phase were excluded.

Study Selection and Data Extraction
Two investigators independently screened the protocols for inclusion and assessed their quality against pre-defined inclusion and exclusion criteria. Any disagreement in the process of study selection was resolved by discussion. Two authors independently extracted the following data from included protocols: (1) basic information: registry number, title, primary sponsor, location, institutional level, study execution time, source of funding; (2) population information: inclusion criteria, exclusion criteria, age, and sample size; (3) interventions: medicine, dosage, usage, course of treatment, and number of study groups; (4) outcomes: definition, time-point of measurement, and method of measurement for primary and secondary outcomes; (5) study design: study type, randomization procedure, allocation concealment, blinding, data collection and management, ethical permit, and informed consensus.

Quality Assessment
Two investigators independently appraised the quality of each included protocols using the modified Standard Protocol Items: Recommendations for Interventional Trials (SPIRIT) 2013 Statement, and any disagreements were resolved by discussion. The modified SPIRIT 2013 was developed following the SPIRIT 2013 Statement (Chan et al., 2013) and the information provided on the ClinicalTrial.gov and the Chinese Clinical Trial Registry. In the modified SPIRIT 2013 checklist, the evaluation items include (1) specific objectives or hypotheses; (2) conflict of interest; (3) clear enrolment schedule; (4) specific participant inclusion and exclusion criteria; (5) sufficient details about interventions for each group, including how and when interventions are applied; (6) matching between grouping and the research purpose; (7) sufficient details about outcome measurement; (8) suitability of the primary outcome; (9) all the collaborating institutions listed in a multicenter study; (10) randomization sequence generation; (11) allocation concealment; (12) blinding; (13) data collection and management methods; (14) ethical permit. We categorized the judgments as low, high, or unclear risk of bias.

Statistical Synthesis
Statistical analysis was conducted using SPSS (version19.0) software. The rate or constituent ratio was used to describe qualitative data.

Search Result and Baseline Characteristics of Included Trial Protocols
A total of 189 trial protocols were retrieved from ClinicalTrials.gov and CCTR. After selection (Figure 1), we included 82 RCT protocols (17 from ClinicalTrials.gov and 65 from CCTR) in the final assessment. The included trials are being conducted in secondary and tertiary hospitals from 16 provincial areas in China, including Beijing, Shanghai, Chongqing, Hubei, Hunan, Hebei, Henan, Guangdong, Zhejiang, Jiangsu, Shandong, Heilongjiang, Liaoning, Anhui, Shanxi, and Sichuan. The estimated study duration of 51 (62.20%) trials will be longer than six months, except for two trials (ChiCTR2000029762 and ChiCTR2000029855) which did not provide details on the estimated date of completion. Fourteen (17.07%) trials are funded by pharmaceutical companies and 32 (39.02%) trials by the government, while no information about funding source is available for the rest 36 (43.90%) trials.

Types of Study Participants
Six (7.32%) trials aim to enroll laboratory-confirmed and suspected or clinically diagnosed COVID-19 cases ( Table 1). Laboratory-confirmed COVID-19 cases could be clinically mild, ordinary, severe, and critical on basis of the classifications of GCVD2019PCDM. Only one (1.22%), nine (10.98%), nine (10.98%), and three (3.66%) trials included mild, ordinary, severe, and critical laboratory-confirmed cases, respectively. The remaining 54 (65.85%) trials plan to recruit more than two subtypes of laboratory-confirmed cases. The participants of 71 (86.58%) trials are adults only and six trials (7.32%) also include children aged 12 years or above, except for five (6.10%) trials without any description of the participant age. Twenty-four (29.27%) trial protocols clearly described the recruitment time     (3 to 14 days). Study sample size ranges from 20 to 600, and a total of 32 (39.02%) trials are small-scale studies with less than 100 subjects.

Types of Study Interventions and Comparators
The majority of the included trials (n = 56; 68.29%) use WM as the study intervention, with the remaining 26 (31.71%) trials evaluating the effects of TCM. For the former, interventions include interferon aerosol inhalation, lopinavir/ritonavir, ribavirin, chloroquine phosphate, arbidol, and remdesivir, with lopinavir/ritonavir being the most common study intervention (n = 13; 15.85%).

Types of Primary Outcomes
We found one trial protocol (ChiCTR2000029755) without specifying a primary outcome measure. Seven (8.54%) protocols included more than three primary outcomes but none set any primary outcomes regarding safety. We obtained 20 primary outcomes from the 82 protocols assessed and classified them into six groups: (1) the prognostic outcome [rate of or time to disease remission or recovery, rate of or time to composite adverse outcome, all-cause mortality or mortality, length of hospitalization, patient prognosis, complication incidence, National Early Warning Score (NEWS) 2, Pneumonia Severity Index (PSI), and Sequential Organ Failure Assessment (SOFA) score]; (2) the etiological outcomes (rate of or time to virus-negative conversion of SARS-CoV-2); (3) outcomes on clinical symptoms (rate of or time to no fever, no cough, no dyspnea, or no myalgia); (4) outcomes about the lung or respiratory function (e.g., rate of or time to lung imaging recovery, lung injury score, oxygenation index, requirements of mechanical ventilation support, etc.); (5) outcome assessed using the TCM symptom score; (6) outcome about the vital physiologic parameters (e.g., body temperature, blood pressure, heart rate, and breathing rate) and routine laboratory tests (e.g., routine blood test, C-reaction protein, procalcitonin, creatine kinase, alanine aminotransferase, CD4, CD8, interleukin, etc.).

Quality Assessment by SPIRIT 2013 Statement
Limitations in terms of methodology existed across all the included protocols ( Figure 2). Although the quality of protocols registered on the ClinicalTrial.gov was better than those registered the on CCTR, their assessment results about five items (No. 2, 10, 11, 13, and 14) could not been performed due to unavailable information regarding funding resource, ethics materials, methods of random sequence generation, allocation concealment, data collection, and management. All included protocols clearly described study objectives. Fifty-six (68.29%) trials investigate both efficacy and safety endpoints while the remaining (31.71%) 26 trials focus only on treatment efficacy. The potential risk of bias due to conflict of interest was noted in 14 (17.07%) trials sponsored by the pharmaceutical companies. A recruitment time of 58 (70.73%) trials was not mentioned, which would cause a high risk of attrition and reporting bias. The inclusion and exclusion criteria of 16 (19.51%) protocols were ambiguous due to undescribed participant age (n = 5), inconsistency between the participants and the study purpose (n = 4), incorrect clinical classifications (n = 4), and absence of overall detailed criteria (n = 3). Fifty-five (67.07%) protocols did not describe the intervention and comparison in detail, particularly regarding the course of treatment, and the selection of control group in two protocols (ChiCTR2000029573 and ChiCTR2000029817) could not match their study purpose. Sixty-five (79.27%) protocols did not define the outcomes, especially in detection timepoint. The primary outcomes of 59 (71.95%) protocols were inappropriate: excessive number of primary outcomes for 7 trials; no safety-related endpoints as primary outcomes for 56 trials evaluating safety (39 did not specify any safety-related outcomes). Among 35 multicenter trials, 9 protocols did not list all the collaborating medical institutions. Despite the accurate methods of random sequence generation (random number table or computer-generated random numbers) in 64 (78.05%) protocols, descriptions of allocation concealment in 63 (76.83%) protocols remained unclear. Therefore, the overall risk of selection bias across these RCTs was unclear. Masking methods in 13 (15.85%) trials varied, from single-blind (participant or outcomes assessor; n = 7), double-blind (participant and care provider; n = 2), triple-blind (participant, care provider and outcomes assessor; n = 1), to quadruple-blind (participant care provider, investigator, and outcomes assessor; n = 3). Nevertheless, 52 (63.41%) trials are of open-label design which is associated with a high risk of performance bias and detection bias. Twenty-nine (35.36%) trials have the Data Management Committee but 20 (24.39%) trials do not. The ethics materials of 26 (31.71%) protocols were incomplete: 11 protocols were not approved by the Ethics Committee, 6 protocols were without available approved file, and 9 protocols did not mention the informed consensus.

DISCUSSION
This survey assessed the quality of 82 RCT protocols regarding treatments for COVID-19 in 16 provincial areas in China. We found that: (1) The study duration of most trials (62.20%) is more than six months which might be too long to enroll enough participants as the COVID-19 will be gradually controlled. Recently, 4 protocols (ChiCTR2000029760, ChiCTR2000029759, ChiCTR2000029762, and ChiCTR2000029761) were withdrawn due to inadequate numbers of patients. (2) These trials mainly focus on COVID-19 mild, ordinary, and severe cases, and the number of trials for WM is more than that for TCM. (3) Eighty- two protocols set 20 different primary outcomes, indicating considerable controversy in the primary outcome for evaluating the efficacy of COVID-19 treatments. Furthermore, only 17 trials (20.73%) considered the safety profile of therapies. (4) The protocols should be improved from several aspects, such as the selection and definition of outcomes, intervention and comparison, recruitment time, inclusion and exclusion criteria, and allocation concealment method. In addition, it is necessary to standardize basic treatments and select appropriate outcomes to reduce the high risk of performance and detection bias in the context of a non-blind design.
Participants of the 82 included trials are suspected, clinically diagnosed, or laboratory-confirmed cases; the majority (73.17%) of these trials recruit more than two subtypes of laboratory-confirmed cases. Given the differences in clinical characteristics, basic treatments, and prognosis of different subtypes, subgroup analyses are strongly suggestedinevaluatingtheefficacyofstudyinterventions.Nevertheless, the sample size of 26 trials are less than 100 and another 4 updated protocols (ChiCTR2000029496, ChiCTR2000029740, ChiCTR2000029757, and ChiCTR2000029434) greatly cut down their sample size, which might compromise the power of statistical analysis. It is critical to detail interventions for future replication study; however, most protocols (67.07%) did not provide any information about the course of treatment, particularly those related to TCM.
One utmost prerequisite to evaluate the treatments for COVID-19 is to determine appropriate outcomes. First of all, the outcomes related to efficacy and safety profiles are of equal importance for a new type of intervention, whereas 26 protocols focused on efficacy only. Even for those trials designed to evaluate the safety profile, only a few selected the incidence of adverse events or severe adverse events as the secondary outcome. Secondly, the primary outcome should represent the greatest therapeutic benefit and be the most important among the many outcomes (Sedgwick, 2010;Andrade, 2015). However, 9 (10.98%) protocols set intermediate outcomes (vital physiologic parameters or routine laboratory tests) as the primary outcome and 7 (8.54%) protocols adopted more than three primary outcomes. Thirdly, the primary outcomes should be generally similar for different protocols with the same study purpose. Nevertheless, there is a variety of primary outcomes among the present protocols regarding investigational interventions for COVID-19. We noted that the time to clinical improvement within 28 days, the lethality by day 28, the rate of symptom (fever, fatigue, and coughing) recovery, the time to achieve a negative RT-PCR result for SARS-CoV-2 in a nasopharyngeal swab sample were set as the primary outcomes in trials regarding remdesivir (Beigel et al., 2020;, chloroquine diphosphate (Borba et al., 2020), Lianhuaqingwen capsules , and triple combination of interferon beta-1b, lopinavir/ritonavir, and ribavirin (Hung et al., 2020), respectively. In addition, Wang Y et al. ), Cao B et al. (Cao et al., 2020, and Beigel JH et al. (Beigel et al., 2020) used different tools (six-point, seven-category, and eight-category ordinal scale) to measure clinical improvement or recovery which, in effect, was proposed as the most critical endpoint by other trial investigators. The significant heterogeneity and lack of critical outcomes across these COVID-19 studies may lead to a waste of research resources.
We argue that the selection of primary outcome should be based on expert consensus and/or conventional practices. For instance, mortality can be the primary outcome for laboratoryconfirmed critical cases. Importantly, clinical classifications of participants should be considered while using mortality as the primary outcomes since laboratory-confirmed mild or ordinary cases have a better prognosis. As to the laboratory-confirmed mild cases, the rate of or the time to disease recovery might be a better primary outcome compared to lung imaging recovery. For the laboratory-confirmed ordinary or severe cases, two optional primary outcomes could be the rate of/the time to disease remission (improvement from severe cases to ordinary cases) and the rate of/the time to composite adverse outcome (admission to an intensive care unit, the use of mechanical ventilation, or death). Due to the high rate of false-negative results of the nucleic acid test of 2019-nCoV, etiological outcome is not suggested as the unique primary outcome despite its specificity. In fact, 9 (10.98%) protocols selected the etiological outcome as the only primary outcome. Additionally, the results of the nucleic acid test of SARS-CoV-2 is an inappropriate outcome for suspected and clinically diagnosed cases. Although TCM symptom score was adopted as the primary outcome in four (4.88%) protocols, it is still controversial in TCM-relevant studies. A recent study ) demonstrated a core outcome set of different outcome measures for different subtypes of laboratoryconfirmed COVID-19 cases based on two rounds of Delphi survey and one consensus meeting, and we are confident that such efforts to develop core outcome sets would be useful for future evidence synthesis and clinical decision-making.
Furthermore, only 17 (20.73%) protocols clearly described how to measure the primary outcomes but the detection time varied a lot. Most protocols agreed that the first week after treatment is important for laboratory-confirmed severe or critical cases and the second week after treatment is critical to evaluate the outcomes regarding the prognosis of COVID-19. A few protocols suggested a longer time (4 weeks or even longer) for mortality measurement. The first week after treatment was also proposed by most protocols for measuring the etiological outcome, while Chen J et al.  considered weeks necessary to detect the rate of virus-negative conversion.
With the spread of COVID-19, increasing clinical trials will be initiated to evaluate the efficacy and safety of potential therapies. The protocol determines the quality of study methodology and the reliability of conclusion, and is thus fundamental to the design, implementation, report, and assessment of a clinical trial.
A previous study investigated 172 trial protocols regarding COVID-19 and found issues related to necessity, scientific validity, ethics, and quality (Xiang et al., 2020). Another cross-sectional analysis characterized trial intervention, sponsorship, critical design elements, and specified outcomes of 201 clinical trials assessing drugs or plasma treatments for COVID-19 and concluded that many trials lacked features to optimize their scientific value . Unfortunately, they did not assess the methodological quality of these protocols. The 33-item SPIRIT Statement is a powerful tool for assessing the quality of published protocols; however, it does not apply to protocols registered on ClinicalTrials.gov and the CCTR, which often contain incomplete information. Consequently, we modified the original SPIRIT 2013 Statement into a more concise 14-item checklist for preliminary assessment of the methodological quality of trial protocols regarding treatments for COVID-19. In the context of the absence of tool for assessing the quality of registry protocols, our study provides a paradigm for future assessments and also might guide study design of clinical trial. The limitations of this study must be acknowledged. All the included trial protocols were from China and 31.71% were related to TCM, which may reduce the generalizability of our results to clinical trial investigators from around world. Furthermore, we were unable to judge whether the statistical power is sufficient in the absence of information about sample estimation and statistical methods. It is worth highlighting that a recent trial of lopinavir/ritonavir in COVID-19 was statistically underpowered and the findings/conclusions indicating that lopinavir/ritonavir was ineffective for COVID-19 patients should thus be interpreted with caution (Carmona-Bayonas et al., 2020). Therefore, further assessment of the trials in terms of methodological quality will be performed after the trials are completed. In fact, the WHO, NIH, etc., have suspended trials/ part trials (WHO Solidarity trial and UK Recovery Trial) with an arm of chloroquine, hydroxychloroquine, or lopinavir/ ritonavir due to no benefitial effect of these antivirals in patients with COVID-19 (Borba et al., 2020;Boulware et al., 2020;Carmona-Bayonas et al., 2020;Griffin, 2020;Mitja et al., 2020;Skipper et al., 2020;Tang et al., 2020;Torjesen, 2020). In China, four relevant trials (ChiCTR2000029760, ChiCTR2000029759, ChiCTR2000029762, ChiCTR2000029761) have also been suspended. We will continuously follow the progress of these trials and also appeal to improvements of registry protocols in line with SPIRIT 2013.

CONCLUSION
Currently, available RCT protocols on potential therapies for CoVID-19 have significant methodological limitations, especially in selection and detection of primary outcomes. Further assessment of trial quality should be performed after the completion of those trials. If the trials are not designed with strict standards, the effort will be in vain. Therefore, despite eager anticipation from the public on the results of COVID-19 therapeutic trials, we must maintain cautious and rigorous on the trial design.

DATA AVAILABILITY STATEMENT
Publicly available datasets were analyzed in this study. This data can be found here: The datasets generated for this study are available on request to the corresponding author.

AUTHOR CONTRIBUTIONS
JZ, YL, and XL collected the data. JZ and XL involved in statistical analysis. JZ, YL, and XL drafted the manuscript. JZ, JS-WK, WZ, and RH revised the final manuscript. All authors contributed to the article and approved the submitted version.